Our October 2018 meetup - A Report
October 2018 - On October 19, 2018, we hosted a community meetup at LinkedIn focussing on Apache Kafka, Apache Samza, and related streaming technologies. The event featured technical deep dives by engineers from LinkedIn and Uber on the latest in the stream processing space. Here is a brief summary of the talks that were presented.
How LinkedIn navigates Streams Infrastructure using Cruise Control
Speaker: Efe Gencer, LinkedIn
Efe shared our work and experiences towards alleviating the management overhead of large-scale Kafka clusters using Cruise Control at LinkedIn. The first part of this talk provided an overview of Cruise Control, including the operational challenges that it solves, its high-level architecture, and some evaluation results from real-world scenarios. The second part went through a hands-on tutorial to demonstrate how we can manage a real Kafka cluster using Cruise Control.
Stream Analytics Manager
Speaker: Sriharsha Chintalapani, Uber
Stream Analytics Manager provides a simplified UI interface to build complex big data applications. It makes it possible for the end user to not only build but also deploy and monitor streaming applications. It provides pluggable interfaces to provide user supplied business logic through Custom Processors, UDFs. Streamline’s main goal is to let developers build, deploy, manage, monitor streaming applications easily in minutes. In this talk Sriharsha went through how we can add other engines like Flink, Spark, Airflow into Streamline and allow users to build both Batch and Streaming applications.
Operating Samza at LinkedIn
Speaker: Abhishek Shivanna, Stephan Soileau, LinkedIn
Operating Samza at LinkedIn, which, processes around a trillion of messages a day with over several thousand jobs, is a daunting task. Abhishek and Stephan went go over the best practices of running Samza as a managed service and took a look at how SREs at LinkedIn use intelligent automation to operate at LinkedIn scale.