Samza provides multiple programming APIs to fit your use case:
- Java APIs: Samza’s provides two Java programming APIs that are ideal for building advanced Stream Processing applications.
- High Level Streams API: Samza’s flexible High Level Streams API lets you describe your complex stream processing pipeline in the form of a Directional Acyclic Graph (DAG) of operations on message streams. It provides a rich set of built-in operators that simplify common stream processing operations such as filtering, projection, repartitioning, joins, and windows.
- Low Level Task API: Samza’s powerful Low Level Task API lets you write your application in terms of processing logic for each incoming message.
- Samza SQL: Samza SQL provides a declarative query language for describing your stream processing logic. It lets you manipulate streams using SQL predicates and UDFs instead of working with the physical implementation details.
- Apache Beam API: Samza also provides a Apache Beam runner to run applications written using the Apache Beam API. This is considered as an extension to the operators supported by the High Level Streams API in Samza.
The following sections will talk about key concepts in writing your Samza applications in Java.
A SamzaApplication describes the inputs, outputs, state, configuration and the logic for processing data from one or more streaming sources.
- For TaskApplication:
Streams and Table Descriptors
Descriptors let you specify the properties of various aspects of your application from within it.
InputDescriptors and OutputDescriptors can be used for specifying Samza and implementation-specific properties of the streaming inputs and outputs for your application. You can obtain InputDescriptors and OutputDescriptors using a SystemDescriptor for your system. This SystemDescriptor can be used for specify Samza and implementation-specific properties of the producer and consumers for your I/O system. Most Samza system implementations come with their own SystemDescriptors, but if one isn’t available, you can use the GenericSystemDescriptor.
The following example illustrates how you can use input and output descriptors for a Kafka system, and a table descriptor for a local RocksDB table within your application:
The same code in the above describe method applies to TaskApplication as well.
Stream Processing Logic
Samza provides two sets of APIs to define the main stream processing logic, High Level Streams API and Low Level Task API, via StreamApplication and TaskApplication, respectively.
High Level Streams API allows you to describe the processing logic in a connected DAG of transformation operators, like the example below:
Low Level Task API allows you to describe the processing logic in a customized StreamTaskFactory or AsyncStreamTaskFactory, like the example below:
Configuration for a Samza Application
To deploy a Samza application, you need to specify the implementation class for your application and the ApplicationRunner to launch your application. The following is an incomplete example of minimum required configuration to set up the Samza application and the runner. For additional configuration, see the Configuration Reference.