Getting Started

Samza SQL


Samza SQL allows you to define your stream processing logic declaratively as a a SQL query. This allows you to create streaming pipelines without Java code or configuration unless you require user-defined functions (UDFs). Support for SQL internally uses Apache Calcite, which provides SQL parsing and query planning. The query is automatically translated to Samza’s high level API and runs on Samza’s execution engine.

You can run Samza SQL locally on your machine or on a YARN cluster.

Running Samza SQL on your local machine

The Samza SQL console allows you to experiment with Samza SQL locally on your machine.

Setup Kafka

Follow the instructions from the Kafka quickstart to start the zookeeper and Kafka server.

Let us create a Kafka topic named ‚ÄúProfileChangeStream‚ÄĚ for this demo.

>./deploy/kafka/bin/ --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic ProfileChangeStream

Download the Samza tools package from here and use the generate-kafka-events script populate the stream with sample data.

> cd samza-tools-<version>
> ./scripts/ -t ProfileChangeStream -e ProfileChange

Using the Samza SQL Console

The simplest SQL query is to read all events from a Kafka topic ProfileChangeStream and print them to the console.

> ./scripts/ --sql "insert into log.consoleoutput select * from kafka.ProfileChangeStream"

Next, let us project a few fields from the input stream.

> ./scripts/ --sql "insert into log.consoleoutput select Name, OldCompany, NewCompany from kafka.ProfileChangeStream"

You can also filter messages in the input stream based on some predicate. In this example, we filter profiles currently working at LinkedIn, whose previous employer matches the regex .*soft. The function RegexMatch(regex, company) is an example of a UDF that defines a predicate.

> ./scripts/ --sql "insert into log.consoleoutput select Name as __key__, Name, NewCompany, RegexMatch('.*soft', OldCompany) from kafka.ProfileChangeStream where NewCompany = 'LinkedIn'"

Running Samza SQL on YARN

The hello-samza project has examples to get started with Samza on YARN. You can define your SQL query in a configuration file and submit it to a YARN cluster.

> ./deploy/samza/bin/ --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=file://$PWD/deploy/samza/config/

SQL Grammar

Samza SQL’s grammar is a subset of capabilities supported by Calcite’s SQL parser.

  |   insert
  |   query 

  | {

      ( INSERT | UPSERT ) INTO tablePrimary
      [ '(' column [, column ]* ')' ]

  { * | projectItem [, projectItem ]* }
  FROM tableExpression
  [ WHERE booleanExpression ]
  [ GROUP BY { groupItem [, groupItem ]* } ]

  expression [ [ AS ] columnAlias ]
  | tableAlias . *

  tableReference [, tableReference ]*
  | tableExpression [ NATURAL ] [ LEFT | RIGHT | FULL ] JOIN tableExpression [ joinCondition ]

  ON booleanExpression
  | USING '(' column [, column ]* ')'

  [ [ AS ] alias [ '(' columnAlias [, columnAlias ]* ')' ] ]

  [ TABLE ] [ [ catalogName . ] schemaName . ] tableName

  VALUES expression [, expression ]*

Known Limitations

Samza SQL only supports simple stateless queries including selections and projections. We are actively working on supporting stateful operations such as aggregations, windows and joins.