org.apache.samza.system.descriptors

Class SystemDescriptor<SubClass extends SystemDescriptor<SubClass>>

java.lang.Object

org.apache.samza.system.descriptors.SystemDescriptor<SubClass>

Type Parameters:

SubClass - type of the concrete sub-class

Direct Known Subclasses:

GenericSystemDescriptor

public abstract class SystemDescriptor<SubClass extends SystemDescriptor<SubClass>>
extends java.lang.Object

A SystemDescriptor can be used for specifying Samza and system-specific properties of an input/output system. It can also be used for obtaining InputDescriptors and OutputDescriptors, which can be used for specifying Samza and system-specific properties of input/output streams.

System properties provided in configuration override corresponding properties specified using a descriptor.

This is the base descriptor for a system. Use a system-specific descriptor (e.g. KafkaSystemDescriptor) if one is available. Otherwise use the GenericSystemDescriptor.

Systems may provide an InputTransformer to be used for input streams on the system. An InputTransformer transforms an IncomingMessageEnvelope with deserialized key and message to another message that is delivered to the MessageStream. It is applied at runtime in InputOperatorImpl.

Systems may provide a StreamExpander to be used for input streams on the system. A StreamExpander expands the provided InputDescriptor to a sub-DAG of one or more operators on the StreamGraph, and returns a new MessageStream with the combined results. It is called during graph description in StreamGraph#getInputStream.

Systems that support consuming messages from a stream should provide users means of obtaining an InputDescriptor. Recommended interfaces for doing so are TransformingInputDescriptorProvider for systems that support system level InputTransformer, ExpandingInputDescriptorProvider for systems that support system level StreamExpander functions, and SimpleInputDescriptorProvider otherwise.

Systems that support producing messages to a stream should provide users means of obtaining an OutputDescriptor. Recommended interface for doing so is OutputDescriptorProvider.

It is not required for SystemDescriptors to implement one of the Provider interfaces above. System implementers may choose to expose additional or alternate APIs for obtaining Input/Output Descriptors by extending SystemDescriptor directly.

Constructor Summary

Constructors

Constructor and Description
SystemDescriptor(java.lang.String systemName, java.lang.String factoryClassName, InputTransformer transformer, StreamExpander expander) Constructs a SystemDescriptor instance.

Method Summary

Modifier and Type	Method and Description
java.util.Optional<StreamExpander>	getExpander()
java.lang.String	getSystemName()
java.util.Optional<InputTransformer>	getTransformer()
java.util.Map<java.lang.String,java.lang.String>	toConfig()
SubClass	withDefaultStreamConfigs(java.util.Map<java.lang.String,java.lang.String> defaultStreamConfigs) Default properties for any stream obtained using this system descriptor.
SubClass	withDefaultStreamOffsetDefault(SystemStreamMetadata.OffsetType offsetType) If a container starts up without a checkpoint, this property determines where in the input stream we should start consuming.
SubClass	withSystemConfigs(java.util.Map<java.lang.String,java.lang.String> systemConfigs) Additional system-specific properties for this system.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SystemDescriptor

public SystemDescriptor(java.lang.String systemName,
                        java.lang.String factoryClassName,
                        InputTransformer transformer,
                        StreamExpander expander)

Constructs a SystemDescriptor instance.

Parameters:

systemName - name of this system

factoryClassName - name of the SystemFactory class for this system

transformer - the InputTransformer for the system if any, else null

expander - the StreamExpander for the system if any, else null

Method Detail

withDefaultStreamOffsetDefault

public SubClass withDefaultStreamOffsetDefault(SystemStreamMetadata.OffsetType offsetType)

If a container starts up without a checkpoint, this property determines where in the input stream we should start consuming. The value must be an SystemStreamMetadata.OffsetType, one of the following:

• upcoming: Start processing messages that are published after the job starts. Any messages published while the job was not running are not processed.
• oldest: Start processing at the oldest available message in the system, and reprocess the entire available message history.

This property is for all streams obtained using this system descriptor. To set it for an individual stream, see InputDescriptor.withOffsetDefault(org.apache.samza.system.SystemStreamMetadata.OffsetType). If both are defined, the stream-level definition takes precedence.

Parameters:

offsetType - offset type to start processing from

Returns:

this system descriptor

withSystemConfigs

public SubClass withSystemConfigs(java.util.Map<java.lang.String,java.lang.String> systemConfigs)

Additional system-specific properties for this system.

These properties are added under the systems.system-name.* scope.

Parameters:

systemConfigs - system-specific properties for this system

Returns:

this system descriptor

withDefaultStreamConfigs

public SubClass withDefaultStreamConfigs(java.util.Map<java.lang.String,java.lang.String> defaultStreamConfigs)

Default properties for any stream obtained using this system descriptor.

For example, if "systems.kafka-system.default.stream.replication.factor"=2 was configured, then every Kafka stream created on the kafka-system will have a replication factor of 2 unless the property is explicitly overridden using the stream descriptor.

Parameters:

defaultStreamConfigs - default stream properties

Returns:

this system descriptor

getSystemName

public java.lang.String getSystemName()

getTransformer

public java.util.Optional<InputTransformer> getTransformer()

getExpander

public java.util.Optional<StreamExpander> getExpander()

toConfig

public java.util.Map<java.lang.String,java.lang.String> toConfig()