Deploying a Samza job from HDFS
This tutorial uses hello-samza to illustrate how to run a Samza job if you want to publish the Samza job’s .tar.gz package to HDFS.
Build a new Samza job package
Build a new Samza job package to include the hadoop-hdfs-version.jar.
- Add dependency statement in pom.xml of samza-job-package
<dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-hdfs</artifactId> <version>2.2.0</version> </dependency>
- Add the following code to src/main/assembly/src.xml in samza-job-package.
- Create .tar.gz package
mvn clean pacakge
- Make sure hadoop-common-version.jar has the same version as your hadoop-hdfs-version.jar. Otherwise, you may still have errors.
Upload the package
hadoop fs -put ./samza-job-package/target/samza-job-package-0.7.0-dist.tar.gz /path/for/tgz
Add HDFS configuration
Put the hdfs-site.xml file of your cluster into ~/.samza/conf directory. (The same place as the yarn-site.xml)
Change properties file
Change the yarn.package.path in the properties file to your HDFS location.
yarn.package.path=hdfs://<hdfs name node ip>:<hdfs name node port>/path/to/tgz
Then you should be able to run the Samza job as described in hello-samza.