This repository was archived by the owner on Jul 9, 2022. It is now read-only.

Name		Name	Last commit message	Last commit date
.mvn		.mvn
spring-cloud-dataflow-server-yarn-autoconfig		spring-cloud-dataflow-server-yarn-autoconfig
spring-cloud-dataflow-server-yarn-client		spring-cloud-dataflow-server-yarn-client
spring-cloud-dataflow-server-yarn-dist		spring-cloud-dataflow-server-yarn-dist
spring-cloud-dataflow-server-yarn-docs		spring-cloud-dataflow-server-yarn-docs
spring-cloud-dataflow-server-yarn-h2		spring-cloud-dataflow-server-yarn-h2
spring-cloud-dataflow-server-yarn-shell-core		spring-cloud-dataflow-server-yarn-shell-core
spring-cloud-dataflow-server-yarn-shell		spring-cloud-dataflow-server-yarn-shell
spring-cloud-dataflow-server-yarn		spring-cloud-dataflow-server-yarn
src		src
.gitignore		.gitignore
.settings.xml		.settings.xml
.travis.yml		.travis.yml
LICENSE		LICENSE
NOTICE		NOTICE
README.adoc		README.adoc
mvnw		mvnw
mvnw.cmd		mvnw.cmd
pom.xml		pom.xml

Repository files navigation

spring-cloud-dataflow-server-yarn is no longer actively maintained by VMware, Inc.

Spring Cloud Data Flow Server for Apache Yarn

The Spring Cloud Data Flow team have been happy stewards of this project, but we are focusing our efforts towards Local, Kubernetes and Cloud Foundry implementations going forward, so the Spring Cloud Data Flow team will no longer maintain it.

However, we are looking to donate the project to the community. If you are interested in forking the project to continue to develop and maintaining it, we would be happy to cross-link and promote your forked project. Please ping us at Gitter to discuss it.

Building

Build and skip all tests:

./mvnw clean package -DskipTests

Or build project and run tests(tests take long time to run):

./mvnw clean package

Running the Server

All needed artifacts are packaged in a distribution file spring-cloud-dataflow-server-yarn-dist/target/spring-cloud-dataflow-server-yarn-dist-1.0.0.BUILD-SNAPSHOT.zip.

Unzip the dist

# unzip spring-cloud-dataflow-server-yarn-dist-1.0.0.BUILD-SNAPSHOT.zip
# find .
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/bin
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/bin/dataflow-server-yarn
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/bin/dataflow-shell
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/bin/dataflow-server-yarn-h2
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/bin/dataflow-server-yarn-cli
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/lib
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/lib/spring-cloud-deployer-yarn-tasklauncherappmaster-1.0.0.BUILD-SNAPSHOT.jar
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/lib/spring-cloud-dataflow-server-yarn-client-1.0.0.BUILD-SNAPSHOT.jar
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/lib/spring-cloud-deployer-yarn-appdeployerappmaster-1.0.0.BUILD-SNAPSHOT.jar
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/lib/spring-cloud-dataflow-server-yarn-h2-1.0.0.BUILD-SNAPSHOT.jar
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/lib/spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT.jar
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/lib/spring-cloud-dataflow-shell-1.0.0.BUILD-SNAPSHOT.jar
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/config
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/config/servers.yml

Configure servers.yml for hadoop and redis settings.

Start server:

spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/bin/dataflow-server-yarn

Start the Data Flow Shell:

spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/bin/dataflow-shell

Create a stream:

dataflow:>stream create --name ticktock --definition "time|hdfs --rollover=100" --deploy
dataflow:>stream destroy --name ticktock

Killing a running application from the command line:

spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/bin/dataflow-server-yarn-cli shell


$ submitted
  APPLICATION ID                  USER          NAME                            QUEUE    TYPE      STARTTIME       FINISHTIME  STATE    FINALSTATUS  ORIGINAL TRACKING URL
  ------------------------------  ------------  ------------------------------  -------  --------  --------------  ----------  -------  -----------  --------------------------
  application_1447944262603_0003  jvalkealahti  spring-cloud-dataflow-yarn-app  default  DATAFLOW  19/11/15 15:49  N/A         RUNNING  UNDEFINED    http://172.16.14.143:45566

$ kill -a application_1447944262603_0003
Kill request for application_1447944262603_0003 sent