Spring Cloud Data Flow #jsug #springday

1. ‹#›© 2016 Pivotal Software, Inc. All rights reserved. ‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Data Microservices with
Spring Cloud Stream, Task,
and Data Flow
Toshiaki Maki (@making)
2016-11-18 Spring Day 2016
#springday #jsug

2. © 2016 Pivotal Software, Inc. All rights reserved.
Who am I ?
• Toshiaki Maki (@making) http://blog.ik.am
• Sr. Solutions Architect @Pivotal
• Spring Framework enthusiast
bit.ly/hajiboot2

3. © 2016 Pivotal Software, Inc. All rights reserved.
Input
Motivation
HTTP
TCP
S3
JDBC
Rabbit
JMS
Twitter
Syslog
Output
HDFS
Cassandra
HAWQ
Greenplum
JDBC
S3
TCP
Gemfire
❓❔❓
• Real Time Analysis
• Data Ingestion
• ETL Process

4. © 2016 Pivotal Software, Inc. All rights reserved.
Spring XD (eXtreme Data)

5. © 2016 Pivotal Software, Inc. All rights reserved.
Spring XD (eXtreme Data)
🤔

6. © 2016 Pivotal Software, Inc. All rights reserved.
Spring XD Architecture
Container Container
gpfdist
Cassandra
jms
http
ZooKeeper
Message Broker
XD Admin
stream1 = http | cassandra
stream2 = jms | gpfdist
On Metal/VMs

7. © 2016 Pivotal Software, Inc. All rights reserved.
Spring XD Architecture
Container Container
gpfdist
Cassandra
jms
http
ZooKeeper
Message Broker
XD Admin
stream1 = http | cassandra
stream2 = jms | gpfdist
On Metal/VMs
😇

8. © 2016 Pivotal Software, Inc. All rights reserved.
Spring XD Architecture
Container Container
gpfdist
Cassandra
jms
http
ZooKeeper
Message Broker
XD Admin
stream1 = http | cassandra
stream2 = jms | gpfdist
On Metal/VMs

9. © 2016 Pivotal Software, Inc. All rights reserved.
Modern Platforms
Cloud Foundry YARN Kubernetes Mesos
Docker Swarm Nomad OpenShift

10. © 2016 Pivotal Software, Inc. All rights reserved.
Cloud Native Redesign
•From multiple modules embedded in
a container to standalone
executable applications
•From our own runtime to delegating
to existing modern platforms

11. © 2016 Pivotal Software, Inc. All rights reserved.
Data Microservices
$ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' |
tr -d '[:punct:]' |
grep -v '[^a-z]‘ |
sort | uniq -c | sort -rn | head

12. © 2016 Pivotal Software, Inc. All rights reserved.
Data Microservices
$ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' |
tr -d '[:punct:]' |
grep -v '[^a-z]‘ |
sort | uniq -c | sort -rn | head

13. © 2016 Pivotal Software, Inc. All rights reserved.
Data Microservices
$ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' |
tr -d '[:punct:]' |
grep -v '[^a-z]‘ |
sort | uniq -c | sort -rn | head
Microservice for each data processing

14. © 2016 Pivotal Software, Inc. All rights reserved.
Data Microservices
$ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' |
tr -d '[:punct:]' |
grep -v '[^a-z]‘ |
sort | uniq -c | sort -rn | head
Microservice for each data processing

15. © 2016 Pivotal Software, Inc. All rights reserved.
Data Microservices
$ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' |
tr -d '[:punct:]' |
grep -v '[^a-z]‘ |
sort | uniq -c | sort -rn | head
Microservice for each data processing
bound with
Message Brokers

16. © 2016 Pivotal Software, Inc. All rights reserved.
Data Microservices
$ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' |
tr -d '[:punct:]' |
grep -v '[^a-z]‘ |
sort | uniq -c | sort -rn | head
Microservice for each data processing
bound with
Message Brokers

17. © 2016 Pivotal Software, Inc. All rights reserved.
Data Microservices
$ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' |
tr -d '[:punct:]' |
grep -v '[^a-z]‘ |
sort | uniq -c | sort -rn | head
Microservice for each data processing
bound with
Message Brokers
on the modern platform

18. © 2016 Pivotal Software, Inc. All rights reserved.
Data Microservices
$ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' |
tr -d '[:punct:]' |
grep -v '[^a-z]‘ |
sort | uniq -c | sort -rn | head
Microservice for each data processing
bound with
Message Brokers
on the modern platform

19. © 2016 Pivotal Software, Inc. All rights reserved.
Data Microservices
$ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' |
tr -d '[:punct:]' |
grep -v '[^a-z]‘ |
sort | uniq -c | sort -rn | head
Microservice for each data processing
bound with
Message Brokers
on the modern platform

20. © 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Data Flow
• Microservices-based Distributed Data Pipelines
•Long Lived Stream Applications
•Short Lived Task Applications

21. © 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Data Flow
• Microservices-based Distributed Data Pipelines
•Long Lived Stream Applications
•Short Lived Task Applications
Spring Cloud Stream

22. © 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Data Flow
• Microservices-based Distributed Data Pipelines
•Long Lived Stream Applications
•Short Lived Task Applications
Spring Cloud Stream
Spring Cloud Task

23. © 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Data Flow
• Microservices-based Distributed Data Pipelines
•Long Lived Stream Applications
•Short Lived Task Applications
Spring Cloud Stream
Spring Cloud Task
Orchestration
Layer

24. © 2016 Pivotal Software, Inc. All rights reserved.
Structure
Spring Cloud Data Flow
Spring Cloud Deployer (SPI)
Spring Cloud Stream Spring Cloud Task
Spring Integration Spring Boot Spring Batch

25. © 2016 Pivotal Software, Inc. All rights reserved.
Structure
Spring Cloud Data Flow
Spring Cloud Deployer (SPI)
Spring Cloud Stream Spring Cloud Task
Spring Integration Spring Boot Spring Batch
• Spring Cloud Deployer Local
• Spring Cloud Deployer Cloud Foundry
• Spring Cloud Deployer Yarn
• Spring Cloud Deployer Kubernetes
• Spring Cloud Deployer Mesos

26. © 2016 Pivotal Software, Inc. All rights reserved.
SCDF Deployment Platform
Spring Cloud Data Flow Server
Deployer SPI
REST API
SCDFShell SCDF Flo

27. © 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Data Flow Architecture
gpfdist
cassandra
jms
http
stream1 = http | cassandra
stream2 = jms | gpfdist
Message Broker
Data Flow Server DB
Platform Runtime

28. © 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Data Flow Architecture
gpfdist
cassandra
jms
http
stream1 = http | cassandra
stream2 = jms | gpfdist
Message Broker
Data Flow Server DB
Platform Runtime

29. ‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Stream

30. © 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Stream
• Event-driven microservice framework
• Built on battle-tested components (Spring Boot / Spring
Integration)
• Opinionated primitives for streaming applications
• Persistent Pub/Sub
• Consumer Groups
• Partitioning Support
• Pluggable messaging middleware bindings
source | processor | sink

31. © 2016 Pivotal Software, Inc. All rights reserved.
JJUG CCC in 2 weeks

32. © 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Stream Applications
Twitter Stream
Cassandra
java -jar twittersource.jar --server.port=8080
--consumerKey=XYZ --consumerSecret=ABC
--spring.cloud.stream.bindings.
output.destination=ingest
Source
Sink java -jar cassandrasink.jar --server.port=8081
--spring.cassandra.keyspace=tweet
--spring.cloud.stream.bindings.
input.destination=ingest

33. © 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Stream Applications
Twitter Stream
Cassandra
java -jar twittersource.jar --server.port=8080
--consumerKey=XYZ --consumerSecret=ABC
--spring.cloud.stream.bindings.
output.destination=ingest
Source
Sink java -jar cassandrasink.jar --server.port=8081
--spring.cassandra.keyspace=tweet
--spring.cloud.stream.bindings.
input.destination=ingest
Twitter
Stream Cassandraingest

34. © 2016 Pivotal Software, Inc. All rights reserved.
Message Binders
• @EnableBinding
• Binder Implementations
• Production-Ready
• Rabbit MQ
• Apache Kafka
• Experimental
• JMS
• Google PubSub

35. © 2016 Pivotal Software, Inc. All rights reserved.
Programming Model (Sink)
@SpringBootApplication
@EnableBinding(Sink.class)
public class DemoSinkApp {
@StreamListener(Sink.INPUT)
void receive(Message<String> message) {
System.out.println("Received " + message);
}
public static void main(String[] args) {
SpringApplication.run(DemoSinkApp.class, args);
}
}

36. © 2016 Pivotal Software, Inc. All rights reserved.
Sink
Properties (Sink)
spring.cloud.stream.bindings.input.destination=demo-strm
demo-
strm
input

37. © 2016 Pivotal Software, Inc. All rights reserved.
Programming Model (Source)
@SpringBootApplication @RestController
@EnableBinding(Source.class)
public class DemoSourceApp {
@Autowired Source source;
@GetMapping void send(@RequestParam String text) {
source.output()
.send(MessageBuilder.withPayload(text).build());
}
public static void main(String[] args) {
SpringApplication.run(DemoSourceApp.class, args);
}
}

38. © 2016 Pivotal Software, Inc. All rights reserved.
Programming Model (Source)
@SpringBootApplication @RestController
@EnableBinding(Source.class)
public class DemoSourceApp {
@Bean
@InboundChannelAdapter(channel = Source.OUTPUT, poller =
@Poller(fixedDelay = "1000", maxMessagesPerPoll = "1"))
MessageSource<String> source() {
return () -> MessageBuilder.withPayload("Hi").build());
}
public static void main(String[] args) {
SpringApplication.run(DemoSourceApp.class, args);
}}

39. © 2016 Pivotal Software, Inc. All rights reserved.
Properties (Source)
spring.cloud.stream.bindings.output.destination=demo-strm
demo-
strm
Source
output

40. © 2016 Pivotal Software, Inc. All rights reserved.
Binder (RabbitMQ)
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-stream-rabbit</artifactId>
</dependency>
demo-
strm

41. © 2016 Pivotal Software, Inc. All rights reserved.
Binder (Apache Kafka)
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-stream-kafka</artifactId>
</dependency>
demo-
strm

42. © 2016 Pivotal Software, Inc. All rights reserved.
Sink
Pipeline
demo-
strm
input
Source
output
source | sink

43. © 2016 Pivotal Software, Inc. All rights reserved.
Programming Model (Processor)
@SpringBootApplication
@EnableBinding(Processor.class)
public class DemoProcessorApp {
@StreamListener(Processor.INPUT)
@SendTo(Processor.OUTPUT)
void receive(String text) {
return "[[" + text + "]]";
}
public static void main(String[] args) {
SpringApplication.run(DemoProcessorApp.class, args);
}
}

44. © 2016 Pivotal Software, Inc. All rights reserved.
Properties (Processor)
spring.cloud.stream.bindings.output.destination=my-source
spring.cloud.stream.bindings.input.destination=my-source
spring.cloud.stream.bindings.output.destination=my-proc
spring.cloud.stream.bindings.input.destination=my-proc
Source
Processor
Sink

45. © 2016 Pivotal Software, Inc. All rights reserved.
Pipeline
my-
source
Source
output source | processor | sink
Processor
output
input
my-proc
Sink
input

46. © 2016 Pivotal Software, Inc. All rights reserved.
Reactive API Support
@SpringBootApplication
@EnableBinding(Processor.class)
public class DemoProcessorRxApp {
@StreamListener @Output(Processor.OUTPUT)
public Flux<String> receive(@Input(Processor.INPUT)
Flux<String> stream) {
return stream.map(text -> "[[" + text + "]]");
}
public static void main(String[] args) {
SpringApplication.run(DemoProcessorRxApp.class, args);
}
}

47. © 2016 Pivotal Software, Inc. All rights reserved.
Reactive API Support
@StreamListener @Output(Processor.OUTPUT)
public Flux<AverageData> receive(@Input(Processor.INPUT)
Flux<SensorData> stream) {
return stream.window(Duration.ofSecond(20),
Duration.ofSecond(10))
.flatMap(win -> win.groupBy(sensor -> sensor.id))
.flatMap(group -> calcAverage(group));
}

48. © 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Stream App Starters
Source Processor Sink
file
ftp
gemfire
gemfire-cq
http
jdbc
jms
load-generator
loggregator
mail
mongodb
rabbit
s3
sftp
syslog
tcp
tcp-client
time
trigger
triggertask
twitterstream
bridge
filter
groovy-filter
groovy-
transform
httpclient
pommel
scriptable-
transform
splitter
tcp-client
transform
aggregate-
counter
cassandra
counter
field-value-
counter
file
ftp
gemfire
gpfdist
hfs
hdfs-dataset
jdbc
log
rabbit
redis-pubsub
router
s3
sftp
task-launcher-
local
task-lancher-
yarn
tcp
throughput
web socket
http://cloud.spring.io/spring-cloud-stream-app-starters/

49. © 2016 Pivotal Software, Inc. All rights reserved.
Stream Orchestration in SCDF
ingest = twitterstream | cassandra
dataflow:> stream create --name=ingest
--definition="twitterstream | cassandra"
dataflow:> stream deploy --name=ingest
|

50. © 2016 Pivotal Software, Inc. All rights reserved.
Import apps
dataflow:> app import --uri http://bit.ly/1-0-4-
GA-stream-applications-rabbit-maven
rabbit maven
kafka docker
stream-applications-
http://cloud.spring.io/spring-cloud-stream-app-starters/

51. © 2016 Pivotal Software, Inc. All rights reserved.
Register your own apps
dataflow:> app register --name foo --type source
--uri http://example.com/foo-source-1.0.jar
dataflow:> app register --name foo --type source
--uri maven://com.example:foo-source:1.0
dataflow:> app register --name foo --type source
--uri docker:myapps/foo-source:1.0
dataflow:> app register --name foo --type source
--uri file:///tmp/foo-source:1.0

52. © 2016 Pivotal Software, Inc. All rights reserved.
Instance Count
dataflow:> stream create --name=s1
--definition="http | work | hdfs"
dataflow:> stream deploy --name=s1 --
properties="app.http.count=2,app.work.count=3,app
.hdfs.count=4"

53. © 2016 Pivotal Software, Inc. All rights reserved.
Instance Count
http
work
hdfs
http
work
work
hdfs
hdfs
hdfs
LoadBalancer

54. © 2016 Pivotal Software, Inc. All rights reserved.
Resource Management
dataflow:> stream deploy --name=s1 --
properties="app.work.spring.cloud.deployer.cloudf
oundry.memory=2048"

55. © 2016 Pivotal Software, Inc. All rights reserved.
Resource Management
http
http
work
work
work
hdfs
hdfs
hdfs
hdfs

56. © 2016 Pivotal Software, Inc. All rights reserved.
Spring Flo http://projects.spring.io/spring-flo/

57. © 2016 Pivotal Software, Inc. All rights reserved.
Stream Core Features
•Persistent Pub-Sub
•Consumer Group
•Partitioning Support

58. © 2016 Pivotal Software, Inc. All rights reserved.
Persistent Pub-Sub
HTTP Average Top Ns1.http s1.ave
Message Broker

59. © 2016 Pivotal Software, Inc. All rights reserved.
Persistent Pub-Sub
HTTP Average
HDFS
Top Ns1.http s1.ave
Message Broker

60. © 2016 Pivotal Software, Inc. All rights reserved.
Persistent Pub-Sub
HTTP Average
HDFS
Top N
Fault
Detection
s1.http s1.ave
Message Broker

61. © 2016 Pivotal Software, Inc. All rights reserved.
Persistent Pub-Sub
HTTP Average
HDFS
Top N
Fault
Detection
s1.http s1.ave
Message Broker
{"id":1, "temperature":38}

62. © 2016 Pivotal Software, Inc. All rights reserved.
Persistent Pub-Sub
HTTP Average
HDFS
Top N
Fault
Detection
s1.http s1.ave
Message Broker
{"id":1, "temperature":38}{"id":1, "temperature":38}

63. © 2016 Pivotal Software, Inc. All rights reserved.
Persistent Pub-Sub
HTTP Average
HDFS
Top N
Fault
Detection
s1.http s1.ave
Message Broker
{"id":1, "temperature":38}
{"id":1, "temperature":38}
{"id":1, "temperature":38}

64. © 2016 Pivotal Software, Inc. All rights reserved.
Persistent Pub-Sub
HTTP Average
HDFS
Top N
Fault
Detection
s1.http s1.ave
Message Broker
{"id":1, "temperature":
{"id":1, "temperature":38} {"id":1, "temperature":38}

65. © 2016 Pivotal Software, Inc. All rights reserved.
HTTP Average
HDFS
Top N
Fault
Detection
s1.http s1.ave
Message Broker
Average
Average
HDFS
HDFS

66. © 2016 Pivotal Software, Inc. All rights reserved.
HTTP Average
HDFS
Top N
Fault
Detection
s1.http s1.ave
Message Broker
{"id":1, "temperature":38}
Average
Average
HDFS
HDFS

67. © 2016 Pivotal Software, Inc. All rights reserved.
HTTP Average
HDFS
Top N
Fault
Detection
s1.http s1.ave
Message Broker
{"id":1, "temperature":38}{"id":1, "temperature":38}
Average
Average
HDFS
HDFS

68. © 2016 Pivotal Software, Inc. All rights reserved.
HTTP Average
HDFS
Top N
Fault
Detection
s1.http s1.ave
Message Broker
{"id":1, "temperature":38}
{"id":1, "temperature":38}
Average
Average
HDFS
HDFS
{"id":1, "temperature":38}
{"id":1, "temperature":38}
{"id":1, "temperature":38}
{"id":1, "temperature":38}

69. © 2016 Pivotal Software, Inc. All rights reserved.
HTTP Average
HDFS
Top N
Fault
Detection
s1.http s1.ave
Message Broker
{"id":1, "temperature":38}
{"id":1, "temperature":38}
Average
Average
HDFS
HDFS
{"id":1, "temperature":38}
{"id":1, "temperature":38}
{"id":1, "temperature":38}
{"id":1, "temperature":38}
😩

70. © 2016 Pivotal Software, Inc. All rights reserved.
HTTP Average
HDFS
Top N
Fault
Detection
s1.http s1.ave
Message Broker
{"id":1, "temperature":38}
{"id":1, "temperature":38}
Average
Average
HDFS
HDFS
{"id":1, "temperature":38}
{"id":1, "temperature":38}
{"id":1, "temperature":38}
{"id":1, "temperature":38}
😩
Consumer Group

71. © 2016 Pivotal Software, Inc. All rights reserved.
HTTP Average
HDFS
Top N
Fault
Detection
s1.http s1.ave
Message Broker
{"id":1, "temperature":38}
{"id":1, "temperature":38}
Average
Average
HDFS
HDFS
{"id":1, "temperature":38}
{"id":1, "temperature":38}
{"id":1, "temperature":38}
{"id":1, "temperature":38}
😩
spring.cloud.stream.bindings.<channelName>.group=ave
Spring Cloud Stream
group is configured by default 😁
Spring Cloud Data Flow
Consumer Group

72. © 2016 Pivotal Software, Inc. All rights reserved.
Consumer Group
HTTP Average
HDFS
Top N
Fault
Detection
s1.http s1.ave
Message Broker
Average
Average
HDFS
HDFS
group=ave
group=hdfs

73. © 2016 Pivotal Software, Inc. All rights reserved.
Consumer Group
HTTP Average
HDFS
Top N
Fault
Detection
s1.http s1.ave
Message Broker
Average
Average
HDFS
HDFS
group=ave
group=hdfs
{"id":1, "temperature":38}

74. © 2016 Pivotal Software, Inc. All rights reserved.
Consumer Group
HTTP Average
HDFS
Top N
Fault
Detection
s1.http s1.ave
Message Broker
Average
Average
HDFS
HDFS
group=ave
group=hdfs
{"id":1, "temperature":38}{"id":1, "temperature":38}

75. © 2016 Pivotal Software, Inc. All rights reserved.
Consumer Group
HTTP Average
HDFS
Top N
Fault
Detection
s1.http s1.ave
Message Broker
Average
Average
HDFS
HDFS
group=ave
group=hdfs
{"id":1, "temperature":38}
{"id":1, "temperature":38}

76. © 2016 Pivotal Software, Inc. All rights reserved.
Consumer Group
HTTP Average
HDFS
Top N
Fault
Detection
s1.http s1.ave
Message Broker
Average
Average
HDFS
HDFS
group=ave
group=hdfs
{"id":1, "temperature":38}
{"id":1, "temperature":38} 🤓

77. © 2016 Pivotal Software, Inc. All rights reserved.
Consumer Group
HTTP Average
HDFS
Top N
Fault
Detection
s1.http s1.ave
Message Broker
Average
Average
HDFS
HDFS
group=ave
group=hdfs
{"id":1, "temperature":38}
{"id":1, "temperature":38} 🤓consumer group subscriptions
are durable 😁

78. © 2016 Pivotal Software, Inc. All rights reserved.
HTTP
Average
s1.http
Average

79. © 2016 Pivotal Software, Inc. All rights reserved.
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}
{"id":2, "temperature":42}
{"id":1, "temperature":37}

80. © 2016 Pivotal Software, Inc. All rights reserved.
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}
{"id":2, "temperature":42}
{"id":1, "temperature":37}

81. © 2016 Pivotal Software, Inc. All rights reserved.
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}
{"id":2, "temperature":42}
{"id":1, "temperature":37}

82. © 2016 Pivotal Software, Inc. All rights reserved.
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}
{"id":2, "temperature":42}
{"id":1, "temperature":37}

83. © 2016 Pivotal Software, Inc. All rights reserved.
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}
{"id":2, "temperature":42}
{"id":1, "temperature":37}

84. © 2016 Pivotal Software, Inc. All rights reserved.
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}
{"id":2, "temperature":42}
{"id":1, "temperature":37}
😩

85. © 2016 Pivotal Software, Inc. All rights reserved.
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}
{"id":2, "temperature":42}
{"id":1, "temperature":37}
😩
Partitioning Support(Stateful Stream)

86. © 2016 Pivotal Software, Inc. All rights reserved.
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}
{"id":2, "temperature":42}
{"id":1, "temperature":37}
😩stream deploy --name=xxxx --properties=
"app.http.producer.partitionKeyExpression=payload.id"
spring.cloud.stream.bindings.<channelName>.producer.par
titionKeyExpression=payload.id
Spring Cloud Stream
Spring Cloud Data Flow
Partitioning Support(Stateful Stream)

87. © 2016 Pivotal Software, Inc. All rights reserved.
Partitioning Support(Stateful Stream)
HTTP
Average
s1.http
Average

88. © 2016 Pivotal Software, Inc. All rights reserved.
Partitioning Support(Stateful Stream)
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}
{"id":2, "temperature":42}
{"id":1, "temperature":37}

89. © 2016 Pivotal Software, Inc. All rights reserved.
Partitioning Support(Stateful Stream)
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}
{"id":2, "temperature":42}
{"id":1, "temperature":37}

90. © 2016 Pivotal Software, Inc. All rights reserved.
Partitioning Support(Stateful Stream)
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}
{"id":2, "temperature":42}
{"id":1, "temperature":37}

91. © 2016 Pivotal Software, Inc. All rights reserved.
Partitioning Support(Stateful Stream)
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}
{"id":2, "temperature":42}
{"id":1, "temperature":37}

92. © 2016 Pivotal Software, Inc. All rights reserved.
Partitioning Support(Stateful Stream)
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}
{"id":2, "temperature":42}
{"id":1, "temperature":37}

93. © 2016 Pivotal Software, Inc. All rights reserved.
Partitioning Support(Stateful Stream)
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}
{"id":2, "temperature":42}
{"id":1, "temperature":37}
🤓

94. © 2016 Pivotal Software, Inc. All rights reserved.
Simple Real Time Analytics
tweets = twitterstream | hdfs
analytics = :ingest.twitterstream > field-value-counter --fieldName=lang
HTTP
s1.http
HDFS COUNTER
Data Flow Server
REST API

95. ‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Demo (Twitter Stream!)

96. © 2016 Pivotal Software, Inc. All rights reserved.

97. ‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Task

98. © 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Task
• Spring Boot based framework for short lived task
• Spring Boot's CommandLineRunner or Spring Batch
• Result of each process persists in Task Repository
• Well Integrated with Spring Batch (Job Repository)

99. © 2016 Pivotal Software, Inc. All rights reserved.
Programming Model (CLR)
@SpringBootApplication
@EnableTask
public class DemoTaskApp {
@Bean
CommandLineRunner clr() {
return args -> System.out.println("Task!");
}
public static void main(String[] args) {
SpringApplication.run(DemoTaskApp.class, args);
}
} spring.application.name=hello

100. © 2016 Pivotal Software, Inc. All rights reserved.
Task Execution List
Task Name ID Start Time End Time Exit Code
hello 3 Tue Nov 15 16:49:08
JST 2016
Tue Nov 15 16:49:08
JST 2016
0
hello 2 Tue Nov 15 16:48:54
JST 2016
Tue Nov 15 16:48:55
JST 2016
0
hello 1 Tue Nov 15 16:48:23
JST 2016
Tue Nov 15 16:48:23
JST 2016
0

101. © 2016 Pivotal Software, Inc. All rights reserved.
Programming Model (Spring Batch)
@SpringBootApplication @EnableBatchProcessing
@EnableTask
public class DemoBatchApp {
@Autowired JobBuilderFactory jobBuilderFactory;
@Autowired StepBuilderFactory stepBuilderFactory;
@Bean Step step1() { /* ... */ }
@Bean Step step2() { /* ... */ }
@Bean Job job() { return jobBuilderFactory.get("job")
.start(step1()).next(step2()).build()}
public static void main(String[] args) {
SpringApplication.run(DemoTaskApp.class, args);
}}
https://github.com/making/cf-spring-batch-demo
spring.application.name=hello-batch

102. © 2016 Pivotal Software, Inc. All rights reserved.
Task Execution List
Task Name ID Start Time End Time Exit Code
hello-
batch
5 Tue Nov 15 17:28:49
JST 2016
Tue Nov 15 17:28:49
JST 2016
0
hello-
batch
4 Tue Nov 15 17:27:56
JST 2016
Tue Nov 15 17:27:57
JST 2016
0

103. © 2016 Pivotal Software, Inc. All rights reserved.
Job Execution List
ID Task ID Start Time Step Execution Count Definition Status
2 5 Tue Nov 15
17:28:49 JST 2016
2 Created
1 4 Tue Nov 15
17:27:56 JST 2016
2 Created

104. © 2016 Pivotal Software, Inc. All rights reserved.
Step Execution List
ID Step Name Job Exec Id Start Time End Time Status
1 step1 1 Tue Nov 15
17:28:49 JST
2016
Tue Nov 15
17:28:49 JST
2016
COMPLETED
2 step2 1 Tue Nov 15
17:27:56 JST
2016
Tue Nov 15
17:27:57 JST
2016
COMPLETED

105. © 2016 Pivotal Software, Inc. All rights reserved.
Task Orchestration in SCDF
>task create hello --definition="timestamp --format=¥"yyyy¥""
>task launch hello
timestamp
Data Flow Server DB
Task Name
Start Time
End Time
Exit Code
Exit Message
Last Updated Time
Parameters
Message Broker

106. © 2016 Pivotal Software, Inc. All rights reserved.
Task Execution List

107. © 2016 Pivotal Software, Inc. All rights reserved.
Job Execution List

108. © 2016 Pivotal Software, Inc. All rights reserved.
Job Execution Details

109. © 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Data Flow -> Task
http | tasklaunchrequest --uri=... | task-lancher
timestamp
Data Flow Server DB
Message Broker

110. ‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Demo (Scalable Tasks)

111. © 2016 Pivotal Software, Inc. All rights reserved.
Pivotal Cloud Foundry
PDF PDF
PDF PDF
S3
Spring Cloud Data Flow
Task
Task
Task
Task
PDF
PDF
Spring Cloud Task
Spring Batch
index
index
source sink

112. ‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Getting Started with SCDF
on Cloud Foundry

113. © 2016 Pivotal Software, Inc. All rights reserved.
Install PCF Dev
• https://docs.pivotal.io/pcf-dev
• Cloud Foundry on your laptop
• Included
• Redis / RabbitMQ / MySQL
• Spring Cloud Services
• Install with cf dev start

114. © 2016 Pivotal Software, Inc. All rights reserved.
cf create-service p-mysql 512mb df-mysql
cf create-service p-rabbitmq standard df-rebbitmq
cf create-service p-redis shared-vm df-redis
cf push dataflow-server -p spring-cloud-dataflow-
server-cloudfoundry-1.0.1.RELEASE.jar
Deploy SCDF Server on PCF Dev

115. © 2016 Pivotal Software, Inc. All rights reserved.
Tutorial
• 📗 https://blog.ik.am/entries/396
• 💎 https://github.com/making-demo-scdf/spring-cloud-
dataflow-cookbook

116. © 2016 Pivotal Software, Inc. All rights reserved.
Upcoming Features
• Some ‘porting’ from XD
• Batch Job DSL + Designer
• Role based access
• Looking forward
• Spring Cloud Sleuth
• JavaDSL
• In-place application version upgrades with Spinnaker
• Application Groups
• Polyglot
• Expanded analytics with Redis and Python/R ecosystem
• More provided apps/tasks
•SCDF Tile for Pivotal Cloud Foundry

117. © 2016 Pivotal Software, Inc. All rights reserved.
Thanks!
• References
• http://www.slideshare.net/SpringCentral/data-microservices-in-t
cloud ✨
• https://cloud.spring.io/spring-cloud-dataflow/
• http://cloud.spring.io/spring-cloud-stream/
• http://cloud.spring.io/spring-cloud-stream-app-starters/
• https://github.com/spring-cloud/spring-cloud-task/
• http://cloud.spring.io/spring-cloud-dataflow-server-cloudfoundry