SlideShare a Scribd company logo
‹#›© 2016 Pivotal Software, Inc. All rights reserved. ‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Data Microservices with
Spring Cloud Stream, Task,
and Data Flow
Toshiaki Maki (@making)
2016-11-18 Spring Day 2016
#springday #jsug
© 2016 Pivotal Software, Inc. All rights reserved.
Who am I ?
• Toshiaki Maki (@making) http://blog.ik.am
• Sr. Solutions Architect @Pivotal
• Spring Framework enthusiast
bit.ly/hajiboot2
© 2016 Pivotal Software, Inc. All rights reserved.
Input
Motivation
HTTP
TCP
S3
JDBC
Rabbit
JMS
Twitter
Syslog
Output
HDFS
Cassandra
HAWQ
Greenplum
JDBC
S3
TCP
Gemfire
❓❔❓
• Real Time Analysis
• Data Ingestion
• ETL Process
© 2016 Pivotal Software, Inc. All rights reserved.
Spring XD (eXtreme Data)
© 2016 Pivotal Software, Inc. All rights reserved.
Spring XD (eXtreme Data)
🤔
© 2016 Pivotal Software, Inc. All rights reserved.
Spring XD Architecture
Container Container
gpfdist
Cassandra
jms
http
ZooKeeper
Message Broker
XD Admin
stream1 = http | cassandra
stream2 = jms | gpfdist
On Metal/VMs
© 2016 Pivotal Software, Inc. All rights reserved.
Spring XD Architecture
Container Container
gpfdist
Cassandra
jms
http
ZooKeeper
Message Broker
XD Admin
stream1 = http | cassandra
stream2 = jms | gpfdist
On Metal/VMs
😇
© 2016 Pivotal Software, Inc. All rights reserved.
Spring XD Architecture
Container Container
gpfdist
Cassandra
jms
http
ZooKeeper
Message Broker
XD Admin
stream1 = http | cassandra
stream2 = jms | gpfdist
On Metal/VMs
© 2016 Pivotal Software, Inc. All rights reserved.
Modern Platforms
Cloud Foundry YARN Kubernetes Mesos
Docker Swarm Nomad OpenShift
© 2016 Pivotal Software, Inc. All rights reserved.
Cloud Native Redesign
•From multiple modules embedded in
a container to standalone
executable applications
•From our own runtime to delegating
to existing modern platforms
© 2016 Pivotal Software, Inc. All rights reserved.
Data Microservices
$ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' |
tr -d '[:punct:]' |
grep -v '[^a-z]‘ |
sort | uniq -c | sort -rn | head
© 2016 Pivotal Software, Inc. All rights reserved.
Data Microservices
$ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' |
tr -d '[:punct:]' |
grep -v '[^a-z]‘ |
sort | uniq -c | sort -rn | head
© 2016 Pivotal Software, Inc. All rights reserved.
Data Microservices
$ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' |
tr -d '[:punct:]' |
grep -v '[^a-z]‘ |
sort | uniq -c | sort -rn | head
Microservice for each data processing
© 2016 Pivotal Software, Inc. All rights reserved.
Data Microservices
$ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' |
tr -d '[:punct:]' |
grep -v '[^a-z]‘ |
sort | uniq -c | sort -rn | head
Microservice for each data processing
© 2016 Pivotal Software, Inc. All rights reserved.
Data Microservices
$ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' |
tr -d '[:punct:]' |
grep -v '[^a-z]‘ |
sort | uniq -c | sort -rn | head
Microservice for each data processing
bound with
Message Brokers
© 2016 Pivotal Software, Inc. All rights reserved.
Data Microservices
$ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' |
tr -d '[:punct:]' |
grep -v '[^a-z]‘ |
sort | uniq -c | sort -rn | head
Microservice for each data processing
bound with
Message Brokers
© 2016 Pivotal Software, Inc. All rights reserved.
Data Microservices
$ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' |
tr -d '[:punct:]' |
grep -v '[^a-z]‘ |
sort | uniq -c | sort -rn | head
Microservice for each data processing
bound with
Message Brokers
on the modern platform
© 2016 Pivotal Software, Inc. All rights reserved.
Data Microservices
$ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' |
tr -d '[:punct:]' |
grep -v '[^a-z]‘ |
sort | uniq -c | sort -rn | head
Microservice for each data processing
bound with
Message Brokers
on the modern platform
© 2016 Pivotal Software, Inc. All rights reserved.
Data Microservices
$ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' |
tr -d '[:punct:]' |
grep -v '[^a-z]‘ |
sort | uniq -c | sort -rn | head
Microservice for each data processing
bound with
Message Brokers
on the modern platform
© 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Data Flow
• Microservices-based Distributed Data Pipelines
•Long Lived Stream Applications
•Short Lived Task Applications
© 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Data Flow
• Microservices-based Distributed Data Pipelines
•Long Lived Stream Applications
•Short Lived Task Applications
Spring Cloud Stream
© 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Data Flow
• Microservices-based Distributed Data Pipelines
•Long Lived Stream Applications
•Short Lived Task Applications
Spring Cloud Stream
Spring Cloud Task
© 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Data Flow
• Microservices-based Distributed Data Pipelines
•Long Lived Stream Applications
•Short Lived Task Applications
Spring Cloud Stream
Spring Cloud Task
Orchestration
Layer
© 2016 Pivotal Software, Inc. All rights reserved.
Structure
Spring Cloud Data Flow
Spring Cloud Deployer (SPI)
Spring Cloud Stream Spring Cloud Task
Spring Integration Spring Boot Spring Batch
© 2016 Pivotal Software, Inc. All rights reserved.
Structure
Spring Cloud Data Flow
Spring Cloud Deployer (SPI)
Spring Cloud Stream Spring Cloud Task
Spring Integration Spring Boot Spring Batch
• Spring Cloud Deployer Local
• Spring Cloud Deployer Cloud Foundry
• Spring Cloud Deployer Yarn
• Spring Cloud Deployer Kubernetes
• Spring Cloud Deployer Mesos
© 2016 Pivotal Software, Inc. All rights reserved.
SCDF Deployment Platform
Spring Cloud Data Flow Server
Deployer SPI
REST API
SCDFShell SCDF Flo
© 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Data Flow Architecture
gpfdist
cassandra
jms
http
stream1 = http | cassandra
stream2 = jms | gpfdist
Message Broker
Data Flow Server DB
Platform Runtime
© 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Data Flow Architecture
gpfdist
cassandra
jms
http
stream1 = http | cassandra
stream2 = jms | gpfdist
Message Broker
Data Flow Server DB
Platform Runtime
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Stream
© 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Stream
• Event-driven microservice framework
• Built on battle-tested components (Spring Boot / Spring
Integration)
• Opinionated primitives for streaming applications
• Persistent Pub/Sub
• Consumer Groups
• Partitioning Support
• Pluggable messaging middleware bindings
source | processor | sink
© 2016 Pivotal Software, Inc. All rights reserved.
JJUG CCC in 2 weeks
© 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Stream Applications
Twitter Stream
Cassandra
java -jar twittersource.jar --server.port=8080
--consumerKey=XYZ --consumerSecret=ABC
--spring.cloud.stream.bindings.
output.destination=ingest
Source
Sink java -jar cassandrasink.jar --server.port=8081
--spring.cassandra.keyspace=tweet
--spring.cloud.stream.bindings.
input.destination=ingest
© 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Stream Applications
Twitter Stream
Cassandra
java -jar twittersource.jar --server.port=8080
--consumerKey=XYZ --consumerSecret=ABC
--spring.cloud.stream.bindings.
output.destination=ingest
Source
Sink java -jar cassandrasink.jar --server.port=8081
--spring.cassandra.keyspace=tweet
--spring.cloud.stream.bindings.
input.destination=ingest
Twitter
Stream Cassandraingest
© 2016 Pivotal Software, Inc. All rights reserved.
Message Binders
• @EnableBinding
• Binder Implementations
• Production-Ready
• Rabbit MQ
• Apache Kafka
• Experimental
• JMS
• Google PubSub
© 2016 Pivotal Software, Inc. All rights reserved.
Programming Model (Sink)
@SpringBootApplication

@EnableBinding(Sink.class)

public class DemoSinkApp {
@StreamListener(Sink.INPUT)

void receive(Message<String> message) {

System.out.println("Received " + message);

}

public static void main(String[] args) {

SpringApplication.run(DemoSinkApp.class, args);

}

}
© 2016 Pivotal Software, Inc. All rights reserved.
Sink
Properties (Sink)
spring.cloud.stream.bindings.input.destination=demo-strm
demo-
strm
input
© 2016 Pivotal Software, Inc. All rights reserved.
Programming Model (Source)
@SpringBootApplication @RestController
@EnableBinding(Source.class)

public class DemoSourceApp {

@Autowired Source source;
@GetMapping void send(@RequestParam String text) {

source.output()
.send(MessageBuilder.withPayload(text).build());

}

public static void main(String[] args) {

SpringApplication.run(DemoSourceApp.class, args);

}

}
© 2016 Pivotal Software, Inc. All rights reserved.
Programming Model (Source)
@SpringBootApplication @RestController
@EnableBinding(Source.class)

public class DemoSourceApp {
@Bean
@InboundChannelAdapter(channel = Source.OUTPUT, poller =
@Poller(fixedDelay = "1000", maxMessagesPerPoll = "1"))
MessageSource<String> source() {

return () -> MessageBuilder.withPayload("Hi").build());

}

public static void main(String[] args) {

SpringApplication.run(DemoSourceApp.class, args);

}}
© 2016 Pivotal Software, Inc. All rights reserved.
Properties (Source)
spring.cloud.stream.bindings.output.destination=demo-strm
demo-
strm
Source
output
© 2016 Pivotal Software, Inc. All rights reserved.
Binder (RabbitMQ)
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-stream-rabbit</artifactId>
</dependency>
demo-
strm
© 2016 Pivotal Software, Inc. All rights reserved.
Binder (Apache Kafka)
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-stream-kafka</artifactId>
</dependency>
demo-
strm
© 2016 Pivotal Software, Inc. All rights reserved.
Sink
Pipeline
demo-
strm
input
Source
output
source | sink
© 2016 Pivotal Software, Inc. All rights reserved.
Programming Model (Processor)
@SpringBootApplication

@EnableBinding(Processor.class)

public class DemoProcessorApp {
@StreamListener(Processor.INPUT)
@SendTo(Processor.OUTPUT)

void receive(String text) {

return "[[" + text + "]]";

}

public static void main(String[] args) {

SpringApplication.run(DemoProcessorApp.class, args);

}

}
© 2016 Pivotal Software, Inc. All rights reserved.
Properties (Processor)
spring.cloud.stream.bindings.output.destination=my-source
spring.cloud.stream.bindings.input.destination=my-source
spring.cloud.stream.bindings.output.destination=my-proc
spring.cloud.stream.bindings.input.destination=my-proc
Source
Processor
Sink
© 2016 Pivotal Software, Inc. All rights reserved.
Pipeline
my-
source
Source
output source | processor | sink
Processor
output
input
my-proc
Sink
input
© 2016 Pivotal Software, Inc. All rights reserved.
Reactive API Support
@SpringBootApplication

@EnableBinding(Processor.class)

public class DemoProcessorRxApp {
@StreamListener @Output(Processor.OUTPUT)

public Flux<String> receive(@Input(Processor.INPUT)
Flux<String> stream) {

return stream.map(text -> "[[" + text + "]]");

}

public static void main(String[] args) {

SpringApplication.run(DemoProcessorRxApp.class, args);

}

}
© 2016 Pivotal Software, Inc. All rights reserved.
Reactive API Support
@StreamListener @Output(Processor.OUTPUT)

public Flux<AverageData> receive(@Input(Processor.INPUT)
Flux<SensorData> stream) {

return stream.window(Duration.ofSecond(20),
Duration.ofSecond(10))
.flatMap(win -> win.groupBy(sensor -> sensor.id))
.flatMap(group -> calcAverage(group));

}
© 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Stream App Starters
Source Processor Sink
file
ftp
gemfire
gemfire-cq
http
jdbc
jms
load-generator
loggregator
mail
mongodb
rabbit
s3
sftp
syslog
tcp
tcp-client
time
trigger
triggertask
twitterstream
bridge
filter
groovy-filter
groovy-
transform
httpclient
pommel
scriptable-
transform
splitter
tcp-client
transform
aggregate-
counter
cassandra
counter
field-value-
counter
file
ftp
gemfire
gpfdist
hfs
hdfs-dataset
jdbc
log
rabbit
redis-pubsub
router
s3
sftp
task-launcher-
local
task-lancher-
yarn
tcp
throughput
web socket
http://cloud.spring.io/spring-cloud-stream-app-starters/
© 2016 Pivotal Software, Inc. All rights reserved.
Stream Orchestration in SCDF
ingest = twitterstream | cassandra
dataflow:> stream create --name=ingest
--definition="twitterstream | cassandra"
dataflow:> stream deploy --name=ingest
|
© 2016 Pivotal Software, Inc. All rights reserved.
Import apps
dataflow:> app import --uri http://bit.ly/1-0-4-
GA-stream-applications-rabbit-maven
rabbit maven
kafka docker
stream-applications-
http://cloud.spring.io/spring-cloud-stream-app-starters/
© 2016 Pivotal Software, Inc. All rights reserved.
Register your own apps
dataflow:> app register --name foo --type source
--uri http://example.com/foo-source-1.0.jar
dataflow:> app register --name foo --type source
--uri maven://com.example:foo-source:1.0
dataflow:> app register --name foo --type source
--uri docker:myapps/foo-source:1.0
dataflow:> app register --name foo --type source
--uri file:///tmp/foo-source:1.0
© 2016 Pivotal Software, Inc. All rights reserved.
Instance Count
dataflow:> stream create --name=s1
--definition="http | work | hdfs"
dataflow:> stream deploy --name=s1 --
properties="app.http.count=2,app.work.count=3,app
.hdfs.count=4"
© 2016 Pivotal Software, Inc. All rights reserved.
Instance Count
http
work
hdfs
http
work
work
hdfs
hdfs
hdfs
LoadBalancer
© 2016 Pivotal Software, Inc. All rights reserved.
Resource Management
dataflow:> stream deploy --name=s1 --
properties="app.work.spring.cloud.deployer.cloudf
oundry.memory=2048"
© 2016 Pivotal Software, Inc. All rights reserved.
Resource Management
http
http
work
work
work
hdfs
hdfs
hdfs
hdfs
© 2016 Pivotal Software, Inc. All rights reserved.
Spring Flo http://projects.spring.io/spring-flo/
© 2016 Pivotal Software, Inc. All rights reserved.
Stream Core Features
•Persistent Pub-Sub
•Consumer Group
•Partitioning Support
© 2016 Pivotal Software, Inc. All rights reserved.
Persistent Pub-Sub
HTTP Average Top Ns1.http s1.ave
Message Broker
© 2016 Pivotal Software, Inc. All rights reserved.
Persistent Pub-Sub
HTTP Average
HDFS
Top Ns1.http s1.ave
Message Broker
© 2016 Pivotal Software, Inc. All rights reserved.
Persistent Pub-Sub
HTTP Average
HDFS
Top N
Fault
Detection
s1.http s1.ave
Message Broker
© 2016 Pivotal Software, Inc. All rights reserved.
Persistent Pub-Sub
HTTP Average
HDFS
Top N
Fault
Detection
s1.http s1.ave
Message Broker
{"id":1, "temperature":38}
© 2016 Pivotal Software, Inc. All rights reserved.
Persistent Pub-Sub
HTTP Average
HDFS
Top N
Fault
Detection
s1.http s1.ave
Message Broker
{"id":1, "temperature":38}{"id":1, "temperature":38}
© 2016 Pivotal Software, Inc. All rights reserved.
Persistent Pub-Sub
HTTP Average
HDFS
Top N
Fault
Detection
s1.http s1.ave
Message Broker
{"id":1, "temperature":38}
{"id":1, "temperature":38}
{"id":1, "temperature":38}
© 2016 Pivotal Software, Inc. All rights reserved.
Persistent Pub-Sub
HTTP Average
HDFS
Top N
Fault
Detection
s1.http s1.ave
Message Broker
{"id":1, "temperature":
{"id":1, "temperature":38} {"id":1, "temperature":38}
© 2016 Pivotal Software, Inc. All rights reserved.
HTTP Average
HDFS
Top N
Fault
Detection
s1.http s1.ave
Message Broker
Average
Average
HDFS
HDFS
© 2016 Pivotal Software, Inc. All rights reserved.
HTTP Average
HDFS
Top N
Fault
Detection
s1.http s1.ave
Message Broker
{"id":1, "temperature":38}
Average
Average
HDFS
HDFS
© 2016 Pivotal Software, Inc. All rights reserved.
HTTP Average
HDFS
Top N
Fault
Detection
s1.http s1.ave
Message Broker
{"id":1, "temperature":38}{"id":1, "temperature":38}
Average
Average
HDFS
HDFS
© 2016 Pivotal Software, Inc. All rights reserved.
HTTP Average
HDFS
Top N
Fault
Detection
s1.http s1.ave
Message Broker
{"id":1, "temperature":38}
{"id":1, "temperature":38}
Average
Average
HDFS
HDFS
{"id":1, "temperature":38}
{"id":1, "temperature":38}
{"id":1, "temperature":38}
{"id":1, "temperature":38}
© 2016 Pivotal Software, Inc. All rights reserved.
HTTP Average
HDFS
Top N
Fault
Detection
s1.http s1.ave
Message Broker
{"id":1, "temperature":38}
{"id":1, "temperature":38}
Average
Average
HDFS
HDFS
{"id":1, "temperature":38}
{"id":1, "temperature":38}
{"id":1, "temperature":38}
{"id":1, "temperature":38}
😩
© 2016 Pivotal Software, Inc. All rights reserved.
HTTP Average
HDFS
Top N
Fault
Detection
s1.http s1.ave
Message Broker
{"id":1, "temperature":38}
{"id":1, "temperature":38}
Average
Average
HDFS
HDFS
{"id":1, "temperature":38}
{"id":1, "temperature":38}
{"id":1, "temperature":38}
{"id":1, "temperature":38}
😩
Consumer Group
© 2016 Pivotal Software, Inc. All rights reserved.
HTTP Average
HDFS
Top N
Fault
Detection
s1.http s1.ave
Message Broker
{"id":1, "temperature":38}
{"id":1, "temperature":38}
Average
Average
HDFS
HDFS
{"id":1, "temperature":38}
{"id":1, "temperature":38}
{"id":1, "temperature":38}
{"id":1, "temperature":38}
😩
spring.cloud.stream.bindings.<channelName>.group=ave
Spring Cloud Stream
group is configured by default 😁
Spring Cloud Data Flow
Consumer Group
© 2016 Pivotal Software, Inc. All rights reserved.
Consumer Group
HTTP Average
HDFS
Top N
Fault
Detection
s1.http s1.ave
Message Broker
Average
Average
HDFS
HDFS
group=ave
group=hdfs
© 2016 Pivotal Software, Inc. All rights reserved.
Consumer Group
HTTP Average
HDFS
Top N
Fault
Detection
s1.http s1.ave
Message Broker
Average
Average
HDFS
HDFS
group=ave
group=hdfs
{"id":1, "temperature":38}
© 2016 Pivotal Software, Inc. All rights reserved.
Consumer Group
HTTP Average
HDFS
Top N
Fault
Detection
s1.http s1.ave
Message Broker
Average
Average
HDFS
HDFS
group=ave
group=hdfs
{"id":1, "temperature":38}{"id":1, "temperature":38}
© 2016 Pivotal Software, Inc. All rights reserved.
Consumer Group
HTTP Average
HDFS
Top N
Fault
Detection
s1.http s1.ave
Message Broker
Average
Average
HDFS
HDFS
group=ave
group=hdfs
{"id":1, "temperature":38}
{"id":1, "temperature":38}
© 2016 Pivotal Software, Inc. All rights reserved.
Consumer Group
HTTP Average
HDFS
Top N
Fault
Detection
s1.http s1.ave
Message Broker
Average
Average
HDFS
HDFS
group=ave
group=hdfs
{"id":1, "temperature":38}
{"id":1, "temperature":38} 🤓
© 2016 Pivotal Software, Inc. All rights reserved.
Consumer Group
HTTP Average
HDFS
Top N
Fault
Detection
s1.http s1.ave
Message Broker
Average
Average
HDFS
HDFS
group=ave
group=hdfs
{"id":1, "temperature":38}
{"id":1, "temperature":38} 🤓consumer group subscriptions
are durable 😁
© 2016 Pivotal Software, Inc. All rights reserved.
HTTP
Average
s1.http
Average
© 2016 Pivotal Software, Inc. All rights reserved.
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}
{"id":2, "temperature":42}
{"id":1, "temperature":37}
© 2016 Pivotal Software, Inc. All rights reserved.
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}
{"id":2, "temperature":42}
{"id":1, "temperature":37}
© 2016 Pivotal Software, Inc. All rights reserved.
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}
{"id":2, "temperature":42}
{"id":1, "temperature":37}
© 2016 Pivotal Software, Inc. All rights reserved.
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}
{"id":2, "temperature":42}
{"id":1, "temperature":37}
© 2016 Pivotal Software, Inc. All rights reserved.
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}
{"id":2, "temperature":42}
{"id":1, "temperature":37}
© 2016 Pivotal Software, Inc. All rights reserved.
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}
{"id":2, "temperature":42}
{"id":1, "temperature":37}
😩
© 2016 Pivotal Software, Inc. All rights reserved.
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}
{"id":2, "temperature":42}
{"id":1, "temperature":37}
😩
Partitioning Support(Stateful Stream)
© 2016 Pivotal Software, Inc. All rights reserved.
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}
{"id":2, "temperature":42}
{"id":1, "temperature":37}
😩stream deploy --name=xxxx --properties=
"app.http.producer.partitionKeyExpression=payload.id"
spring.cloud.stream.bindings.<channelName>.producer.par
titionKeyExpression=payload.id
Spring Cloud Stream
Spring Cloud Data Flow
Partitioning Support(Stateful Stream)
© 2016 Pivotal Software, Inc. All rights reserved.
Partitioning Support(Stateful Stream)
HTTP
Average
s1.http
Average
© 2016 Pivotal Software, Inc. All rights reserved.
Partitioning Support(Stateful Stream)
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}
{"id":2, "temperature":42}
{"id":1, "temperature":37}
© 2016 Pivotal Software, Inc. All rights reserved.
Partitioning Support(Stateful Stream)
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}
{"id":2, "temperature":42}
{"id":1, "temperature":37}
© 2016 Pivotal Software, Inc. All rights reserved.
Partitioning Support(Stateful Stream)
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}
{"id":2, "temperature":42}
{"id":1, "temperature":37}
© 2016 Pivotal Software, Inc. All rights reserved.
Partitioning Support(Stateful Stream)
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}
{"id":2, "temperature":42}
{"id":1, "temperature":37}
© 2016 Pivotal Software, Inc. All rights reserved.
Partitioning Support(Stateful Stream)
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}
{"id":2, "temperature":42}
{"id":1, "temperature":37}
© 2016 Pivotal Software, Inc. All rights reserved.
Partitioning Support(Stateful Stream)
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}
{"id":2, "temperature":42}
{"id":1, "temperature":37}
🤓
© 2016 Pivotal Software, Inc. All rights reserved.
Simple Real Time Analytics
tweets = twitterstream | hdfs
analytics = :ingest.twitterstream > field-value-counter --fieldName=lang
HTTP
s1.http
HDFS COUNTER
Data Flow Server
REST API
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Demo (Twitter Stream!)
© 2016 Pivotal Software, Inc. All rights reserved.
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Task
© 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Task
• Spring Boot based framework for short lived task
• Spring Boot's CommandLineRunner or Spring Batch
• Result of each process persists in Task Repository
• Well Integrated with Spring Batch (Job Repository)
© 2016 Pivotal Software, Inc. All rights reserved.
Programming Model (CLR)
@SpringBootApplication

@EnableTask
public class DemoTaskApp {

@Bean

CommandLineRunner clr() {

return args -> System.out.println("Task!");

}


public static void main(String[] args) {

SpringApplication.run(DemoTaskApp.class, args);

}

} spring.application.name=hello
© 2016 Pivotal Software, Inc. All rights reserved.
Task Execution List
Task Name ID Start Time End Time Exit Code
hello 3 Tue Nov 15 16:49:08
JST 2016
Tue Nov 15 16:49:08
JST 2016
0
hello 2 Tue Nov 15 16:48:54
JST 2016
Tue Nov 15 16:48:55
JST 2016
0
hello 1 Tue Nov 15 16:48:23
JST 2016
Tue Nov 15 16:48:23
JST 2016
0
© 2016 Pivotal Software, Inc. All rights reserved.
Programming Model (Spring Batch)
@SpringBootApplication @EnableBatchProcessing

@EnableTask
public class DemoBatchApp {
@Autowired JobBuilderFactory jobBuilderFactory;
@Autowired StepBuilderFactory stepBuilderFactory;

@Bean Step step1() { /* ... */ }
@Bean Step step2() { /* ... */ }
@Bean Job job() { return jobBuilderFactory.get("job")
.start(step1()).next(step2()).build()}

public static void main(String[] args) {

SpringApplication.run(DemoTaskApp.class, args);

}}
https://github.com/making/cf-spring-batch-demo
spring.application.name=hello-batch
© 2016 Pivotal Software, Inc. All rights reserved.
Task Execution List
Task Name ID Start Time End Time Exit Code
hello-
batch
5 Tue Nov 15 17:28:49
JST 2016
Tue Nov 15 17:28:49
JST 2016
0
hello-
batch
4 Tue Nov 15 17:27:56
JST 2016
Tue Nov 15 17:27:57
JST 2016
0
© 2016 Pivotal Software, Inc. All rights reserved.
Job Execution List
ID Task ID Start Time Step Execution Count Definition Status
2 5 Tue Nov 15
17:28:49 JST 2016
2 Created
1 4 Tue Nov 15
17:27:56 JST 2016
2 Created
© 2016 Pivotal Software, Inc. All rights reserved.
Step Execution List
ID Step Name Job Exec Id Start Time End Time Status
1 step1 1 Tue Nov 15
17:28:49 JST
2016
Tue Nov 15
17:28:49 JST
2016
COMPLETED
2 step2 1 Tue Nov 15
17:27:56 JST
2016
Tue Nov 15
17:27:57 JST
2016
COMPLETED
© 2016 Pivotal Software, Inc. All rights reserved.
Task Orchestration in SCDF
>task create hello --definition="timestamp --format=¥"yyyy¥""
>task launch hello
timestamp
Data Flow Server DB
Task Name
Start Time
End Time
Exit Code
Exit Message
Last Updated Time
Parameters
Message Broker
© 2016 Pivotal Software, Inc. All rights reserved.
Task Execution List
© 2016 Pivotal Software, Inc. All rights reserved.
Job Execution List
© 2016 Pivotal Software, Inc. All rights reserved.
Job Execution Details
© 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Data Flow -> Task
http | tasklaunchrequest --uri=... | task-lancher
timestamp
Data Flow Server DB
Message Broker
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Demo (Scalable Tasks)
© 2016 Pivotal Software, Inc. All rights reserved.
Pivotal Cloud Foundry
PDF PDF
PDF PDF
S3
Spring Cloud Data Flow
Task
Task
Task
Task
PDF
PDF
Spring Cloud Task
Spring Batch
index
index
source sink
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Getting Started with SCDF
on Cloud Foundry
© 2016 Pivotal Software, Inc. All rights reserved.
Install PCF Dev
• https://docs.pivotal.io/pcf-dev
• Cloud Foundry on your laptop
• Included
• Redis / RabbitMQ / MySQL
• Spring Cloud Services
• Install with cf dev start
© 2016 Pivotal Software, Inc. All rights reserved.
cf create-service p-mysql 512mb df-mysql
cf create-service p-rabbitmq standard df-rebbitmq
cf create-service p-redis shared-vm df-redis
cf push dataflow-server -p spring-cloud-dataflow-
server-cloudfoundry-1.0.1.RELEASE.jar
Deploy SCDF Server on PCF Dev
© 2016 Pivotal Software, Inc. All rights reserved.
Tutorial
• 📗 https://blog.ik.am/entries/396
• 💎 https://github.com/making-demo-scdf/spring-cloud-
dataflow-cookbook
© 2016 Pivotal Software, Inc. All rights reserved.
Upcoming Features
• Some ‘porting’ from XD
• Batch Job DSL + Designer
• Role based access
• Looking forward
• Spring Cloud Sleuth
• JavaDSL
• In-place application version upgrades with Spinnaker
• Application Groups
• Polyglot
• Expanded analytics with Redis and Python/R ecosystem
• More provided apps/tasks
•SCDF Tile for Pivotal Cloud Foundry
© 2016 Pivotal Software, Inc. All rights reserved.
Thanks!
• References
• http://www.slideshare.net/SpringCentral/data-microservices-in-t
cloud ✨
• https://cloud.spring.io/spring-cloud-dataflow/
• http://cloud.spring.io/spring-cloud-stream/
• http://cloud.spring.io/spring-cloud-stream-app-starters/
• https://github.com/spring-cloud/spring-cloud-task/
• http://cloud.spring.io/spring-cloud-dataflow-server-cloudfoundry

More Related Content

What's hot

こんなに使える!今どきのAPIドキュメンテーションツール
こんなに使える!今どきのAPIドキュメンテーションツールこんなに使える!今どきのAPIドキュメンテーションツール
こんなに使える!今どきのAPIドキュメンテーションツール
dcubeio
 
組織の問題も解決するアーキテクチャ BackendsForFrontends
組織の問題も解決するアーキテクチャ BackendsForFrontends組織の問題も解決するアーキテクチャ BackendsForFrontends
組織の問題も解決するアーキテクチャ BackendsForFrontends
PIXTA Inc.
 
分散トレーシング技術について(Open tracingやjaeger)
分散トレーシング技術について(Open tracingやjaeger)分散トレーシング技術について(Open tracingやjaeger)
分散トレーシング技術について(Open tracingやjaeger)
NTT Communications Technology Development
 
NGINXをBFF (Backend for Frontend)として利用した話
NGINXをBFF (Backend for Frontend)として利用した話NGINXをBFF (Backend for Frontend)として利用した話
NGINXをBFF (Backend for Frontend)として利用した話
Hitachi, Ltd. OSS Solution Center.
 
コスト最適化概論
コスト最適化概論コスト最適化概論
コスト最適化概論
RikiMakita
 
CyberAgent における OSS の CI/CD 基盤開発 myshoes #CICD2021
CyberAgent における OSS の CI/CD 基盤開発 myshoes #CICD2021CyberAgent における OSS の CI/CD 基盤開発 myshoes #CICD2021
CyberAgent における OSS の CI/CD 基盤開発 myshoes #CICD2021
whywaita
 
より速く より運用しやすく 進化し続けるJVM(Java Developers Summit Online 2023 発表資料)
より速く より運用しやすく 進化し続けるJVM(Java Developers Summit Online 2023 発表資料)より速く より運用しやすく 進化し続けるJVM(Java Developers Summit Online 2023 発表資料)
より速く より運用しやすく 進化し続けるJVM(Java Developers Summit Online 2023 発表資料)
NTT DATA Technology & Innovation
 
NTTデータ流Infrastructure as Code~ 大規模プロジェクトを通して考え抜いた基盤自動化の新たな姿~(NTTデータ テクノロジーカンフ...
NTTデータ流Infrastructure as Code~ 大規模プロジェクトを通して考え抜いた基盤自動化の新たな姿~(NTTデータ テクノロジーカンフ...NTTデータ流Infrastructure as Code~ 大規模プロジェクトを通して考え抜いた基盤自動化の新たな姿~(NTTデータ テクノロジーカンフ...
NTTデータ流Infrastructure as Code~ 大規模プロジェクトを通して考え抜いた基盤自動化の新たな姿~(NTTデータ テクノロジーカンフ...
NTT DATA Technology & Innovation
 
deep dive distributed tracing
deep dive distributed tracingdeep dive distributed tracing
deep dive distributed tracing
Takayoshi Tanaka
 
Kubernetesのしくみ やさしく学ぶ 内部構造とアーキテクチャー
Kubernetesのしくみ やさしく学ぶ 内部構造とアーキテクチャーKubernetesのしくみ やさしく学ぶ 内部構造とアーキテクチャー
Kubernetesのしくみ やさしく学ぶ 内部構造とアーキテクチャー
Toru Makabe
 
Ceph アーキテクチャ概説
Ceph アーキテクチャ概説Ceph アーキテクチャ概説
Ceph アーキテクチャ概説
Emma Haruka Iwao
 
コンテナネットワーキング(CNI)最前線
コンテナネットワーキング(CNI)最前線コンテナネットワーキング(CNI)最前線
コンテナネットワーキング(CNI)最前線
Motonori Shindo
 
[CEDEC 2021] 運用中タイトルでも怖くない! 『メルクストーリア』におけるハイパフォーマンス・ローコストなリアルタイム通信技術の導入事例
[CEDEC 2021] 運用中タイトルでも怖くない! 『メルクストーリア』におけるハイパフォーマンス・ローコストなリアルタイム通信技術の導入事例[CEDEC 2021] 運用中タイトルでも怖くない! 『メルクストーリア』におけるハイパフォーマンス・ローコストなリアルタイム通信技術の導入事例
[CEDEC 2021] 運用中タイトルでも怖くない! 『メルクストーリア』におけるハイパフォーマンス・ローコストなリアルタイム通信技術の導入事例
Naoya Kishimoto
 
データレイクを基盤としたAWS上での機械学習サービス構築
データレイクを基盤としたAWS上での機械学習サービス構築データレイクを基盤としたAWS上での機械学習サービス構築
データレイクを基盤としたAWS上での機械学習サービス構築
Amazon Web Services Japan
 
コンテナ未経験新人が学ぶコンテナ技術入門
コンテナ未経験新人が学ぶコンテナ技術入門コンテナ未経験新人が学ぶコンテナ技術入門
コンテナ未経験新人が学ぶコンテナ技術入門
Kohei Tokunaga
 
リッチなドメインモデル 名前探し
リッチなドメインモデル 名前探しリッチなドメインモデル 名前探し
リッチなドメインモデル 名前探し
増田 亨
 
え、まって。その並列分散処理、Kafkaのしくみでもできるの? Apache Kafkaの機能を利用した大規模ストリームデータの並列分散処理
え、まって。その並列分散処理、Kafkaのしくみでもできるの? Apache Kafkaの機能を利用した大規模ストリームデータの並列分散処理え、まって。その並列分散処理、Kafkaのしくみでもできるの? Apache Kafkaの機能を利用した大規模ストリームデータの並列分散処理
え、まって。その並列分散処理、Kafkaのしくみでもできるの? Apache Kafkaの機能を利用した大規模ストリームデータの並列分散処理
NTT DATA Technology & Innovation
 
Docker入門 - 基礎編 いまから始めるDocker管理
Docker入門 - 基礎編 いまから始めるDocker管理Docker入門 - 基礎編 いまから始めるDocker管理
Docker入門 - 基礎編 いまから始めるDocker管理
Masahito Zembutsu
 
OpenTelemetry For Developers
OpenTelemetry For DevelopersOpenTelemetry For Developers
OpenTelemetry For Developers
Kevin Brockhoff
 
Opentelemetry - From frontend to backend
Opentelemetry - From frontend to backendOpentelemetry - From frontend to backend
Opentelemetry - From frontend to backend
Sebastian Poxhofer
 

What's hot (20)

こんなに使える!今どきのAPIドキュメンテーションツール
こんなに使える!今どきのAPIドキュメンテーションツールこんなに使える!今どきのAPIドキュメンテーションツール
こんなに使える!今どきのAPIドキュメンテーションツール
 
組織の問題も解決するアーキテクチャ BackendsForFrontends
組織の問題も解決するアーキテクチャ BackendsForFrontends組織の問題も解決するアーキテクチャ BackendsForFrontends
組織の問題も解決するアーキテクチャ BackendsForFrontends
 
分散トレーシング技術について(Open tracingやjaeger)
分散トレーシング技術について(Open tracingやjaeger)分散トレーシング技術について(Open tracingやjaeger)
分散トレーシング技術について(Open tracingやjaeger)
 
NGINXをBFF (Backend for Frontend)として利用した話
NGINXをBFF (Backend for Frontend)として利用した話NGINXをBFF (Backend for Frontend)として利用した話
NGINXをBFF (Backend for Frontend)として利用した話
 
コスト最適化概論
コスト最適化概論コスト最適化概論
コスト最適化概論
 
CyberAgent における OSS の CI/CD 基盤開発 myshoes #CICD2021
CyberAgent における OSS の CI/CD 基盤開発 myshoes #CICD2021CyberAgent における OSS の CI/CD 基盤開発 myshoes #CICD2021
CyberAgent における OSS の CI/CD 基盤開発 myshoes #CICD2021
 
より速く より運用しやすく 進化し続けるJVM(Java Developers Summit Online 2023 発表資料)
より速く より運用しやすく 進化し続けるJVM(Java Developers Summit Online 2023 発表資料)より速く より運用しやすく 進化し続けるJVM(Java Developers Summit Online 2023 発表資料)
より速く より運用しやすく 進化し続けるJVM(Java Developers Summit Online 2023 発表資料)
 
NTTデータ流Infrastructure as Code~ 大規模プロジェクトを通して考え抜いた基盤自動化の新たな姿~(NTTデータ テクノロジーカンフ...
NTTデータ流Infrastructure as Code~ 大規模プロジェクトを通して考え抜いた基盤自動化の新たな姿~(NTTデータ テクノロジーカンフ...NTTデータ流Infrastructure as Code~ 大規模プロジェクトを通して考え抜いた基盤自動化の新たな姿~(NTTデータ テクノロジーカンフ...
NTTデータ流Infrastructure as Code~ 大規模プロジェクトを通して考え抜いた基盤自動化の新たな姿~(NTTデータ テクノロジーカンフ...
 
deep dive distributed tracing
deep dive distributed tracingdeep dive distributed tracing
deep dive distributed tracing
 
Kubernetesのしくみ やさしく学ぶ 内部構造とアーキテクチャー
Kubernetesのしくみ やさしく学ぶ 内部構造とアーキテクチャーKubernetesのしくみ やさしく学ぶ 内部構造とアーキテクチャー
Kubernetesのしくみ やさしく学ぶ 内部構造とアーキテクチャー
 
Ceph アーキテクチャ概説
Ceph アーキテクチャ概説Ceph アーキテクチャ概説
Ceph アーキテクチャ概説
 
コンテナネットワーキング(CNI)最前線
コンテナネットワーキング(CNI)最前線コンテナネットワーキング(CNI)最前線
コンテナネットワーキング(CNI)最前線
 
[CEDEC 2021] 運用中タイトルでも怖くない! 『メルクストーリア』におけるハイパフォーマンス・ローコストなリアルタイム通信技術の導入事例
[CEDEC 2021] 運用中タイトルでも怖くない! 『メルクストーリア』におけるハイパフォーマンス・ローコストなリアルタイム通信技術の導入事例[CEDEC 2021] 運用中タイトルでも怖くない! 『メルクストーリア』におけるハイパフォーマンス・ローコストなリアルタイム通信技術の導入事例
[CEDEC 2021] 運用中タイトルでも怖くない! 『メルクストーリア』におけるハイパフォーマンス・ローコストなリアルタイム通信技術の導入事例
 
データレイクを基盤としたAWS上での機械学習サービス構築
データレイクを基盤としたAWS上での機械学習サービス構築データレイクを基盤としたAWS上での機械学習サービス構築
データレイクを基盤としたAWS上での機械学習サービス構築
 
コンテナ未経験新人が学ぶコンテナ技術入門
コンテナ未経験新人が学ぶコンテナ技術入門コンテナ未経験新人が学ぶコンテナ技術入門
コンテナ未経験新人が学ぶコンテナ技術入門
 
リッチなドメインモデル 名前探し
リッチなドメインモデル 名前探しリッチなドメインモデル 名前探し
リッチなドメインモデル 名前探し
 
え、まって。その並列分散処理、Kafkaのしくみでもできるの? Apache Kafkaの機能を利用した大規模ストリームデータの並列分散処理
え、まって。その並列分散処理、Kafkaのしくみでもできるの? Apache Kafkaの機能を利用した大規模ストリームデータの並列分散処理え、まって。その並列分散処理、Kafkaのしくみでもできるの? Apache Kafkaの機能を利用した大規模ストリームデータの並列分散処理
え、まって。その並列分散処理、Kafkaのしくみでもできるの? Apache Kafkaの機能を利用した大規模ストリームデータの並列分散処理
 
Docker入門 - 基礎編 いまから始めるDocker管理
Docker入門 - 基礎編 いまから始めるDocker管理Docker入門 - 基礎編 いまから始めるDocker管理
Docker入門 - 基礎編 いまから始めるDocker管理
 
OpenTelemetry For Developers
OpenTelemetry For DevelopersOpenTelemetry For Developers
OpenTelemetry For Developers
 
Opentelemetry - From frontend to backend
Opentelemetry - From frontend to backendOpentelemetry - From frontend to backend
Opentelemetry - From frontend to backend
 

Viewers also liked

SpringOne 2GX 2014 参加報告 & Spring 4.1について #jsug
SpringOne 2GX 2014 参加報告 & Spring 4.1について #jsugSpringOne 2GX 2014 参加報告 & Spring 4.1について #jsug
SpringOne 2GX 2014 参加報告 & Spring 4.1について #jsug
Toshiaki Maki
 
Event Driven Microservices with Spring Cloud Stream #jjug_ccc #ccc_ab3
Event Driven Microservices with Spring Cloud Stream #jjug_ccc #ccc_ab3Event Driven Microservices with Spring Cloud Stream #jjug_ccc #ccc_ab3
Event Driven Microservices with Spring Cloud Stream #jjug_ccc #ccc_ab3
Toshiaki Maki
 
Spring Day 2016 - Web API アクセス制御の最適解
Spring Day 2016 - Web API アクセス制御の最適解Spring Day 2016 - Web API アクセス制御の最適解
Spring Day 2016 - Web API アクセス制御の最適解
都元ダイスケ Miyamoto
 
Lineにおけるspring frameworkの活用
Lineにおけるspring frameworkの活用Lineにおけるspring frameworkの活用
Lineにおけるspring frameworkの活用
Tokuhiro Matsuno
 
Spring Day 2016 springの現在過去未来
Spring Day 2016 springの現在過去未来Spring Day 2016 springの現在過去未来
Spring Day 2016 springの現在過去未来
Yuichi Hasegawa
 
Spring bootで学ぶ初めてのwebアプリ開発
Spring bootで学ぶ初めてのwebアプリ開発Spring bootで学ぶ初めてのwebアプリ開発
Spring bootで学ぶ初めてのwebアプリ開発
terahide
 
アメブロの大規模システム刷新と それを支えるSpring
アメブロの大規模システム刷新と それを支えるSpringアメブロの大規模システム刷新と それを支えるSpring
アメブロの大規模システム刷新と それを支えるSpring
Takuya Hattori
 
Spring 5に備えるリアクティブプログラミング入門
Spring 5に備えるリアクティブプログラミング入門Spring 5に備えるリアクティブプログラミング入門
Spring 5に備えるリアクティブプログラミング入門
Takuya Iwatsuka
 
Springを使ったwebアプリにリファクタリングしよう
Springを使ったwebアプリにリファクタリングしようSpringを使ったwebアプリにリファクタリングしよう
Springを使ったwebアプリにリファクタリングしよう
土岐 孝平
 
Application Re-Architecture Technology ~ StrutsからSpring MVCへ ~
Application Re-Architecture Technology ~ StrutsからSpring MVCへ ~Application Re-Architecture Technology ~ StrutsからSpring MVCへ ~
Application Re-Architecture Technology ~ StrutsからSpring MVCへ ~
Yuichi Hasegawa
 
Grailsでドメイン駆動設計を実践する時の勘所
Grailsでドメイン駆動設計を実践する時の勘所Grailsでドメイン駆動設計を実践する時の勘所
Grailsでドメイン駆動設計を実践する時の勘所
Takuma Watabiki
 
Cloud Foundry x Wagby
Cloud Foundry x WagbyCloud Foundry x Wagby
Cloud Foundry x Wagby
Yoshinori Nie
 
楽天トラベルとSpring(Spring Day 2016)
楽天トラベルとSpring(Spring Day 2016)楽天トラベルとSpring(Spring Day 2016)
楽天トラベルとSpring(Spring Day 2016)
Rakuten Group, Inc.
 
ビッグじゃなくても使えるSpark Streaming
ビッグじゃなくても使えるSpark Streamingビッグじゃなくても使えるSpark Streaming
ビッグじゃなくても使えるSpark Streaming
chibochibo
 
Distributed Tracing Velocity2016
Distributed Tracing Velocity2016Distributed Tracing Velocity2016
Distributed Tracing Velocity2016
Reshmi Krishna
 
Distributed tracing - get a grasp on your production
Distributed tracing - get a grasp on your productionDistributed tracing - get a grasp on your production
Distributed tracing - get a grasp on your production
nklmish
 
Spring CloudとZipkinを利用した分散トレーシング
Spring CloudとZipkinを利用した分散トレーシングSpring CloudとZipkinを利用した分散トレーシング
Spring CloudとZipkinを利用した分散トレーシング
Rakuten Group, Inc.
 
形態素解析
形態素解析形態素解析
形態素解析
Works Applications
 
WalB: Real-time and Incremental Backup System for Block Devices
WalB: Real-time and Incremental Backup System for Block DevicesWalB: Real-time and Incremental Backup System for Block Devices
WalB: Real-time and Incremental Backup System for Block Devices
uchan_nos
 
3000社の業務データ絞り込みを支える技術
3000社の業務データ絞り込みを支える技術3000社の業務データ絞り込みを支える技術
3000社の業務データ絞り込みを支える技術
Ryo Mitoma
 

Viewers also liked (20)

SpringOne 2GX 2014 参加報告 & Spring 4.1について #jsug
SpringOne 2GX 2014 参加報告 & Spring 4.1について #jsugSpringOne 2GX 2014 参加報告 & Spring 4.1について #jsug
SpringOne 2GX 2014 参加報告 & Spring 4.1について #jsug
 
Event Driven Microservices with Spring Cloud Stream #jjug_ccc #ccc_ab3
Event Driven Microservices with Spring Cloud Stream #jjug_ccc #ccc_ab3Event Driven Microservices with Spring Cloud Stream #jjug_ccc #ccc_ab3
Event Driven Microservices with Spring Cloud Stream #jjug_ccc #ccc_ab3
 
Spring Day 2016 - Web API アクセス制御の最適解
Spring Day 2016 - Web API アクセス制御の最適解Spring Day 2016 - Web API アクセス制御の最適解
Spring Day 2016 - Web API アクセス制御の最適解
 
Lineにおけるspring frameworkの活用
Lineにおけるspring frameworkの活用Lineにおけるspring frameworkの活用
Lineにおけるspring frameworkの活用
 
Spring Day 2016 springの現在過去未来
Spring Day 2016 springの現在過去未来Spring Day 2016 springの現在過去未来
Spring Day 2016 springの現在過去未来
 
Spring bootで学ぶ初めてのwebアプリ開発
Spring bootで学ぶ初めてのwebアプリ開発Spring bootで学ぶ初めてのwebアプリ開発
Spring bootで学ぶ初めてのwebアプリ開発
 
アメブロの大規模システム刷新と それを支えるSpring
アメブロの大規模システム刷新と それを支えるSpringアメブロの大規模システム刷新と それを支えるSpring
アメブロの大規模システム刷新と それを支えるSpring
 
Spring 5に備えるリアクティブプログラミング入門
Spring 5に備えるリアクティブプログラミング入門Spring 5に備えるリアクティブプログラミング入門
Spring 5に備えるリアクティブプログラミング入門
 
Springを使ったwebアプリにリファクタリングしよう
Springを使ったwebアプリにリファクタリングしようSpringを使ったwebアプリにリファクタリングしよう
Springを使ったwebアプリにリファクタリングしよう
 
Application Re-Architecture Technology ~ StrutsからSpring MVCへ ~
Application Re-Architecture Technology ~ StrutsからSpring MVCへ ~Application Re-Architecture Technology ~ StrutsからSpring MVCへ ~
Application Re-Architecture Technology ~ StrutsからSpring MVCへ ~
 
Grailsでドメイン駆動設計を実践する時の勘所
Grailsでドメイン駆動設計を実践する時の勘所Grailsでドメイン駆動設計を実践する時の勘所
Grailsでドメイン駆動設計を実践する時の勘所
 
Cloud Foundry x Wagby
Cloud Foundry x WagbyCloud Foundry x Wagby
Cloud Foundry x Wagby
 
楽天トラベルとSpring(Spring Day 2016)
楽天トラベルとSpring(Spring Day 2016)楽天トラベルとSpring(Spring Day 2016)
楽天トラベルとSpring(Spring Day 2016)
 
ビッグじゃなくても使えるSpark Streaming
ビッグじゃなくても使えるSpark Streamingビッグじゃなくても使えるSpark Streaming
ビッグじゃなくても使えるSpark Streaming
 
Distributed Tracing Velocity2016
Distributed Tracing Velocity2016Distributed Tracing Velocity2016
Distributed Tracing Velocity2016
 
Distributed tracing - get a grasp on your production
Distributed tracing - get a grasp on your productionDistributed tracing - get a grasp on your production
Distributed tracing - get a grasp on your production
 
Spring CloudとZipkinを利用した分散トレーシング
Spring CloudとZipkinを利用した分散トレーシングSpring CloudとZipkinを利用した分散トレーシング
Spring CloudとZipkinを利用した分散トレーシング
 
形態素解析
形態素解析形態素解析
形態素解析
 
WalB: Real-time and Incremental Backup System for Block Devices
WalB: Real-time and Incremental Backup System for Block DevicesWalB: Real-time and Incremental Backup System for Block Devices
WalB: Real-time and Incremental Backup System for Block Devices
 
3000社の業務データ絞り込みを支える技術
3000社の業務データ絞り込みを支える技術3000社の業務データ絞り込みを支える技術
3000社の業務データ絞り込みを支える技術
 

Similar to Data Microservices with Spring Cloud Stream, Task, and Data Flow #jsug #springday

Spring Cloud Servicesの紹介 #pcf_tokyo
Spring Cloud Servicesの紹介 #pcf_tokyoSpring Cloud Servicesの紹介 #pcf_tokyo
Spring Cloud Servicesの紹介 #pcf_tokyo
Toshiaki Maki
 
Meetup: Streaming Data Pipeline Development
Meetup:  Streaming Data Pipeline DevelopmentMeetup:  Streaming Data Pipeline Development
Meetup: Streaming Data Pipeline Development
Timothy Spann
 
DBA Fundamentals Group: Continuous SQL with Kafka and Flink
DBA Fundamentals Group: Continuous SQL with Kafka and FlinkDBA Fundamentals Group: Continuous SQL with Kafka and Flink
DBA Fundamentals Group: Continuous SQL with Kafka and Flink
Timothy Spann
 
2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...
2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...
2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...
Timothy Spann
 
Spring Framework 5.0による Reactive Web Application #JavaDayTokyo
Spring Framework 5.0による Reactive Web Application #JavaDayTokyoSpring Framework 5.0による Reactive Web Application #JavaDayTokyo
Spring Framework 5.0による Reactive Web Application #JavaDayTokyo
Toshiaki Maki
 
Delivering the power of data using Spring Cloud DataFlow and DataStax Enterpr...
Delivering the power of data using Spring Cloud DataFlow and DataStax Enterpr...Delivering the power of data using Spring Cloud DataFlow and DataStax Enterpr...
Delivering the power of data using Spring Cloud DataFlow and DataStax Enterpr...
VMware Tanzu
 
Using the SDACK Architecture on Security Event Inspection by Yu-Lun Chen and ...
Using the SDACK Architecture on Security Event Inspection by Yu-Lun Chen and ...Using the SDACK Architecture on Security Event Inspection by Yu-Lun Chen and ...
Using the SDACK Architecture on Security Event Inspection by Yu-Lun Chen and ...
Docker, Inc.
 
Hadoop application architectures - using Customer 360 as an example
Hadoop application architectures - using Customer 360 as an exampleHadoop application architectures - using Customer 360 as an example
Hadoop application architectures - using Customer 360 as an example
hadooparchbook
 
Real-time Analysis of Data Processing Pipelines with Spring Cloud Data Flow a...
Real-time Analysis of Data Processing Pipelines with Spring Cloud Data Flow a...Real-time Analysis of Data Processing Pipelines with Spring Cloud Data Flow a...
Real-time Analysis of Data Processing Pipelines with Spring Cloud Data Flow a...
VMware Tanzu
 
Episode 3: Kubernetes and Big Data Services
Episode 3: Kubernetes and Big Data ServicesEpisode 3: Kubernetes and Big Data Services
Episode 3: Kubernetes and Big Data Services
Mesosphere Inc.
 
Azure HDlnsight에서 R 및 Spark를 이용하여 확장 가능한 머신러닝
Azure HDlnsight에서 R 및 Spark를 이용하여 확장 가능한 머신러닝Azure HDlnsight에서 R 및 Spark를 이용하여 확장 가능한 머신러닝
Azure HDlnsight에서 R 및 Spark를 이용하여 확장 가능한 머신러닝
OSS On Azure
 
Big Data Analytics with Spark
Big Data Analytics with SparkBig Data Analytics with Spark
Big Data Analytics with Spark
DataStax Academy
 
Monitoring as Code: Getting to Monitoring-Driven Development - DEV314 - re:In...
Monitoring as Code: Getting to Monitoring-Driven Development - DEV314 - re:In...Monitoring as Code: Getting to Monitoring-Driven Development - DEV314 - re:In...
Monitoring as Code: Getting to Monitoring-Driven Development - DEV314 - re:In...
Amazon Web Services
 
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache KafkaThe Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
Kai Wähner
 
Analyzing and processing streaming data with Amazon EMR - ADB204 - New York A...
Analyzing and processing streaming data with Amazon EMR - ADB204 - New York A...Analyzing and processing streaming data with Amazon EMR - ADB204 - New York A...
Analyzing and processing streaming data with Amazon EMR - ADB204 - New York A...
Amazon Web Services
 
Data Science at Scale on MPP databases - Use Cases & Open Source Tools
Data Science at Scale on MPP databases - Use Cases & Open Source ToolsData Science at Scale on MPP databases - Use Cases & Open Source Tools
Data Science at Scale on MPP databases - Use Cases & Open Source Tools
Esther Vasiete
 
Fraud Detection for Israel BigThings Meetup
Fraud Detection  for Israel BigThings MeetupFraud Detection  for Israel BigThings Meetup
Fraud Detection for Israel BigThings Meetup
Gwen (Chen) Shapira
 
DevOps Days Tel Aviv - Serverless Architecture
DevOps Days Tel Aviv - Serverless ArchitectureDevOps Days Tel Aviv - Serverless Architecture
DevOps Days Tel Aviv - Serverless Architecture
Antons Kranga
 
Spark Streaming with Azure Databricks
Spark Streaming with Azure DatabricksSpark Streaming with Azure Databricks
Spark Streaming with Azure Databricks
Dustin Vannoy
 
Spark Summit EU talk by Christos Erotocritou
Spark Summit EU talk by Christos ErotocritouSpark Summit EU talk by Christos Erotocritou
Spark Summit EU talk by Christos Erotocritou
Spark Summit
 

Similar to Data Microservices with Spring Cloud Stream, Task, and Data Flow #jsug #springday (20)

Spring Cloud Servicesの紹介 #pcf_tokyo
Spring Cloud Servicesの紹介 #pcf_tokyoSpring Cloud Servicesの紹介 #pcf_tokyo
Spring Cloud Servicesの紹介 #pcf_tokyo
 
Meetup: Streaming Data Pipeline Development
Meetup:  Streaming Data Pipeline DevelopmentMeetup:  Streaming Data Pipeline Development
Meetup: Streaming Data Pipeline Development
 
DBA Fundamentals Group: Continuous SQL with Kafka and Flink
DBA Fundamentals Group: Continuous SQL with Kafka and FlinkDBA Fundamentals Group: Continuous SQL with Kafka and Flink
DBA Fundamentals Group: Continuous SQL with Kafka and Flink
 
2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...
2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...
2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...
 
Spring Framework 5.0による Reactive Web Application #JavaDayTokyo
Spring Framework 5.0による Reactive Web Application #JavaDayTokyoSpring Framework 5.0による Reactive Web Application #JavaDayTokyo
Spring Framework 5.0による Reactive Web Application #JavaDayTokyo
 
Delivering the power of data using Spring Cloud DataFlow and DataStax Enterpr...
Delivering the power of data using Spring Cloud DataFlow and DataStax Enterpr...Delivering the power of data using Spring Cloud DataFlow and DataStax Enterpr...
Delivering the power of data using Spring Cloud DataFlow and DataStax Enterpr...
 
Using the SDACK Architecture on Security Event Inspection by Yu-Lun Chen and ...
Using the SDACK Architecture on Security Event Inspection by Yu-Lun Chen and ...Using the SDACK Architecture on Security Event Inspection by Yu-Lun Chen and ...
Using the SDACK Architecture on Security Event Inspection by Yu-Lun Chen and ...
 
Hadoop application architectures - using Customer 360 as an example
Hadoop application architectures - using Customer 360 as an exampleHadoop application architectures - using Customer 360 as an example
Hadoop application architectures - using Customer 360 as an example
 
Real-time Analysis of Data Processing Pipelines with Spring Cloud Data Flow a...
Real-time Analysis of Data Processing Pipelines with Spring Cloud Data Flow a...Real-time Analysis of Data Processing Pipelines with Spring Cloud Data Flow a...
Real-time Analysis of Data Processing Pipelines with Spring Cloud Data Flow a...
 
Episode 3: Kubernetes and Big Data Services
Episode 3: Kubernetes and Big Data ServicesEpisode 3: Kubernetes and Big Data Services
Episode 3: Kubernetes and Big Data Services
 
Azure HDlnsight에서 R 및 Spark를 이용하여 확장 가능한 머신러닝
Azure HDlnsight에서 R 및 Spark를 이용하여 확장 가능한 머신러닝Azure HDlnsight에서 R 및 Spark를 이용하여 확장 가능한 머신러닝
Azure HDlnsight에서 R 및 Spark를 이용하여 확장 가능한 머신러닝
 
Big Data Analytics with Spark
Big Data Analytics with SparkBig Data Analytics with Spark
Big Data Analytics with Spark
 
Monitoring as Code: Getting to Monitoring-Driven Development - DEV314 - re:In...
Monitoring as Code: Getting to Monitoring-Driven Development - DEV314 - re:In...Monitoring as Code: Getting to Monitoring-Driven Development - DEV314 - re:In...
Monitoring as Code: Getting to Monitoring-Driven Development - DEV314 - re:In...
 
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache KafkaThe Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
 
Analyzing and processing streaming data with Amazon EMR - ADB204 - New York A...
Analyzing and processing streaming data with Amazon EMR - ADB204 - New York A...Analyzing and processing streaming data with Amazon EMR - ADB204 - New York A...
Analyzing and processing streaming data with Amazon EMR - ADB204 - New York A...
 
Data Science at Scale on MPP databases - Use Cases & Open Source Tools
Data Science at Scale on MPP databases - Use Cases & Open Source ToolsData Science at Scale on MPP databases - Use Cases & Open Source Tools
Data Science at Scale on MPP databases - Use Cases & Open Source Tools
 
Fraud Detection for Israel BigThings Meetup
Fraud Detection  for Israel BigThings MeetupFraud Detection  for Israel BigThings Meetup
Fraud Detection for Israel BigThings Meetup
 
DevOps Days Tel Aviv - Serverless Architecture
DevOps Days Tel Aviv - Serverless ArchitectureDevOps Days Tel Aviv - Serverless Architecture
DevOps Days Tel Aviv - Serverless Architecture
 
Spark Streaming with Azure Databricks
Spark Streaming with Azure DatabricksSpark Streaming with Azure Databricks
Spark Streaming with Azure Databricks
 
Spark Summit EU talk by Christos Erotocritou
Spark Summit EU talk by Christos ErotocritouSpark Summit EU talk by Christos Erotocritou
Spark Summit EU talk by Christos Erotocritou
 

More from Toshiaki Maki

From Spring Boot 2.2 to Spring Boot 2.3 #jsug
From Spring Boot 2.2 to Spring Boot 2.3 #jsugFrom Spring Boot 2.2 to Spring Boot 2.3 #jsug
From Spring Boot 2.2 to Spring Boot 2.3 #jsug
Toshiaki Maki
 
Concourse x Spinnaker #concourse_tokyo
Concourse x Spinnaker #concourse_tokyoConcourse x Spinnaker #concourse_tokyo
Concourse x Spinnaker #concourse_tokyo
Toshiaki Maki
 
Serverless with Spring Cloud Function, Knative and riff #SpringOneTour #s1t
Serverless with Spring Cloud Function, Knative and riff #SpringOneTour #s1tServerless with Spring Cloud Function, Knative and riff #SpringOneTour #s1t
Serverless with Spring Cloud Function, Knative and riff #SpringOneTour #s1t
Toshiaki Maki
 
決済システムの内製化への旅 - SpringとPCFで作るクラウドネイティブなシステム開発 #jsug #sf_h1
決済システムの内製化への旅 - SpringとPCFで作るクラウドネイティブなシステム開発 #jsug #sf_h1決済システムの内製化への旅 - SpringとPCFで作るクラウドネイティブなシステム開発 #jsug #sf_h1
決済システムの内製化への旅 - SpringとPCFで作るクラウドネイティブなシステム開発 #jsug #sf_h1
Toshiaki Maki
 
Spring Boot Actuator 2.0 & Micrometer #jjug_ccc #ccc_a1
Spring Boot Actuator 2.0 & Micrometer #jjug_ccc #ccc_a1Spring Boot Actuator 2.0 & Micrometer #jjug_ccc #ccc_a1
Spring Boot Actuator 2.0 & Micrometer #jjug_ccc #ccc_a1
Toshiaki Maki
 
Spring Boot Actuator 2.0 & Micrometer
Spring Boot Actuator 2.0 & MicrometerSpring Boot Actuator 2.0 & Micrometer
Spring Boot Actuator 2.0 & Micrometer
Toshiaki Maki
 
Open Service Broker APIとKubernetes Service Catalog #k8sjp
Open Service Broker APIとKubernetes Service Catalog #k8sjpOpen Service Broker APIとKubernetes Service Catalog #k8sjp
Open Service Broker APIとKubernetes Service Catalog #k8sjp
Toshiaki Maki
 
Spring Cloud Function & Project riff #jsug
Spring Cloud Function & Project riff #jsugSpring Cloud Function & Project riff #jsug
Spring Cloud Function & Project riff #jsug
Toshiaki Maki
 
Introduction to Spring WebFlux #jsug #sf_a1
Introduction to Spring WebFlux #jsug #sf_a1Introduction to Spring WebFlux #jsug #sf_a1
Introduction to Spring WebFlux #jsug #sf_a1
Toshiaki Maki
 
BOSH / CF Deployment in modern ways #cf_tokyo
BOSH / CF Deployment in modern ways #cf_tokyoBOSH / CF Deployment in modern ways #cf_tokyo
BOSH / CF Deployment in modern ways #cf_tokyo
Toshiaki Maki
 
Why PCF is the best platform for Spring Boot
Why PCF is the best platform for Spring BootWhy PCF is the best platform for Spring Boot
Why PCF is the best platform for Spring Boot
Toshiaki Maki
 
Zipkin Components #zipkin_jp
Zipkin Components #zipkin_jpZipkin Components #zipkin_jp
Zipkin Components #zipkin_jp
Toshiaki Maki
 
マイクロサービスに必要な技術要素はすべてSpring Cloudにある #DO07
マイクロサービスに必要な技術要素はすべてSpring Cloudにある #DO07マイクロサービスに必要な技術要素はすべてSpring Cloudにある #DO07
マイクロサービスに必要な技術要素はすべてSpring Cloudにある #DO07
Toshiaki Maki
 
実例で学ぶ、明日から使えるSpring Boot Tips #jsug
実例で学ぶ、明日から使えるSpring Boot Tips #jsug実例で学ぶ、明日から使えるSpring Boot Tips #jsug
実例で学ぶ、明日から使えるSpring Boot Tips #jsug
Toshiaki Maki
 
Spring ❤️ Kotlin #jjug
Spring ❤️ Kotlin #jjugSpring ❤️ Kotlin #jjug
Spring ❤️ Kotlin #jjug
Toshiaki Maki
 
Managing your Docker image continuously with Concourse CI
Managing your Docker image continuously with Concourse CIManaging your Docker image continuously with Concourse CI
Managing your Docker image continuously with Concourse CI
Toshiaki Maki
 
Short Lived Tasks in Cloud Foundry #cfdtokyo
Short Lived Tasks in Cloud Foundry #cfdtokyoShort Lived Tasks in Cloud Foundry #cfdtokyo
Short Lived Tasks in Cloud Foundry #cfdtokyo
Toshiaki Maki
 
今すぐ始めるCloud Foundry #hackt #hackt_k
今すぐ始めるCloud Foundry #hackt #hackt_k今すぐ始めるCloud Foundry #hackt #hackt_k
今すぐ始めるCloud Foundry #hackt #hackt_k
Toshiaki Maki
 
Team Support in Concourse CI 2.0 #concourse_tokyo
Team Support in Concourse CI 2.0 #concourse_tokyoTeam Support in Concourse CI 2.0 #concourse_tokyo
Team Support in Concourse CI 2.0 #concourse_tokyo
Toshiaki Maki
 
From Zero to Hero with REST and OAuth2 #jjug
From Zero to Hero with REST and OAuth2 #jjugFrom Zero to Hero with REST and OAuth2 #jjug
From Zero to Hero with REST and OAuth2 #jjug
Toshiaki Maki
 

More from Toshiaki Maki (20)

From Spring Boot 2.2 to Spring Boot 2.3 #jsug
From Spring Boot 2.2 to Spring Boot 2.3 #jsugFrom Spring Boot 2.2 to Spring Boot 2.3 #jsug
From Spring Boot 2.2 to Spring Boot 2.3 #jsug
 
Concourse x Spinnaker #concourse_tokyo
Concourse x Spinnaker #concourse_tokyoConcourse x Spinnaker #concourse_tokyo
Concourse x Spinnaker #concourse_tokyo
 
Serverless with Spring Cloud Function, Knative and riff #SpringOneTour #s1t
Serverless with Spring Cloud Function, Knative and riff #SpringOneTour #s1tServerless with Spring Cloud Function, Knative and riff #SpringOneTour #s1t
Serverless with Spring Cloud Function, Knative and riff #SpringOneTour #s1t
 
決済システムの内製化への旅 - SpringとPCFで作るクラウドネイティブなシステム開発 #jsug #sf_h1
決済システムの内製化への旅 - SpringとPCFで作るクラウドネイティブなシステム開発 #jsug #sf_h1決済システムの内製化への旅 - SpringとPCFで作るクラウドネイティブなシステム開発 #jsug #sf_h1
決済システムの内製化への旅 - SpringとPCFで作るクラウドネイティブなシステム開発 #jsug #sf_h1
 
Spring Boot Actuator 2.0 & Micrometer #jjug_ccc #ccc_a1
Spring Boot Actuator 2.0 & Micrometer #jjug_ccc #ccc_a1Spring Boot Actuator 2.0 & Micrometer #jjug_ccc #ccc_a1
Spring Boot Actuator 2.0 & Micrometer #jjug_ccc #ccc_a1
 
Spring Boot Actuator 2.0 & Micrometer
Spring Boot Actuator 2.0 & MicrometerSpring Boot Actuator 2.0 & Micrometer
Spring Boot Actuator 2.0 & Micrometer
 
Open Service Broker APIとKubernetes Service Catalog #k8sjp
Open Service Broker APIとKubernetes Service Catalog #k8sjpOpen Service Broker APIとKubernetes Service Catalog #k8sjp
Open Service Broker APIとKubernetes Service Catalog #k8sjp
 
Spring Cloud Function & Project riff #jsug
Spring Cloud Function & Project riff #jsugSpring Cloud Function & Project riff #jsug
Spring Cloud Function & Project riff #jsug
 
Introduction to Spring WebFlux #jsug #sf_a1
Introduction to Spring WebFlux #jsug #sf_a1Introduction to Spring WebFlux #jsug #sf_a1
Introduction to Spring WebFlux #jsug #sf_a1
 
BOSH / CF Deployment in modern ways #cf_tokyo
BOSH / CF Deployment in modern ways #cf_tokyoBOSH / CF Deployment in modern ways #cf_tokyo
BOSH / CF Deployment in modern ways #cf_tokyo
 
Why PCF is the best platform for Spring Boot
Why PCF is the best platform for Spring BootWhy PCF is the best platform for Spring Boot
Why PCF is the best platform for Spring Boot
 
Zipkin Components #zipkin_jp
Zipkin Components #zipkin_jpZipkin Components #zipkin_jp
Zipkin Components #zipkin_jp
 
マイクロサービスに必要な技術要素はすべてSpring Cloudにある #DO07
マイクロサービスに必要な技術要素はすべてSpring Cloudにある #DO07マイクロサービスに必要な技術要素はすべてSpring Cloudにある #DO07
マイクロサービスに必要な技術要素はすべてSpring Cloudにある #DO07
 
実例で学ぶ、明日から使えるSpring Boot Tips #jsug
実例で学ぶ、明日から使えるSpring Boot Tips #jsug実例で学ぶ、明日から使えるSpring Boot Tips #jsug
実例で学ぶ、明日から使えるSpring Boot Tips #jsug
 
Spring ❤️ Kotlin #jjug
Spring ❤️ Kotlin #jjugSpring ❤️ Kotlin #jjug
Spring ❤️ Kotlin #jjug
 
Managing your Docker image continuously with Concourse CI
Managing your Docker image continuously with Concourse CIManaging your Docker image continuously with Concourse CI
Managing your Docker image continuously with Concourse CI
 
Short Lived Tasks in Cloud Foundry #cfdtokyo
Short Lived Tasks in Cloud Foundry #cfdtokyoShort Lived Tasks in Cloud Foundry #cfdtokyo
Short Lived Tasks in Cloud Foundry #cfdtokyo
 
今すぐ始めるCloud Foundry #hackt #hackt_k
今すぐ始めるCloud Foundry #hackt #hackt_k今すぐ始めるCloud Foundry #hackt #hackt_k
今すぐ始めるCloud Foundry #hackt #hackt_k
 
Team Support in Concourse CI 2.0 #concourse_tokyo
Team Support in Concourse CI 2.0 #concourse_tokyoTeam Support in Concourse CI 2.0 #concourse_tokyo
Team Support in Concourse CI 2.0 #concourse_tokyo
 
From Zero to Hero with REST and OAuth2 #jjug
From Zero to Hero with REST and OAuth2 #jjugFrom Zero to Hero with REST and OAuth2 #jjug
From Zero to Hero with REST and OAuth2 #jjug
 

Recently uploaded

Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
SOFTTECHHUB
 
Video Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the FutureVideo Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the Future
Alpen-Adria-Universität
 
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
Neo4j
 
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AIMind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
Kumud Singh
 
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdfUni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems S.M.S.A.
 
GridMate - End to end testing is a critical piece to ensure quality and avoid...
GridMate - End to end testing is a critical piece to ensure quality and avoid...GridMate - End to end testing is a critical piece to ensure quality and avoid...
GridMate - End to end testing is a critical piece to ensure quality and avoid...
ThomasParaiso2
 
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
James Anderson
 
DevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA ConnectDevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA Connect
Kari Kakkonen
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
Octavian Nadolu
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
DianaGray10
 
A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...
sonjaschweigert1
 
20240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 202420240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 2024
Matthew Sinclair
 
Full-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalizationFull-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalization
Zilliz
 
By Design, not by Accident - Agile Venture Bolzano 2024
By Design, not by Accident - Agile Venture Bolzano 2024By Design, not by Accident - Agile Venture Bolzano 2024
By Design, not by Accident - Agile Venture Bolzano 2024
Pierluigi Pugliese
 
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionGenerative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Aggregage
 
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FMEEssentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
Safe Software
 
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptxHow to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
danishmna97
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Malak Abu Hammad
 
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
Neo4j
 
Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1
DianaGray10
 

Recently uploaded (20)

Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
 
Video Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the FutureVideo Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the Future
 
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
 
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AIMind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
 
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdfUni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdf
 
GridMate - End to end testing is a critical piece to ensure quality and avoid...
GridMate - End to end testing is a critical piece to ensure quality and avoid...GridMate - End to end testing is a critical piece to ensure quality and avoid...
GridMate - End to end testing is a critical piece to ensure quality and avoid...
 
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
 
DevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA ConnectDevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA Connect
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
 
A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...
 
20240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 202420240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 2024
 
Full-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalizationFull-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalization
 
By Design, not by Accident - Agile Venture Bolzano 2024
By Design, not by Accident - Agile Venture Bolzano 2024By Design, not by Accident - Agile Venture Bolzano 2024
By Design, not by Accident - Agile Venture Bolzano 2024
 
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionGenerative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
 
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FMEEssentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
 
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptxHow to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
 
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
 
Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1
 

Data Microservices with Spring Cloud Stream, Task, and Data Flow #jsug #springday

  • 1. ‹#›© 2016 Pivotal Software, Inc. All rights reserved. ‹#›© 2016 Pivotal Software, Inc. All rights reserved. Data Microservices with Spring Cloud Stream, Task, and Data Flow Toshiaki Maki (@making) 2016-11-18 Spring Day 2016 #springday #jsug
  • 2. © 2016 Pivotal Software, Inc. All rights reserved. Who am I ? • Toshiaki Maki (@making) http://blog.ik.am • Sr. Solutions Architect @Pivotal • Spring Framework enthusiast bit.ly/hajiboot2
  • 3. © 2016 Pivotal Software, Inc. All rights reserved. Input Motivation HTTP TCP S3 JDBC Rabbit JMS Twitter Syslog Output HDFS Cassandra HAWQ Greenplum JDBC S3 TCP Gemfire ❓❔❓ • Real Time Analysis • Data Ingestion • ETL Process
  • 4. © 2016 Pivotal Software, Inc. All rights reserved. Spring XD (eXtreme Data)
  • 5. © 2016 Pivotal Software, Inc. All rights reserved. Spring XD (eXtreme Data) 🤔
  • 6. © 2016 Pivotal Software, Inc. All rights reserved. Spring XD Architecture Container Container gpfdist Cassandra jms http ZooKeeper Message Broker XD Admin stream1 = http | cassandra stream2 = jms | gpfdist On Metal/VMs
  • 7. © 2016 Pivotal Software, Inc. All rights reserved. Spring XD Architecture Container Container gpfdist Cassandra jms http ZooKeeper Message Broker XD Admin stream1 = http | cassandra stream2 = jms | gpfdist On Metal/VMs 😇
  • 8. © 2016 Pivotal Software, Inc. All rights reserved. Spring XD Architecture Container Container gpfdist Cassandra jms http ZooKeeper Message Broker XD Admin stream1 = http | cassandra stream2 = jms | gpfdist On Metal/VMs
  • 9. © 2016 Pivotal Software, Inc. All rights reserved. Modern Platforms Cloud Foundry YARN Kubernetes Mesos Docker Swarm Nomad OpenShift
  • 10. © 2016 Pivotal Software, Inc. All rights reserved. Cloud Native Redesign •From multiple modules embedded in a container to standalone executable applications •From our own runtime to delegating to existing modern platforms
  • 11. © 2016 Pivotal Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head
  • 12. © 2016 Pivotal Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head
  • 13. © 2016 Pivotal Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing
  • 14. © 2016 Pivotal Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing
  • 15. © 2016 Pivotal Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing bound with Message Brokers
  • 16. © 2016 Pivotal Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing bound with Message Brokers
  • 17. © 2016 Pivotal Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing bound with Message Brokers on the modern platform
  • 18. © 2016 Pivotal Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing bound with Message Brokers on the modern platform
  • 19. © 2016 Pivotal Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing bound with Message Brokers on the modern platform
  • 20. © 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Data Flow • Microservices-based Distributed Data Pipelines •Long Lived Stream Applications •Short Lived Task Applications
  • 21. © 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Data Flow • Microservices-based Distributed Data Pipelines •Long Lived Stream Applications •Short Lived Task Applications Spring Cloud Stream
  • 22. © 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Data Flow • Microservices-based Distributed Data Pipelines •Long Lived Stream Applications •Short Lived Task Applications Spring Cloud Stream Spring Cloud Task
  • 23. © 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Data Flow • Microservices-based Distributed Data Pipelines •Long Lived Stream Applications •Short Lived Task Applications Spring Cloud Stream Spring Cloud Task Orchestration Layer
  • 24. © 2016 Pivotal Software, Inc. All rights reserved. Structure Spring Cloud Data Flow Spring Cloud Deployer (SPI) Spring Cloud Stream Spring Cloud Task Spring Integration Spring Boot Spring Batch
  • 25. © 2016 Pivotal Software, Inc. All rights reserved. Structure Spring Cloud Data Flow Spring Cloud Deployer (SPI) Spring Cloud Stream Spring Cloud Task Spring Integration Spring Boot Spring Batch • Spring Cloud Deployer Local • Spring Cloud Deployer Cloud Foundry • Spring Cloud Deployer Yarn • Spring Cloud Deployer Kubernetes • Spring Cloud Deployer Mesos
  • 26. © 2016 Pivotal Software, Inc. All rights reserved. SCDF Deployment Platform Spring Cloud Data Flow Server Deployer SPI REST API SCDFShell SCDF Flo
  • 27. © 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Data Flow Architecture gpfdist cassandra jms http stream1 = http | cassandra stream2 = jms | gpfdist Message Broker Data Flow Server DB Platform Runtime
  • 28. © 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Data Flow Architecture gpfdist cassandra jms http stream1 = http | cassandra stream2 = jms | gpfdist Message Broker Data Flow Server DB Platform Runtime
  • 29. ‹#›© 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Stream
  • 30. © 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Stream • Event-driven microservice framework • Built on battle-tested components (Spring Boot / Spring Integration) • Opinionated primitives for streaming applications • Persistent Pub/Sub • Consumer Groups • Partitioning Support • Pluggable messaging middleware bindings source | processor | sink
  • 31. © 2016 Pivotal Software, Inc. All rights reserved. JJUG CCC in 2 weeks
  • 32. © 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Stream Applications Twitter Stream Cassandra java -jar twittersource.jar --server.port=8080 --consumerKey=XYZ --consumerSecret=ABC --spring.cloud.stream.bindings. output.destination=ingest Source Sink java -jar cassandrasink.jar --server.port=8081 --spring.cassandra.keyspace=tweet --spring.cloud.stream.bindings. input.destination=ingest
  • 33. © 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Stream Applications Twitter Stream Cassandra java -jar twittersource.jar --server.port=8080 --consumerKey=XYZ --consumerSecret=ABC --spring.cloud.stream.bindings. output.destination=ingest Source Sink java -jar cassandrasink.jar --server.port=8081 --spring.cassandra.keyspace=tweet --spring.cloud.stream.bindings. input.destination=ingest Twitter Stream Cassandraingest
  • 34. © 2016 Pivotal Software, Inc. All rights reserved. Message Binders • @EnableBinding • Binder Implementations • Production-Ready • Rabbit MQ • Apache Kafka • Experimental • JMS • Google PubSub
  • 35. © 2016 Pivotal Software, Inc. All rights reserved. Programming Model (Sink) @SpringBootApplication
 @EnableBinding(Sink.class)
 public class DemoSinkApp { @StreamListener(Sink.INPUT)
 void receive(Message<String> message) {
 System.out.println("Received " + message);
 }
 public static void main(String[] args) {
 SpringApplication.run(DemoSinkApp.class, args);
 }
 }
  • 36. © 2016 Pivotal Software, Inc. All rights reserved. Sink Properties (Sink) spring.cloud.stream.bindings.input.destination=demo-strm demo- strm input
  • 37. © 2016 Pivotal Software, Inc. All rights reserved. Programming Model (Source) @SpringBootApplication @RestController @EnableBinding(Source.class)
 public class DemoSourceApp {
 @Autowired Source source; @GetMapping void send(@RequestParam String text) {
 source.output() .send(MessageBuilder.withPayload(text).build());
 }
 public static void main(String[] args) {
 SpringApplication.run(DemoSourceApp.class, args);
 }
 }
  • 38. © 2016 Pivotal Software, Inc. All rights reserved. Programming Model (Source) @SpringBootApplication @RestController @EnableBinding(Source.class)
 public class DemoSourceApp { @Bean @InboundChannelAdapter(channel = Source.OUTPUT, poller = @Poller(fixedDelay = "1000", maxMessagesPerPoll = "1")) MessageSource<String> source() {
 return () -> MessageBuilder.withPayload("Hi").build());
 }
 public static void main(String[] args) {
 SpringApplication.run(DemoSourceApp.class, args);
 }}
  • 39. © 2016 Pivotal Software, Inc. All rights reserved. Properties (Source) spring.cloud.stream.bindings.output.destination=demo-strm demo- strm Source output
  • 40. © 2016 Pivotal Software, Inc. All rights reserved. Binder (RabbitMQ) <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-stream-rabbit</artifactId> </dependency> demo- strm
  • 41. © 2016 Pivotal Software, Inc. All rights reserved. Binder (Apache Kafka) <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-stream-kafka</artifactId> </dependency> demo- strm
  • 42. © 2016 Pivotal Software, Inc. All rights reserved. Sink Pipeline demo- strm input Source output source | sink
  • 43. © 2016 Pivotal Software, Inc. All rights reserved. Programming Model (Processor) @SpringBootApplication
 @EnableBinding(Processor.class)
 public class DemoProcessorApp { @StreamListener(Processor.INPUT) @SendTo(Processor.OUTPUT)
 void receive(String text) {
 return "[[" + text + "]]";
 }
 public static void main(String[] args) {
 SpringApplication.run(DemoProcessorApp.class, args);
 }
 }
  • 44. © 2016 Pivotal Software, Inc. All rights reserved. Properties (Processor) spring.cloud.stream.bindings.output.destination=my-source spring.cloud.stream.bindings.input.destination=my-source spring.cloud.stream.bindings.output.destination=my-proc spring.cloud.stream.bindings.input.destination=my-proc Source Processor Sink
  • 45. © 2016 Pivotal Software, Inc. All rights reserved. Pipeline my- source Source output source | processor | sink Processor output input my-proc Sink input
  • 46. © 2016 Pivotal Software, Inc. All rights reserved. Reactive API Support @SpringBootApplication
 @EnableBinding(Processor.class)
 public class DemoProcessorRxApp { @StreamListener @Output(Processor.OUTPUT)
 public Flux<String> receive(@Input(Processor.INPUT) Flux<String> stream) {
 return stream.map(text -> "[[" + text + "]]");
 }
 public static void main(String[] args) {
 SpringApplication.run(DemoProcessorRxApp.class, args);
 }
 }
  • 47. © 2016 Pivotal Software, Inc. All rights reserved. Reactive API Support @StreamListener @Output(Processor.OUTPUT)
 public Flux<AverageData> receive(@Input(Processor.INPUT) Flux<SensorData> stream) {
 return stream.window(Duration.ofSecond(20), Duration.ofSecond(10)) .flatMap(win -> win.groupBy(sensor -> sensor.id)) .flatMap(group -> calcAverage(group));
 }
  • 48. © 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Stream App Starters Source Processor Sink file ftp gemfire gemfire-cq http jdbc jms load-generator loggregator mail mongodb rabbit s3 sftp syslog tcp tcp-client time trigger triggertask twitterstream bridge filter groovy-filter groovy- transform httpclient pommel scriptable- transform splitter tcp-client transform aggregate- counter cassandra counter field-value- counter file ftp gemfire gpfdist hfs hdfs-dataset jdbc log rabbit redis-pubsub router s3 sftp task-launcher- local task-lancher- yarn tcp throughput web socket http://cloud.spring.io/spring-cloud-stream-app-starters/
  • 49. © 2016 Pivotal Software, Inc. All rights reserved. Stream Orchestration in SCDF ingest = twitterstream | cassandra dataflow:> stream create --name=ingest --definition="twitterstream | cassandra" dataflow:> stream deploy --name=ingest |
  • 50. © 2016 Pivotal Software, Inc. All rights reserved. Import apps dataflow:> app import --uri http://bit.ly/1-0-4- GA-stream-applications-rabbit-maven rabbit maven kafka docker stream-applications- http://cloud.spring.io/spring-cloud-stream-app-starters/
  • 51. © 2016 Pivotal Software, Inc. All rights reserved. Register your own apps dataflow:> app register --name foo --type source --uri http://example.com/foo-source-1.0.jar dataflow:> app register --name foo --type source --uri maven://com.example:foo-source:1.0 dataflow:> app register --name foo --type source --uri docker:myapps/foo-source:1.0 dataflow:> app register --name foo --type source --uri file:///tmp/foo-source:1.0
  • 52. © 2016 Pivotal Software, Inc. All rights reserved. Instance Count dataflow:> stream create --name=s1 --definition="http | work | hdfs" dataflow:> stream deploy --name=s1 -- properties="app.http.count=2,app.work.count=3,app .hdfs.count=4"
  • 53. © 2016 Pivotal Software, Inc. All rights reserved. Instance Count http work hdfs http work work hdfs hdfs hdfs LoadBalancer
  • 54. © 2016 Pivotal Software, Inc. All rights reserved. Resource Management dataflow:> stream deploy --name=s1 -- properties="app.work.spring.cloud.deployer.cloudf oundry.memory=2048"
  • 55. © 2016 Pivotal Software, Inc. All rights reserved. Resource Management http http work work work hdfs hdfs hdfs hdfs
  • 56. © 2016 Pivotal Software, Inc. All rights reserved. Spring Flo http://projects.spring.io/spring-flo/
  • 57. © 2016 Pivotal Software, Inc. All rights reserved. Stream Core Features •Persistent Pub-Sub •Consumer Group •Partitioning Support
  • 58. © 2016 Pivotal Software, Inc. All rights reserved. Persistent Pub-Sub HTTP Average Top Ns1.http s1.ave Message Broker
  • 59. © 2016 Pivotal Software, Inc. All rights reserved. Persistent Pub-Sub HTTP Average HDFS Top Ns1.http s1.ave Message Broker
  • 60. © 2016 Pivotal Software, Inc. All rights reserved. Persistent Pub-Sub HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker
  • 61. © 2016 Pivotal Software, Inc. All rights reserved. Persistent Pub-Sub HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38}
  • 62. © 2016 Pivotal Software, Inc. All rights reserved. Persistent Pub-Sub HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38}{"id":1, "temperature":38}
  • 63. © 2016 Pivotal Software, Inc. All rights reserved. Persistent Pub-Sub HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38}
  • 64. © 2016 Pivotal Software, Inc. All rights reserved. Persistent Pub-Sub HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature": {"id":1, "temperature":38} {"id":1, "temperature":38}
  • 65. © 2016 Pivotal Software, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS
  • 66. © 2016 Pivotal Software, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38} Average Average HDFS HDFS
  • 67. © 2016 Pivotal Software, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38}{"id":1, "temperature":38} Average Average HDFS HDFS
  • 68. © 2016 Pivotal Software, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38} {"id":1, "temperature":38} Average Average HDFS HDFS {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38}
  • 69. © 2016 Pivotal Software, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38} {"id":1, "temperature":38} Average Average HDFS HDFS {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} 😩
  • 70. © 2016 Pivotal Software, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38} {"id":1, "temperature":38} Average Average HDFS HDFS {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} 😩 Consumer Group
  • 71. © 2016 Pivotal Software, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38} {"id":1, "temperature":38} Average Average HDFS HDFS {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} 😩 spring.cloud.stream.bindings.<channelName>.group=ave Spring Cloud Stream group is configured by default 😁 Spring Cloud Data Flow Consumer Group
  • 72. © 2016 Pivotal Software, Inc. All rights reserved. Consumer Group HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS group=ave group=hdfs
  • 73. © 2016 Pivotal Software, Inc. All rights reserved. Consumer Group HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS group=ave group=hdfs {"id":1, "temperature":38}
  • 74. © 2016 Pivotal Software, Inc. All rights reserved. Consumer Group HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS group=ave group=hdfs {"id":1, "temperature":38}{"id":1, "temperature":38}
  • 75. © 2016 Pivotal Software, Inc. All rights reserved. Consumer Group HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS group=ave group=hdfs {"id":1, "temperature":38} {"id":1, "temperature":38}
  • 76. © 2016 Pivotal Software, Inc. All rights reserved. Consumer Group HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS group=ave group=hdfs {"id":1, "temperature":38} {"id":1, "temperature":38} 🤓
  • 77. © 2016 Pivotal Software, Inc. All rights reserved. Consumer Group HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS group=ave group=hdfs {"id":1, "temperature":38} {"id":1, "temperature":38} 🤓consumer group subscriptions are durable 😁
  • 78. © 2016 Pivotal Software, Inc. All rights reserved. HTTP Average s1.http Average
  • 79. © 2016 Pivotal Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
  • 80. © 2016 Pivotal Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
  • 81. © 2016 Pivotal Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
  • 82. © 2016 Pivotal Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
  • 83. © 2016 Pivotal Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
  • 84. © 2016 Pivotal Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37} 😩
  • 85. © 2016 Pivotal Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37} 😩 Partitioning Support(Stateful Stream)
  • 86. © 2016 Pivotal Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37} 😩stream deploy --name=xxxx --properties= "app.http.producer.partitionKeyExpression=payload.id" spring.cloud.stream.bindings.<channelName>.producer.par titionKeyExpression=payload.id Spring Cloud Stream Spring Cloud Data Flow Partitioning Support(Stateful Stream)
  • 87. © 2016 Pivotal Software, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average
  • 88. © 2016 Pivotal Software, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
  • 89. © 2016 Pivotal Software, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
  • 90. © 2016 Pivotal Software, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
  • 91. © 2016 Pivotal Software, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
  • 92. © 2016 Pivotal Software, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
  • 93. © 2016 Pivotal Software, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37} 🤓
  • 94. © 2016 Pivotal Software, Inc. All rights reserved. Simple Real Time Analytics tweets = twitterstream | hdfs analytics = :ingest.twitterstream > field-value-counter --fieldName=lang HTTP s1.http HDFS COUNTER Data Flow Server REST API
  • 95. ‹#›© 2016 Pivotal Software, Inc. All rights reserved. Demo (Twitter Stream!)
  • 96. © 2016 Pivotal Software, Inc. All rights reserved.
  • 97. ‹#›© 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Task
  • 98. © 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Task • Spring Boot based framework for short lived task • Spring Boot's CommandLineRunner or Spring Batch • Result of each process persists in Task Repository • Well Integrated with Spring Batch (Job Repository)
  • 99. © 2016 Pivotal Software, Inc. All rights reserved. Programming Model (CLR) @SpringBootApplication
 @EnableTask public class DemoTaskApp {
 @Bean
 CommandLineRunner clr() {
 return args -> System.out.println("Task!");
 } 
 public static void main(String[] args) {
 SpringApplication.run(DemoTaskApp.class, args);
 }
 } spring.application.name=hello
  • 100. © 2016 Pivotal Software, Inc. All rights reserved. Task Execution List Task Name ID Start Time End Time Exit Code hello 3 Tue Nov 15 16:49:08 JST 2016 Tue Nov 15 16:49:08 JST 2016 0 hello 2 Tue Nov 15 16:48:54 JST 2016 Tue Nov 15 16:48:55 JST 2016 0 hello 1 Tue Nov 15 16:48:23 JST 2016 Tue Nov 15 16:48:23 JST 2016 0
  • 101. © 2016 Pivotal Software, Inc. All rights reserved. Programming Model (Spring Batch) @SpringBootApplication @EnableBatchProcessing
 @EnableTask public class DemoBatchApp { @Autowired JobBuilderFactory jobBuilderFactory; @Autowired StepBuilderFactory stepBuilderFactory;
 @Bean Step step1() { /* ... */ } @Bean Step step2() { /* ... */ } @Bean Job job() { return jobBuilderFactory.get("job") .start(step1()).next(step2()).build()}
 public static void main(String[] args) {
 SpringApplication.run(DemoTaskApp.class, args);
 }} https://github.com/making/cf-spring-batch-demo spring.application.name=hello-batch
  • 102. © 2016 Pivotal Software, Inc. All rights reserved. Task Execution List Task Name ID Start Time End Time Exit Code hello- batch 5 Tue Nov 15 17:28:49 JST 2016 Tue Nov 15 17:28:49 JST 2016 0 hello- batch 4 Tue Nov 15 17:27:56 JST 2016 Tue Nov 15 17:27:57 JST 2016 0
  • 103. © 2016 Pivotal Software, Inc. All rights reserved. Job Execution List ID Task ID Start Time Step Execution Count Definition Status 2 5 Tue Nov 15 17:28:49 JST 2016 2 Created 1 4 Tue Nov 15 17:27:56 JST 2016 2 Created
  • 104. © 2016 Pivotal Software, Inc. All rights reserved. Step Execution List ID Step Name Job Exec Id Start Time End Time Status 1 step1 1 Tue Nov 15 17:28:49 JST 2016 Tue Nov 15 17:28:49 JST 2016 COMPLETED 2 step2 1 Tue Nov 15 17:27:56 JST 2016 Tue Nov 15 17:27:57 JST 2016 COMPLETED
  • 105. © 2016 Pivotal Software, Inc. All rights reserved. Task Orchestration in SCDF >task create hello --definition="timestamp --format=¥"yyyy¥"" >task launch hello timestamp Data Flow Server DB Task Name Start Time End Time Exit Code Exit Message Last Updated Time Parameters Message Broker
  • 106. © 2016 Pivotal Software, Inc. All rights reserved. Task Execution List
  • 107. © 2016 Pivotal Software, Inc. All rights reserved. Job Execution List
  • 108. © 2016 Pivotal Software, Inc. All rights reserved. Job Execution Details
  • 109. © 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Data Flow -> Task http | tasklaunchrequest --uri=... | task-lancher timestamp Data Flow Server DB Message Broker
  • 110. ‹#›© 2016 Pivotal Software, Inc. All rights reserved. Demo (Scalable Tasks)
  • 111. © 2016 Pivotal Software, Inc. All rights reserved. Pivotal Cloud Foundry PDF PDF PDF PDF S3 Spring Cloud Data Flow Task Task Task Task PDF PDF Spring Cloud Task Spring Batch index index source sink
  • 112. ‹#›© 2016 Pivotal Software, Inc. All rights reserved. Getting Started with SCDF on Cloud Foundry
  • 113. © 2016 Pivotal Software, Inc. All rights reserved. Install PCF Dev • https://docs.pivotal.io/pcf-dev • Cloud Foundry on your laptop • Included • Redis / RabbitMQ / MySQL • Spring Cloud Services • Install with cf dev start
  • 114. © 2016 Pivotal Software, Inc. All rights reserved. cf create-service p-mysql 512mb df-mysql cf create-service p-rabbitmq standard df-rebbitmq cf create-service p-redis shared-vm df-redis cf push dataflow-server -p spring-cloud-dataflow- server-cloudfoundry-1.0.1.RELEASE.jar Deploy SCDF Server on PCF Dev
  • 115. © 2016 Pivotal Software, Inc. All rights reserved. Tutorial • 📗 https://blog.ik.am/entries/396 • 💎 https://github.com/making-demo-scdf/spring-cloud- dataflow-cookbook
  • 116. © 2016 Pivotal Software, Inc. All rights reserved. Upcoming Features • Some ‘porting’ from XD • Batch Job DSL + Designer • Role based access • Looking forward • Spring Cloud Sleuth • JavaDSL • In-place application version upgrades with Spinnaker • Application Groups • Polyglot • Expanded analytics with Redis and Python/R ecosystem • More provided apps/tasks •SCDF Tile for Pivotal Cloud Foundry
  • 117. © 2016 Pivotal Software, Inc. All rights reserved. Thanks! • References • http://www.slideshare.net/SpringCentral/data-microservices-in-t cloud ✨ • https://cloud.spring.io/spring-cloud-dataflow/ • http://cloud.spring.io/spring-cloud-stream/ • http://cloud.spring.io/spring-cloud-stream-app-starters/ • https://github.com/spring-cloud/spring-cloud-task/ • http://cloud.spring.io/spring-cloud-dataflow-server-cloudfoundry