Presentation by Maruis Bogoevici, Pivotal

1. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Cloud Native Data Streaming
Microservices
with Spring Cloud and Kafka
Marius Bogoevici, Pivotal
@mariusbogoevici

2. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Who am I ?
• Software Engineer with Pivotal
– Project Lead, Spring Cloud Stream
• Spring contributor since 2008
– Spring Integration, Spring XD, Spring Integration Kafka, Spring Cloud
Stream
• Co-author, “Spring Integration in Action”, Manning, 2012

3. From Platform to Microservices
3
Spring Cloud Stream2015
Spring XD Spring Cloud Data Flow
Spring Cloud Task

4. Microservices for data processing
• Stand-alone, production grade applications
• Architectural advantages:
• Bounded contexts
• Development agility
• Immutable infrastructure
• Failure isolation
• Process-specific resource tuning:
• memory, CPU, instance count, network
• But also, added complexity (“distributed systems are hard”)
4

5. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Event-driven microservices
● Messaging vs HTTP/“fire-and-forget” vs. request-reply
o Decoupling
o Discoverability
o Availability
● Eventual consistency vs distributed transactions
o especially over heterogenous resources
● Strong similarities between:
o data streaming - ‘big data’: ingestion, analytics
o event streaming - async interaction, event sourcing, etc

6. Cloud-native data processing with microservices
• Rely on platform capabilities for scalability/resilience requirements
• Resource management
• memory, CPU, instance count
• Scaling up/down
• Health monitoring and failover
• Routing and load balancing
6
Apache Mesos Kubernetes

7. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Kafka in the Spring Ecosystem
• Spring Kafka
• Spring Boot AutoConfiguration
• Spring Integration Kafka
• Spring Cloud Stream
• Reactor Kafka

8. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Spring for Apache Kafka
• Spring concepts to Kafka messaging applications
• Lifecycle management
– Configuration
– Dependency Injection
• Kafka Template for sending messages
• Listener container for receiving messages
• Kafka Streams support
– Container-based configuration
– Management of Kafka Streams Lifecycle
•Automatic start-stop
•Automatic management of KStreamBuilder

9. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Spring Boot + Spring Kafka
• Spring Boot - simplified Spring configuration for microservices
– Uberjar
• Simplified support for simple Kafka applications

10. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Spring Integration Kafka
• Spring Integration - Enterprise Application Integration with Spring
– Enterprise Integration Patterns
• Kafka extensions for Spring Integration
– Inbound channel adapters
– Outbound channel adapters

11. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Data and Event Streaming: Conceptually Similar

12. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Spring Cloud Stream in a nutshell

13. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
… 10000 ft nutshell

14. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Binder abstraction

15. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Programming model
@EnableBinding + Binder Implementation
Apache Kafka
JMS Google PubSub
Production-ready:
Experimental

16. Programming model - individual message
handling
16
@SpringBootApplication
@EnableBinding(Processor.class)
public class UppercaseProcessor {
@StreamListener(“input”)
@SendTo(“output”)
public String process(String s) {
return s.toUpperCase();
}
}

17. [dumb] Pipes and [smart] filters
17
http averag
es
topN
ingest
data
average value
over last 5s
hottest sensors
over last 10 seconds

18. Programming model : Kafka Streams
18
@StreamListener(“input”)
@SendTo(“output”)
public KStream<Integer,String> addByKey(KStream<Integer,String> input)
{
return input.map((i,s) -> new KeyValue<>(i, s.toUpperCase()))
.reduceByKey((String value1, String value2) -> value1 + value2,
TimeWindows.of("windowName", 1000))
.toStream()
}

19. Programming model : Kafka Streams (2)
19
@StreamListener
@SendTo(“joined”)
public KStream<byte[],ClickedImpressions> join(
@Input(“clicks”) KStream<byte[],Clicks> clicks,
@Input(“impressions”) KStream<byte[],Impressions> impressions){
// join clicks and impressions
}

20. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Spring Cloud Stream + Kafka Streams
• 1:1 match - KStreams are inputs and outputs
• Spring Cloud Stream
• subscribes input KStreams to topics
• connects the output KStream
• Spring Boot configuration
• Exporting metrics
• Spring Cloud Stream content type negociation
• Underlying Spring Boot
• Flexible configuration: program arguments, environment variables
• Actuator endpoints: health, metrics

21. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Spring Cloud Stream + Kafka Streams
clicks
gadgets
joined
Spring Cloud Stream KStream Processor
clicks-us-east
impressions-us-east
joined-us-east
Kafka topics
Spring Boot
KStream API
Spring Cloud Stream
Kafka topics

22. Spring Cloud Data Flow
• Orchestration:
• DSL for Stream topologies
• REST API
• Shell
• UI
• Portable Deployment SPI
• OOTB apps for common integration use-
cases
22

23. Simple topologies : (relatively) easy to deploy
…
23
http
HDF
S
spring.cloud.stream.bindings.output.destinatio
n=httphdfs.1
spring.cloud.stream.bindings.input.destination=h
spring.cloud.stream.bindings.input.group=httphd
httphdfs.1

24. … but how about complex topologies ?
24
http
raw-sensor-data
averag
es
top-n
Calculat
or
Failure
detector
averag
es
averag
es
HDFSHDFSHDFS

25. Spring Cloud Data Flow - Stream DSL
25
Stream definition
Boot Apps built
on top of Spring
Ecosystem
httpfile = http | file
|

26. Spring Cloud Data Flow Deployment Platforms
26
Data Flow Server
REST API
Deployer SPI
SCDF FloSCDF Shell

27. 27
cassandra
gpfdist
http
stream1 = http | cassandra
stream2 = jms | gpfdist
Kafka
Data Flow Server DB
Platform Runtime
jms

28. Deployment: Partitioning and Instance Count
28
http
http
work
work
work
hdfs
hdfs
hdfs
hdfs
LoadBalancer
stream create s1 --definition “http | work | hdfs”
stream deploy s1 --propertiesFile ingest.properties
app.http.count=2
app.work.count=3
app.hdfs.count=4
app.http.producer.partitionKeyExpression=payload.id

29. Deployment: Resource Management
29
http
http
work
work
work
app.work.spring.cloud.deployer.cloudfoundry.memory=2048

30. Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Demo (Video?)
• Creating a Spring Cloud Stream Kafka Streams Processor from scratch
• Start the processor locally and show processing data from an http source
• Start Spring Cloud Data Flow
• Install the newly created processor as a processor application
• Create and deploy a stream on Google Cloud Platform