Imagine a world where operational data is continuously flowing from applications and devices at an extremely high rate. Now imagine services intercepting this data and analyzing it real time. Sounds futuristic? It's not—it's here today. Mark Richards describes what streaming architecture is all about—what it is, when to use it, and how to implement it in a microservices ecosystem. Mark describes the overall ecosystem for streaming architecture—including a brief discussion about the differences in Apache Spark, Flink, and Hadoop—and then explains how Apache Kafka works. Using live coding examples in Kafka, Mark demonstrates some techniques for leveraging streaming data such as metrics gathering for monitoring, distributed logging, request distribution analysis, and threshold analysis in a microservices ecosystem. Join Mark as he discusses some of the implications and limitations of Kafka and the important topic of when to choose Kafka over standard messaging such as JMS, AMQP, and MSMQ.

1. BW5
Concurrent Session
11/8/17 1:30 PM
Leverage Streaming Data in a
Microservices Ecosystem
Presented by:
Mark Richards
Independent Consultant
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073
888---268---8770 ·· 904---278---0524 - info@techwell.com - https://www.techwell.com/

2. Mark Richards
Independent Consultant
In the software industry since 1983, Mark Richards is an experienced, hands-on
software architect involved in the architecture, design, and implementation of
microservices architectures, service-oriented architectures, and distributed
systems. He has significant experience and expertise in application, integration,
and enterprise architecture. Mark is the author of O'Reilly books on
microservices; the Software Architecture Fundamentals video series; Enterprise
Messaging video series; Java Message Service, 2nd Edition; and a contributing
author to 97 Things Every Software Architect Should Know. Mark is a frequent
speaker on enterprise-related technical topics at conferences and user groups
worldwide. www.wmrichards.com

3. NFJS Software Symposium Series 2016
Author of Software Architecture Fundamentals Video Series (O’Reilly)
Author of Microservices Pitfalls and AntiPatterns (O’Reilly)
Author of Microservices vs. Service-Oriented Architecture (O’Reilly)
Author of Enterprise Messaging Video Series (O’Reilly)
Author of Java Message Service 2nd Edition (O’Reilly)
Independent Consultant
Hands-on So*ware Architect
Published Author / Conference Speaker
www.wmrichards.com
Mark Richards
Leveraging Streaming Data in a
Microservices Ecosystem
agenda
kafka producers and consumers
streaming architecture patterns
real-world examples of streaming data
kafka vs. standard messaging
kafka overview

4. https://github.com/wmr513/streaming
source code
Streaming Architecture
Patterns

5. streaming architecture patterns
service
service
service
analytics
capture
service
service
service
analytics
capture
streaming architecture patterns
very high data volume and
throughput rate

6. streaming architecture patterns
data loss is acceptable
streaming architecture patterns
data duplication is acceptable

7. streaming architecture patterns
not all data needs to be
persisted
streaming architecture patterns

8. streaming architecture patterns
service
capture and store
filter and store
analyze and store
Kafka Overview

9. kafka overview
publish and subscribe hybrid messaging model
messages are always persisted in a partitioned file by topics
message throughput can be upwards to 1,000,000+/sec
record size (bytes)
throughput(records/second)
200K
400K
600K
800K
1K10 100K10K100
kafka overview
topic structure
partition 0
producer
consumer 1 consumer 2
msg
4
msg
1
msg
2
msg
3
msg
5
msg
6
msg
7
offset 0offset 1offset 2offset 3 offset 3offset 4offset 5
…

10. kafka overview
topic structure
partition 0
msg
8
msg
1
msg
2
msg
4
msg
9
msg
11
partition 1
msg
3
msg
7
msg
10
partition 2
msg
13
msg
5
msg
6
msg
12
msg
14
msg
15
msg
16
msg
17
msg
18
msg
19
producer
Kafka Producers and
Consumers

11. kafka producers and consumers
service
(producer)
client
metrics
(consumer)
offsets (0.11)
service
(producer)
kafka producers and consumers
client
metrics
(consumer)
offsets (0.11)

12. kafka producers and consumers
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "…kafka…StringSerializer");
props.put("value.serializer", "…kafka…StringSerializer");
KafkaProducer<String, String> producer =
new KafkaProducer<String, String>(props);
String topic = "customer_comment_service_metrics";
String key = "duration";
String value = "320";
ProducerRecord<String, String> msg =
new ProducerRecord<>(topic, key, value);
producer.send(msg);
producer.flush();
producer.close(); messages are sent in a batch
within a separate thread
service (producer)
kafka producers and consumers
client
service
(producer)
metrics
(consumer)
offsets (0.11)

13. kafka producers and consumers
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "CG1");
props.put("key.deserializer", "…kafka…StringDeserializer");
props.put("value.deserializer", "…kafka…StringDeserializer");
KafkaConsumer<String, String> consumer =
new KafkaConsumer<String, String>(props);
consumer.subscribe(Arrays.asList(
"customer_comment_service_metrics"));
metrics (consumer)
kafka producers and consumers
try {
while (true) {
ConsumerRecords<String, String> msgs = consumer.poll(100);
for (ConsumerRecord<String, String> msg : msgs) {
System.out.println("topic: " + msg.topic());
System.out.println("key: " + msg.key());
System.out.println("value: " + msg.value());
System.out.println("partition: " + msg.partition());
System.out.println("offset: " + msg.offset());
}
} finally {
consumer.close();
}
metrics (consumer)
we are using auto commit of
our offset sync point (5 sec)

14. kafka producers and consumers
try {
while (true) {
ConsumerRecords<String, String> msgs = consumer.poll(100);
for (ConsumerRecord<String, String> msg : msgs) {
System.out.println("topic: " + msg.topic());
System.out.println("key: " + msg.key());
System.out.println("value: " + msg.value());
System.out.println("partition: " + msg.partition());
System.out.println("offset: " + msg.offset());
}
try {
consumer.commitSync();
} catch (CommitFailedException e) {
log.error("rats - I have no idea what to do now!");
}
} finally {
consumer.close();
}
metrics (consumer)
kafka producers and consumers

15. Real-World Examples of
Streaming Data
microservices metrics analytics
trade
generator
trade
validation

16. microservices metrics analytics
trade
generator
trade
validation
metrics
analyzer
trade_validation_service_metrics
trade_gen_service_metrics
microservices metrics analytics
trade
validation trade_validation_service_metrics
trade_gen_service_metrics
trade
generator
metrics
analyzer
threshold
analyzer

17. microservices metrics analytics
trade
validation trade_validation_service_metrics
trade_gen_service_metrics
trade
generator
metrics
analyzer
threshold
analyzer
trade_gen_service_error
error
analyzer
microservices metrics analytics
trade
validation trade_validation_service_metrics
trade_gen_service_metrics
trade
generator
metrics
analyzer
threshold
analyzer
trade_gen_service_error
error
analyzer
trade_gen_service_symbol
symbol
analyzer

18. microservices metrics analytics
trade
validation trade_validation_service_metrics
trade_gen_service_metrics
trade
generator
metrics
analyzer
threshold
analyzer
trade_gen_service_error
error
analyzer
trade_gen_service_symbol
symbol
analyzer
Kafka vs. Messaging

19. primary data type
throughput
payload
data loss
data duplication
msg confirmation
operational data transactional data
up to 1 million/sec up to 4K/sec (10K/sec)
single name/value pair aggregate data
possible* rare
possible* rare
consumer managed broker managed
apache kafka vs. standard messaging
* StreamsAPI significantly reduces data loss and duplication
load balancing
message order
msg properties
messaging models
msg persistence
supported supported
consumer control preserved (fifo)*
supported** supported
pub/sub pub/sub, p2p, hybrid
always optional
guaranteed delivery not supported** supported
apache kafka vs. standard messaging
** StreamsAPI provides some support for these
* use of message priority can change message order

20. Summary

21. NFJS Software Symposium Series 2016
Author of Software Architecture Fundamentals Video Series (O’Reilly)
Author of Microservices Pitfalls and AntiPatterns (O’Reilly)
Author of Microservices vs. Service-Oriented Architecture (O’Reilly)
Author of Enterprise Messaging Video Series (O’Reilly)
Author of Java Message Service 2nd Edition (O’Reilly)
Independent Consultant
Hands-on So*ware Architect
Published Author / Conference Speaker
www.wmrichards.com
Mark Richards
Leveraging Streaming Data in a
Microservices Ecosystem