use cases and architecture from the real world of using Kafka

1. Using Streaming APIs in Production:
use cases and architecture from the real
world
Luca Ferrari - EMEA Solution Architect
Red Hat

2. Red Hat2
Agenda:
❏ Why am I here?
❏ Why are you here?
❏ Short Kafka intro
❏ Kafka & microservices
❏ Kafka & streaming
❏ Kafka in production
❏ Demo

3. Red Hat3
KAFKA INTRODUCTION

4. Red Hat
pub/sub
4

5. Red Hat
topic, partitions and offset
5

6. Red Hat
truck gps example
6

7. Red Hat
broker and partitions
7

8. Red Hat
replication and leaders
8

9. Red Hat
producers
9

10. Red Hat
consumers
10

11. Red Hat
delivery semantics
11

12. Red Hat
broker discovery
12

13. Red Hat
zookeeper
13

14. Red Hat
full picture
14

15. Red Hat15
KAFKA &
MICROSERVICES

16. Red Hat
monolithic approach
16

17. Red Hat
SOA approach
17

18. Red Hat
REST approach
18

19. Red Hat
API Management approach
19

20. Red Hat
observations
20
What we want:
● Improve decoupling
● Improve scalability
What we don’t want:
● Service coupling
● Ripple effect
● Service overload
● Slow response time

21. Red Hat
event driven approach
21

22. Red Hat22
KAFKA &
STREAMING

23. Red Hat
including streaming data
23

24. Red Hat
Kafka Streams
24

25. Red Hat
Kafka Streams &
Streaming APIs
25
Kafka Strengths:
● Real Time Data Pipeline
● High-throughput
● Fault-tolerant
● Low Latency
● Scalability
Kafka Limits:
Kafka is not optimized to provide API consumers with friendly access to real time data.
Difficult to expose and re-use API oriented tools for Kafka

26. Red Hat
Kafka Streams &
Streaming APIs
26
Protocol bridge
The way to present the information is not friendly but for the REST all is good
Is there an easy Kafka way to bridge Kafka and REST?
There are several: Apache Camel to the rescue!

27. Red Hat27
KAFKA
IN PRODUCTION

28. Red Hat
Industries
28
➔ Travel companies
➔ Finance and fintech companies
➔ Retailers and online shopping
➔ Automotive and manufacturing companies
➔ Video Streaming companies
➔ Social networks
➔ Transportation
...
[ https://kafka.apache.org/powered-by]

29. Red Hat
Travel: Singapore Airlines
29
Problem:
Many
airplanes
Many components
per airplane
Each airplane
different
flight-plan
! Predictive maintenance is a hard !

30. Red Hat
Travel: Singapore Airlines
30
Solution:
Kafka
Connector
Kafka
Streams App ML model
Web App

31. Red Hat
Transportation: Uber
31
Problem:
Multiple
locations
Multiple riders Multiple drivers
All of this in real time…
… and if this sounds easy, let’s make fares
variable in real time as well!

32. Red Hat
Transportation: Uber
32
Solution:

33. Red Hat
Transportation: Uber
33
Speaking about Kafka scalability:
Trillions of
messages
per day
PBs of data Tens of
thousands of
topics

34. Red Hat
Finance: ING
34
https://www.slideshare.net/ConfluentInc/kafka-summit-nyc-2017-the-realtime-event-driven-bank-a-kafka-story?qid=39eb
618f-fdcc-4b26-bdbe-6de13b4e8d85&v=&b=&from_search=6
Challenges to traditional banks:
➔ Offering actionable insights in real time to customer
➔ Differentiating user experience
➔ Convergence of channels
➔ Make the insights personal

35. Red Hat
Finance: ING
35
Started with Fraud Detection process:

36. Red Hat
Finance: ING
36
Extended the architecture globally:

37. Red Hat
News: New York Times
37
Problems:
● Consumers need to know everything about Producers
● Producer API has to live forever
● Every API has a different information schema
● No easy way to read historical info in bulk
● Monoliths that tried to do everything for everyone
● Hard to develop new products and change old ones
Typical API oriented architecture
[ https://www.slideshare.net/ConfluentInc/apache-kafka-delivers-a-single-source-of-truth-for-the-new-york-times ]

38. Red Hat
News: New York Times
38
Solution:
[ https://www.slideshare.net/ConfluentInc/apache-kafka-delivers-a-single-source-of-truth-for-the-new-york-times ]

39. Red Hat
News: New York Times
39
Solution:
1. Log is the source of truth
2. All assets are protobuf binaries
3. Use a Monolog (single partition, completely ordered, infinite
retention
[ https://www.slideshare.net/ConfluentInc/apache-kafka-delivers-a-single-source-of-truth-for-the-new-york-times ]

40. Red Hat
Leisure: Tinder
40
Problems:
● Delayed notifications
● Scheduled content moderation
● Managing inactive users, likely to drop out
● Schedule feature drop correctly
Didn’t realize dating was so complex nowadays

41. Red Hat
Leisure: Tinder
41
Delayed
scheduling:

42. Red Hat
Leisure: Tinder
42
Content
moderation:

43. Red Hat
Leisure: Tinder
43
User
reactivation:

44. Red Hat44
KAFKA DEMO

45. Red Hat
3 microservices:
1. Order to create orders. This services sends messages to Kafka. It uses the
KafkaTemplate.
2. Shipment receives the orders and extract the information needed to ship the items.
3. Invoicing receives the messages, too. It extracts all information to send out an invoice.
It uses @KafkaListener just like Shipment.
How much Kafka do you want in your life?
45 [ https://github.com/ewolff/microservice-kafka ]

46. Red Hat46
SHORT SURVEY

47. Red Hat
http://tiny.cc/brvz6y
How much Kafka do you want in your life?
47

48. THANK YOU !
plus.google.com/+RedHat
linkedin.com/company/red-hat
youtube.com/user/RedHatVideos
facebook.com/redhatinc
twitter.com/RedHatNews
48