Date: 15th November 2017 Location: Fast Data Theatre Time: 15:50 - 16:20 Speaker: Ben Stopford Organisation: Confluent

1. The Streaming
Transformation
Ben Stopford
@benstopford

2. Build
Features
Build for
the Future

3. Evolution!

4. KAFKA
Serving
Layer
(Cassandra etc)
Kafka Streams /
KSQL
Streaming Platforms
Data is embedded in
each engine
High Throughput
Messaging
Clustered
Java App

5. authorization_attempts possible_fraud
Streaming Example

6. CREATE STREAM possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 MINUTE)
GROUP BY card_number
HAVING count(*) > 3;
authorization_attempts possible_fraud

7. CREATE STREAM possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 MINUTE)
GROUP BY card_number
HAVING count(*) > 3;
authorization_attempts possible_fraud

8. CREATE STREAM possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 MINUTE)
GROUP BY card_number
HAVING count(*) > 3;
authorization_attempts possible_fraud

9. CREATE STREAM possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 MINUTE)
GROUP BY card_number
HAVING count(*) > 3;
authorization_attempts possible_fraud

10. CREATE STREAM possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 MINUTE)
GROUP BY card_number
HAVING count(*) > 3;
authorization_attempts possible_fraud

11. CREATE STREAM possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 MINUTE)
GROUP BY card_number
HAVING count(*) > 3;
authorization_attempts possible_fraud

12. Streaming == Manipulating Data in Flight
• Join
• Aggregate
• Map
• Reduce
• Peek
• Transform
• (any arbitrary code)
• Window
• Transactions

13. Kafka: a Streaming Platform
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine

14. Kafka: a Streaming Platform
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine

15. What is a Distributed Log?

16. Shard on the way in
Producing
Services
Kafka
Consuming
Services

17. Each shard is a queue
Producing
Services
Kafka
Consuming
Services

18. Share Load / Fault Tollerant
Producing
Services
Kafka
Consuming
Services

19. Retain datasets in the log. “Rewind & Replay”.
Rewind & Replay

20. Kafka: a Streaming Platform
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine

21. Kafka Connect
Kafka
Connect
Kafka
Connect
Kafka

22. Kafka: a Streaming Platform
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine

23. SELECT card_number, count(*)
FROM authorization_attempts
WINDOW (SIZE 5 MINUTE)
GROUP BY card_number
HAVING count(*) > 3;
KSQL is SQL over Kafka Streams

24. Kafka Streams is just an API
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
builder.stream(”caterpillars")
.map((k, v) -> coolTransformation(k, v))
.to(“butterflies”);
new KafkaStreams(builder.build(), props()).start();
}
24

25. KAFKA
Buffer 10 mins
Windows

26. Windows / Retention – Handle Late Events
KAFKA
Buffer 5 mins
Join by Key
KStream orders = builder.stream(“Orders”);
KStream payments = builder.stream(“Payments”);
orders.join(payments, KeyValue::new, JoinWindows.of(10 * MIN))
.peek((key, pair) -> emailer.sendMail(pair));

27. Lookup tables
KAFKA

28. A KTable is just a stream with infinite retention
KStream orders = builder.stream(“Orders”);
KStream payments = builder.stream(“Payments”);
KTable customers = builder.table(“Customers”);
orders.join(payments, EmailTuple::new, JoinWindows.of(10*MIN))
.join(customers, (tuple, cust) -> tuple.setCust(cust))
.peek((key, tuple) -> emailer.sendMail(tuple));
KAFKA
Join
Materialize a
table in two
lines of code!
Dataset Moves
to Client

29. Streaming is about
1. Processing data incrementally
2. Moving data to where it needs to be
processed (quickly and efficiently)

30. Kafka: a Streaming Platform
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine

31. Business
Applications

32. EcosystemsApp
Increasingly we build
ecosystems

33. SOA / Microservices / EDA
Customer
Service
Shipping
Service

34. The Problem is DATA

35. Most services share the same core facts.
Catalog
Most services
live in here

36. Buying an iPad with REST
Submit
Order
shipOrder() getCustomer()
Orders
Service
Shipping
Service
Customer
Service
Webserver

37. Buying an iPad with Events
Message Broker (Kafka)
Notification Data is
replicated
(incrementally)
Submit
Order
Order
Created
Customer
Updated
Orders
Service
Shipping
Service
Customer
Service
Webserver
KAFKA

38. Events for Notification Only
Message Broker (Kafka)
Submit
Order
Order
Created
getCustomer()
REST
Notification
Orders
Service
Shipping
Service
Customer
Service
Webserver
KAFKA

39. Events for Data Locality
Customer
Updated
Submit
Order
Order
Created
Data is
replicated
(incrementally)Orders
Service
Shipping
Service
Customer
Service
Webserver
KAFKA

40. Events have two hats
Notification Data
replication

41. Apply Stream Processing
Streaming is about manipulating data in flight

42. Kafka is a high throughput
Messaging & Storage System
Orders
Service
Shipping
Service
Customer
Service
KAFKA
Web App

43. Kafka Streams API (or KSQL)
An embedded API for data in flight
Orders
Service
Shipping
Service
Customer
Service
KAFKA
Web App

44. Streaming platforms
optimize for moving
data to code!?!

45. Add a more data
intensive use case

46. A Scrollable Grid
Orders
Service
Customer
Service
Web App
Scrollable Grid
Customer & Order in
each row
Many rows

47. Add caching -> problems of its own
Orders
Service
Customer
Service
Web App
Scrollable Grid

48. The Streaming Way
Orders
Service
Customer
Service
KAFKA
Web App
Scrollable Grid
(with RocksDB)
KStreams API
Select * from
orders, customers
where…

49. Streams & Tables
Orders
Service
Customer
Service
KAFKA
Web App
Scrollable Grid
Orders
provide
Notification
Customers
are replicated

50. Add Payments Service & Window
Orders
Service
Customer
Service
KAFKA
Web App
Scrollable Grid
Orders
provide
Notification
Customers
are replicated
Payments
Service
Buffer /
Window

51. Orders
Service
Customer
Service
KAFKA
Web App
Scrollable Grid
Payments
Service
Query Runs INSIDE the webserver

52. Orders
Service
Customer
Service
KAFKA
Web App
Scrollable Grid
Payments
Service
Events are stored in Kafka (e.g. Customers)

53. Orders
Service
Customer
Service
KAFKA
Web App
Scrollable Grid
Payments
Service
Streaming is about Data Movement
Materialized View

54. POST
GET
Load
Balancer
ORDERSORDERS
OVTOPIC
Order
Validations
KAFKA
INVENTORY
Orders
Inventory
Fraud
Service
Order
Details
Service
Inventory
Service
(see previous figure)
Order
Created
Order
Validated
Orders
View
Q in CQRS
Orders
Service
C is CQRS
Services in the Micro: Orders Service
Find the
code online!

55. Orders Customers
Payments
Stock
Query Engine (Kstreams/KSQL)
Larger Ecosystems
HISTORICAL
EVENT STREAMS

56. Kafka
KAFKA
New York
Tokyo
London
Global / Disconnected Ecosystems

57. WIRED Principals
• Windowed: Use an API built for async events
• Immutable: Store events in an immutable log
• Repeatable: Compose from side-effect free functions
• Evolutionary: Be pluggable. Have data available in the log.
• Data-Enabled: Push data to services where necessary

58. Makes it easier to evolve!

59. The Streaming
Transformation

60. References
• Confluent Microservices Series:
https://www.confluent.io/blog/tag/microservices
• Code examples:
https://github.com/confluentinc/kafka-streams-
examples

61. Twitter:
@benstopford