Speaker: Ben Stopford, Technologist, Office of the CTO, Confluent
Are events the new API? Event driven systems provide some unique properties, particularly for microservice architectures, as they can be used both for notification as well as for state transfer. This lets systems run in a broad range of use cases that cross geographies, clouds and devices.
In this talk we will look at what event driven systems are; how they provide a unique contract for services to communicate and share data and how stream processing tools can be used to simplify the interaction between different services, be them closely coupled or largely disconnected.
Ben is a technologist working in the Office of the CTO at Confluent Inc (the company behind Apache Kafka®). He’s worked on a wide range of projects, from implementing the latest version of Kafka’s replication protocol through to developing strategies for streaming applications. Before Confluent Ben led the design and build of a company-wide data platform for a large investment bank. His earlier career spanned a variety of projects at ThoughtWorks and UK-based enterprise companies. He is the author of the book “Designing Event Driven Systems,” O’Reilly, 2018.
Watch the recording: https://videos.confluent.io/watch/8MLuNHnE3uSZPgstdzSk4Q?.
2. About me:
- Office of the CTO @Confluent
- Kafka Core (replication protocol,
throttling, data balancing…)
- Built a central data service for a large
financial intuition
- Background in High Performance
Computing
- Ex-Thoughtworks
http://bit.ly/designing-event-driven-systems
3. What is an API?
Ordering fish and chips
Chef
Customer
8. An API just describes this
interchange
• Choose what you want
• Ask for your food
• Wait for it to be prepared
• Get your food
9. With Events
Events are just facts that describe the real world:
• User: ordered fish and chips
• Kitchen: fish and chips prepared
• User: fish and chips consumed
10. Use a ‘Broker’: A middle man.
KitchenTable
The waiter records what you
chose to eat
The waiter brings your
food back to you
37. Businesses have an implicit
flow
Order
Requested
Order
Validated
Payment
Processed
Shipment
Dispatched
Shipment
Delivered
Shipment
Prepared
Order
Completed
Order
Confirmed
38. Fit services around this flow
Order
Requested
Order
Validated
Payment
Processed
Shipment
Dispatched
Shipment
Delivered
Shipment
Prepared
Order
Completed
Order
Confirmed
Basket Service
Order Service
Shipment Service
Payment Service
39. Let’s explore this event driven
approach with an example
Orders
Service
Shipping
Service
Customer
Service
Webserver
40. Buying an iPad (with REST)
Submit
Order
shipOrder() getCustomer()
Orders
Service
Shipping
Service
Customer
Service
Webserver
41. Events for Notification Only
Message Broker (Kafka)
Submit
Order
Order
Created
getCustomer()
REST
Notification
Orders
Service
Shipping
Service
Customer
Service
Webserver
KAFKA
43. Events for Data Locality
Customer
Updated
Submit
Order
Order
Created
Data is
replicated
Orders
Service
Shipping
Service
Customer
Service
Webserver
KAFKA
70. SELECT card_number, count(*)
FROM authorization_attempts
WINDOW (SIZE 5 MINUTE)
GROUP BY card_number
HAVING count(*) > 3;
Continuously Running Queries
75. Streaming leverages the
two hats
1. Processing data incrementally
2. Moving data to where it needs
to be processed (quickly and
efficiently)
On Notification
Data Replication
76. Key Features
• ‘Big data’ throughput & storage
• High availability and always on execution
• Messaging separated from processing/routing
• Routing/inspection/analytics provided via SQL
77. Kafka: a Streaming Platform
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine
78. So what do real-
world streaming
ecosystems look
like?