Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
1
Building Event-Driven Services
with Apache Kafka
Ben Stopford
@benstopford
2
In this talk
1.  How we traditionally build services with
REST
2.  How Event Driven Services are different
3.  Building ...
3
Companies evolve in different
ways
•  Internet companies
•  Enterprise companies
4
Internet companies typically grow
from front-facing websites.
pic
Growing fast
5
Carve pieces off
6
If stateful, they might share a DB
7
Or they might use their own
8
Over time => more complex
9
Over time => more async
(async w.r.t. user)
10
Chained blocking commands
Buffering, handling failure, backpressure, scaling etc all get
pushed into service’s responsi...
11
The synchronous world of
request response protocols
leads to tight, point-to-point
couplings.
12
Enterprises are typically different
13
pic
Enterprise companies form as
heterogeneous application
conglomerates, often in sylos
14
The start life as independent islands
15
Linked together by files or via
messaging
pic
16
Interactions are async from day-1
pic
17
Forwarding of data gets messy:
Chinese Whispers / Telephone game
pic
18
Both these approaches often end up at
the same point, but for different
reasons
19
Data is herded into specific services
Enterprise
20
Data is herded into specific services
Data Services
Stateless
Services
Internet Co
21
Lots of Data Services
=> Distributed Join Problem
pic
This is a problem databases find hard,
and they’re highly tuned f...
22
Lots of Data Services
=> Distributed Join Problem
pic
TESTING BECOMES DIFFICULT!
Client Service
performs join
Payments ...
23
In short: the high degree of “connectedness”
makes it hard to evolve independently and to scale
24
LETS TRY
EVENT DRIVEN!
25
Interact through
Events,
Don't talk to
Services
26
This
Payments
Service
Purchase Requested Events
Listen & React to Events
Payment Completed Events
27
Not this
Service Payment
Service
http://payments/ID/42
Don’t Request/Command from individual services.
28
Let’s take an example...
29
UI
Service
Payment
Service
Stock
Service
(1) Process
Payment (2) Contact
Payment
Provider
(3) Update
stock
(4) Maybe
or...
30
The Event Driven Way
UI
Service
Payment
Service
Stock
Service
Payment
Provider
Maybe order
more stock
31
UI
Service
Payment
Service
Stock
Service
Payment
Provider
Kafka Topics:
(1)  Purchases
(2)  Payments
Maybe order
more s...
32
UI
Service
Payment
Service
Stock
Service
Payment
Provider
Maybe order
more stock
New Order -> PurchaseRequested Event
R...
33
UI
Service
Payment
Service
Stock
Service
Payment
Provider
Maybe order
more stock
Payment Service Listens In
Raise
Purch...
34
UI
Service
Payment
Service
Stock
Service
Payment
Provider
Maybe order
more stock
When Complete -> PaymentProcessed
Rais...
35
UI
Service
Payment
Service
Stock
Service
Payment
Provider
Maybe order
more stock
Stock service hooks in too!
Raise
Purc...
36
UI
Service
Payment
Service
Stock
Service
Payment
Provider
Maybe order
more stock
Payment service listens
Raise
Purchase...
37
UI
Service
Payment
Service
Stock
Service
Payment
Provider
Stock service entirely
owns its flow control
Maybe order
more...
38
UI
Service
Payment
Service
Stock
Service
Quite different to Synchronous Model
Stock service flow
control owned by
payme...
39
1. Events are Immutable Facts
(state changes taken from the real world and journalled)
UI
Service
Purchases
Payments
Pu...
40
2. Services couple only to data flows
No knowledge of contributing services (RDFC)
UI
Service
Purchases
Payments
Purcha...
41
Because coupling is only to Events, not
services
Purchase Requests
PurchaseRequest TopicUI
Service
Payment
Service
Stoc...
42
The architecture is “Pluggable”
Purchase Requests
PurchaseRequest TopicUI
Service
Payment
Service
Stock
Service
Fraud
S...
43
Communication & State Concepts Merge
•  The concept of Data & Events
become one.
•  Kafka is both Event Store and
Commu...
44
The Events form a Canonical, Shared
Narrative
Payments
Orders
Shipments Customers
Evolving state of the system over time
45
Scaling is a concern of the broker, not
“upstream” services
Purchase Requests
PurchaseRequest TopicUI
Service
Payment
S...
46
But there is something missing!
How do we look things up?
(i.e. the Q in CQRS)
•  What is the address for this customer...
47
With REST Lookups are natural, if
Remote
UI
Service
Customer
Service
http://customers/ID/42
Go to the relevant service ...
48
Which creates the dependency problem
Many services tightly coupled to one another
49
With Event Driven we create “views” or
“projections” inside each bounded context
Projection of
data from
Outside
Contex...
50
Pattern 1: Local KTables
Shipping Service
Customers
Events
Customers By ID
(saved by KStreams
using RocksDB)
Kafka
51
Pattern 1: Local KTables
Shipping Service
Customers
Events
Customers By ID
KTable customers =
builder.table(
CustomerId...
52
As we’re using a streaming engine, we
can translate into any Domain Model
(projection) we wish
Shipping Service
Custome...
53
Combine different streams from
different services
Fraud
Service
Payment
Events
Purchase
Events
PurchasePayments
payment...
54
Pattern 1: Local KTables
•  Simple to create and query
•  Local to each service so fast
•  Powerful DSL for transformat...
55
Pattern 2(a): Queryable Interface
Fraud
Service
Kafka
Fraud Service
(Creates streaming projection)
Query via Rest Inter...
56
Pattern 2(b): Query Service
Kafka
Bespoke View Service
(Creates streaming
projection)
Rest Interface
Fraud
Service
Cont...
57
Pattern 2: Exposing Materialised
Projections
•  Similar to Pattern 1 but with a
“Query Interface” (using Kafka’s
Querya...
58
Pattern 3: External DB
DB of choice
Kafka
Connect
APIFraud
Service
Context
Boundary
59
Pattern 3: External DB
•  Again similar, but heavier weight
•  Use when you cannot pre-compute
the view (i.e. to do ad ...
60
Pattern 4: Hybrid
Fraud
Service
Kafka
Rest Interface
Context
Boundary
Booking
Service
NB this breaks the async model, b...
61
Pattern 4: Hybrid
•  Quite common in practice
•  Allows you to break out of the
async world
•  Use sparingly, but do us...
62
Benefits
•  Forms a central, immutable narrative
•  Communication Protocol and Event Store
become one
•  Better decoupl...
63
Interact through
Events,
Don't talk to
Services
64
But it's ok to break
the rules!
65
Pulling it all together
66
All order logic
Orders-by-Customer view
(KStreams + Queryable State API)
UI ServiceStream
Maintenance
Fulfillment uses ...
67
Thank You!
@benstopford
68
Discount code: kafcom17
‪Use the Apache Kafka community discount
code to get $50 off
‪www.kafka-summit.org
Kafka Summit...
Upcoming SlideShare
Loading in …5
×

Building Event-Driven Services with Apache Kafka

4,430 views

Published on

Should you use REST to sew services together? Is it better to use a richer, brokered protocol? This practical talk will dig into how we piece services together in event driven systems, how we we use a distributed log to create a central, persistent narrative and what benefits we reap from doing so.

Published in: Software
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Building Event-Driven Services with Apache Kafka

  1. 1. 1 Building Event-Driven Services with Apache Kafka Ben Stopford @benstopford
  2. 2. 2 In this talk 1.  How we traditionally build services with REST 2.  How Event Driven Services are different 3.  Building an Immutable, Shared Narrative. 4.  Leveraging Materialized Views. 5.  Relating to CQRS & Event Sourcing 6.  Pulling it all together
  3. 3. 3 Companies evolve in different ways •  Internet companies •  Enterprise companies
  4. 4. 4 Internet companies typically grow from front-facing websites. pic Growing fast
  5. 5. 5 Carve pieces off
  6. 6. 6 If stateful, they might share a DB
  7. 7. 7 Or they might use their own
  8. 8. 8 Over time => more complex
  9. 9. 9 Over time => more async (async w.r.t. user)
  10. 10. 10 Chained blocking commands Buffering, handling failure, backpressure, scaling etc all get pushed into service’s responsibilities
  11. 11. 11 The synchronous world of request response protocols leads to tight, point-to-point couplings.
  12. 12. 12 Enterprises are typically different
  13. 13. 13 pic Enterprise companies form as heterogeneous application conglomerates, often in sylos
  14. 14. 14 The start life as independent islands
  15. 15. 15 Linked together by files or via messaging pic
  16. 16. 16 Interactions are async from day-1 pic
  17. 17. 17 Forwarding of data gets messy: Chinese Whispers / Telephone game pic
  18. 18. 18 Both these approaches often end up at the same point, but for different reasons
  19. 19. 19 Data is herded into specific services Enterprise
  20. 20. 20 Data is herded into specific services Data Services Stateless Services Internet Co
  21. 21. 21 Lots of Data Services => Distributed Join Problem pic This is a problem databases find hard, and they’re highly tuned for it! Payments Service Orders Service Customers Service Client Service performs join
  22. 22. 22 Lots of Data Services => Distributed Join Problem pic TESTING BECOMES DIFFICULT! Client Service performs join Payments Service Orders Service Customers Service
  23. 23. 23 In short: the high degree of “connectedness” makes it hard to evolve independently and to scale
  24. 24. 24 LETS TRY EVENT DRIVEN!
  25. 25. 25 Interact through Events, Don't talk to Services
  26. 26. 26 This Payments Service Purchase Requested Events Listen & React to Events Payment Completed Events
  27. 27. 27 Not this Service Payment Service http://payments/ID/42 Don’t Request/Command from individual services.
  28. 28. 28 Let’s take an example...
  29. 29. 29 UI Service Payment Service Stock Service (1) Process Payment (2) Contact Payment Provider (3) Update stock (4) Maybe order more stock Request Response Example
  30. 30. 30 The Event Driven Way UI Service Payment Service Stock Service Payment Provider Maybe order more stock
  31. 31. 31 UI Service Payment Service Stock Service Payment Provider Kafka Topics: (1)  Purchases (2)  Payments Maybe order more stock The Event Driven Way
  32. 32. 32 UI Service Payment Service Stock Service Payment Provider Maybe order more stock New Order -> PurchaseRequested Event Raise PurchaseRequested Event
  33. 33. 33 UI Service Payment Service Stock Service Payment Provider Maybe order more stock Payment Service Listens In Raise PurchaseRequested Event Listen to PurchaseRequested Events
  34. 34. 34 UI Service Payment Service Stock Service Payment Provider Maybe order more stock When Complete -> PaymentProcessed Raise PurchaseRequested Event Listen to PurchaseRequested Events Raise PaymentProcessed Event *Some might use the Command/Query Pattern
  35. 35. 35 UI Service Payment Service Stock Service Payment Provider Maybe order more stock Stock service hooks in too! Raise PurchaseRequested Event Listen to PurchaseRequested Events Raise PaymentProcessed Event Listen to PaymentProcessed events
  36. 36. 36 UI Service Payment Service Stock Service Payment Provider Maybe order more stock Payment service listens Raise PurchaseRequested Event Listen to PurchaseRequested Events Raise PaymentProcessed Event Listen to PaymentProcessed events Listen to PaymentProcessed Event
  37. 37. 37 UI Service Payment Service Stock Service Payment Provider Stock service entirely owns its flow control Maybe order more stock Flow Control is Receiver Driven
  38. 38. 38 UI Service Payment Service Stock Service Quite different to Synchronous Model Stock service flow control owned by payments service
  39. 39. 39 1. Events are Immutable Facts (state changes taken from the real world and journalled) UI Service Purchases Payments Purchase Requested Payment Completed All the benefits of “Event Sourcing”
  40. 40. 40 2. Services couple only to data flows No knowledge of contributing services (RDFC) UI Service Purchases Payments Purchase Requested Payment Completed
  41. 41. 41 Because coupling is only to Events, not services Purchase Requests PurchaseRequest TopicUI Service Payment Service Stock Service
  42. 42. 42 The architecture is “Pluggable” Purchase Requests PurchaseRequest TopicUI Service Payment Service Stock Service Fraud Service
  43. 43. 43 Communication & State Concepts Merge •  The concept of Data & Events become one. •  Kafka is both Event Store and Communication Channel
  44. 44. 44 The Events form a Canonical, Shared Narrative Payments Orders Shipments Customers Evolving state of the system over time
  45. 45. 45 Scaling is a concern of the broker, not “upstream” services Purchase Requests PurchaseRequest TopicUI Service Payment Service Stock Service Fraud Service Kafka •  Linearly scalable •  Fault Tolerant •  Multi Tennant
  46. 46. 46 But there is something missing! How do we look things up? (i.e. the Q in CQRS) •  What is the address for this customer? •  Is this user allowed to view this stock? •  What is the contents of this user’s shopping basket?
  47. 47. 47 With REST Lookups are natural, if Remote UI Service Customer Service http://customers/ID/42 Go to the relevant service and ask!
  48. 48. 48 Which creates the dependency problem Many services tightly coupled to one another
  49. 49. 49 With Event Driven we create “views” or “projections” inside each bounded context Projection of data from Outside Context Boundary Shipping Service
  50. 50. 50 Pattern 1: Local KTables Shipping Service Customers Events Customers By ID (saved by KStreams using RocksDB) Kafka
  51. 51. 51 Pattern 1: Local KTables Shipping Service Customers Events Customers By ID KTable customers = builder.table( CustomerId, Customer, “customers-topic”, “customer-store”); customers = streams.store( "customers-store”…); customer = customers.get(42)
  52. 52. 52 As we’re using a streaming engine, we can translate into any Domain Model (projection) we wish Shipping Service Customers Events ShippingCustomers By CustomerName
  53. 53. 53 Combine different streams from different services Fraud Service Payment Events Purchase Events PurchasePayments payments.join(“purchases”,..) .groupByKey() .reduce(newValueReducer, “PurcahsePaymentsStore”)
  54. 54. 54 Pattern 1: Local KTables •  Simple to create and query •  Local to each service so fast •  Powerful DSL for transformation •  Inbuilt high availability •  No external database •  Controlled entirely within service’s bounded context •  KTable and GlobalKTable allow scale out
  55. 55. 55 Pattern 2(a): Queryable Interface Fraud Service Kafka Fraud Service (Creates streaming projection) Query via Rest Interface User Interface Known as queryable state in Kafka Context Boundary
  56. 56. 56 Pattern 2(b): Query Service Kafka Bespoke View Service (Creates streaming projection) Rest Interface Fraud Service Context Boundary
  57. 57. 57 Pattern 2: Exposing Materialised Projections •  Similar to Pattern 1 but with a “Query Interface” (using Kafka’s Queryable State feature) •  Commonly used with UI’s
  58. 58. 58 Pattern 3: External DB DB of choice Kafka Connect APIFraud Service Context Boundary
  59. 59. 59 Pattern 3: External DB •  Again similar, but heavier weight •  Use when you cannot pre-compute the view (i.e. to do ad hoc queries) •  Trick: start with only the data (fields) you need today. You can always go back for more.
  60. 60. 60 Pattern 4: Hybrid Fraud Service Kafka Rest Interface Context Boundary Booking Service NB this breaks the async model, but can be a useful compromise in some cases
  61. 61. 61 Pattern 4: Hybrid •  Quite common in practice •  Allows you to break out of the async world •  Use sparingly, but do use when appropriate (e.g. login service)
  62. 62. 62 Benefits •  Forms a central, immutable narrative •  Communication Protocol and Event Store become one •  Better decoupling (RDFC) •  Excellent scalability •  Several approaches for managing queries, lightweight => heavy weight •  Event sourcing & CQRS at its core
  63. 63. 63 Interact through Events, Don't talk to Services
  64. 64. 64 But it's ok to break the rules!
  65. 65. 65 Pulling it all together
  66. 66. 66 All order logic Orders-by-Customer view (KStreams + Queryable State API) UI ServiceStream Maintenance Fulfillment uses local state store, which is replicated to a redundant instance UI uses Orders-by- Customer View directly via Queryable State History Service pushes data to a local Elastic Search Instance Orders Service Derived View UI joins data Tables & Streams Fulfillment Service History Service Fully redundant instance OrdersProduct Custo- mers KTables KTables KTables Schemas
  67. 67. 67 Thank You! @benstopford
  68. 68. 68 Discount code: kafcom17 ‪Use the Apache Kafka community discount code to get $50 off ‪www.kafka-summit.org Kafka Summit New York: May 8 Kafka Summit San Francisco: August 28 Presented by

×