Building Event-Driven Services with Apache Kafka
•
21 likes•6,838 views
Should you use REST to sew services together? Is it better to use a richer, brokered protocol? This practical talk will dig into how we piece services together in event driven systems, how we we use a distributed log to create a central, persistent narrative and what benefits we reap from doing so.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Building Event-Driven Services with Apache Kafka
Similar to Building Event-Driven Services with Apache Kafka (20)
More from confluent
More from confluent (20)
Recently uploaded
Recently uploaded (20)
Building Event-Driven Services with Apache Kafka
- 1. 1 Building Event-Driven Services with Apache Kafka Ben Stopford @benstopford
- 2. 2 In this talk 1. How we traditionally build services with REST 2. How Event Driven Services are different 3. Building an Immutable, Shared Narrative. 4. Leveraging Materialized Views. 5. Relating to CQRS & Event Sourcing 6. Pulling it all together
- 3. 3 Companies evolve in different ways • Internet companies • Enterprise companies
- 4. 4 Internet companies typically grow from front-facing websites. pic Growing fast
- 6. 6 If stateful, they might share a DB
- 7. 7 Or they might use their own
- 8. 8 Over time => more complex
- 9. 9 Over time => more async (async w.r.t. user)
- 10. 10 Chained blocking commands Buffering, handling failure, backpressure, scaling etc all get pushed into service’s responsibilities
- 11. 11 The synchronous world of request response protocols leads to tight, point-to-point couplings.
- 12. 12 Enterprises are typically different
- 13. 13 pic Enterprise companies form as heterogeneous application conglomerates, often in sylos
- 14. 14 The start life as independent islands
- 15. 15 Linked together by files or via messaging pic
- 16. 16 Interactions are async from day-1 pic
- 17. 17 Forwarding of data gets messy: Chinese Whispers / Telephone game pic
- 18. 18 Both these approaches often end up at the same point, but for different reasons
- 19. 19 Data is herded into specific services Enterprise
- 20. 20 Data is herded into specific services Data Services Stateless Services Internet Co
- 21. 21 Lots of Data Services => Distributed Join Problem pic This is a problem databases find hard, and they’re highly tuned for it! Payments Service Orders Service Customers Service Client Service performs join
- 22. 22 Lots of Data Services => Distributed Join Problem pic TESTING BECOMES DIFFICULT! Client Service performs join Payments Service Orders Service Customers Service
- 23. 23 In short: the high degree of “connectedness” makes it hard to evolve independently and to scale
- 25. 25 Interact through Events, Don't talk to Services
- 26. 26 This Payments Service Purchase Requested Events Listen & React to Events Payment Completed Events
- 27. 27 Not this Service Payment Service http://payments/ID/42 Don’t Request/Command from individual services.
- 28. 28 Let’s take an example...
- 29. 29 UI Service Payment Service Stock Service (1) Process Payment (2) Contact Payment Provider (3) Update stock (4) Maybe order more stock Request Response Example
- 30. 30 The Event Driven Way UI Service Payment Service Stock Service Payment Provider Maybe order more stock
- 31. 31 UI Service Payment Service Stock Service Payment Provider Kafka Topics: (1) Purchases (2) Payments Maybe order more stock The Event Driven Way
- 32. 32 UI Service Payment Service Stock Service Payment Provider Maybe order more stock New Order -> PurchaseRequested Event Raise PurchaseRequested Event
- 33. 33 UI Service Payment Service Stock Service Payment Provider Maybe order more stock Payment Service Listens In Raise PurchaseRequested Event Listen to PurchaseRequested Events
- 34. 34 UI Service Payment Service Stock Service Payment Provider Maybe order more stock When Complete -> PaymentProcessed Raise PurchaseRequested Event Listen to PurchaseRequested Events Raise PaymentProcessed Event *Some might use the Command/Query Pattern
- 35. 35 UI Service Payment Service Stock Service Payment Provider Maybe order more stock Stock service hooks in too! Raise PurchaseRequested Event Listen to PurchaseRequested Events Raise PaymentProcessed Event Listen to PaymentProcessed events
- 36. 36 UI Service Payment Service Stock Service Payment Provider Maybe order more stock Payment service listens Raise PurchaseRequested Event Listen to PurchaseRequested Events Raise PaymentProcessed Event Listen to PaymentProcessed events Listen to PaymentProcessed Event
- 37. 37 UI Service Payment Service Stock Service Payment Provider Stock service entirely owns its flow control Maybe order more stock Flow Control is Receiver Driven
- 38. 38 UI Service Payment Service Stock Service Quite different to Synchronous Model Stock service flow control owned by payments service
- 39. 39 1. Events are Immutable Facts (state changes taken from the real world and journalled) UI Service Purchases Payments Purchase Requested Payment Completed All the benefits of “Event Sourcing”
- 40. 40 2. Services couple only to data flows No knowledge of contributing services (RDFC) UI Service Purchases Payments Purchase Requested Payment Completed
- 41. 41 Because coupling is only to Events, not services Purchase Requests PurchaseRequest TopicUI Service Payment Service Stock Service
- 42. 42 The architecture is “Pluggable” Purchase Requests PurchaseRequest TopicUI Service Payment Service Stock Service Fraud Service
- 43. 43 Communication & State Concepts Merge • The concept of Data & Events become one. • Kafka is both Event Store and Communication Channel
- 44. 44 The Events form a Canonical, Shared Narrative Payments Orders Shipments Customers Evolving state of the system over time
- 45. 45 Scaling is a concern of the broker, not “upstream” services Purchase Requests PurchaseRequest TopicUI Service Payment Service Stock Service Fraud Service Kafka • Linearly scalable • Fault Tolerant • Multi Tennant
- 46. 46 But there is something missing! How do we look things up? (i.e. the Q in CQRS) • What is the address for this customer? • Is this user allowed to view this stock? • What is the contents of this user’s shopping basket?
- 47. 47 With REST Lookups are natural, if Remote UI Service Customer Service http://customers/ID/42 Go to the relevant service and ask!
- 48. 48 Which creates the dependency problem Many services tightly coupled to one another
- 49. 49 With Event Driven we create “views” or “projections” inside each bounded context Projection of data from Outside Context Boundary Shipping Service
- 50. 50 Pattern 1: Local KTables Shipping Service Customers Events Customers By ID (saved by KStreams using RocksDB) Kafka
- 51. 51 Pattern 1: Local KTables Shipping Service Customers Events Customers By ID KTable customers = builder.table( CustomerId, Customer, “customers-topic”, “customer-store”); customers = streams.store( "customers-store”…); customer = customers.get(42)
- 52. 52 As we’re using a streaming engine, we can translate into any Domain Model (projection) we wish Shipping Service Customers Events ShippingCustomers By CustomerName
- 53. 53 Combine different streams from different services Fraud Service Payment Events Purchase Events PurchasePayments payments.join(“purchases”,..) .groupByKey() .reduce(newValueReducer, “PurcahsePaymentsStore”)
- 54. 54 Pattern 1: Local KTables • Simple to create and query • Local to each service so fast • Powerful DSL for transformation • Inbuilt high availability • No external database • Controlled entirely within service’s bounded context • KTable and GlobalKTable allow scale out
- 55. 55 Pattern 2(a): Queryable Interface Fraud Service Kafka Fraud Service (Creates streaming projection) Query via Rest Interface User Interface Known as queryable state in Kafka Context Boundary
- 56. 56 Pattern 2(b): Query Service Kafka Bespoke View Service (Creates streaming projection) Rest Interface Fraud Service Context Boundary
- 57. 57 Pattern 2: Exposing Materialised Projections • Similar to Pattern 1 but with a “Query Interface” (using Kafka’s Queryable State feature) • Commonly used with UI’s
- 58. 58 Pattern 3: External DB DB of choice Kafka Connect APIFraud Service Context Boundary
- 59. 59 Pattern 3: External DB • Again similar, but heavier weight • Use when you cannot pre-compute the view (i.e. to do ad hoc queries) • Trick: start with only the data (fields) you need today. You can always go back for more.
- 60. 60 Pattern 4: Hybrid Fraud Service Kafka Rest Interface Context Boundary Booking Service NB this breaks the async model, but can be a useful compromise in some cases
- 61. 61 Pattern 4: Hybrid • Quite common in practice • Allows you to break out of the async world • Use sparingly, but do use when appropriate (e.g. login service)
- 62. 62 Benefits • Forms a central, immutable narrative • Communication Protocol and Event Store become one • Better decoupling (RDFC) • Excellent scalability • Several approaches for managing queries, lightweight => heavy weight • Event sourcing & CQRS at its core
- 63. 63 Interact through Events, Don't talk to Services
- 64. 64 But it's ok to break the rules!
- 65. 65 Pulling it all together
- 66. 66 All order logic Orders-by-Customer view (KStreams + Queryable State API) UI ServiceStream Maintenance Fulfillment uses local state store, which is replicated to a redundant instance UI uses Orders-by- Customer View directly via Queryable State History Service pushes data to a local Elastic Search Instance Orders Service Derived View UI joins data Tables & Streams Fulfillment Service History Service Fully redundant instance OrdersProduct Custo- mers KTables KTables KTables Schemas
- 68. 68 Discount code: kafcom17 Use the Apache Kafka community discount code to get $50 off www.kafka-summit.org Kafka Summit New York: May 8 Kafka Summit San Francisco: August 28 Presented by