Download as PDF, PPTX
Uploaded byconfluent
Building Event-Driven Services with Apache Kafka
The document discusses the transition from traditional REST services to event-driven architectures using Apache Kafka, highlighting the challenges of data management in both internet and enterprise companies. It emphasizes the benefits of decoupling services through event interactions, utilizing immutability and shared narratives, and presents various patterns for querying data in event-driven systems. Ultimately, the talk promotes an architecture that allows for better scalability, flexibility, and fault tolerance in managing complex service interactions.
More Related Content
Similar to Building Event-Driven Services with Apache Kafka
Building Event-Driven Services with Apache Kafka
- 1. 1 Building Event-Driven Services withApache Kafka Ben Stopford @benstopford
- 2. 2 In this talk 1. How we traditionally build services with REST 2. How Event Driven Services are different 3. Building an Immutable, Shared Narrative. 4. Leveraging Materialized Views. 5. Relating to CQRS & Event Sourcing 6. Pulling it all together
- 3. 3 Companies evolve indifferent ways • Internet companies • Enterprise companies
- 4. 4 Internet companies typicallygrow from front-facing websites. pic Growing fast
- 5.
- 6. 6 If stateful, theymight share a DB
- 7. 7 Or they mightuse their own
- 8. 8 Over time =>more complex
- 9. 9 Over time =>more async (async w.r.t. user)
- 10. 10 Chained blocking commands Buffering,handling failure, backpressure, scaling etc all get pushed into service’s responsibilities
- 11. 11 The synchronous worldof request response protocols leads to tight, point-to-point couplings.
- 12. 12 Enterprises are typicallydifferent
- 13. 13 pic Enterprise companies formas heterogeneous application conglomerates, often in sylos
- 14. 14 The start lifeas independent islands
- 15. 15 Linked together byfiles or via messaging pic
- 16. 16 Interactions are asyncfrom day-1 pic
- 17. 17 Forwarding of datagets messy: Chinese Whispers / Telephone game pic
- 18. 18 Both these approachesoften end up at the same point, but for different reasons
- 19. 19 Data is herdedinto specific services Enterprise
- 20. 20 Data is herdedinto specific services Data Services Stateless Services Internet Co
- 21. 21 Lots of DataServices => Distributed Join Problem pic This is a problem databases find hard, and they’re highly tuned for it! Payments Service Orders Service Customers Service Client Service performs join
- 22. 22 Lots of DataServices => Distributed Join Problem pic TESTING BECOMES DIFFICULT! Client Service performs join Payments Service Orders Service Customers Service
- 23. 23 In short: thehigh degree of “connectedness” makes it hard to evolve independently and to scale
- 24.
- 25. 25 Interact through Events, Don't talkto Services
- 26. 26 This Payments Service Purchase Requested Events Listen& React to Events Payment Completed Events
- 27. 27 Not this Service Payment Service http://payments/ID/42 Don’tRequest/Command from individual services.
- 28. 28 Let’s take anexample...
- 29. 29 UI Service Payment Service Stock Service (1) Process Payment (2)Contact Payment Provider (3) Update stock (4) Maybe order more stock Request Response Example
- 30. 30 The Event DrivenWay UI Service Payment Service Stock Service Payment Provider Maybe order more stock
- 31. 31 UI Service Payment Service Stock Service Payment Provider Kafka Topics: (1) Purchases (2) Payments Maybe order more stock The Event Driven Way
- 32. 32 UI Service Payment Service Stock Service Payment Provider Maybe order more stock NewOrder -> PurchaseRequested Event Raise PurchaseRequested Event
- 33. 33 UI Service Payment Service Stock Service Payment Provider Maybe order more stock PaymentService Listens In Raise PurchaseRequested Event Listen to PurchaseRequested Events
- 34. 34 UI Service Payment Service Stock Service Payment Provider Maybe order more stock WhenComplete -> PaymentProcessed Raise PurchaseRequested Event Listen to PurchaseRequested Events Raise PaymentProcessed Event *Some might use the Command/Query Pattern
- 35. 35 UI Service Payment Service Stock Service Payment Provider Maybe order more stock Stockservice hooks in too! Raise PurchaseRequested Event Listen to PurchaseRequested Events Raise PaymentProcessed Event Listen to PaymentProcessed events
- 36. 36 UI Service Payment Service Stock Service Payment Provider Maybe order more stock Paymentservice listens Raise PurchaseRequested Event Listen to PurchaseRequested Events Raise PaymentProcessed Event Listen to PaymentProcessed events Listen to PaymentProcessed Event
- 37. 37 UI Service Payment Service Stock Service Payment Provider Stock service entirely ownsits flow control Maybe order more stock Flow Control is Receiver Driven
- 38. 38 UI Service Payment Service Stock Service Quite different toSynchronous Model Stock service flow control owned by payments service
- 39. 39 1. Events areImmutable Facts (state changes taken from the real world and journalled) UI Service Purchases Payments Purchase Requested Payment Completed All the benefits of “Event Sourcing”
- 40. 40 2. Services coupleonly to data flows No knowledge of contributing services (RDFC) UI Service Purchases Payments Purchase Requested Payment Completed
- 41. 41 Because coupling isonly to Events, not services Purchase Requests PurchaseRequest TopicUI Service Payment Service Stock Service
- 42. 42 The architecture is“Pluggable” Purchase Requests PurchaseRequest TopicUI Service Payment Service Stock Service Fraud Service
- 43. 43 Communication & StateConcepts Merge • The concept of Data & Events become one. • Kafka is both Event Store and Communication Channel
- 44. 44 The Events forma Canonical, Shared Narrative Payments Orders Shipments Customers Evolving state of the system over time
- 45. 45 Scaling is aconcern of the broker, not “upstream” services Purchase Requests PurchaseRequest TopicUI Service Payment Service Stock Service Fraud Service Kafka • Linearly scalable • Fault Tolerant • Multi Tennant
- 46. 46 But there issomething missing! How do we look things up? (i.e. the Q in CQRS) • What is the address for this customer? • Is this user allowed to view this stock? • What is the contents of this user’s shopping basket?
- 47. 47 With REST Lookupsare natural, if Remote UI Service Customer Service http://customers/ID/42 Go to the relevant service and ask!
- 48. 48 Which creates thedependency problem Many services tightly coupled to one another
- 49. 49 With Event Drivenwe create “views” or “projections” inside each bounded context Projection of data from Outside Context Boundary Shipping Service
- 50. 50 Pattern 1: LocalKTables Shipping Service Customers Events Customers By ID (saved by KStreams using RocksDB) Kafka
- 51. 51 Pattern 1: LocalKTables Shipping Service Customers Events Customers By ID KTable customers = builder.table( CustomerId, Customer, “customers-topic”, “customer-store”); customers = streams.store( "customers-store”…); customer = customers.get(42)
- 52. 52 As we’re usinga streaming engine, we can translate into any Domain Model (projection) we wish Shipping Service Customers Events ShippingCustomers By CustomerName
- 53. 53 Combine different streamsfrom different services Fraud Service Payment Events Purchase Events PurchasePayments payments.join(“purchases”,..) .groupByKey() .reduce(newValueReducer, “PurcahsePaymentsStore”)
- 54. 54 Pattern 1: LocalKTables • Simple to create and query • Local to each service so fast • Powerful DSL for transformation • Inbuilt high availability • No external database • Controlled entirely within service’s bounded context • KTable and GlobalKTable allow scale out
- 55. 55 Pattern 2(a): QueryableInterface Fraud Service Kafka Fraud Service (Creates streaming projection) Query via Rest Interface User Interface Known as queryable state in Kafka Context Boundary
- 56. 56 Pattern 2(b): QueryService Kafka Bespoke View Service (Creates streaming projection) Rest Interface Fraud Service Context Boundary
- 57. 57 Pattern 2: ExposingMaterialised Projections • Similar to Pattern 1 but with a “Query Interface” (using Kafka’s Queryable State feature) • Commonly used with UI’s
- 58. 58 Pattern 3: ExternalDB DB of choice Kafka Connect APIFraud Service Context Boundary
- 59. 59 Pattern 3: ExternalDB • Again similar, but heavier weight • Use when you cannot pre-compute the view (i.e. to do ad hoc queries) • Trick: start with only the data (fields) you need today. You can always go back for more.
- 60. 60 Pattern 4: Hybrid Fraud Service Kafka RestInterface Context Boundary Booking Service NB this breaks the async model, but can be a useful compromise in some cases
- 61. 61 Pattern 4: Hybrid • Quite common in practice • Allows you to break out of the async world • Use sparingly, but do use when appropriate (e.g. login service)
- 62. 62 Benefits • Forms acentral, immutable narrative • Communication Protocol and Event Store become one • Better decoupling (RDFC) • Excellent scalability • Several approaches for managing queries, lightweight => heavy weight • Event sourcing & CQRS at its core
- 63. 63 Interact through Events, Don't talkto Services
- 64. 64 But it's okto break the rules!
- 65. 65 Pulling it alltogether
- 66. 66 All order logic Orders-by-Customerview (KStreams + Queryable State API) UI ServiceStream Maintenance Fulfillment uses local state store, which is replicated to a redundant instance UI uses Orders-by- Customer View directly via Queryable State History Service pushes data to a local Elastic Search Instance Orders Service Derived View UI joins data Tables & Streams Fulfillment Service History Service Fully redundant instance OrdersProduct Custo- mers KTables KTables KTables Schemas
- 67.
- 68. 68 Discount code: kafcom17 Usethe Apache Kafka community discount code to get $50 off www.kafka-summit.org Kafka Summit New York: May 8 Kafka Summit San Francisco: August 28 Presented by