Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Microservices, Events, and Breaking the Data Monolith with Kafka

491 views

Published on

One of the trickiest problems with microservices is dealing with data as it becomes spread across many different bounded contexts. An event architecture and event-streaming platform like Kafka provide a respite to this problem. Event-first thinking has a plethora of other advantages too, pulling in concepts from event sourcing, stream processing, and domain-driven design.

In this talk, Ben and Cornelia will tackle how to do the following:

● Transform the data monolith to microservices
● Manage bounded contexts for data fields that overlap
● Use event architectures that apply streaming technologies like Kafka to address the challenges of distributed data

Speakers:
Cornelia Davis, Author & VP, Technology, Pivotal
Ben Stopford, Author & Technologist, Office of CTO, Confluent

Published in: Technology
  • Be the first to comment

Microservices, Events, and Breaking the Data Monolith with Kafka

  1. 1. © Copyright 2019 Pivotal Software, Inc. All rights Reserved. Version 1.0 Cornelia Davis Vice President, Technology, Pivotal @cdavisafc February 2019 Microservices, Events, and Breaking the Data Monolith Webinar Ben Stopford Office of the CTO, Confluent @benstopford
  2. 2. Cornelia Developer (wasn’t Ops) Web architectures for >10 years Cloud-native for 6+ years Cloud Foundry for 6+ years Discount code 40% off!: 40cloudnat https://www.manning.com/books/cloud-native @cdavisafc
  3. 3. Ben Stopford • Leads Office of the CTO team at Confluent. • 20 years in engineering, mostly distributed data. • Worked on Kafka’s replication protocol, rebalancing, throttling. Download book: bit.ly/benstopford @benstopford
  4. 4. @cdavisafc Where we’re going today Microservices Goodness Microservices Challenges How Event-driven Approaches Help
  5. 5. Why Microservices ● Scale Applications ● Scale Teams ● Independent Development Cycles ● Experimentation ● Resilience
  6. 6. http://highscalability.com/blog/2014/4/8/microservices-not-a-free-lunch.html
  7. 7. The Microservices Death-star
  8. 8. Assumption: Each microservice boasts 99.999% availability (and there are 100 of them) Assumption: The network is reliable Result: Home Page achieves 99.9% availability https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing
  9. 9. How we try to add a level of resilience?
  10. 10. Retries ▸ Client must consider failure ▸ Decide on fall-back behavior ▸ Likely including retries ▸ But then we need to handle downstream consequences of these (retry) behaviors CLIENT SERVICE Timeouts? If we don’t hear back, try again
  11. 11. But… CLIENT SERVICE CLIENT CLIENT CLIENT ▸ Client must consider failure ▸ Decide on fall-back behavior ▸ Likely including retries ▸ But then we need to handle downstream consequences of these (retry) behaviors
  12. 12. Circuit Breakers ▸ Client must consider failure ▸ Decide on fall-back behavior ▸ Likely including retries ▸ But then we need to handle downstream consequences of these (retry) behaviors CLIENT SERVICE CLIENT CLIENT CLIENT
  13. 13. But then, is stale data better than no data? - often it is ▸ Client must consider failure ▸ Decide on fall-back behavior ▸ Likely including caches ▸ But then we need to handle things like cache expiry SERVICE APP SERVICE APP APP SERVICE APP SERVICE APP SERVICE APP
  14. 14. And there’s more…
  15. 15. New LIGHTWEIGHT ARCHITECTURES are emerging
 Microservices addressing speed to market and cloud scale Monolithic / Layered Microservices
  16. 16. Pattern: Bounded Context ● Domain-Driven Design ● Each bounded context has a single, unified model ● Relationships between models are explicitly defined ● A product team usually has a strong correlation to a bounded context ● Ideal pattern for Data APIs - do not fall into the trap of simply projecting current data models
  17. 17. Pattern: Database per Service ● Supports Polyglot persistence ● Independent availability, backup/restore, access patterns, etc.
  18. 18. Pattern: Database per Service ● Supports Polyglot persistence ● Independent availability, backup/restore, access patterns, etc. But then what about… ● Overlapping data ● Data ownership ● Data synchronization
  19. 19. More patterns to the rescue
  20. 20. Request/Response - root to leaf; Events - leaf to roots
  21. 21. Connections’ Posts API Abundant Sunshine Your source for healthy cooking New From Your Network Seasonal Favorites ! Response " Response # Response $ Request %Request &Request Connections API User: - id - username - name Connections: - id - follower - followed Posts API Posts: - id - date - userId - title - body Starting with the familiar (Root to Leaf)
  22. 22. % $ ! & " # Connections’ Posts API Abundant Sunshine Your source for healthy cooking New From Your Network Seasonal Favorites Response Response Response Request Request Request Connections API User: - id - username - name Connections: - id - follower - followed Posts API Posts: - id - date - userId - title - body Layer in caching to improve performance Cache How do we handle out of data data?
  23. 23. µService Event Consumer µService Event Producer event store … … Introduce a broker/event store to decouple microservices Key Properties: - Messaging - Scalability/HA - Event Storage & Replay - Stream Processing
  24. 24. Connections’ Posts API Abundant Sunshine Your source for healthy cooking New From Your Network Seasonal Favorites Connections API Posts API Response Request 1 2 event store … … 1 Turn the flow on its head (leaves to root) 1 Connection Created Event - id - follower - followed 1 Post Created Event - id - date - userId - title - body Cache (Materialized View) When the request comes in, the Connections’ Post service already has the answer!
  25. 25. Connections’ Posts API Abundant Sunshine Your source for healthy cooking New From Your Network Seasonal Favorites Connections API Posts API Response Request 1 1 1 1 2 event store … … Cache is populated when things change, not on request Materialized View View is formed to match what we need: “New from your Network”
  26. 26. Event Sourcing
  27. 27. Traditional systems use mutable state DB This isn’t wrong, it’s just lossy
  28. 28. Events record the user’s journey Shopping Cart Events 2 Trousers added 1 Jumper added 1 Trousers removed 1 Hat added Checkout Shopping Cart Event User Journey 12.42 12.44 12.49 12.50 12.59
  29. 29. Stored as a stream Stored statefully (think DB) 12.42 12.44 12.49 12.50 12.59 Information lost! Event User Journey
  30. 30. 12.42 12.44 12.49 12.50 12.59 We can derive the current state (but not the other way around) DERIVE Stream Processor
  31. 31. Connections’ Posts API Abundant Sunshine Your source for healthy cooking New From Your Network Seasonal Favorites Connections API Posts API Response Request event store … … Our “Materialized View” is a form of Event Sourcing View Cache the view somewhere convenient Define the view in our code
  32. 32. Connections’ Posts API Abundant Sunshine Your source for healthy cooking New From Your Network Seasonal Favorites Connections API Posts API Response Request event store … … When the applications starts it loads the cache from the log View
  33. 33. Abundant Sunshine Your source for healthy cooking New From Your Network Seasonal Favorites event store … … Event Store becomes the Source of Truth Service A Service E C Service D DB Apps Search NoSQL Apps Apps S T R E A M I N G P L A T F O R M Service C DB Apps Search NoSQL Apps App DWH Hadoop S T R E A M I N G P L A T F O R M C Service B DB Apps Search NoSQL Apps Apps DWH Hadoop S T R E A M I N G P L A T F O R M
  34. 34. Event Streaming
  35. 35. Connections’ Posts API Abundant Sunshine Your source for healthy cooking New From Your Network Seasonal Favorites Connections API Posts API Response Request event store … … Event Streaming View Recommendations Service 1 2 Recommendation Service plugs right into the existing flow
  36. 36. event store … … Event Streaming Recommendations Service KafkaStreamsAPI Posts Connections Page Views by User Recommendation Kafka Streams wraps up the . solutions to these problems in a single API: - Joins real-time streams - Tables (from streams) - Summarises streams
  37. 37. Connections’ Posts API Abundant Sunshine Your source for healthy cooking New From Your Network Seasonal Favorites Connections API Posts API Response Request event store … … Event Streaming View Recommendations Service Create ‘views’ derived from events in the log Build event streaming services using KStreams API Decouple services from one another using the event store
  38. 38. Summing it up…
  39. 39. Evolution of software systems App AppApp App AppApp App AppApp App AppApp App Kafka Monolith Distributed Monolith Microservices Event-Driven Microservices Microservices or? AppApp App AppApp App
  40. 40. Kafka Event-Driven Microservices AppApp App AppApp App
  41. 41. Kafka Event-Driven Microservices AppApp App AppApp App
  42. 42. AppApp App AppApp App Kafka Event-Driven Microservices
  43. 43. Service Bindings AppApp App AppApp App Kafka Event-Driven Microservices Service Bindings
  44. 44. © Copyright 2019 Pivotal Software, Inc. All rights Reserved. Version 1.0 Cornelia Davis Vice President, Technology, Pivotal @cdavisafc February 2019 Microservices, Events, and Breaking the Data Monolith Webinar Ben Stopford Office of the CTO, Confluent @benstopford
  45. 45. Join us! Keynote Speaker: James Watters Senior Vice President of Pivotal’s Product and Business Development Organizations  Register now at Kafka-Summit.org, and get a special 25% off using this code: KSNYPivotal25

×