Uploaded byconfluent
Kafka Summit NYC 2017 - The Data Dichotomy: Rethinking Data and Services with Streams
The document discusses rethinking data and services using streams. It argues that microservices are about independence but this comes at a cost of data divergence. It proposes using streams as a shared data backbone, with immutable event streams providing a central narrative. This allows services to remain independent while accessing shared data, and for data to be sliced and diced within each service boundary using stream processing. It concludes that embracing shared streaming data can balance the "data dichotomy" between exposing data in databases and hiding it within services.
More Related Content
Similar to Kafka Summit NYC 2017 - The Data Dichotomy: Rethinking Data and Services with Streams
Kafka Summit NYC 2017 - The Data Dichotomy: Rethinking Data and Services with Streams
- 1. The Data Dichotomy: RethinkingData and Services with Streams Ben Stopford @benstopford
- 3.
- 4. GUI UI Service Orders Service Returns Service Fulfilment Service Payment Service Stock Service Splitting the Monolith? SingleProcess /Code base /Deployment Many Processes /Code bases /Deployments
- 5.
- 6.
- 7. Independence is where servicesget their value
- 8.
- 9. Scaling in termsof people What happens when we grow?
- 10. Companies are inevitablya collection of applications They must work together to some degree
- 11. Interconnection is an afterthought FTP/ Enterprise Messaging etc
- 12. Internal World External world Externalworld External World External world Service Boundary Services Force Us To Consider The External World
- 13. Internal World External world Externalworld External World External world Service Boundary External World is something we should Design For
- 14. But this independence comes ata cost $$$
- 15. Orders Object Statement ObjectgetOpenOrders(), Less Surface Area= Less Coupling Encapsulate State Encapsulate Functionality Consider Two Objects in one address space Encapsulation => Loose Coupling
- 16.
- 17. Orders Service StatementServicegetOpenOrders(), Independently Deployable Independently Deployable Synchronized changes are painful
- 18. Services work best whererequirements are isolated in a single bounded context
- 19. Single Sign OnBusiness Serviceauthorise(), It’s unlikely that a Business Service would need the internal SSO state / function to change Single Sign On
- 20. SSO has atightly bounded context
- 21.
- 22. Catalog Authorisation Most businessservices share the same core stream of facts. Most services live in here
- 23. The futures of businessservices are far more tightly intertwined.
- 24. We need encapsulationto hide internal state. Be loosely coupled. But we need the freedom to slice & dice shared data like any other dataset
- 25. But data systems havelittle to do with encapsulation
- 26. Service Database Data on inside Dataon outside Data on inside Data on outside Interface hides data Interface amplifies data Databases amplify the data they hold
- 27. The data dichotomy Datasystems are about exposing data. Services are about hiding it.
- 28. Service Database Data on inside Dataon outside Data on inside Data on outside Encapsulation protects us against future change. Databases provide ultimate flexibility for the now These approaches serve different goals
- 29. This affects services inone of two ways
- 30. getOpenOrders( fulfilled=false, deliveryLocation=CA, orderValue=100, operator=GreatherThan) Either (1) weconstantly add to the interface, as datasets grow
- 31. getOrder(Id) getOrder(UserId) getAllOpenOrders() getAllOrdersUnfulfilled(Id) getAllOrders() (1) Services canend up looking like kookie home-grown databases
- 32. ...and DATA amplifies this“God Service” problem
- 33. (2) Give upand move whole datasets en masse getAllOrders() Data Copied Internally
- 34. This leads toa different problem
- 35. Data diverges overtime Orders Service Stock Service Email Service Fulfill- ment Service
- 36. The more mutable copies, themore data will diverge over time
- 37. Nice neat services Service contract too limiting Canwe change both services together, easily? NO Broaden Contract Eek $$$ NO YES Is it a shared Database? Frack it! Just give me ALL the data Data diverges. (many different versions of the same facts) Lets encapsulate Lets centralise to one copy YES Start here Cycle of inadequacy:
- 38. These forces competein the systems we build Divergence Accessibility Encapsulation
- 39.
- 40. Is there abetter way?
- 41. Data on theInside Data on the Outside Data on the Outside Data on the Outside Data on the Outside Service Boundary
- 42.
- 43. Separate Reads & Writeswith Immutable Streams
- 44. Orders Service Stock Service Kafka Fulfilment Service UI Service Fraud Service Event Driven +CQRS + Stateful Stream Processing Commands go to owning services Queries are interpreted entirely inside in each bounded context
- 45. Event Driven +CQRS + Stateful Stream Processing Orders Service Stock Service Orders Fulfilment Service UI Service Fraud Service • Events form a central Shared Narrative • Writes/Commands => Go to “owning” service • Reads/Queries => Wholly owned by each Service • Stream processing toolkit
- 46. Kafka is aStreaming Platform
- 47. Kafka: a StreamingPlatform The Log ConnectorsConnectors Producer Consumer Streaming Engine
- 48.
- 49. Kafka is atype of messaging system Writers Kafka Readers Important Properties: • Petabytes of data • Scales out linearly • Built in fault tolerance • Log can be replayed
- 50. Scaling becomes a concernof the broker, not the service
- 51.
- 52. This provides FaultTolerance
- 53. Reset to anypoint in the shared narrative Rewind & Replay
- 54. Compacted Log (retains onlylatest version) Version 3 Version 2 Version 1 Version 2 Version 1 Version 5 Version 4 Version 3 Version 2 Version 1
- 55. Service Backbone Scalable, FaultTolerant, Concurrent, Strongly Ordered, Retentive The Log ConnectorsConnectors Producer Consumer Streaming Engine
- 56. A place tokeep the data-on-the-outside as an immutable narrative
- 57.
- 58. Kafka’s Streaming API Ageneral, embeddable streaming engine The Log ConnectorsConnectors Producer Consumer Streaming Engine
- 59. What is StreamProcessing? A machine for combining and processing streams of events
- 60. Features: similar todatabase query engine Join Filter Aggr- egate View Window
- 61. Stateful Stream Processingadds Tables & Local State stream Compacted stream Join Stream Data Stream-Tabular Data Domain Logic Tables / Views Kafka Event Driven Service
- 62. Avg(o.time – p.time) Fromorders, payment, user Group by user.region over 1 day window emitting every second Stateful Stream Processing Streams Stream Processing Engine Derived “Table” Table Domain Logic
- 63. Tools make iteasy to join streams of events (or tables of data) from different services Email Service Legacy App Orders Payments Stock KSTREAMS Kafka
- 64. So we mightjoin two event streams and create a local store in our own Domain Model Orders Service Payment Events Purchase Events PurchasePayments
- 65. Make the projectionsqueryable Orders Service Kafka Queryable State API Context Boundary
- 66. Blend the conceptsof services communicating and services sharing state customer orders catalogue Pay- ments
- 67. Very different tosharing a database customer orders catalogue Pay- ments Services communicate and share data via the log Slicing & Dicing is embedded in each bounded context
- 68. But sometimes youhave to move data
- 69. Easy to dropout to an external database if needed DB of choice Kafka Connect APIFraud Service Context Boundary
- 70. Oracle Connector API &Open Source Ecosystem make this easier The Log ConnectorConnector Producer Consumer Streaming Engine Cassandra
- 71. WIRED Principals • Wimpy:Start simple, lightweight and fault tolerant. • Immutable: Build a retentive, shared narrative. • Reactive: Leverage Asynchronicity. Focus on the now. • Evolutionary: Use only the data you need today. • Decentralized: Receiver driven. Coordination avoiding. No God services
- 72.
- 73. Architecture is reallyabout designing for change
- 74. The data dichotomy Datasystems are about exposing data. Services are about hiding it. Remember:
- 75. Shared data isa reality for most organisations Catalog
- 76. Embrace data thatlives and flows between services
- 77. Canonical Immutable Streams Embedthe ability to slice and dice into each bounded context Distributed LogEvent Driven Service
- 78.
- 79. Create a canonical sharednarrative
- 80. Use stream processing tohandle data-on-the- outside whilst prioritizing the NOW
- 81. Event Driven Services TheLog (streams & tables) Ingestion Services Services with Polyglotic persistence Simple Services Streaming Services
- 82. Twitter: @benstopford Blog of thistalk at http://confluent.io/blog