Strata Software Architecture NY: The Data Dichotomy
Report
Share
•4 likes•2,607 views
Strata Software Architecture NY: The Data Dichotomy
•4 likes•2,607 views
Report
Share
Download to read offline
Ben Stopford is an engineer and architect working on the Apache Kafka Core Team at Confluent (the company behind Apache Kafka). A specialist in data, both from a technology and an organizational perspective, Ben previously spent five years leading data integration at a large investment bank, using a central streaming database. His earlier career spanned a variety of projects at Thoughtworks and UK-based enterprise companies. He writes at Benstopford.com.
Recommended
More Related Content
What's hot
What's hot(20)
Similar to Strata Software Architecture NY: The Data Dichotomy
Similar to Strata Software Architecture NY: The Data Dichotomy(20)
More from Ben Stopford
More from Ben Stopford(17)
Recently uploaded
Recently uploaded(20)
Strata Software Architecture NY: The Data Dichotomy
- 1. The Data Dichotomy: Rethinking Data and Services with Streams Ben Stopford @benstopford
- 4. GUI UI Service Orders Service Returns Service Fulfilment Service Payment Service Stock Service Splitting the Monolith? Single Process /Code base /Deployment Many Processes /Code bases /Deployments
- 7. Independence is where services get their value
- 9. Scaling in terms of people What happens when we grow?
- 10. Companies are inevitably a collection of applications They must work together to some degree
- 11. Interconnection is an afterthought FTP / Enterprise Messaging etc
- 12. Internal World External world External world External World External world Service Boundary Services Force Us To Consider The External World
- 13. Internal World External world External world External World External world Service Boundary External World is something we should Design For
- 14. But this independence comes at a cost $$$
- 15. Orders Object Statement ObjectgetOpenOrders(), Less Surface Area = Less Coupling Encapsulate State Encapsulate Functionality Consider Two Objects in one address space Encapsulation => Loose Coupling
- 16. Change 1 Change 2 Redeploy Singly-deployable apps are easy
- 17. Orders Service Statement ServicegetOpenOrders(), Independently Deployable Independently Deployable Synchronized changes are painful
- 18. Services work best where requirements are isolated in a single bounded context
- 19. Single Sign On Business Serviceauthorise(), It’s unlikely that a Business Service would need the internal SSO state / function to change Single Sign On
- 20. SSO has a tightly bounded context
- 22. Catalog Authorisation Most business services share the same core stream of facts. Most services live in here
- 23. The futures of business services are far more tightly intertwined.
- 24. We need encapsulation to hide internal state. Be loosely coupled. But we need the freedom to slice & dice shared data like any other dataset
- 25. But data systems have little to do with encapsulation
- 26. Service Database Data on inside Data on outside Data on inside Data on outside Interface hides data Interface amplifies data Databases amplify the data they hold
- 27. The data dichotomy Data systems are about exposing data. Services are about hiding it.
- 28. Service Database Data on inside Data on outside Data on inside Data on outside Encapsulation protects us against future change. Databases provide ultimate flexibility for the now These approaches serve different goals
- 29. This affects services in one of two ways
- 30. getOpenOrders( fulfilled=false, deliveryLocation=CA, orderValue=100, operator=GreatherThan) Either (1) we constantly add to the interface, as datasets grow
- 31. getOrder(Id) getOrder(UserId) getAllOpenOrders() getAllOrdersUnfulfilled(Id) getAllOrders() (1) Services can end up looking like kookie home-grown databases
- 32. ...and DATA amplifies this “God Service” problem
- 33. (2) Give up and move whole datasets en masse getAllOrders() Data Copied Internally
- 34. This leads to a different problem
- 35. Data diverges over time Orders Service Stock Service Email Service Fulfill- ment Service
- 36. The more mutable copies, the more data will diverge over time
- 37. Nice neat services Service contract too limiting Can we change both services together, easily? NO Broaden Contract Eek $$$ NO YES Is it a shared Database? Frack it! Just give me ALL the data Data diverges. (many different versions of the same facts) Lets encapsulate Lets centralise to one copy YES Start here Cycle of inadequacy:
- 38. These forces compete in the systems we build Divergence Accessibility Encapsulation
- 40. Is there a better way?
- 41. Data on the Inside Data on the Outside Data on the Outside Data on the Outside Data on the Outside Service Boundary
- 42. Make data-on-the-outside a 1st class citizen
- 43. Separate Reads & Writes with Immutable Streams
- 44. Orders Service Stock Service Kafka Fulfilment Service UI Service Fraud Service Event Driven + CQRS + Stateful Stream Processing Commands go to owning services Queries are interpreted entirely inside in each bounded context
- 45. Event Driven + CQRS + Stateful Stream Processing Orders Service Stock Service Orders Fulfilment Service UI Service Fraud Service • Events form a central Shared Narrative • Writes/Commands => Go to “owning” service • Reads/Queries => Wholly owned by each Service • Stream processing toolkit
- 46. Kafka is a Streaming Platform
- 47. Kafka: a Streaming Platform The Log ConnectorsConnectors Producer Consumer Streaming Engine
- 48. The Log
- 49. Kafka is a type of messaging system Writers Kafka Readers Important Properties: • Petabytes of data • Scales out linearly • Built in fault tolerance • Log can be replayed
- 50. Scaling becomes a concern of the broker, not the service
- 52. This provides Fault Tolerance
- 53. Reset to any point in the shared narrative Rewind & Replay
- 54. Compacted Log (retains only latest version) Version 3 Version 2 Version 1 Version 2 Version 1 Version 5 Version 4 Version 3 Version 2 Version 1
- 55. Service Backbone Scalable, Fault Tolerant, Concurrent, Strongly Ordered, Retentive The Log ConnectorsConnectors Producer Consumer Streaming Engine
- 56. A place to keep the data-on-the-outside as an immutable narrative
- 58. Kafka’s Streaming API A general, embeddable streaming engine The Log ConnectorsConnectors Producer Consumer Streaming Engine
- 59. What is Stream Processing? A machine for combining and processing streams of events
- 60. Features: similar to database query engine Join Filter Aggr- egate View Window
- 61. Stateful Stream Processing adds Tables & Local State stream Compacted stream Join Stream Data Stream-Tabular Data Domain Logic Tables / Views Kafka Event Driven Service
- 62. Avg(o.time – p.time) From orders, payment, user Group by user.region over 1 day window emitting every second Stateful Stream Processing Streams Stream Processing Engine Derived “Table” Table Domain Logic
- 63. Tools make it easy to join streams of events (or tables of data) from different services Email Service Legacy App Orders Payments Stock KSTREAMS Kafka
- 64. So we might join two event streams and create a local store in our own Domain Model Orders Service Payment Events Purchase Events PurchasePayments
- 65. Make the projections queryable Orders Service Kafka Queryable State API Context Boundary
- 66. Blend the concepts of services communicating and services sharing state customer orders catalogue Pay- ments
- 67. Very different to sharing a database customer orders catalogue Pay- ments Services communicate and share data via the log Slicing & Dicing is embedded in each bounded context
- 68. But sometimes you have to move data
- 69. Easy to drop out to an external database if needed DB of choice Kafka Connect APIFraud Service Context Boundary
- 70. Oracle Connector API & Open Source Ecosystem make this easier The Log ConnectorConnector Producer Consumer Streaming Engine Cassandra
- 71. WIRED Principals • Wimpy: Start simple, lightweight and fault tolerant. • Immutable: Build a retentive, shared narrative. • Reactive: Leverage Asynchronicity. Focus on the now. • Evolutionary: Use only the data you need today. • Decentralized: Receiver driven. Coordination avoiding. No God services
- 72. So…
- 73. Architecture is really about designing for change
- 74. The data dichotomy Data systems are about exposing data. Services are about hiding it. Remember:
- 75. Shared data is a reality for most organisations Catalog
- 76. Embrace data that lives and flows between services
- 77. Canonical Immutable Streams Embed the ability to slice and dice into each bounded context Distributed LogEvent Driven Service
- 78. Shared database Service Interfaces Better Accessibility Better Independence Balance the Data Dichotomy
- 79. Create a canonical shared narrative
- 80. Use stream processing to handle data-on-the- outside whilst prioritizing the NOW
- 81. Event Driven Services The Log (streams & tables) Ingestion Services Services with Polyglotic persistence Simple Services Streaming Services
- 82. Twitter: @benstopford Blog of this talk at http://confluent.io/blog