JAX London Slides
•6 likes•5,375 views
Report
Share
Download to read offline
Presentation for JAX London 2016 on Kafka, Streaming and Microservices.
Recommended
More Related Content
What's hot
What's hot(20)
Viewers also liked
Viewers also liked(20)
Similar to JAX London Slides
Similar to JAX London Slides(20)
More from Ben Stopford
More from Ben Stopford(12)
Recently uploaded
Recently uploaded(20)
JAX London Slides
- 7. Internal World External world External world External World External world Service Boundary
- 8. User-Orders StatementgetOpenOrders(), Less Surface Area = Less Coupling Encapsulate State Encapsulate Functionality Encapsulation => Loose Coupling
- 11. It’s hard to scale individual applications What happens when we grow?
- 12. Companies are inevitably a collection of applications They must work together to some degree
- 13. Inverse Conway Maneuver The 'Inverse Conway Maneuver' recommends evolving your team and organizational structure to promote your desired architecture. ! Org Structure Software Architecture
- 14. But independence comes at a cost $$$
- 15. Change 1 Change 2 Redeploy Singly-deployable apps are easy
- 16. User-Orders StatementgetOpenOrders(), Independently Deployable Independently Deployable Synchronized changes are painful
- 17. Single Sign On Business Serviceauthorise(), It’s unlikely that a Business Service would need the internal SSO state / function to change Single Sign On
- 18. SSO has a tightly bounded context
- 20. Catalog Authorisation Most business services share the same core stream of facts. Most services live in here
- 21. We need encapsulation to hide internal state But we need the freedom to slice & dice shared data like any other dataset
- 22. But data systems have little to do with encapsulation
- 23. Service Database Data on inside Data on outside Data on inside Data on outside Interface hides data Interface amplifies data Databases amplify the data they hold
- 24. The data dichotomy Data systems are about exposing data. Services are about hiding it.
- 25. This affects services in one of two ways
- 26. getOpenOrders( fulfilled=false, deliveryLocation=CA, orderValue=100, operator=GreatherThan) Either (1) we constantly add to the interface, as datasets grow
- 27. getOrder(Id) getOrder(UserId) getAllOpenOrders() getAllOrdersUnfulfilled(Id) getAllOrders() Services can end up looking like kookie home-grown databases
- 28. ...and data amplifies this service boundary problem
- 29. (2) Give up and move whole datasets en masse getAllOrders() Data Copied Internally
- 30. This leads to a different problem
- 31. Data diverges over time Orders Service Stock Service Email Service Fulfill- ment Service
- 32. The more mutable copies, the more data will diverge over time
- 33. Nice neat services Service contract too limiting Can we change both services together, easily? NO Broaden Contract Eek $$$ NO YES Is it a shared Database? Frack it! Just give me ALL the data Data diverges. (many different versions of the same facts) Lets encapsulate Lets centralise to one copy YES Start here
- 34. There is competition between these forces DivergenceAccessibilityEncapsulation
- 36. Is there a better way?
- 37. Data on the Inside Data on the Outside Data on the Outside Data on the Outside Data on the Outside Service Boundary
- 38. Make data-on-the-outside a 1st class citizen
- 39. Use a Stream Processing tool-kit
- 40. Kafka is a Streaming Platform
- 41. Kafka: a Streaming Platform The Log ConnectorsConnectors Producer Consumer Streaming Engine
- 42. The Log?
- 43. Shard on the way in Producing Services Kafka Consuming Services
- 44. Each shard is a queue Producing Services Kafka Consuming Services
- 48. Build ‘Always On’ Services Rely on Fault Tolerant Broker
- 49. Services can “Rewind & Replay” the log Rewind & Replay
- 50. Compacted Log (retains only latest version) Version 3 Version 2 Version 1 Version 2 Version 1 Version 5 Version 4 Version 3 Version 2 Version 1
- 51. Two primitives: (a) Log (append only) (b) Compact-Log (update)
- 52. Service Backbone Scalable, Fault Tolerant, Concurrent, Strongly Ordered, Stateful The Log ConnectorsConnectors Producer Consumer Streaming Engine
- 53. A place to keep data-on-the- outside
- 55. KStreams: A database engine for data-in-flight The Log ConnectorsConnectors Producer Consumer Streaming Engine
- 56. Max(price) From orders where ccy=‘GBP’ over 1 day window emitting every second Continuously Running Queries
- 57. Features: similar to database query engine JoinFilter Aggr- egate View Window
- 58. What is Stream Processing? Table Index Query Engine Query Enginevs Database Finite source Stream Processor Infinite source
- 59. What is Stateful Stream Processing? Table Index Query Engine Query Engine vs Database Finite source Stateful Stream Processor Infinite & Finite source Table Compacted Stream KStreams Kafka
- 60. Stateful Stream Processing stream Compacted stream Join Stream Data Stream-Tabular Data Windowed Stream Locally Cached Table (disk resident) KafkaKafka Streams
- 61. Email Confirmation Example Orders Customer (Compacted) Join Customer Stream Create Confirmation Emails by Joining Order “Events” with Customer information Kafka Email Service Orders Stream
- 62. Scales Out
- 64. Analytic function, Event Driven, all in one. Analytics Event Driven Business Services
- 65. Join shared datasets without putting pressure on source services Email Service Legacy App Orders Payments Stock KSTREAMS
- 66. Shared State is only cached in the service, so there is no way to diverge Email Service Orders Payments Stock Lives here! Cached here! KSTREAMS
- 67. Services work “Event Driven”, but with the power to join, filter and aggregate embedded in each service The Log ConnectorsConnectors Producer Consumer Streaming Engine
- 68. Services work “Event Driven”, but with the power to join, filter and aggregate embedded in each service customer orders catalogue Pay- ments
- 69. But sometimes you have to move data
- 70. Replicate it, so both copies are identical Keep Immutable
- 72. Regenerate from the Log Fix at source If you have to move data en masse fix at source and regenerate from the log
- 73. Oracle Connectors make this easier The Log ConnectorConnector Producer Consumer Streaming Engine Cassandra
- 74. So…
- 75. When building services consider more than just REST
- 76. The data dichotomy Data systems are about exposing data. Services are about hiding it. Remember:
- 77. Shared data is a reality for most organisations Catalog
- 78. Embrace data that lives in between services
- 79. Avoid data services that have complex / amplifying interfaces
- 80. Share Data via Immutable Streams Embed Function into each service Distributed LogEmbedded KStreams
- 81. Always on, Event-Driven Services The Log (streams & tables) Ingestion Services Services with Polyglotic persistence Simple Services Streaming Services
- 82. Keep it simple, Keep it moving Twitter: @benstopford Slides at benstopford.com Blog series arriving end of the month at http://confluent.io/blog