Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

JAX London Slides

4,333 views

Published on

Presentation for JAX London 2016 on Kafka, Streaming and Microservices.

Published in: Technology

JAX London Slides

  1. 1. Microservices in a Streaming World
  2. 2. What are microservices really about?
  3. 3. GUI UI Service Orders Service Returns Service Fulfilment Service Payment Service Stock Service Collaboration?
  4. 4. Orders Service Autonomy?
  5. 5. Internal World External world External world External World External world Service Boundary
  6. 6. User-Orders StatementgetOpenOrders(), Less Surface Area = Less Coupling Encapsulate State Encapsulate Functionality Encapsulation => Loose Coupling
  7. 7. Independence gives services their value
  8. 8. Orders Service Stock Service Email Service Fulfillment Service Independently Deployable Independently Deployable Independently Deployable Independently Deployable
  9. 9. It’s hard to scale individual applications What happens when we grow?
  10. 10. Companies are inevitably a collection of applications They must work together to some degree
  11. 11. Inverse Conway Maneuver The 'Inverse Conway Maneuver' recommends evolving your team and organizational structure to promote your desired architecture. ! Org Structure Software Architecture
  12. 12. But independence comes at a cost $$$
  13. 13. Change 1 Change 2 Redeploy Singly-deployable apps are easy
  14. 14. User-Orders StatementgetOpenOrders(), Independently Deployable Independently Deployable Synchronized changes are painful
  15. 15. Single Sign On Business Serviceauthorise(), It’s unlikely that a Business Service would need the internal SSO state / function to change Single Sign On
  16. 16. SSO has a tightly bounded context
  17. 17. Business services are different
  18. 18. Catalog Authorisation Most business services share the same core stream of facts. Most services live in here
  19. 19. We need encapsulation to hide internal state But we need the freedom to slice & dice shared data like any other dataset
  20. 20. But data systems have little to do with encapsulation
  21. 21. Service Database Data on inside Data on outside Data on inside Data on outside Interface hides data Interface amplifies data Databases amplify the data they hold
  22. 22. The data dichotomy Data systems are about exposing data. Services are about hiding it.
  23. 23. This affects services in one of two ways
  24. 24. getOpenOrders( fulfilled=false, deliveryLocation=CA, orderValue=100, operator=GreatherThan) Either (1) we constantly add to the interface, as datasets grow
  25. 25. getOrder(Id) getOrder(UserId) getAllOpenOrders() getAllOrdersUnfulfilled(Id) getAllOrders() Services can end up looking like kookie home-grown databases
  26. 26. ...and data amplifies this service boundary problem
  27. 27. (2) Give up and move whole datasets en masse getAllOrders() Data Copied Internally
  28. 28. This leads to a different problem
  29. 29. Data diverges over time Orders Service Stock Service Email Service Fulfill- ment Service
  30. 30. The more mutable copies, the more data will diverge over time
  31. 31. Nice neat services Service contract too limiting Can we change both services together, easily? NO Broaden Contract Eek $$$ NO YES Is it a shared Database? Frack it! Just give me ALL the data Data diverges. (many different versions of the same facts) Lets encapsulate Lets centralise to one copy YES Start here
  32. 32. There is competition between these forces DivergenceAccessibilityEncapsulation
  33. 33. Shared database Service Interfaces Better Accessibility Better Independence No perfect solution
  34. 34. Is there a better way?
  35. 35. Data on the Inside Data on the Outside Data on the Outside Data on the Outside Data on the Outside Service Boundary
  36. 36. Make data-on-the-outside a 1st class citizen
  37. 37. Use a Stream Processing tool-kit
  38. 38. Kafka is a Streaming Platform
  39. 39. Kafka: a Streaming Platform The Log ConnectorsConnectors Producer Consumer Streaming Engine
  40. 40. The Log?
  41. 41. Shard on the way in Producing Services Kafka Consuming Services
  42. 42. Each shard is a queue Producing Services Kafka Consuming Services
  43. 43. Consumers share load Producing Services Kafka Consuming Services
  44. 44. Load Balanced Services
  45. 45. Fault Tolerant Services
  46. 46. Build ‘Always On’ Services Rely on Fault Tolerant Broker
  47. 47. Services can “Rewind & Replay” the log Rewind & Replay
  48. 48. Compacted Log (retains only latest version) Version 3 Version 2 Version 1 Version 2 Version 1 Version 5 Version 4 Version 3 Version 2 Version 1
  49. 49. Two primitives: (a) Log (append only) (b) Compact-Log (update)
  50. 50. Service Backbone Scalable, Fault Tolerant, Concurrent, Strongly Ordered, Stateful The Log ConnectorsConnectors Producer Consumer Streaming Engine
  51. 51. A place to keep data-on-the- outside
  52. 52. Now add Stream Processing
  53. 53. KStreams: A database engine for data-in-flight The Log ConnectorsConnectors Producer Consumer Streaming Engine
  54. 54. Max(price) From orders where ccy=‘GBP’ over 1 day window emitting every second Continuously Running Queries
  55. 55. Features: similar to database query engine JoinFilter Aggr- egate View Window
  56. 56. What is Stream Processing? Table Index Query Engine Query Enginevs Database Finite source Stream Processor Infinite source
  57. 57. What is Stateful Stream Processing? Table Index Query Engine Query Engine vs Database Finite source Stateful Stream Processor Infinite & Finite source Table Compacted Stream KStreams Kafka
  58. 58. Stateful Stream Processing stream Compacted stream Join Stream Data Stream-Tabular Data Windowed Stream Locally Cached Table (disk resident) KafkaKafka Streams
  59. 59. Email Confirmation Example Orders Customer (Compacted) Join Customer Stream Create Confirmation Emails by Joining Order “Events” with Customer information Kafka Email Service Orders Stream
  60. 60. Scales Out
  61. 61. Embeddable Orders Service Kafka Orders Service Orders Service
  62. 62. Analytic function, Event Driven, all in one. Analytics Event Driven Business Services
  63. 63. Join shared datasets without putting pressure on source services Email Service Legacy App Orders Payments Stock KSTREAMS
  64. 64. Shared State is only cached in the service, so there is no way to diverge Email Service Orders Payments Stock Lives here! Cached here! KSTREAMS
  65. 65. Services work “Event Driven”, but with the power to join, filter and aggregate embedded in each service The Log ConnectorsConnectors Producer Consumer Streaming Engine
  66. 66. Services work “Event Driven”, but with the power to join, filter and aggregate embedded in each service customer orders catalogue Pay- ments
  67. 67. But sometimes you have to move data
  68. 68. Replicate it, so both copies are identical Keep Immutable
  69. 69. Iterate via view regeneration
  70. 70. Regenerate from the Log Fix at source If you have to move data en masse fix at source and regenerate from the log
  71. 71. Oracle Connectors make this easier The Log ConnectorConnector Producer Consumer Streaming Engine Cassandra
  72. 72. So…
  73. 73. When building services consider more than just REST
  74. 74. The data dichotomy Data systems are about exposing data. Services are about hiding it. Remember:
  75. 75. Shared data is a reality for most organisations Catalog
  76. 76. Embrace data that lives in between services
  77. 77. Avoid data services that have complex / amplifying interfaces
  78. 78. Share Data via Immutable Streams Embed Function into each service Distributed LogEmbedded KStreams
  79. 79. Always on, Event-Driven Services The Log (streams & tables) Ingestion Services Services with Polyglotic persistence Simple Services Streaming Services
  80. 80. Keep it simple, Keep it moving Twitter: @benstopford Slides at benstopford.com Blog series arriving end of the month at http://confluent.io/blog

×