Microservices for a Streaming World
•
0 likes•425 views
Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1QN63P7. Ben Stopford looks at the implications of mixing toolsets from the stream processing world into real-time business applications: how to effectively handle infinite streams, how to leverage a high throughput, persistent Log and deploy dynamic, fault tolerant, and streaming services. Filmed at qconlondon.com. Ben Stopford is a specialist in data technologies. He’s worked in Finance, at Thoughtworks and is now at Confluent. His experience cover a variety of fields including online trading, retail, High Performance Computing and building the central data platform for a large investment bank.
Recommended
Recommended
More Related Content
More from C4Media
More from C4Media (20)
Recently uploaded
Recently uploaded (20)
Microservices for a Streaming World
- 1. Microservices in a Streaming World
- 2. InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /microservices-streaming
- 3. Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon London www.qconlondon.com
- 4. There are many good reasons for building service-based systems • Loose Coupling • Bounded Contexts • Autonomy • Ease of scaling • Composability
- 5. But when we do, we’re building a distributed system
- 6. This can be a bit tricky
- 7. Monolithic & Centralised Approaches Shared, mutable state
- 10. Stream Processing is a bit different batch analytics => real time => at scale => accurately
- 11. and comes with an interesting toolset
- 13. Some fundamental patterns of distributed systems
- 16. Event Driven Async / Fire and Forget
- 17. Event Based • Simple • Synchronous • Event Driven • Good decoupling • Requires Broker • Fire & Forget • Polling • Full decoupling Request/Response vs.
- 18. SOA / Microservices Message Broker Event Based Request/Response
- 20. Combinations Withdraw £100 Account Service General Ledger Customer Statements Fraud Detection Check Funds Async Message Broker I need moneyReST
- 21. Services generally eschew shared, mutable state
- 22. How do we put these things together?
- 23. Request/Response
- 27. Queues
- 28. Point to Point Service A Service B
- 29. Load Balancing Instance 2 Instance 1 Single message allocation has scalability issues
- 30. Batched Allocation Instance 1 Instance 2 Throughput!
- 31. Lose Ordering Guarantees Fail! Instance 1 Instance 2
- 32. Topics
- 33. Topics are Broadcast Consumer Consumer Broker broadcast
- 34. Topics Retain Ordering Trades Buys Sells Broker Instance 1 Instance 2
- 35. Even when services fail Trades Buys Sells Fail! Broker We retain ordering, but we have to detect & reprovision Instance 1 Instance 2
- 37. Queues Lose Ordering Guarantees at Scale Fail! Worker 1 Worker 2
- 40. Is there another way?
- 41. A Distributed Log Kafka is one example
- 42. Think back to the queue example Batch Batch
- 43. Shard on the way in
- 44. Each shard is a queue Strong Ordering (in shard). Good concurrency.
- 45. Each consuming service is assigned a “personal set” of queues each little queue is sent to only one service in a group
- 46. Services instances naturally rebalance on failure Service instance dies, data is redirected, ordering guarantees remain
- 47. Very Scalable, Very High Throughput Sharded In, Sharded Out
- 48. Reduces to a globally ordered queue
- 49. Fault Tolerance
- 50. The Log Single seek & scan Append only messages don’t need to be transient!
- 51. Cleaning the Log Delete old segments
- 52. Cleaning the Log Delete old versions that share the same key K1 K1 K1 K2 K2 K2 K1 V1 V1 V2 V3 V2 V4 V3
- 53. • Scalable multiprocessing • Strong partition-based ordering • Efficient data retention • Always on
- 54. So how is this useful for microservices?
- 55. Build ‘Always On’ Services Rely on Fault Tolerant Broker
- 56. Load Balance Services Load Balance Services (with strong ordering)
- 57. Fault Tolerant Services Services automatically fail over (retaining ordering)
- 58. Services can return back to old messages in the log Rewind & Replay
- 60. Lets take a little example
- 61. Getting Exchange Rates Exchange Rate Service USD/GBP = 0.71 EUR/GBP = 0.77 USD/INR = 67.7 USD/AUD = 1.38 EUR/JPY = 114.41 … I need exchange rates!
- 62. Option1: Request Response rate for USD/GBP? 0.71 Exchange Rate Service I need exchange rates!
- 63. Option 2: Publish Subscribe Exchange Rate Service Accumulate current state ETL I need exchange rates!
- 64. Option 3: Accumulate in Compacted Stream Exchange Rate Service Get all exchange rates Publish to clients USD/GBP = 0.71 EUR/GBP = 0.77 USD/INR = 67.7 USD/AUD = 1.38 EUR/JPY = 114.41 … Broker retains latest versions Publish all rate events
- 65. Is it a stream or is it a table? transitory stateful
- 66. Datasets can live in the broker! trades books risk results ex- rates
- 67. Service Backbone Scalable, Fault Tolerant, Concurrent, Strongly Ordered, Stateful
- 68. … lets add in stream processing
- 69. Max(price) From orders where ccy=‘GBP’ over 1 day window emitting every second What is stream processing? Continuous Queries.
- 70. What is stream processing engine? Data Index Query Engine Query Engine vs Database Finite, well defined source Stream Processor Infinite, poorly defined source
- 71. Windowing For unordered or unpredictable streams Sliding Fixed (tumbling)
- 72. Features: similar to database query engine JoinFilter Aggr- egate View Window
- 73. KStreams & KTables stream Compacted stream Join Streaming Data Stored Data KStream KTable
- 75. Buying Lunch Abroad Payments Service Exchange Rates Service Buy Notification Service Amount in ££ $$ $$ Text Message: ££ $$
- 76. Request-Response Option Payments Service Exchange Rates Service Buy Amount in ££ Join etc Text Message: ££ Iterative join over the network
- 77. ETL Option Payments Service Exchange Rates Service Buy Amount in ££ ETL ETL Join etc Text Message: ££
- 78. Stream Processor Option Payments Service Exchange Rates Service Buy Stream Processor join etc Text Message: ££
- 79. Buying Lunch Abroad Payments Exchange Rates Looks like a table (compacted stream) Looks like an infinite stream KStream KTable
- 80. Buying Lunch Abroad Payments Exchange Rates • Filter(ccy<>’GBP’) • Join on ccy • Calculate GBP • Send text message buffering
- 81. Local DB (fast joins) Topic Compacted Topic KStream pre-populate
- 82. KTables can also be written to - they’re backed by the broker Manage intermediary state KStream KTable Topic Compacted Topic
- 83. Scales Out (MPP)
- 84. These tools are pretty handy for managing decentralised services
- 85. Talk our own data model Data Stream View Query
- 86. Handle Unpredictability 9am 5pm Late trades
- 88. Duality between Stream and Table Join KStream KTable
- 89. More Complex Use Cases Trades Valuations Books Customers General Ledger
- 90. trades books risk results ex- rates Practical mechanism for managing data intensive, loosely coupled services • Stateful streams live inside the Log • Data extracted quickly! • Fast, local joins, over large datasets • HA pre-caching • Manage intermediary state • Just a simple library (over Kafka)
- 91. There is much more to stream processing it is grounded in the world of big-data analytics
- 92. Simple Approaches Just a library (over Kafka)
- 94. Big Global Bag of State in the Sky Problem: No BGBSS
- 95. How to you provide the accuracy of this
- 96. In this?
- 97. Centralised vs Federated Centralised consistency model Distributed consistency model
- 98. One problem is failure
- 99. Duplicate messages are inevitable have I seen this before?
- 100. Make Services Idempotent try 1 try 2 try 3 try 4
- 101. Stream processors have to solve this problem
- 102. Exactly Once not available in Kafka… yet
- 103. So what do we have?
- 105. Queued Delivery System Ordered queue
- 106. Scales Horizontally
- 107. Scales Horizontally
- 108. Scales Horizontally
- 109. Scales Horizontally
- 110. Built In Fault Tolerance
- 111. Runs Always On
- 112. For Services Too
- 115. Scales Horizontally with history stored in the Log
- 116. Scales Horizontally Extending to any number of services
- 117. Scales Horizontally With any data throughput
- 118. Scales Horizontally With any data throughput
- 119. Scales Horizontally With any data throughput
- 120. Scales Horizontally powerful tools for slicing and dicing streams
- 121. Scales Horizontally the declarative processing of data join filter aggregate
- 122. at any throughput
- 124. Scales Horizontally backed up to the log
- 127. trades books risk results ex- rates with data living in the stream
- 128. but retaining loose coupling trades books risk results ex- rates
- 129. Scales Horizontally with strong ordering and repeatability guarantees (eventually)
- 130. so…
- 131. Microservices push us away from shared, mutable state
- 132. Big Global Bag of State in the Sky Away from BGBSS’s
- 133. This means data is increasingly remote
- 134. Sure, you can collect it all copy copy copy copy copy copy copy ETL ETL ETL ETL ETL ETL
- 135. can be a lot of work
- 136. Or you can look it all up get get get get get get get get, get, get, get
- 137. but that doesn’t scale well (with system complexity or with data throughput)
- 139. We need a decentralised toolset to do this trades books risk results ex- rates
- 140. Keep it simple, Keep it moving
- 141. Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations/ microservices-streaming