More Related Content Similar to Real-Time Market Data Analytics Using Kafka Streams (20) Real-Time Market Data Analytics Using Kafka Streams1. © 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Lei Chen
Software Engineer, Team Lead
lchen576@bloomberg.net
Real-time* Market Data Processing
Using Kafka Streams
* Actually, just low-latency
2. © 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Agenda
• Use cases and challenges
• Why Kafka Streams
• Deep dive into our implementation
• Some other tips & tricks
3. © 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Who we are & What we do
• Derivative market data group in Bloomberg
• Builds market data pipelines
• Apply big data & AI technologies in financial domain
4. © 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Streaming use cases
Market movement
(bid/ask/trade)
Composite price
(Bloomberg Generated Market Indicator)
Option price
(Calculate option price using
calculated volatilities)
5. © 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Challenges and why Kafka Streams
• Zero data loss
• Ultra-Low latency
• Huge data volume
• Large state
• Corporate DR compliance
• Maintenance
‘Exactly once’ delivery
Super fast
Highly scalable
State store
Fault tolerant
Minimal management overhead
*Data already in Kafka!
6. © 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Deep dive into our implementation
7. © 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Key takeaways
• transform()/process() combines the best of both DSL and PAPI
• Kryo for state serialization/deserialization
• Think stream/table duality
• Avoid unnecessary DSL call by accessing state directly
• Use Kubernetes as runtime
• Monitoring is important (leverage built-in metrics)
8. © 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
DSL vs Processor API
• Declarative vs Imperative
• Usability vs Flexibility
• High-level API vs low-level programming model
9. © 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Monitoring
• Built-in webserver
• Queryable state
• Metrics
• Internal topics
• Lags (topic/partition)
10. © 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Hot load configuration
KStream
KTable
Stream & Table Join
Joined Stream
KStream
GlobalKTable
Joined Stream
Stream & Table Join
Conf. topic
Data topic
Comdb2 Kafka Connect
Conf. topic
Data topic
DB Kafka Connect
KStream
KTable state
11. © 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Community Involvement
• KIP-362 - Dynamic Gap Session Window
— Versus fixed-gap session window
session1 session2 session3
gap3gap2gap1
13. © 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Thread model and depth first topology
APP
KAFKA
STREAMS
topology
14. © 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Trick 1 - Batch Processing In Kafka Streams
10/15/18
Micro batch?
Possible!
Batch?
Harder!
watermark
State
App2
App1
15. © 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Trick 2 - Chain Multiple Kafka Streams App
• Kafka as message bus
• Compose pipeline using multiple Kafka Streams apps
• Could leverage third-party pipeline framework – Spring Cloud Data Flow, etc.
KSTREAMS
APP1
topology
KSTREAMS
APPN
topology
KSTREAMS
APP2
topology
16. © 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Our Streaming Platform
17. © 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
We are hiring!
Questions?
https://tinyurl.com/y7bepre9