Are you a Kafka Streams application developer who needs a faster, more efficient way to reproduce a bug or issue from past events? Do you need to test a new algorithm patch near a discrete point-in-time? Do you have a standard methodology for investigating these issues, or does each team member devise an ad hoc solution? How much time does your simulation take? It is technically impossible to solve this challenge using only changelog topics, since a compact, topic-based changelog doesn't capture the full change history. The Derivatives Data team at Bloomberg augments this through periodic snapshots. To make the snapshotted state accessible to our replay system, we built a query service that leverages Interactive Queries using the Kafka Streams API, along with gRPC-based coordination, to fetch the distributed snapshot states from different snapshotting instances. In this talk, we will cover: ● Overview of this system architecture ● Deep dive into the mechanics of snapshots with Kafka Streams, state-store changelogs, and a query service to serve replay requests ● How two modes, normal and replay, are used on the Kafka Streams application runtime ● Some use cases that benefit from this replay system

1. © 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2020 Bloomberg Finance L.P. All rights reserved.
Replaying KStreams Apps
Using State Snapshots
Kafka Summit 2020
August 25, 2020
Nish Sinha, Software Engineer
Yan Wang, Software Engineer

2. © 2018 Bloomberg Finance L.P. All rights reserved.
transformfilter
State
transform
Market data
(bid/ask/trade)
Kafka Streams Processors
Algorithm-
generated price
State
filter
LOGGING + METRICS + ALARMS
Kafka Streams Stateful Pipelines

3. © 2018 Bloomberg Finance L.P. All rights reserved.
© 2020 Bloomberg Finance L.P. All rights reserved.
Motivation: Real Scenarios
• Debug a bad output that appeared on last Thursday @ 2:30 PM
• Backtest a new model
• Backfill by correcting bad data

4. © 2018 Bloomberg Finance L.P. All rights reserved.
© 2020 Bloomberg Finance L.P. All rights reserved.
Logging Your Problem Away

5. © 2018 Bloomberg Finance L.P. All rights reserved.
Another Way: Play The Point In Time From Start

6. © 2018 Bloomberg Finance L.P. All rights reserved.
Yet Another Way: Play The Point In Time From Nearby Snapshots

7. © 2018 Bloomberg Finance L.P. All rights reserved.
© 2020 Bloomberg Finance L.P. All rights reserved.
Wish list
• Make Real Replay (align with the states) achievable
• Minimal load on Kafka clusters
• Minimal impact on production system
• Whole Nine Yards: Replay the application itself

8. © 2018 Bloomberg Finance L.P. All rights reserved.
© 2020 Bloomberg Finance L.P. All rights reserved.
Design Considerations
• Snapshotting vs. continuous data capture
• How much historical time?
• Scale out?

9. © 2018 Bloomberg Finance L.P. All rights reserved.
External Non-Intrusive Snapshots

10. © 2018 Bloomberg Finance L.P. All rights reserved.
System Overview

11. © 2018 Bloomberg Finance L.P. All rights reserved.
Snapshotter

12. © 2018 Bloomberg Finance L.P. All rights reserved.
Storage Layer

13. © 2018 Bloomberg Finance L.P. All rights reserved.
Query Snapshots

14. © 2018 Bloomberg Finance L.P. All rights reserved.
State Orchestrator

15. © 2018 Bloomberg Finance L.P. All rights reserved.
© 2020 Bloomberg Finance L.P. All rights reserved.
State Exposer

16. © 2018 Bloomberg Finance L.P. All rights reserved.
Align Data with States

17. © 2018 Bloomberg Finance L.P. All rights reserved.
© 2020 Bloomberg Finance L.P. All rights reserved.
Summary

18. © 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2020 Bloomberg Finance L.P. All rights reserved.
Questions?
Nish Sinha, nsinha35@bloomberg.net
Yan Wang, ywang1445@bloomberg.net

19. © 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2020 Bloomberg Finance L.P. All rights reserved.
We are hiring!
Thank you!
https://www.bloomberg.com/careers