Verifying Apache Kafka-Based Data Pipelines With Subhangi Agarwala | Current 2022 Trading activity is at an all time high in the financial markets. In order to handle increasing volumes/velocity of data in real time to meet clients' expectations of capturing market moves with minimal latency, we recently migrated our financial data generation pipelines from batch to stream processing. It is important to have Integration Tests in place to ensure that our system is behaving as intended and provides greater confidence in maintaining the system over time, as it can simulate potential issues that could arise in production. A single software or network glitch in processing critical financial information can lead to erroneous data and outages. Testing stream-based applications to avoid these glitches is challenging because data can come in from an array of sources and in a variety of formats like json, avro or others. In this talk, we aim to highlight the importance of integration testing, a critical verification method for stable and reliable large-scale distributed streaming applications. We will also provide a high level overview of our system, challenges faced in moving to a streaming infrastructure, explore alternatives to our proposed solution and talk through the lessons learned while working with TestContainers library to test our KafkaStreams application.

1. © 2018 Bloomberg Finance L.P. All rights reserved.
© 2022 Bloomberg Finance L.P. All rights reserved.
Verifying Apache Kafka-based
Data Pipelines
Current 2022
October 5, 2022
Subhangi Agarwala
Senior Software Engineer

2. © 2022 Bloomberg Finance L.P. All rights reserved.
• Bloomberg
• Derivatives Data Engineering
— Data ingestion, storage and aggregation
• We build real-time financial market data pipelines
About Us

3. © 2018 Bloomberg Finance L.P. All rights reserved.
© 2022 Bloomberg Finance L.P. All rights reserved.
Agenda
● Background and our use case
● Why we need to test streaming applications in finance
● Integration test framework
● Conclusion

4. © 2022 Bloomberg Finance L.P. All rights reserved.
Real-Time Data Pipelines in
Finance – and at Bloomberg

5. © 2022 Bloomberg Finance L.P. All rights reserved.
Application Architecture
Config
Publisher
Trading Data
Publisher
Kafka
Connect
Persistent
data store
Kafka
Streams
Client App
Analytical Engine
Real Time
analysis
Config
Updates
Config
Changes
Trading
Data

6. © 2018 Bloomberg Finance L.P. All rights reserved.
© 2022 Bloomberg Finance L.P. All rights reserved.
● Unit Testing
○ Easily test new functional logic in isolation
○ Insufficient for larger workflows
● Integration Testing
○ Handle exceptions and failures where different moving parts are involved
○ Reliability
Unit Testing vs. Integration Testing

7. © 2018 Bloomberg Finance L.P. All rights reserved.
© 2022 Bloomberg Finance L.P. All rights reserved.
Testing Framework
Test
Validator
TestContainers
using Docker
(Producer)
TestContainers
using Docker
(Consumer)
TestContainers using
Docker
(test Kafka Connect)
Kafka Kafka
KConnect

8. © 2018 Bloomberg Finance L.P. All rights reserved.
© 2022 Bloomberg Finance L.P. All rights reserved.
● Asynchronous nature of Test Containers
○ Increase in test runtime
● Ensuring consistency between components
○ Add a marker at the end of the input record
Challenges
m0
Producer m0
m0
c
m1 m2
m1
c
c

9. © 2018 Bloomberg Finance L.P. All rights reserved.
© 2022 Bloomberg Finance L.P. All rights reserved.
● Testing interaction between a Kafka producer and a Kafka consumer.
● Producing and consuming raw records and JSON records.
● Combining REST API testing with Kafka testing.
● Spinning up Kafka components - Kafka, KConnect and KStreams in a TestContainer.
Conclusion

10. © 2018 Bloomberg Finance L.P. All rights reserved.
© 2022 Bloomberg Finance L.P. All rights reserved.
Thank you!
https://www.bloomberg.com/careers
Learn more about our Derivatives team:
https://www.bloomberg.com/company/stories/meet-the-team-derivativ
es-engineering/
Subhangi Agarwala: sagarwala10@bloomberg.net