Improving the Reliability of Market Data Subscription Feeds With Ruchir Vani | Current 2022 Nasdaq Cloud Data Service has democratized access to financial data not only for companies but also for researchers and educators. When providing data feeds as a subscription service, there is no out of the box standard for subscribers’ Kafka client configurations or system designs. As you might imagine, ensuing optimal performance with an endless variety of clients comes with many challenges such as: Preventing Client Side Data Loss Performance Inefficiencies Client Recovery Data Feed Discovery Schema Management In this talk we will discuss those challenges and introduce the Nasdaq Cloud Data Service SDK, an Open Source library for Kafka Consumers that tackles these issues and allows for uniform resilience, performance and operations among varied client configurations. https://github.com/Nasdaq/NasdaqCloudDataService-SDK-Java

1. Better Consumer Experience for
Market Data Feeds in the Cloud
Ruchir Vani
Director of Software Engineering
Kafka Summit 2022

2. Agenda
• Nasdaq & Data Distribution
• Nasdaq Data Link Streaming
• Software Development Kit (SDK)
• Key Learnings

3. Nasdaq Exchange
Nasdaq Technology
(Matching, Data & APIs)

4. Market Data Feeds
Nasdaq & Data Distribution
Multicast/UDP
Gap Filling
Colocation
FPGA
Proprietary Protocols
Market Data Feeds
sub-15 µ second latency

5. Capturing More Market Share
• Ease of use, ease of access
• Time to market/value
• Rise of new open standards
• Accessibility of open-source technologies
• Cloud native solution
Nasdaq & Data Distribution

6. Nasdaq Data Link Streaming
Efficient Delivery of Real-time Market Data in the Cloud
Products
Proprietary data
Partner data
Alternative data
Capabilities
Accessible over public cloud
Streaming data & analytics
Real-time & historical data
Open-source SDK

7. Scope & Scale
Nasdaq Data Link Streaming
Products
Sizes
Spikes
Volume
Storage
250+ Datasets & Analytics
1-800kb messages
1-300 M messages/second
100’s-6B messages/day
PB’s

8. Nasdaq Cloud Data Link
Streaming Architecture

9. Architecture
Nasdaq Data Link Streaming
Public Cloud
Historic Data Store Monitoring, Alerting
Nasdaq Data Centers
Exchange Data Feed
Feed Adapter Data Stream Analytics Stream Processing Engine
3rd
Party
Data Feed
AuthN/AuthZ
Streaming/Rest APIs
SDK
Containers & Orchestration

10. Architecture
Nasdaq Data Link Streaming
Public Cloud
Historic Data Store Ops — Logging,
Monitoring, Alerting
Nasdaq Data Centers
Exchange Data Feed
Feed Adapter Data Stream Analytics Stream Processing Engine
3rd
Party
Data Feed
AuthN/AuthZ
Streaming/Rest APIs
SDK
Containers & Orchestration
• 50+ Kafka brokers
• 100+ Topics
• 200+ Consumers

12. Why Kafka
Guaranteed Order
Data Protection
Scalability
Heterogenous Feeds
Nasdaq Data Link Streaming
Single Partition Topics
Replicas Set
Horizontally Scale Cluster
Topics

13. Improving the Consumer
Experience

14. Software Development Kit (SDK)
Improving the Customer Experience
Eliminate
Barriers
Security
Standards
Market Data
Nuances

15. Schema Management
Eliminate Barriers
Producers
Connectors Consumer
Consumer
Consumer
1 2 3 5 6 7
Data Topic
1 2 3 5 6 7
Data Topic
1 2 3 5 6 7
Control Topic
1 2 3 5 6 7
Data Topic
1 2 3 5 6 7
Data Topic
Kafka Cluster

16. Avro Desearlizer
Eliminate Barriers

17. Kafka Consumer Tuning
Eliminate Barriers
request.timeout.ms=5000
retry.backoff.ms=500
fetch.max.wait.ms=30
fetch.min.bytes=131072
fetch.max.bytes=8388608
max.partition.fetch.bytes=8388608
enable.auto.commit=true
auto.commit.interval.ms=1000
isolation.level=read_committed
session.timeout.ms=10000
heartbeat.interval.ms=3000
auto.offset.reset=latest
max.poll.interval.ms=30000
max.poll.records=1000000

18. Oauth Handler
Security Standards
Java Kafka client oauth handler classes
Set following properties in config files
oauth.token.endpoint.uri=
oauth.client.id=
oauth.client.secret=
NasdaqDataLink nasdaqDataLink =
new
NasdaqDataLink(securityCfg,kafkaConf
ig);
Nasdaq Data Link Configuration

19. Convenient Functions
Market Data Nuances

20. Testing Framework
Market Data Nuances

21. SDK Demo

23. Key Learnings

24. Consumer Quotas
• Multi tenant cluster
• Heterogeneous types of clients/data
• Poorly configured consumer
• Market Data is volatile
Key Learnings

25. Rebalancing Kafka Cluster
• Topic load change
• Evolving customer demand
• New data feeds
Key Learnings

26. Open Source is Challenging
• Expert Support
• Lot of moving parts
• Tooling and Monitoring
Key Learnings

27. Questions?