1. Using Machine Learning to Govern
Kafka Clients
UmaMahesh Sistu & Shu Wang
Event Streaming Platform Team, Fidelity Investments

2. PART 1 - Introduction
Streaming platform
Application structure
Initial ideas
Application governance
PART 3 – Machine Learning
Machine learning
Continuous learning
Optimization
PART 2 - Implementation
Implementation
Challenges
Kafka linting tool
Rules engine
Demo
Objectives
Using Machine Learning to Govern Kafka Clients

3. 3
Part 1
Introduction
Streaming Platform Ecosystem of Services

4. Fidelity Management &
Research Company
Founded
Fidelity Purchases
First Computer
Fidelity Becomes an
Internet Pioneer with
First Mutual Fund to
Create a Home Page
Today
6,500+ Applications on Public Cloud
Fidelity’s First
Product Application
Deployed to the
Cloud
1965
1946
1995
2016
2019
Fidelity Launches Multi-Cloud Hybrid Strategy
Fidelity’s Cloud Journey

5. Event Streaming Platform
4
Years in
public cloud
6B+
Events per day
72+
16k+
Producer and consumer
Applications
Self-service APIs
300+
Observability Metrics

6. Event Streaming Platform
Core and Common Capabilities
Schema Registry
Connectors
Query and Analyze Streaming
Data
Management Plane
Events and Streams
Control
Plane
Data Plane
Tech
Stack
Telemetry
Database
CRM
Other
Data
Warehouse
Down Stream
Apps
DATA
INTEGRATION
App Events Log Events Mobile IoT Events
Business
Events
Analytics & ML
Inventory
Management
Other
Fraud
Detection
Transactional
Apps
REAL-TIME
APPLICATIONS
Producer
Consumer
Platform capabilities and
standards
Enhance developer experience
FinOps management
Consistent observability
Security and governance
LARGE DATA SETS
Multi
Cloud

7. Streaming Applications
• Labyrinth of streaming paths
• Many producers and
consumers
• Multi-cloud service
providers
• Polyglot of programming
languages
• Complex event mesh

8. Governance Challenges
Increase
throughput
Avoid
common
pitfalls
Producer:
1. buffer.memory
2. batch.size
3. compression.type
4. linger.ms
5. retries
Consumer:
1. fetch.min.bytes
2. fetch.max.wait.ms
3. max.poll.interval.ms
4. max.poll.records
5. enable.auto.commit

9. Initial Approach
Documentation
Best practices
Training
Workshops

10. Initial Approach
Scale and efficiency?

11. Application Governance
Introduction of Application / Client Governance Gates
Application Observability
Performance Testing
Resiliency Testing
Client Config Linting
Client Optimization Release to Prod
Release to Prod

12. Application Governance
Introduction of Application / Client Governance Gates
Application Observability
Performance Testing
Resiliency Testing
Client Config Linting
Client Optimization Release to Prod
Release to Prod

13. Part 2
Implementation
Challenges and Opportunities
13

14. Challenges
• Thousands of producers/consumers
• 170+ client configurations
How can we help producer/consumer teams configure client apps
properly?

15. How can we convert our experiences to a tool that can help users?
Analyze

16. Rule Engine & Linting API
What feedback can we provide?
• Rule/Policy engine
• Decouple decision making logic from business code
• Maintain the rules/policies centrally as code
• Serve as a linting API to client application to provide
feedback

17. How do we integrate with the client applications?
Demo

18. What’s next?
What about the parameters that have range value?
What’s the best value for them?
Integration Details

19. Looking Forward
How to find the relationship between the parameters and the throughput?
Performance Testing
• Cost / expensive
• Too many teams

20. Part 3
Machine Learning
Continuous Learning and Optimization
20

21. How to use the model to figure out the best value of parameter
Inferring Insights using Machine Learning Models

22. Where should we start?
Use Model to Find the Better Result

23. How to find the best possible performance and the corresponding parameters
Determine the Starting Point

24. Is the model’s prediction perfect?
Inference

25. Determine the optimal values based on performance testing
Performance Testing

26. How to deal with changes and improve the model
Default allowed values
Rule base, security/best practice
Get start value to do further tuning
Inference to find better values
Much less performance testing
The optimal values of client configurations
Fine-Tuning Stages

27. Self-Learning Cycle
Keep adding new data
Keep removing obsolete data
Keep identifying the new factors
Keep improving the ML model
ML Model Optimization

28. Summary
Application Observability
Performance Testing
Resiliency Testing
Client Config Linting
Client Optimization

29. Thank You!
Kafka Summit London 2023