Kafka is a VERY flexible and configurable system. That flexibility has undoubtedly been key to its adoption, but it's also a double-edged sword. How can you ensure applications use cluster resources effectively, knowing that much configuration can be performed client-side? Hidden away from the watchful eye of operators. An incorrectly chosen value hidden in a config file somewhere can have drastic effects on application and Kafka cluster performance. At Conduktor, we have observed many examples of such issues reported by our customers. Common issues include producers with inefficiently configured batch size and linger, consumer groups with more members than subscribed partitions, and topics that unintentionally mix data with schema and without. In many of these cases, the impact of the issue is invisible to the client application but its impact on Kafka cluster performance is significant. Join us for a detective hunt as we discuss tools and metrics used to detect misconfigurations in clients, how to address them once discovered, and ways in which to ensure that new occurrences are prevented from arising in the future.

1. Safeguarding - protecting your clusters
from misbehaving clients.
Tom Scott

2. ● Principal Engineer - Conduktor
Gateway
● 6 years in Kafka across many roles
● 15+ yrs in all things data
Who am I?

3. ● To minimise outages
● To respond gracefully to change
● To reduce friction between domains
Result: Velocity!
Start with Why

4. The practice of protecting your data and
infrastructure from intentional and
unintentional threats, both internal and
external.
What is Safeguarding?
Data Governance
“
Technical Best
Practices
Safeguarding

5. A collection of loose, negotiable contracts enforced by
humans.
Case in point: The “production readiness” check
These suffer from:
● Relevance - scoped for multiple applications in multiple technologies
● Timeliness - Seen as a box check to be completed just before go live
● Perception - Rules to be bent/ignored, “what can we get away with?”
Stage 1: safeguarding by habit

6. “Your application is behaving badly”
Now what?
● Uncomfortable decisions
● Temporary solutions (that become permanent)
● Tech debt
Stage 2: reactive automation

7. ● Enforced at the earliest point
● Applied in every environment
● Specific and detailed information on breaches
Examples include:
● Schema validation
● Partition count limits
● Automatic encryption
Stage 3: proactive automation

8. How? (but actually where)
A Client Plugin
A Broker Plugin
A Proxy

9. Application
Proxy
Application
Encryption
Interceptor
Schema Validation
Interceptor
Ring fenced
Conduktor Gateway - a Kafka
Proxy
● Interceptors - functions that perform
transformation/filtering/projection on Kafka
requests/responses
● Gateway embeds one or more interceptors
● To a client:
Gateway == Kafka Brokers

10. ● Proactive safeguarding:
○ Prevents outages
○ Resets perception
● Enforced inter-domain contracts help:
○ React gracefully to change
○ Reach broader technical goals
○ Reduce friction
● Proactive safeguarding does not:
○ Define governance/best practice policies
○ Fully replace reactive solutions
In summary

11. Q&A