Building Event-Driven Microservices using Kafka Streams (Stathis Souris, ThousandEyes)

Building event-driven
microservices with
Kafka Streams
Stathis Souris
Lead Software Engineer

2
Copyright ©2020 ThousandEyes, Inc. All Rights Reserved. @ThousandEyes
Agenda
• Kafka
• Kafka Streams
• Endpoint Agent
• Kafka Streams Use Cases
• Production Issues
• Takeaways
• Q&A

3
Why Kafka
• Simple at ﬁrst!

4
Why Kafka
• Complicated

5
Decoupling of data streams

6
Why Kafka
• Distributed, resilient architecture, fault tolerant
• Horizontal scalability
• High performance (latency of less than 10ms) - real time
• User by known companies
– LinkedIn, Netﬂix, AirBnb etc

7
Apache Kafka: Use cases
• Messaging System
• Activity Tracking tool
• Gather metrics from diﬀerent locations
• Application logs
• Stream processing (Kafka Streams or Spark e.g.)
• Decoupling of systems
• Works with Spark, Flink, Hadoop etc

8
What is Kafka Streams?
• Easy data processing and
transformation library within
Kafka
• Standard Java Application
• No need to create a separate
cluster
• Highly scalable, elastic and fault
tolerant (inherits from Kafka)
• Exactly Once Capabilities
• One record at a time processing
(no batching)
• Works for any application size

9
Kafka Streams Architecture Design
•

10
Kafka Streams history
• The API / Library was introduced as part of Kafka 0.10 (2016)
• Serious contender to other processing frameworks such as
Spark, Flink, NiFi etc

11
About the Endpoint Agent
• Agents that run on users laptops or desktops
• Collect metrics from customer’s browser interactions
• Perform network tests e.g. ping, pathtrace against various targets
• Checks-in every 10 minutes
• Alerts & Reports

12
High-level Architecture Overview

13
Why event-driven microservices?
• Operate at large scale 100K agents
• Complex logic that needs to run at scale
• As real time as possible
• Asynchronous communication

14
Why Kafka Streams?
✓ Inherits Kafka Streams properties
✓ Simple DSL for
– Aggregations
– Windowing
✓ Streams & Tables
✓ <Key, Value>

16
Use case
Synthetic tests at an interval
Schedule tests on agents dynamically
Powerful visualization and filtering capabilities

17
Batch Job approach
• Agent checks-in every 10 minutes
• Batch job runs to assign tests every 15 minutes
• Pull state from various DBs
• Run business logic
• Save assignments
After stress testing:
■ Latency increase as we added more agents
■ Could only scale vertically - not an option at
that point

18
Event Driven approach
• Stream of check-ins
• Use that stream to power the Scheduler
• Assign tasks on check-in event

19
Event Driven approach
✓ Application scales with number of
Kafka partitions
✓ Join with GlobalKTables
✓ Run the business logic
✓ Save assignments in KTable
Facts:
➢ All state lives in Kafka
➢ At least once delivery
➢ Materialize assignments in MongoDB:
○ Historical queries
○ Timeline of assignments

20
Interactive Queries
• Query in-memory KTable for assignments
directly
• Expose through a Rest API
• Very fast
• When State store is temporarily unavailable
use MongoDB query
– zero-downtime deployments

Checkin Reconciler:
React on application
events

22
Checkin Reconciler

23
Problem:
■ updating the KTable on every event
■ creating hot partitions that took too long to process
After 20K agents

24
Use KTable cache
Reduce the commit interval of the application.
StreamsConfig.COMMIT_INTERVAL_MS_CONFIG
Temporary solution

25
Long term ﬁx
Removed repartitioning step and stored active check-ins in Redis instead

27
Browser Session Metrics
✓ Real User Monitoring events coupled with network
tests
✓ No set interval
✓ Alerter needs binned data
✓ One minute window and emit aggregated metrics

28
Window operator
Problem:
Alerting use case needs aggregated event to be emitted at the end, not on every update.

29
Suppress operator
Problems:
Windowed aggregates took to long to reach the Alerter.

30
Aggregation was delayed?
Closing a window is driven by
events, that advance the stream
time.
Solution:
Created a cron job to generate
events every close window +
grace period to force the window
to close.

31
Production issues
• Compaction wasn’t working in some cases
• Avoid repartitioning to hot keys
• Interactive queries misbehavior
– Metadata incorrect
– Created loop between services

32
Key Takeaways
✓ Use KTable cache to de-duplicate events before
sending downstream. Use “commit.interval” to
your advantage.
✓ Avoid hot partition keys if possible especially when
you are going big.
✓ Make sure compaction works for your topics
✓ If you don’t really use RocksDB disable it
✓ Use binary format from the beginning if you are
going big
✓ Kafka as a DB is possible, but don’t overdo it
✓ Small latencies on the processor level can add up
once you have lag (100ms * 10.000 ~= 16min)

33
Q&A
Twitter: @efsouris
Blogpost:
https://medium.com/thousandeyes-engine
ering/kafka-streams-in-the-endpoint-agent
-670a098ae7a4

Building Event-Driven Microservices using Kafka Streams (Stathis Souris, ThousandEyes)

Building Event-Driven Microservices using Kafka Streams (Stathis Souris, ThousandEyes)

More Related Content

What's hot

Similar to Building Event-Driven Microservices using Kafka Streams (Stathis Souris, ThousandEyes)

More from London Microservices

Recently uploaded

Building Event-Driven Microservices using Kafka Streams (Stathis Souris, ThousandEyes)