(Mike Graham + Dan Carroll, Comcast) Kafka Summit SF 2018
Comcast manages over 2 million miles of fiber and coax, and over 40 million in home devices. This “outside plant” is subject to adverse conditions from severe weather to power grid outages to construction-related disruptions. Maintaining the health of this large and important infrastructure requires a distributed, scalable, reliable and fast information system capable of real-time processing and rapid analysis and response. Using Apache Kafka and the Kafka Streams Processor API, Comcast built an innovative new system for monitoring, problem analysis, metrics reporting and action response for the outside plant.
In this talk, you’ll learn how topic partitions, state stores, key mapping, source and sink topics and processors from the Kafka Streams Processor API work together to build a powerful dynamic system. We will dive into the details about the inner workings of the state store—how it is backed by a Kafka “changelog” topic, how it is scaled horizontally by partition and how the instances are rebuilt on startup or on processor failure. We will discuss how these state stores essentially become like materialized views in a SQL database but are updated incrementally as data flows through the system, and how this allows the developers to maintain the data in the optimal structures for performing the processing. The best part is that the data is readily available when needed by the processors. You will see how a REST API using Kafka Streams “interactive queries” can be used to retrieve the data in the state stores. We will explore the deployment and monitoring mechanisms used to deliver this system as a set of independently deployed components.
4. 4
TOPOLOGY
Copy
XX%
STREAMING APPLICATION APIS
KAFKA STREAMING TECH STACK
CONCEPTS
INFORMATION SYSTEM
CLIENTPROCESSOR
API
STREAMS
API
KSQL
STREAMING APPLICATIONS
KAFKA SERVER PLATFORM
ANALYSIS EVENTS NODE SCORE
BROKERS ZOOKEEPER CONNECT
PRODUCER TOPIC CONSUMER
CLIENTJAR
ADMIN CLIENT
STREAMSJARKSQLJAR
SOURCE PROCESSOR SINK
STATE STORE
BUILDER
KSTREAM OPERATIONS
KTABLE
STREAM KSQL
TABLE
5. 5
PARTITION MECHANICS
KAFKA CONSUMERS & PRODUCERS
CONCEPTS
PRODUCERS
TOPIC CONSUMER GROUP
BROKERBROKERBROKERBROKER
P 0
P 0
P 0
P 1
P 2
P 1
P 1
P 2
P 2
KEY
VALUE
KEY
VALUE
KEY
VALUE
KEY
VALUE
PRODUCER
PRODUCER
PRODUCER
PRODUCER
H
A
S
H
CONSUMER
CONSUMER
CONSUMER
CONSUMER
CONSUMER GROUP
CONSUMER
6. 6
PARTITION MECHANICS IN THE PROCESSOR API
KAFKA STREAMS - PROCESSOR API - PARTITIONING
CONCEPTS
SOURCE-TOPICS
P 0
P 1
P 2
SINK-TOPICS
P 0
P 1
P 2
STATE-STORE-CHANGELOG-TOPICS
P 0 P 1 P 2
TOPOLOGY
SOURCES SINKS
STATE STORES
INSTANCE 0
SOURCES PROCESSORS SINKS
STATE STORES
INSTANCE 1
SOURCES PROCESSOR SINKS
STATE STORES
INSTANCE 2
SOURCES PROCESSORS SINKS
STATE STORES
PROCESSORS
8. 8
TODAY’S DISCUSSION – PROCESSOR API PATTERNS
Monitoring Outside Plant
Streaming Concepts
Processor API Patterns
Tech Notes
9. 9
STATE STORE USE
STATE STORES ARE USED FOR
• Deduplication of requests
• Materialized view of table data
• Rolling aggregates
10. 1 0
STATE STORE
DE-DEPLICATION OF NODE REQUESTS
• Use Case: Do not have duplicate node polls running concurrently
• This architecture easily prevents duplicate node requests and reduces
resource stress and avoids database locking
• Works in conjunction with sockets to return results to multiple requestors
(UI), or will publish results to a state store
11. 1 1
STATE STORE
DE-DUPLICATION OF NODE REQUESTS
NA
Publish
processor
Node
State
store
request
Set to “running”
Node LK1 “at rest”
Node AnalysisNA
Request
processor
Set to “at rest”
Node Poll Results
DE-DUPLICATION OF NODE REQUESTS
UI (or
other)
12. 1 2
STATE STORE
MATERIALIZED VIEW OF TABLE DATA
• Use Case: Need to have a current list of device data (location & device type),
also ref data, poll data and other needs. We are reshaping the raw data for
use, this is faster that retrieving and reshaping constantly.
• We use the UI, a timer or an external system “push” to load data to a topic
and then a state store. In the future, we would like to get all changes in data
streamed to our application
• We query the app not the DB
13. 1 3
STATE STORE
MATERIALIZED VIEW OF TABLE DATA
Topic with
raw data
processor
State store
Device &
location
State
store
stats
State store
performance
Timer
UI App
Rest
Endpoint
Rest of
application
14. 1 4
STATE STORE
ROLLING AGGREGATES
• Use Case: Keeping track of the steps of a node request
• Can pull stats from this data, and is a powerful tool for checking health of our
application
• Gives us a window into the node analysis steps
15. 1 5
STATE STORE
ROLLING AGGREGATES
Our UI displays the results
from each stage of a node
request. This is a useful tool
for operations.
16. 1 6
TIMERS AND STREAM LOGIC
TIMERS AND STREAMING SOLUTION
• Data Pump
• Plant Validator
17. 1 7
TIMERS AND STREAM LOGIC
DATA PUMP
• Use Case: Load data from other systems/ sources on a schedule
• Most other systems don’t push change messages to us, and generally make
the data available through rest APIs
• We use a timer to kick off getting the data
• We stream the data to a topic and a processor takes the data, shapes it to the
needed use, and populates a state store or topic.
19. 1 9
TIMERS AND STREAM LOGIC
EVENT VALIDATOR FUNCTION
• Use Case: Some issues found in the outside plant need a “soaking” period to
confirm the issue to determine the correct action (if any)
• We need to move “soaking” events to “confirmed”, “closed” or “outbound”
• We use a one minute timer, check the status of an event, check if event is still
valid and update the status
• Possible through the use of a state store for the soaking events. No need to
query a DB
20. 2 0
TIMERS AND STREAM LOGIC
EVENT VALIDATOR PROCESS
Validation
Result
Processor
Soaking
Event
store
request
Get soaking events
Node AnalysisValidation
Processor
Event Actions
Timer
Set to “outbound”
21. 2 1
ASYNC REST SERVICE CALLS
USED IN COMMUNICATION WITH LEGACY SYSTEM
• Use Case: Remove latency when calling rest web services
• One thread available per stream topology, so we use Flux (from Reactor Web)
to make async rest service calls to prevent blocking
• To poll a node, we call a webservice that retrieves all the SNMP data for the
devices. The call takes 10+ seconds and the streaming
• When the async calls returns, we are no longer in streams processing, so we
use a producer to create the next topic message.
25. 2 5
Copy
XX%
Copy
XX%
Copy
XX%
Copy
XX%
CODE SAMPLE PROCESSING THE ASYNC RETURN, GETTING BACK INTO STREAMS USING A PRODUCER
ASYNC REST SERVICE CALLS
Response returns
We are out of
streaming framework
Need to use a producer
26. 2 6
ADVANTAGES OF STREAMING
ADVANTAGES INCLUDE:
• Looked for solutions to refactor our current application code base
• Faster than a traditional database for this use case
• Streams solution is not monolithic. We’ll be able to update the app quicker
• Modular, once we had the basic framework for Node Analysis developed.
• Programming model is very straightforward
• We only need a few code patterns to solve many of our needs
• Kafka streams is fundamentally different from the other solutions we had
available
28. 2 8
INSTANCE2
REST API ßà STATE STORE
INTERACTIVE QUERY
TECH NOTES
STATE-STORE-CHANGELOG-TOPIC
P 0 P 1 P 2
INSTANCE0
REST API
STATE STORE
INSTANCE1
REST API
STATE STORE STATE STORE
PROPERTY: SERVER & PORT
INJECT TOPOLOGY BUILDER
GET STREAMS FROM TOPOLOGY
STREAMS.STORE (LOOP / TIMER)
META: SERVER MAP WITH PARTITIONS
REST API
33. 3 3
OPEN SOURCE AT COMCAST
APACHE TRAFFIC
CONTROL
Graduated to TLP May 16, 2018
Build a large scale content
delivery network using open
source.
MIRROR-TOOL-FOR-KAFKA-
CONNECT
Comcast’s Rhys McCaig
Mirroring Kafka topics between
clusters using the Kafka
Connect framework.
145 OPEN SOURCE
REPOSITORIES
Comcast is committed to open
source software.
We use OSS to build products,
attract talent and evolve the
technology we use to improve
the customer experience.
comcast.github.io
github.com/comcast
labs.comcast.com