(Mike Graham + Dan Carroll, Comcast) Kafka Summit SF 2018 Comcast manages over 2 million miles of fiber and coax, and over 40 million in home devices. This “outside plant” is subject to adverse conditions from severe weather to power grid outages to construction-related disruptions. Maintaining the health of this large and important infrastructure requires a distributed, scalable, reliable and fast information system capable of real-time processing and rapid analysis and response. Using Apache Kafka and the Kafka Streams Processor API, Comcast built an innovative new system for monitoring, problem analysis, metrics reporting and action response for the outside plant. In this talk, you’ll learn how topic partitions, state stores, key mapping, source and sink topics and processors from the Kafka Streams Processor API work together to build a powerful dynamic system. We will dive into the details about the inner workings of the state store—how it is backed by a Kafka “changelog” topic, how it is scaled horizontally by partition and how the instances are rebuilt on startup or on processor failure. We will discuss how these state stores essentially become like materialized views in a SQL database but are updated incrementally as data flows through the system, and how this allows the developers to maintain the data in the optimal structures for performing the processing. The best part is that the data is readily available when needed by the processors. You will see how a REST API using Kafka Streams “interactive queries” can be used to retrieve the data in the state stores. We will explore the deployment and monitoring mechanisms used to deliver this system as a set of independently deployed components.

1. INSIDE KAFKA STREAMS -
MONITORING COMCAST’S OUTSIDE PLANT
October 17, 2018

2. 2
MONITORING
”OUTSIDE PLANT”
EXTERNAL FACTORS
• Weather
• Power
• Construction
REQUIREMENTS
• Fast
• Resilient
• Evolvable

3. 3
NODENODENODE
MONITORING ”OUTSIDE PLANT”
CMTS
ANALYZE
NODES
GENERATE
EVENTS
SCORE
NODES
PRIORITIZE
WORK
POLL
POLL
THRESHOLDS
DEVICES
USE KAFKA
STREAMS TO
MONITOR
AND
RESPOND TO
OUTSIDE
PLANT
CONDITIONS

4. 4
TOPOLOGY
Copy
XX%
STREAMING APPLICATION APIS
KAFKA STREAMING TECH STACK
CONCEPTS
INFORMATION SYSTEM
CLIENTPROCESSOR
API
STREAMS
API
KSQL
STREAMING APPLICATIONS
KAFKA SERVER PLATFORM
ANALYSIS EVENTS NODE SCORE
BROKERS ZOOKEEPER CONNECT
PRODUCER TOPIC CONSUMER
CLIENTJAR
ADMIN CLIENT
STREAMSJARKSQLJAR
SOURCE PROCESSOR SINK
STATE STORE
BUILDER
KSTREAM OPERATIONS
KTABLE
STREAM KSQL
TABLE

5. 5
PARTITION MECHANICS
KAFKA CONSUMERS & PRODUCERS
CONCEPTS
PRODUCERS
TOPIC CONSUMER GROUP
BROKERBROKERBROKERBROKER
P 0
P 0
P 0
P 1
P 2
P 1
P 1
P 2
P 2
KEY
VALUE
KEY
VALUE
KEY
VALUE
KEY
VALUE
PRODUCER
PRODUCER
PRODUCER
PRODUCER
H
A
S
H
CONSUMER
CONSUMER
CONSUMER
CONSUMER
CONSUMER GROUP
CONSUMER

6. 6
PARTITION MECHANICS IN THE PROCESSOR API
KAFKA STREAMS - PROCESSOR API - PARTITIONING
CONCEPTS
SOURCE-TOPICS
P 0
P 1
P 2
SINK-TOPICS
P 0
P 1
P 2
STATE-STORE-CHANGELOG-TOPICS
P 0 P 1 P 2
TOPOLOGY
SOURCES SINKS
STATE STORES
INSTANCE 0
SOURCES PROCESSORS SINKS
STATE STORES
INSTANCE 1
SOURCES PROCESSOR SINKS
STATE STORES
INSTANCE 2
SOURCES PROCESSORS SINKS
STATE STORES
PROCESSORS

7. 7
PROCESSOR
STATE STORE DURABILITY
KAFKA STREAMS - PROCESSOR API – STATE STORE
CONCEPTS
STATE STORE (P 0)
STATE-STORE-CHANGELOG-TOPIC (P 0)
K:1
V:a
K:1
V:a
K:2
V:b
K:2
V:b
K:3
V:c
K:3
V:c
K:1
V:a
K:2
V:b
K:3
V:c
K:1
V:d
K:1
V:d
K:1
V:d
K:2
V:e
K:1
V:f
K:2
V:e
K:2
V:e
K:1
V:f
K:1
V:f
LOG COMPACTION
K:1
V:a
K:2
V:b
K:1
V:d
SOURCE TOPIC (P 0)
K:1
V:a
K:2
V:b
K:3
V:c
K:1
V:d
K:2
V:e
K:1
V:f

8. 8
TODAY’S DISCUSSION – PROCESSOR API PATTERNS
Monitoring Outside Plant
Streaming Concepts
Processor API Patterns
Tech Notes

9. 9
STATE STORE USE
STATE STORES ARE USED FOR
• Deduplication of requests
• Materialized view of table data
• Rolling aggregates

10. 1 0
STATE STORE
DE-DEPLICATION OF NODE REQUESTS
• Use Case: Do not have duplicate node polls running concurrently
• This architecture easily prevents duplicate node requests and reduces
resource stress and avoids database locking
• Works in conjunction with sockets to return results to multiple requestors
(UI), or will publish results to a state store

11. 1 1
STATE STORE
DE-DUPLICATION OF NODE REQUESTS
NA
Publish
processor
Node
State
store
request
Set to “running”
Node LK1 “at rest”
Node AnalysisNA
Request
processor
Set to “at rest”
Node Poll Results
DE-DUPLICATION OF NODE REQUESTS
UI (or
other)

12. 1 2
STATE STORE
MATERIALIZED VIEW OF TABLE DATA
• Use Case: Need to have a current list of device data (location & device type),
also ref data, poll data and other needs. We are reshaping the raw data for
use, this is faster that retrieving and reshaping constantly.
• We use the UI, a timer or an external system “push” to load data to a topic
and then a state store. In the future, we would like to get all changes in data
streamed to our application
• We query the app not the DB

13. 1 3
STATE STORE
MATERIALIZED VIEW OF TABLE DATA
Topic with
raw data
processor
State store
Device &
location
State
store
stats
State store
performance
Timer
UI App
Rest
Endpoint
Rest of
application

14. 1 4
STATE STORE
ROLLING AGGREGATES
• Use Case: Keeping track of the steps of a node request
• Can pull stats from this data, and is a powerful tool for checking health of our
application
• Gives us a window into the node analysis steps

15. 1 5
STATE STORE
ROLLING AGGREGATES
Our UI displays the results
from each stage of a node
request. This is a useful tool
for operations.

16. 1 6
TIMERS AND STREAM LOGIC
TIMERS AND STREAMING SOLUTION
• Data Pump
• Plant Validator

17. 1 7
TIMERS AND STREAM LOGIC
DATA PUMP
• Use Case: Load data from other systems/ sources on a schedule
• Most other systems don’t push change messages to us, and generally make
the data available through rest APIs
• We use a timer to kick off getting the data
• We stream the data to a topic and a processor takes the data, shapes it to the
needed use, and populates a state store or topic.

18. 1 8
Copy
XX%
Copy
XX%
Copy
XX%
Copy
XX%
CODE SAMPLE FOR A TIMER
TIMER DATA LOAD
Spring application
@EnableScheduling
Use @Scheduled
Runs the data load
this one uses web service call

19. 1 9
TIMERS AND STREAM LOGIC
EVENT VALIDATOR FUNCTION
• Use Case: Some issues found in the outside plant need a “soaking” period to
confirm the issue to determine the correct action (if any)
• We need to move “soaking” events to “confirmed”, “closed” or “outbound”
• We use a one minute timer, check the status of an event, check if event is still
valid and update the status
• Possible through the use of a state store for the soaking events. No need to
query a DB

20. 2 0
TIMERS AND STREAM LOGIC
EVENT VALIDATOR PROCESS
Validation
Result
Processor
Soaking
Event
store
request
Get soaking events
Node AnalysisValidation
Processor
Event Actions
Timer
Set to “outbound”

21. 2 1
ASYNC REST SERVICE CALLS
USED IN COMMUNICATION WITH LEGACY SYSTEM
• Use Case: Remove latency when calling rest web services
• One thread available per stream topology, so we use Flux (from Reactor Web)
to make async rest service calls to prevent blocking
• To poll a node, we call a webservice that retrieves all the SNMP data for the
devices. The call takes 10+ seconds and the streaming
• When the async calls returns, we are no longer in streams processing, so we
use a producer to create the next topic message.

22. 2 2
ASYNC REST SERVICE CALLS
NODENODENODE
CMTS
ANALYZE
NODES
GENERATE
EVENTS
SCORE
NODES
PRIORITIZE
WORK
POLL
POLL
THRESHOLDS
ACCOUNTS

23. 2 3
Copy
XX%
CODE SAMPLE CREATING THE ASYNC CALL
ASYNC REST SERVICE CALLS
WebService call
callClient is a
WebClient
Returns a Mono

24. 2 4
Copy
XX%
Copy
XX%
Copy
XX%
Copy
XX%
CODE SAMPLE PROCESSING THE ASYNC RETURN
ASYNC REST SERVICE CALLS
Code runs when
call completes
receives the
response object
runs business
logic

25. 2 5
Copy
XX%
Copy
XX%
Copy
XX%
Copy
XX%
CODE SAMPLE PROCESSING THE ASYNC RETURN, GETTING BACK INTO STREAMS USING A PRODUCER
ASYNC REST SERVICE CALLS
Response returns
We are out of
streaming framework
Need to use a producer

26. 2 6
ADVANTAGES OF STREAMING
ADVANTAGES INCLUDE:
• Looked for solutions to refactor our current application code base
• Faster than a traditional database for this use case
• Streams solution is not monolithic. We’ll be able to update the app quicker
• Modular, once we had the basic framework for Node Analysis developed.
• Programming model is very straightforward
• We only need a few code patterns to solve many of our needs
• Kafka streams is fundamentally different from the other solutions we had
available

27. 2 7
TODAY’S DISCUSSION – TECH NOTES
Monitoring Outside Plant
Streaming Concepts
Processor API Patterns
Tech Notes

28. 2 8
INSTANCE2
REST API ßà STATE STORE
INTERACTIVE QUERY
TECH NOTES
STATE-STORE-CHANGELOG-TOPIC
P 0 P 1 P 2
INSTANCE0
REST API
STATE STORE
INSTANCE1
REST API
STATE STORE STATE STORE
PROPERTY: SERVER & PORT
INJECT TOPOLOGY BUILDER
GET STREAMS FROM TOPOLOGY
STREAMS.STORE (LOOP / TIMER)
META: SERVER MAP WITH PARTITIONS
REST API

29. 2 9
PROCESSOR API - DIAGRAMING & NAMING
DIAGRAMING
TECH NOTES

30. 3 0
MICROSERVICES & DEPLOYMENT
TECH NOTES
APPLICATION INTEGRATION
STREAMING APPLICATIONS
KAFKA SERVER PLATFORM
APP 1 APP 2
TOPICS

31. 3 1
SCRIPT TO MANAGE TOPICS
TOPICS SCRIPT
TECH NOTES
ENVIRONMENT PREFIX
TOPOLOGIES, CONSUMER GROUPS, & TOPICS
KAFKA TOPICS
MIKE- DAN- DEV-
MIKE- DAN- DEV- TEST-
TEST-
Topic Prefix
Partition Count
Target Kafka Cluster
Delete Existing Topics?
Create New Topics?

32. 3 2
GRAFANA & PROMETHEUS
MONITORING
TECH NOTES
KAFKA SERVER
PLATFORM
BROKER
JMX
EXPORTER
GRAFANA DASHBOARDS
YAML
PROMETHEUS PLATFORM
JSON
JNDI(:1234)

33. 3 3
OPEN SOURCE AT COMCAST
APACHE TRAFFIC
CONTROL
Graduated to TLP May 16, 2018
Build a large scale content
delivery network using open
source.
MIRROR-TOOL-FOR-KAFKA-
CONNECT
Comcast’s Rhys McCaig
Mirroring Kafka topics between
clusters using the Kafka
Connect framework.
145 OPEN SOURCE
REPOSITORIES
Comcast is committed to open
source software.
We use OSS to build products,
attract talent and evolve the
technology we use to improve
the customer experience.
comcast.github.io
github.com/comcast
labs.comcast.com

34. comcast.github.io
github.com/comcast
Mike Graham
twitter datumgeek
github datumgeek
charles_graham@comcast.com
Dan Carroll
twitter dcarroll
github dcarroll
daniel_carroll@comcast.com
Thank You !!!!
J J