This document discusses using Kafka Streams to transform operational metrics data from Priceline applications before loading it into Splunk. It describes how the legacy monitoring system worked, and the motives for moving to Kafka and Kafka Streams. It then explains how data is collected from applications into Kafka topics, and how various transformations like formatting, key application, and aggregation are performed using Kafka Streams before loading to Splunk. It also discusses testing, monitoring, and debugging Kafka Streams applications.

1. Data Transformations on Operational Metrics using Kafka Streams
Vidhya Ramachandran
Mukund Murthy

2. About Priceline
Priceline.com is part of the
Booking Holdings Group. The
Booking Holdings Group is
the world leader in online
travel & related services. The
Group’s mission is to help
people experience the world.

3. About Us
Vidhya Ramachandran Mukund Murthy
Director Engineering, Common Platform Services Software Engineer , Common Platform Services

4. Legacy Monitoring System

5. Legacy Architecture
Custom Alerting System
Custom Dashboards
Application Servers

6. Kafka - The Only Constant!

7. Current Architecture

8. Motive behind going with Kafka & Kafka Streams

9. Kafka Infrastructure & new Monitoring Solution & other Sinks
Config Reader Event Listener
Applications
Message Router
Embedded Agent
Asynchronous Message Dispatcher
Kafka Consumer
Cluster
Infrastructure Code embedded in
the Products to Produce Events
to the Kafka Topic Splunk HEC
Add other sinks
Data Collection Console

10. Priceline Data Collection Console

11. Priceline Streams Multiplexing
• Stream = Events emitted by some source (for e.g. a Priceline Application)
• Multiple Streams can come from the same Application and be written into the
same topic
• A stream can be emitted by multiple Applications
Stream 1
Stream 3
Stream 2
Topic

12. Priceline Data Collection Console

13. Streams in Splunk before & after transformations

14. License Reduction from Transformations
Operational metrics data flowing through Kafka are converted to Pipe
separated format and the keys are applied back as search time
extractions.

15. Transformations for PCI/PII

16. Statsd Conversion

17. Topology – Kafka Streams

18. Windowed Events

19. Custom Partitioner

20. Custom TimeStamp Extractor

21. License Reduction from Summarizations

22. Late Arriving Events
Windowed Event Entries
| T1 | T2 | T3 | ……
|1 2 3 |5 6 7 |9 10 11 |
--------> 4 (T1)
--------- > 8 (T2)
Key Value
T1 1,2,3
T1 1,2,3,4
T2 5,6,7
T2 5,6,7,8
T3 9,10,11

23. Processing too long…
Measure & make sure to keep your Processors simple and
straightforward
Maintain SLA in Monitoring system < 5seconds from Event
Time…

24. Exception Handling

25. Gotchas…

26. Testing Kafka Streams
• Unit testing individual
processors and transformers
• Integration testing with an
embedded Kafka cluster,
multiple instances
• Integration Testing the
aggregation by repeating the
concatenated raw messages
in the final aggregated event

27. Debugging / Testing

28. 28
• Splunk Add-on to Poll local or remote JMX management Servers
• Index MBean attributes, outputs from MBean operations, and
MBean notifications.
• Configurable Templates to selectively monitor JMX stats
Kafka Monitoring – JMX Stats

29. Kafka Monitoring - Dashboards

30. Kafka Monitoring - Alerts

31. Monitoring Kafka Streams
• Thread metrics
• Average time for commits, poll, process operations
• Tasks created per second, tasked closed per second
• Task metrics
• Average number of commits per second
• Average commit time
• Processor node metrics
• Average and max processing time
• Average number of process operations per second Forward rate
• State store metrics
• Average execution time for put, get, and flush operations
• Average number put, get, and flush operations per second

33. Conclusion & Next Steps

34. Acknowledgements
• Our Core Platform team for their contributions to the Streaming library
and Data Collection Console
• To Confluent support and sales team

35. We are Hiring!!! https://careers.priceline.com/

36. Questions