Kafka on Yarn (KOYA) is an open-source integration initiative aimed at enhancing Apache Kafka's deployment and management within Hadoop's YARN framework, providing scalability, fault tolerance, and ease of use. It addresses challenges such as complex deployment and operational inefficiencies by automating broker recovery and centralizing resource management. Future enhancements include improved status information, UI options for Kafka management, and better control over server placement.

1. Kafka On YARN (KOYA)
An Open Source Initiative to integrate Kafka & YARN
Thomas Weise – thomas@datatorrent.com
April 15th, 2015

2. Scalable, High Performance, Fault Tolerant In-memory Stream Processing Platform
Physical | Virtual | Cloud
Hadoop 2.x – YARN + HDFS
Re-usable Stream Operator Library
Ingestion Transformation Analytics
Alert-
Action
Visualization
&
Distribution
Management & Monitoring Graphical App Design &
Launch
Real-Time Data
Visualization
Architecting for Stream Processing…

3. High volume auto-scaling fault
tolerant event stream across
multiple data centers globally.
Dimensional computation.
Use Case: Online Advertising Analytics
Ad Servers
DC1
Ad Servers
DC2
Ad Servers
DC3
Real-time
Dashboard
Metrics
Impressions
Impressions
Impressions

4. “A high-throughput distributed messaging system.”
“Fast, Scalable, Durable, Distributed”
Kafka is a natural fit to deliver events
into a our stream processing platform.
Apache Kafka

5. Kafka feeds Stream Processing
Kafka Cluster
Server-1
P1 P2 P3
Server-2
P1 P2 P3
Server-3
P1 P2 P3
YARN Cluster
Node Manager
DT Container
…
Node Manager
DT AppMaster
DT Container
… …
Resource
Manager
…

6. Problem?
• It is not easy to get started with Kafka
– Initial deployment difficult (bring your own tools)
• It is not easy to keep it running
– No central management (status, configuration changes,…)
– No automatic replacement for failed broker
• Operational Inefficiencies
– Resource fragmentation, underutilization
– Common infrastructure not leveraged, extra skill sets
• Adaption Barrier!

7. Why Kafka on YARN
• YARN enables:
– Horizontal scalability with commodity hardware
– Central resource management with queues, limits and locality constraints
– Framework for achieving fault tolerance and security
• Automate:
– Broker recovery
– Deployment of Kafka clusters
• Integrate:
– User friendly management (alternative to Kafka command line utilities)

8. YARN Cluster
Kafka on YARN through Slider
Node Manager
…
Node Manager
DT AppMaster
DT Container
… …
Resource
Manager
…
Node Manager
…
Node Manager
Slider AM
DT Container
…
Server-1
P
1
P
2
P
3
Server-2
P
1
P
2
P
3
Slider Agent
Slider Agent

9. Why Slider?
• Automates deployment and configuration of components
– Simplify on-demand cluster creation
• Generic AM for long running services
– Management of container failures – automates recovery
– Sticky allocation of components to hosts across AM restart
– Isolation: node labels to pin components to specific set of machines
• Central status
– View all servers in one place
• Areas for improvement
– Anti-affinity support (YARN limitation)
– Agent API documentation
– Flexibility in component instance specification

10. Configuration Example

11. Demo

18. Project Status
• Open Source: https://github.com/DataTorrent/koya
• Python Scripts + Configuration
• Works on Hadoop 2.6 through Slider 0.6
• Install: Embedded Slider or Application Package
• First Release by Q2
• Future Enhancements
– Expanded Status Info through Slider AM
– Explore Kafka management UI options
– Support for Disk as a Resource in YARN - YARN-2139
– Better control over server placement (anti-affinity)
– Slider-799

19. Q and & A
For more information: www.datatorrent.com/blogs