The document introduces Hortonworks Streams Messaging Manager (SMM), a tool designed for managing and monitoring Kafka streams to address 'Kafka blindness'. It offers a unified monitoring dashboard for various Kafka entities, integrates with Apache Atlas for data lineage, and supports hybrid cloud deployments. Key features include enhanced troubleshooting capabilities, end-to-end stream visualization, and developer-friendly REST API access.

1. 1 © Hortonworks Inc. 2011–2018. All rights reserved
Curing Kafka Blindness with
Hortonworks Streams Messaging
Manager (SMM)
George Vetticaden, VP Product Management, @gvetticaden
Dinesh Chandrasekhar, Director, Product Marketing, @AppInt4All

2. 2 © Hortonworks Inc. 2011–2018. All rights reserved
Streaming Analytics Reference Architecture
Data Flow Apps
Powered by NiFi
Kafka is Everywhere. Critical Component of Streaming Architectures
Kafka Producers Kafka Topics Kafka TopicsKafka Consumers & Producers Kafka Consumers
US West Fleet
Truck Sensors C++
Agent
US Central Fleet
Truck Sensors C++
Agent
US East Fleet
Truck Sensors C++
Agent
Analytics App 1
Analytics App 2
Analytics App 5
Analytics App 3
Analytics App 4

3. 3 © Hortonworks Inc. 2011–2018. All rights reserved
Distinct Needs of 3 Personas/Teams using Kafka
Concerned with monitoring
the overall health of the
cluster and the infrastructure
it runs on
Concerned with monitoring
the Kafka entities associated
with their apps
Concerned with audit,
compliance, access control &
chain of custody requirements

4. 4 © Hortonworks Inc. 2011–2018. All rights reserved
HORTONWORKS
DATA PLATFORM (HDP®)
DATA-AT-REST
PRODUCERS CONSUMERS
TOPICS
BROKERS
SMM REST Server
HORTONWORKS
DATAFLOW (HDF)
DATA-IN-MOTION
Optimize your
Kafka Clusters
Troubleshoot
Kafka Streams
Trace end-to-
end Kafka flows
Streams Messaging Manager (SMM)

5. 5 © Hortonworks Inc. 2011–2018. All rights reserved
Cure is Here: Hortonworks Streams Messaging Manager (SMM)
What is SMM?
 Kafka Management and Monitoring tool
 Cure the “Kafka Blindness”
 Single Monitoring Dashboard for all your
Kafka Clusters across 4 entities
– Broker
– Producer
– Topic
– Consumer
 Supports multiple HDP and/or HDF Kafka
Clusters
 REST as a First Class Citizen
 Delivered as a DataPlane Service

6. 6 © Hortonworks Inc. 2011–2018. All rights reserved
Simplify troubleshooting Kafka environments
• Visibility and control to the
Kafka Operations teams
• Intelligent filtering and sorting
• Grafana integration
• Ambari integration

7. 7 © Hortonworks Inc. 2011–2018. All rights reserved
Visualize end-to-end Kafka stream flows
• Interactive visualization
• All streams/flows across all
Kafka clusters
• Analyze the stream dynamics
between producers and
consumers
• Intelligent filtering
• Throughput analysis

8. 8 © Hortonworks Inc. 2011–2018. All rights reserved
Track data lineage and governance from edge-to-enterprise
• Tight integration with Apache
Atlas
• Metadata about Kafka Topic in
Atlas
• Complete data lineage across
multiple Kafka hops

9. 9 © Hortonworks Inc. 2011–2018. All rights reserved
Extend monitoring / management capabilities with REST
• Developer / DevOps friendly
• Search for specific topics
• Drill down for messages
• SMM REST Server
• All capabilities exposed as
REST services

10. 10 © Hortonworks Inc. 2011–2018. All rights reserved
Adopt hybrid cloud deployments for Kafka
• SMM as a DataPlane Platform
app
• Manage multiple Kafka clusters
from a single SMM instance
• Hybrid cloud model

11. 11 © Hortonworks Inc. 2011–2018. All rights reserved
Demo

12. 12 © Hortonworks Inc. 2011–2018. All rights reserved
Recap: What Did We Just Show? Tracking the flow of data across multiple Kafka
Hops with SMM & Atlas Integration-Powerful
Data Flow Apps
Powered by NiFi
US West Fleet
Truck Sensors C++
Agent
US Central Fleet
Truck Sensors C++
Agent
US East Fleet
Truck Sensors C++
Agent
Analytics App 3
Analytics App 2
Analytics App 1

13. 13 © Hortonworks Inc. 2011–2018. All rights reserved
www.hortonworks.com/smm