Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Curing Kafka Blindness with Hortonworks Streams Messaging Manager

429 views

Published on

With the growth of Apache Kafka adoption in all major streaming initiatives across large organizations, the operational and visibility challenges associated with Kafka are on the rise as well. Kafka users want better visibility in understanding what is going on in the clusters as well as within the stream flows across producers, topics, brokers, and consumers.

With no tools in the market that readily address the challenges of the Kafka Ops teams, the development teams, and the security/governance teams, Hortonworks Streams Messaging Manager is a game-changer.

https://hortonworks.com/webinar/curing-kafka-blindness-hortonworks-streams-messaging-manager/

Published in: Technology
  • Be the first to comment

Curing Kafka Blindness with Hortonworks Streams Messaging Manager

  1. 1. 1 © Hortonworks Inc. 2011–2018. All rights reserved Curing Kafka Blindness with Hortonworks Streams Messaging Manager (SMM) George Vetticaden, VP Product Management, @gvetticaden Dinesh Chandrasekhar, Director, Product Marketing, @AppInt4All
  2. 2. 2 © Hortonworks Inc. 2011–2018. All rights reserved Streaming Analytics Reference Architecture Data Flow Apps Powered by NiFi Kafka is Everywhere. Critical Component of Streaming Architectures Kafka Producers Kafka Topics Kafka TopicsKafka Consumers & Producers Kafka Consumers US West Fleet Truck Sensors C++ Agent US Central Fleet Truck Sensors C++ Agent US East Fleet Truck Sensors C++ Agent Analytics App 1 Analytics App 2 Analytics App 5 Analytics App 3 Analytics App 4
  3. 3. 3 © Hortonworks Inc. 2011–2018. All rights reserved Distinct Needs of 3 Personas/Teams using Kafka Concerned with monitoring the overall health of the cluster and the infrastructure it runs on Concerned with monitoring the Kafka entities associated with their apps Concerned with audit, compliance, access control & chain of custody requirements
  4. 4. 4 © Hortonworks Inc. 2011–2018. All rights reserved HORTONWORKS DATA PLATFORM (HDP®) DATA-AT-REST PRODUCERS CONSUMERS TOPICS BROKERS SMM REST Server HORTONWORKS DATAFLOW (HDF) DATA-IN-MOTION Optimize your Kafka Clusters Troubleshoot Kafka Streams Trace end-to- end Kafka flows Streams Messaging Manager (SMM)
  5. 5. 5 © Hortonworks Inc. 2011–2018. All rights reserved Cure is Here: Hortonworks Streams Messaging Manager (SMM) What is SMM?  Kafka Management and Monitoring tool  Cure the “Kafka Blindness”  Single Monitoring Dashboard for all your Kafka Clusters across 4 entities – Broker – Producer – Topic – Consumer  Supports multiple HDP and/or HDF Kafka Clusters  REST as a First Class Citizen  Delivered as a DataPlane Service
  6. 6. 6 © Hortonworks Inc. 2011–2018. All rights reserved Simplify troubleshooting Kafka environments • Visibility and control to the Kafka Operations teams • Intelligent filtering and sorting • Grafana integration • Ambari integration
  7. 7. 7 © Hortonworks Inc. 2011–2018. All rights reserved Visualize end-to-end Kafka stream flows • Interactive visualization • All streams/flows across all Kafka clusters • Analyze the stream dynamics between producers and consumers • Intelligent filtering • Throughput analysis
  8. 8. 8 © Hortonworks Inc. 2011–2018. All rights reserved Track data lineage and governance from edge-to-enterprise • Tight integration with Apache Atlas • Metadata about Kafka Topic in Atlas • Complete data lineage across multiple Kafka hops
  9. 9. 9 © Hortonworks Inc. 2011–2018. All rights reserved Extend monitoring / management capabilities with REST • Developer / DevOps friendly • Search for specific topics • Drill down for messages • SMM REST Server • All capabilities exposed as REST services
  10. 10. 10 © Hortonworks Inc. 2011–2018. All rights reserved Adopt hybrid cloud deployments for Kafka • SMM as a DataPlane Platform app • Manage multiple Kafka clusters from a single SMM instance • Hybrid cloud model
  11. 11. 11 © Hortonworks Inc. 2011–2018. All rights reserved Demo
  12. 12. 12 © Hortonworks Inc. 2011–2018. All rights reserved Recap: What Did We Just Show? Tracking the flow of data across multiple Kafka Hops with SMM & Atlas Integration-Powerful Data Flow Apps Powered by NiFi US West Fleet Truck Sensors C++ Agent US Central Fleet Truck Sensors C++ Agent US East Fleet Truck Sensors C++ Agent Analytics App 3 Analytics App 2 Analytics App 1
  13. 13. 13 © Hortonworks Inc. 2011–2018. All rights reserved www.hortonworks.com/smm

×