SlideShare a Scribd company logo
1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Streams Messaging Manager
Paris Future Of Data Meetup
https://bit.ly/2g36oWc
2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Next Generation Data Problems
Multiple clusters and data sources
“Enterprises demand self-service data preparation solutions to empower business users like data scientists
to directly engage with big data platforms, reducing the involvement of technology.”
Forrester Report: Big Data Fabric
Business Analysts & Data Scientists
Ability to quickly understand data models and
reliably access and search across data stores
Data Engineers
Ability to consistently create, move and backup
data across multiple locations
Data Steward (Architect/Security)
Ability to consistently apply security and
governance policy across all types of data
3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Core Capabilities
Provide basic functions to be used by extensible
data services for for multiple types and tiers
⬢ Data Source Integration
Ability to register and/or create data sources to
allow consolidated access
⬢ Data Services Catalog
Full configuration and management utilities for the
enablement of new services
⬢ Security Controls
Full definition of security access controls including
persona definitions
Hortonworks DataPlane Service
4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
DataPlane: ‘Data Fabric’ for the Hybrid Cloud
Understand, Secure and Govern across Data Lakes
In the Cloud
On Premises
(Structured)
(Structured)
Cluster 1 Dublin
(Unstructured) (Structured) (Unstructured)
Cluster 2 Las Vegas
(Unstructured)(Structured) (Structured)
Cluster 3 Bangkok
Apache Ranger
(Structured)
Apache Ranger Apache Ranger
(Unstructured)(Structured) (Unstructured) (Structured)
Apache Ranger Apache Ranger
DPS
Cluster
Visibility
Data Services
Delivery
Role-based
Administration
5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
DPS Console: One place to visualize Virtual Data Lakes
DPS PlatformData Plane Service Console
6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Hortonworks DataPlane Service
7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Streams Messaging
Manager (SMM)
8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Kafka’s Omnipresence Has Led to the Onset of “Kafka Blindness”
à What is “Kafka Blindness”?
– Customers who use Kafka today struggle with monitoring / “seeing”/troubleshooting what is
happening in their clusters
à Who is Affected?
– Platform Operation Teams
– Developers / DevOps Teams
– Security / Governance Teams
à What are the Symptoms?
– Difficulty seeing who is producing and consuming data
– Difficulty understanding the flow of data from producers -> topics à consumers
– Difficulty troubleshooting/monitoring.
9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Cure is Here: Hortonworks Streams Messaging Manager (SMM)
à New Open Source project led by
Hortonworks to Cure the “Kafka Blindness”
à Single Monitoring Dashboard for all your
Kafka Clusters across 4 entities
– Broker
– Topic
– Producer
– Consumer
à Designed for the Enterprise
– Support for Secure/Kerborized Kafka cluster
– Rich Access Control Policies (ACLS)
– Supports multiple HDP and/or HDF Kafka Clusters
à REST as a First Class Citizen
à Delivered as a DataPlane Service
10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
SMM features (1/2)
à Intelligent filtering on all entities
à Important KPIs such as lag & throughput
Simplify troubleshooting Kafka
environments
Visualize end-to-end Kafka stream flows
à End to end vision from producers to
consumers passing by topics and brokers
11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
SMM features (2/2)
à End-to-end integration with enterprise
services on Hortonworks platforms :
Ambari, Grafana, Ranger & Atlas
Data lineage and governance from edge-to-
enterprise
monitoring / management capabilities
with REST
à Comprehensive REST service for open
integration with APM and ticketing tools
12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Data Plane
Core Services
Ambari
Kafka ZK
SMM
Agent
HDP Cluster
…
Data Plane DB
Knox
LDAP / AD
Ambari
Kafka ZK
SMM
Agent
HDF Cluster
Knox
Ambari
Kafka ZK
SMM
Agent
HDP Cluster
Knox
DP
UI
SMM
UI
Single User Store
High Level Architecture of DP/SMM
AMS SRAMS SRAMS SR
13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
DP Core (Micro) Services and DP Apps are Docker Containers
SMM Agent
Kafka Broker
Zookeeper
AMS
SMM Rest Server
14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
DataPlane Components - Access services
à Apache Knox
– Provides authentication for DPS
– Has to be configured with the same LDAP / AD user identity store as the managed HDP clusters
à Consul
– Service registry for DPS components
– Services register with Consul so they can be discovered by other services
– Allows services to be distributed in future to multiple machines
à Zuul
– Zuul enables dynamic routing, monitoring, resiliency and security
– Used to intercept & authenticate all API calls (redirecting to Knox if required)
– Proxy to redirect calls to physical location of components (using Consul to discover their location)
15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
SMM REST Admin Server
What is SMM REST Server?
à SMM UI powered by first class REST Services
via the SMM REST Admin Server
à Monitoring Rest Endpoints can be used to
integrate with APM/Alerting/Ticketing
solutions
à Powered by Apache Knox
à Installed via an Ambari Management Pack
on a target HDP/HDF cluster where Kafka
Service is running
à Supports both HDP & HDF Platforms
à Open Source / AGPL licensed projected led
by Hortonworks
16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
SMM Demo
17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Questions

More Related Content

What's hot

Cloud Foundry - How Service broker integrates with AppDirect to provide catal...
Cloud Foundry - How Service broker integrates with AppDirect to provide catal...Cloud Foundry - How Service broker integrates with AppDirect to provide catal...
Cloud Foundry - How Service broker integrates with AppDirect to provide catal...
Nima Badiey
 
APITalkMeetupSharable
APITalkMeetupSharableAPITalkMeetupSharable
APITalkMeetupSharable
Obaidur (OB) Rashid
 
The use case of a scalable architecture
The use case of a scalable architectureThe use case of a scalable architecture
The use case of a scalable architecture
Toru Wonyoung Choi
 
Mulethenewtechnology 12549172699166-phpapp03-160421133841
Mulethenewtechnology 12549172699166-phpapp03-160421133841Mulethenewtechnology 12549172699166-phpapp03-160421133841
Mulethenewtechnology 12549172699166-phpapp03-160421133841
ppts123456
 
Mule developing a cloud hub application
Mule developing a cloud hub applicationMule developing a cloud hub application
Mule developing a cloud hub application
D.Rajesh Kumar
 
Anypoint data gateway
Anypoint data gatewayAnypoint data gateway
Anypoint data gateway
Mohammed246
 
iPlanet presentation
iPlanet presentationiPlanet presentation
iPlanet presentation
rogerkellerman
 
Comprehensive Information on Platform as a Service
Comprehensive Information on Platform as a ServiceComprehensive Information on Platform as a Service
Comprehensive Information on Platform as a Service
HTS Hosting
 
Role of integration in Digital Transformation
Role of integration in Digital TransformationRole of integration in Digital Transformation
Role of integration in Digital Transformation
WSO2
 
I Planet Overview
I Planet OverviewI Planet Overview
I Planet Overview
rogerkellerman
 
Mule architecture
Mule architectureMule architecture
Mule architecture
Khasim Saheb
 
What's New with Anypoint Platform? Unified Platform Management
What's New with Anypoint Platform? Unified Platform ManagementWhat's New with Anypoint Platform? Unified Platform Management
What's New with Anypoint Platform? Unified Platform Management
MuleSoft
 
Introduction to mule soft
Introduction to mule softIntroduction to mule soft
Introduction to mule soft
rajnishjha29
 
M2M Integration Platform as a Service iPaaS
M2M Integration Platform as a Service iPaaSM2M Integration Platform as a Service iPaaS
M2M Integration Platform as a Service iPaaS
Eurotech
 
WSO2Con 2011: Introduction to Stratos
WSO2Con 2011: Introduction to StratosWSO2Con 2011: Introduction to Stratos
WSO2Con 2011: Introduction to Stratos
Afkham Azeez
 
EdgeIQ platform datasheet
EdgeIQ platform datasheetEdgeIQ platform datasheet
EdgeIQ platform datasheet
EdgeIQ
 
Cloud Side: REST APIs - Best practices
Cloud Side: REST APIs - Best practicesCloud Side: REST APIs - Best practices
Cloud Side: REST APIs - Best practices
Nicolas FOATA
 
Anypoint platform security components
Anypoint platform security componentsAnypoint platform security components
Anypoint platform security components
D.Rajesh Kumar
 
Service mesh in action with onap
Service mesh in action with onapService mesh in action with onap
Service mesh in action with onap
Huabing Zhao
 
The Overview of Microservices Architecture
The Overview of Microservices ArchitectureThe Overview of Microservices Architecture
The Overview of Microservices Architecture
Paria Heidari
 

What's hot (20)

Cloud Foundry - How Service broker integrates with AppDirect to provide catal...
Cloud Foundry - How Service broker integrates with AppDirect to provide catal...Cloud Foundry - How Service broker integrates with AppDirect to provide catal...
Cloud Foundry - How Service broker integrates with AppDirect to provide catal...
 
APITalkMeetupSharable
APITalkMeetupSharableAPITalkMeetupSharable
APITalkMeetupSharable
 
The use case of a scalable architecture
The use case of a scalable architectureThe use case of a scalable architecture
The use case of a scalable architecture
 
Mulethenewtechnology 12549172699166-phpapp03-160421133841
Mulethenewtechnology 12549172699166-phpapp03-160421133841Mulethenewtechnology 12549172699166-phpapp03-160421133841
Mulethenewtechnology 12549172699166-phpapp03-160421133841
 
Mule developing a cloud hub application
Mule developing a cloud hub applicationMule developing a cloud hub application
Mule developing a cloud hub application
 
Anypoint data gateway
Anypoint data gatewayAnypoint data gateway
Anypoint data gateway
 
iPlanet presentation
iPlanet presentationiPlanet presentation
iPlanet presentation
 
Comprehensive Information on Platform as a Service
Comprehensive Information on Platform as a ServiceComprehensive Information on Platform as a Service
Comprehensive Information on Platform as a Service
 
Role of integration in Digital Transformation
Role of integration in Digital TransformationRole of integration in Digital Transformation
Role of integration in Digital Transformation
 
I Planet Overview
I Planet OverviewI Planet Overview
I Planet Overview
 
Mule architecture
Mule architectureMule architecture
Mule architecture
 
What's New with Anypoint Platform? Unified Platform Management
What's New with Anypoint Platform? Unified Platform ManagementWhat's New with Anypoint Platform? Unified Platform Management
What's New with Anypoint Platform? Unified Platform Management
 
Introduction to mule soft
Introduction to mule softIntroduction to mule soft
Introduction to mule soft
 
M2M Integration Platform as a Service iPaaS
M2M Integration Platform as a Service iPaaSM2M Integration Platform as a Service iPaaS
M2M Integration Platform as a Service iPaaS
 
WSO2Con 2011: Introduction to Stratos
WSO2Con 2011: Introduction to StratosWSO2Con 2011: Introduction to Stratos
WSO2Con 2011: Introduction to Stratos
 
EdgeIQ platform datasheet
EdgeIQ platform datasheetEdgeIQ platform datasheet
EdgeIQ platform datasheet
 
Cloud Side: REST APIs - Best practices
Cloud Side: REST APIs - Best practicesCloud Side: REST APIs - Best practices
Cloud Side: REST APIs - Best practices
 
Anypoint platform security components
Anypoint platform security componentsAnypoint platform security components
Anypoint platform security components
 
Service mesh in action with onap
Service mesh in action with onapService mesh in action with onap
Service mesh in action with onap
 
The Overview of Microservices Architecture
The Overview of Microservices ArchitectureThe Overview of Microservices Architecture
The Overview of Microservices Architecture
 

Similar to Paris FOD meetup - Streams Messaging Manager

Curing Kafka Blindness with Hortonworks Streams Messaging Manager
Curing Kafka Blindness with Hortonworks Streams Messaging ManagerCuring Kafka Blindness with Hortonworks Streams Messaging Manager
Curing Kafka Blindness with Hortonworks Streams Messaging Manager
Hortonworks
 
Unlocking insights in streaming data
Unlocking insights in streaming dataUnlocking insights in streaming data
Unlocking insights in streaming data
Carolyn Duby
 
Its Finally Here! Building Complex Streaming Analytics Apps in under 10 min w...
Its Finally Here! Building Complex Streaming Analytics Apps in under 10 min w...Its Finally Here! Building Complex Streaming Analytics Apps in under 10 min w...
Its Finally Here! Building Complex Streaming Analytics Apps in under 10 min w...
DataWorks Summit
 
Streaming analytics manager
Streaming analytics managerStreaming analytics manager
Streaming analytics manager
Sriharsha Chintalapani
 
SAM - Streaming Analytics Made Easy
SAM - Streaming Analytics Made EasySAM - Streaming Analytics Made Easy
SAM - Streaming Analytics Made Easy
DataWorks Summit
 
Druid: Sub-Second OLAP queries over Petabytes of Streaming Data
Druid: Sub-Second OLAP queries over Petabytes of Streaming DataDruid: Sub-Second OLAP queries over Petabytes of Streaming Data
Druid: Sub-Second OLAP queries over Petabytes of Streaming Data
DataWorks Summit
 
Managing enterprise users in Hadoop ecosystem
Managing enterprise users in Hadoop ecosystemManaging enterprise users in Hadoop ecosystem
Managing enterprise users in Hadoop ecosystem
DataWorks Summit
 
Data at Scales and the Values of Starting Small with Apache NiFi & MiNiFi
Data at Scales and the Values of Starting Small with Apache NiFi & MiNiFiData at Scales and the Values of Starting Small with Apache NiFi & MiNiFi
Data at Scales and the Values of Starting Small with Apache NiFi & MiNiFi
Aldrin Piri
 
Schema Registry & Stream Analytics Manager
Schema Registry  & Stream Analytics ManagerSchema Registry  & Stream Analytics Manager
Schema Registry & Stream Analytics Manager
Sriharsha Chintalapani
 
Apache Atlas: Why Big Data Management Requires Hierarchical Taxonomies
Apache Atlas: Why Big Data Management Requires Hierarchical Taxonomies Apache Atlas: Why Big Data Management Requires Hierarchical Taxonomies
Apache Atlas: Why Big Data Management Requires Hierarchical Taxonomies
DataWorks Summit/Hadoop Summit
 
Streamline - Stream Analytics for Everyone
Streamline - Stream Analytics for EveryoneStreamline - Stream Analytics for Everyone
Streamline - Stream Analytics for Everyone
DataWorks Summit/Hadoop Summit
 
Internet of things Crash Course Workshop
Internet of things Crash Course WorkshopInternet of things Crash Course Workshop
Internet of things Crash Course Workshop
DataWorks Summit
 
Internet of Things Crash Course Workshop at Hadoop Summit
Internet of Things Crash Course Workshop at Hadoop SummitInternet of Things Crash Course Workshop at Hadoop Summit
Internet of Things Crash Course Workshop at Hadoop Summit
DataWorks Summit
 
Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level
Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next LevelHortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level
Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level
Hortonworks
 
Real-Time Processing in Hadoop for IoT Use Cases - Phoenix HUG
Real-Time Processing in Hadoop for IoT Use Cases - Phoenix HUGReal-Time Processing in Hadoop for IoT Use Cases - Phoenix HUG
Real-Time Processing in Hadoop for IoT Use Cases - Phoenix HUG
skumpf
 
Future of Data New Jersey - HDF 3.0 Deep Dive
Future of Data New Jersey - HDF 3.0 Deep DiveFuture of Data New Jersey - HDF 3.0 Deep Dive
Future of Data New Jersey - HDF 3.0 Deep Dive
Aldrin Piri
 
Harnessing Data-in-Motion with HDF 2.0, introduction to Apache NIFI/MINIFI
Harnessing Data-in-Motion with HDF 2.0, introduction to Apache NIFI/MINIFIHarnessing Data-in-Motion with HDF 2.0, introduction to Apache NIFI/MINIFI
Harnessing Data-in-Motion with HDF 2.0, introduction to Apache NIFI/MINIFI
Haimo Liu
 
Hortonworks Data in Motion Webinar Series - Part 1
Hortonworks Data in Motion Webinar Series - Part 1Hortonworks Data in Motion Webinar Series - Part 1
Hortonworks Data in Motion Webinar Series - Part 1
Hortonworks
 
Fortifying Multi-Cluster Hybrid Cloud Data Lakes using Apache Knox
Fortifying Multi-Cluster Hybrid Cloud Data Lakes using Apache KnoxFortifying Multi-Cluster Hybrid Cloud Data Lakes using Apache Knox
Fortifying Multi-Cluster Hybrid Cloud Data Lakes using Apache Knox
DataWorks Summit
 
Cloudy with a chance of Hadoop - real world considerations
Cloudy with a chance of Hadoop - real world considerationsCloudy with a chance of Hadoop - real world considerations
Cloudy with a chance of Hadoop - real world considerations
DataWorks Summit
 

Similar to Paris FOD meetup - Streams Messaging Manager (20)

Curing Kafka Blindness with Hortonworks Streams Messaging Manager
Curing Kafka Blindness with Hortonworks Streams Messaging ManagerCuring Kafka Blindness with Hortonworks Streams Messaging Manager
Curing Kafka Blindness with Hortonworks Streams Messaging Manager
 
Unlocking insights in streaming data
Unlocking insights in streaming dataUnlocking insights in streaming data
Unlocking insights in streaming data
 
Its Finally Here! Building Complex Streaming Analytics Apps in under 10 min w...
Its Finally Here! Building Complex Streaming Analytics Apps in under 10 min w...Its Finally Here! Building Complex Streaming Analytics Apps in under 10 min w...
Its Finally Here! Building Complex Streaming Analytics Apps in under 10 min w...
 
Streaming analytics manager
Streaming analytics managerStreaming analytics manager
Streaming analytics manager
 
SAM - Streaming Analytics Made Easy
SAM - Streaming Analytics Made EasySAM - Streaming Analytics Made Easy
SAM - Streaming Analytics Made Easy
 
Druid: Sub-Second OLAP queries over Petabytes of Streaming Data
Druid: Sub-Second OLAP queries over Petabytes of Streaming DataDruid: Sub-Second OLAP queries over Petabytes of Streaming Data
Druid: Sub-Second OLAP queries over Petabytes of Streaming Data
 
Managing enterprise users in Hadoop ecosystem
Managing enterprise users in Hadoop ecosystemManaging enterprise users in Hadoop ecosystem
Managing enterprise users in Hadoop ecosystem
 
Data at Scales and the Values of Starting Small with Apache NiFi & MiNiFi
Data at Scales and the Values of Starting Small with Apache NiFi & MiNiFiData at Scales and the Values of Starting Small with Apache NiFi & MiNiFi
Data at Scales and the Values of Starting Small with Apache NiFi & MiNiFi
 
Schema Registry & Stream Analytics Manager
Schema Registry  & Stream Analytics ManagerSchema Registry  & Stream Analytics Manager
Schema Registry & Stream Analytics Manager
 
Apache Atlas: Why Big Data Management Requires Hierarchical Taxonomies
Apache Atlas: Why Big Data Management Requires Hierarchical Taxonomies Apache Atlas: Why Big Data Management Requires Hierarchical Taxonomies
Apache Atlas: Why Big Data Management Requires Hierarchical Taxonomies
 
Streamline - Stream Analytics for Everyone
Streamline - Stream Analytics for EveryoneStreamline - Stream Analytics for Everyone
Streamline - Stream Analytics for Everyone
 
Internet of things Crash Course Workshop
Internet of things Crash Course WorkshopInternet of things Crash Course Workshop
Internet of things Crash Course Workshop
 
Internet of Things Crash Course Workshop at Hadoop Summit
Internet of Things Crash Course Workshop at Hadoop SummitInternet of Things Crash Course Workshop at Hadoop Summit
Internet of Things Crash Course Workshop at Hadoop Summit
 
Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level
Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next LevelHortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level
Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level
 
Real-Time Processing in Hadoop for IoT Use Cases - Phoenix HUG
Real-Time Processing in Hadoop for IoT Use Cases - Phoenix HUGReal-Time Processing in Hadoop for IoT Use Cases - Phoenix HUG
Real-Time Processing in Hadoop for IoT Use Cases - Phoenix HUG
 
Future of Data New Jersey - HDF 3.0 Deep Dive
Future of Data New Jersey - HDF 3.0 Deep DiveFuture of Data New Jersey - HDF 3.0 Deep Dive
Future of Data New Jersey - HDF 3.0 Deep Dive
 
Harnessing Data-in-Motion with HDF 2.0, introduction to Apache NIFI/MINIFI
Harnessing Data-in-Motion with HDF 2.0, introduction to Apache NIFI/MINIFIHarnessing Data-in-Motion with HDF 2.0, introduction to Apache NIFI/MINIFI
Harnessing Data-in-Motion with HDF 2.0, introduction to Apache NIFI/MINIFI
 
Hortonworks Data in Motion Webinar Series - Part 1
Hortonworks Data in Motion Webinar Series - Part 1Hortonworks Data in Motion Webinar Series - Part 1
Hortonworks Data in Motion Webinar Series - Part 1
 
Fortifying Multi-Cluster Hybrid Cloud Data Lakes using Apache Knox
Fortifying Multi-Cluster Hybrid Cloud Data Lakes using Apache KnoxFortifying Multi-Cluster Hybrid Cloud Data Lakes using Apache Knox
Fortifying Multi-Cluster Hybrid Cloud Data Lakes using Apache Knox
 
Cloudy with a chance of Hadoop - real world considerations
Cloudy with a chance of Hadoop - real world considerationsCloudy with a chance of Hadoop - real world considerations
Cloudy with a chance of Hadoop - real world considerations
 

More from Abdelkrim Hadjidj

Disaster Recovery and High Availability with Kafka, SRM and MM2
Disaster Recovery and High Availability with Kafka, SRM and MM2Disaster Recovery and High Availability with Kafka, SRM and MM2
Disaster Recovery and High Availability with Kafka, SRM and MM2
Abdelkrim Hadjidj
 
Hive 3 a new horizon
Hive 3  a new horizonHive 3  a new horizon
Hive 3 a new horizon
Abdelkrim Hadjidj
 
Paris FOD meetup - koordinator
Paris FOD meetup - koordinatorParis FOD meetup - koordinator
Paris FOD meetup - koordinator
Abdelkrim Hadjidj
 
Paris FOD meetup - kafka security 101
Paris FOD meetup - kafka security 101Paris FOD meetup - kafka security 101
Paris FOD meetup - kafka security 101
Abdelkrim Hadjidj
 
FOD Paris Meetup - Global Data Management with DataPlane Services (DPS)
FOD Paris Meetup -  Global Data Management with DataPlane Services (DPS)FOD Paris Meetup -  Global Data Management with DataPlane Services (DPS)
FOD Paris Meetup - Global Data Management with DataPlane Services (DPS)
Abdelkrim Hadjidj
 
Paris FOD Meetup #5 Hortonworks Presentation
Paris FOD Meetup #5 Hortonworks PresentationParis FOD Meetup #5 Hortonworks Presentation
Paris FOD Meetup #5 Hortonworks Presentation
Abdelkrim Hadjidj
 
Paris FOD Meetup #5 Cognizant Presentation
Paris FOD Meetup #5 Cognizant PresentationParis FOD Meetup #5 Cognizant Presentation
Paris FOD Meetup #5 Cognizant Presentation
Abdelkrim Hadjidj
 
Apache NiFi: latest developments for flow management at scale
Apache NiFi: latest developments for flow management at scaleApache NiFi: latest developments for flow management at scale
Apache NiFi: latest developments for flow management at scale
Abdelkrim Hadjidj
 
Future of Data Meetup : Boontadata
Future of Data Meetup : BoontadataFuture of Data Meetup : Boontadata
Future of Data Meetup : Boontadata
Abdelkrim Hadjidj
 

More from Abdelkrim Hadjidj (9)

Disaster Recovery and High Availability with Kafka, SRM and MM2
Disaster Recovery and High Availability with Kafka, SRM and MM2Disaster Recovery and High Availability with Kafka, SRM and MM2
Disaster Recovery and High Availability with Kafka, SRM and MM2
 
Hive 3 a new horizon
Hive 3  a new horizonHive 3  a new horizon
Hive 3 a new horizon
 
Paris FOD meetup - koordinator
Paris FOD meetup - koordinatorParis FOD meetup - koordinator
Paris FOD meetup - koordinator
 
Paris FOD meetup - kafka security 101
Paris FOD meetup - kafka security 101Paris FOD meetup - kafka security 101
Paris FOD meetup - kafka security 101
 
FOD Paris Meetup - Global Data Management with DataPlane Services (DPS)
FOD Paris Meetup -  Global Data Management with DataPlane Services (DPS)FOD Paris Meetup -  Global Data Management with DataPlane Services (DPS)
FOD Paris Meetup - Global Data Management with DataPlane Services (DPS)
 
Paris FOD Meetup #5 Hortonworks Presentation
Paris FOD Meetup #5 Hortonworks PresentationParis FOD Meetup #5 Hortonworks Presentation
Paris FOD Meetup #5 Hortonworks Presentation
 
Paris FOD Meetup #5 Cognizant Presentation
Paris FOD Meetup #5 Cognizant PresentationParis FOD Meetup #5 Cognizant Presentation
Paris FOD Meetup #5 Cognizant Presentation
 
Apache NiFi: latest developments for flow management at scale
Apache NiFi: latest developments for flow management at scaleApache NiFi: latest developments for flow management at scale
Apache NiFi: latest developments for flow management at scale
 
Future of Data Meetup : Boontadata
Future of Data Meetup : BoontadataFuture of Data Meetup : Boontadata
Future of Data Meetup : Boontadata
 

Recently uploaded

Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Speck&Tech
 
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practicesNational Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
Quotidiano Piemontese
 
Data structures and Algorithms in Python.pdf
Data structures and Algorithms in Python.pdfData structures and Algorithms in Python.pdf
Data structures and Algorithms in Python.pdf
TIPNGVN2
 
Introduction to CHERI technology - Cybersecurity
Introduction to CHERI technology - CybersecurityIntroduction to CHERI technology - Cybersecurity
Introduction to CHERI technology - Cybersecurity
mikeeftimakis1
 
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing DaysClimate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
Kari Kakkonen
 
RESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for studentsRESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for students
KAMESHS29
 
Building RAG with self-deployed Milvus vector database and Snowpark Container...
Building RAG with self-deployed Milvus vector database and Snowpark Container...Building RAG with self-deployed Milvus vector database and Snowpark Container...
Building RAG with self-deployed Milvus vector database and Snowpark Container...
Zilliz
 
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfObservability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Paige Cruz
 
Microsoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdfMicrosoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdf
Uni Systems S.M.S.A.
 
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
James Anderson
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
Octavian Nadolu
 
20 Comprehensive Checklist of Designing and Developing a Website
20 Comprehensive Checklist of Designing and Developing a Website20 Comprehensive Checklist of Designing and Developing a Website
20 Comprehensive Checklist of Designing and Developing a Website
Pixlogix Infotech
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
DianaGray10
 
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...
Zilliz
 
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionGenerative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Aggregage
 
A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...
sonjaschweigert1
 
How to use Firebase Data Connect For Flutter
How to use Firebase Data Connect For FlutterHow to use Firebase Data Connect For Flutter
How to use Firebase Data Connect For Flutter
Daiki Mogmet Ito
 
UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
DianaGray10
 
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 202420240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
Matthew Sinclair
 
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
Neo4j
 

Recently uploaded (20)

Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
 
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practicesNational Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
 
Data structures and Algorithms in Python.pdf
Data structures and Algorithms in Python.pdfData structures and Algorithms in Python.pdf
Data structures and Algorithms in Python.pdf
 
Introduction to CHERI technology - Cybersecurity
Introduction to CHERI technology - CybersecurityIntroduction to CHERI technology - Cybersecurity
Introduction to CHERI technology - Cybersecurity
 
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing DaysClimate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
 
RESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for studentsRESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for students
 
Building RAG with self-deployed Milvus vector database and Snowpark Container...
Building RAG with self-deployed Milvus vector database and Snowpark Container...Building RAG with self-deployed Milvus vector database and Snowpark Container...
Building RAG with self-deployed Milvus vector database and Snowpark Container...
 
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfObservability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
 
Microsoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdfMicrosoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdf
 
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
 
20 Comprehensive Checklist of Designing and Developing a Website
20 Comprehensive Checklist of Designing and Developing a Website20 Comprehensive Checklist of Designing and Developing a Website
20 Comprehensive Checklist of Designing and Developing a Website
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
 
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...
 
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionGenerative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
 
A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...
 
How to use Firebase Data Connect For Flutter
How to use Firebase Data Connect For FlutterHow to use Firebase Data Connect For Flutter
How to use Firebase Data Connect For Flutter
 
UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
 
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 202420240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
 
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
 

Paris FOD meetup - Streams Messaging Manager

  • 1. 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Streams Messaging Manager Paris Future Of Data Meetup https://bit.ly/2g36oWc
  • 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Next Generation Data Problems Multiple clusters and data sources “Enterprises demand self-service data preparation solutions to empower business users like data scientists to directly engage with big data platforms, reducing the involvement of technology.” Forrester Report: Big Data Fabric Business Analysts & Data Scientists Ability to quickly understand data models and reliably access and search across data stores Data Engineers Ability to consistently create, move and backup data across multiple locations Data Steward (Architect/Security) Ability to consistently apply security and governance policy across all types of data
  • 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Core Capabilities Provide basic functions to be used by extensible data services for for multiple types and tiers ⬢ Data Source Integration Ability to register and/or create data sources to allow consolidated access ⬢ Data Services Catalog Full configuration and management utilities for the enablement of new services ⬢ Security Controls Full definition of security access controls including persona definitions Hortonworks DataPlane Service
  • 4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved DataPlane: ‘Data Fabric’ for the Hybrid Cloud Understand, Secure and Govern across Data Lakes In the Cloud On Premises (Structured) (Structured) Cluster 1 Dublin (Unstructured) (Structured) (Unstructured) Cluster 2 Las Vegas (Unstructured)(Structured) (Structured) Cluster 3 Bangkok Apache Ranger (Structured) Apache Ranger Apache Ranger (Unstructured)(Structured) (Unstructured) (Structured) Apache Ranger Apache Ranger DPS Cluster Visibility Data Services Delivery Role-based Administration
  • 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved DPS Console: One place to visualize Virtual Data Lakes DPS PlatformData Plane Service Console
  • 6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Hortonworks DataPlane Service
  • 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Streams Messaging Manager (SMM)
  • 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Kafka’s Omnipresence Has Led to the Onset of “Kafka Blindness” à What is “Kafka Blindness”? – Customers who use Kafka today struggle with monitoring / “seeing”/troubleshooting what is happening in their clusters à Who is Affected? – Platform Operation Teams – Developers / DevOps Teams – Security / Governance Teams à What are the Symptoms? – Difficulty seeing who is producing and consuming data – Difficulty understanding the flow of data from producers -> topics à consumers – Difficulty troubleshooting/monitoring.
  • 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Cure is Here: Hortonworks Streams Messaging Manager (SMM) à New Open Source project led by Hortonworks to Cure the “Kafka Blindness” à Single Monitoring Dashboard for all your Kafka Clusters across 4 entities – Broker – Topic – Producer – Consumer à Designed for the Enterprise – Support for Secure/Kerborized Kafka cluster – Rich Access Control Policies (ACLS) – Supports multiple HDP and/or HDF Kafka Clusters à REST as a First Class Citizen à Delivered as a DataPlane Service
  • 10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved SMM features (1/2) à Intelligent filtering on all entities à Important KPIs such as lag & throughput Simplify troubleshooting Kafka environments Visualize end-to-end Kafka stream flows à End to end vision from producers to consumers passing by topics and brokers
  • 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved SMM features (2/2) Ã End-to-end integration with enterprise services on Hortonworks platforms : Ambari, Grafana, Ranger & Atlas Data lineage and governance from edge-to- enterprise monitoring / management capabilities with REST Ã Comprehensive REST service for open integration with APM and ticketing tools
  • 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Data Plane Core Services Ambari Kafka ZK SMM Agent HDP Cluster … Data Plane DB Knox LDAP / AD Ambari Kafka ZK SMM Agent HDF Cluster Knox Ambari Kafka ZK SMM Agent HDP Cluster Knox DP UI SMM UI Single User Store High Level Architecture of DP/SMM AMS SRAMS SRAMS SR
  • 13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved DP Core (Micro) Services and DP Apps are Docker Containers SMM Agent Kafka Broker Zookeeper AMS SMM Rest Server
  • 14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved DataPlane Components - Access services à Apache Knox – Provides authentication for DPS – Has to be configured with the same LDAP / AD user identity store as the managed HDP clusters à Consul – Service registry for DPS components – Services register with Consul so they can be discovered by other services – Allows services to be distributed in future to multiple machines à Zuul – Zuul enables dynamic routing, monitoring, resiliency and security – Used to intercept & authenticate all API calls (redirecting to Knox if required) – Proxy to redirect calls to physical location of components (using Consul to discover their location)
  • 15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved SMM REST Admin Server What is SMM REST Server? à SMM UI powered by first class REST Services via the SMM REST Admin Server à Monitoring Rest Endpoints can be used to integrate with APM/Alerting/Ticketing solutions à Powered by Apache Knox à Installed via an Ambari Management Pack on a target HDP/HDF cluster where Kafka Service is running à Supports both HDP & HDF Platforms à Open Source / AGPL licensed projected led by Hortonworks
  • 16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved SMM Demo
  • 17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Questions