Hortonworks Data In Motion Webinar Series Pt. 2

Make Your Big Data Ecosystem
Work Better for You
@MarkLochbihler
Partner Engineering
October 12th, 2016
© Hortonworks Inc. 2011 – 2015. All Rights Reserved

2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda • Hortonworks Connected Data Platforms
• HDF 2.0 Platform
• Data Ingestion into Hadoop made EASY
• HDF 2.0 Platform Use Cases
• HDF 2.0 Product Integration Certification
• HDF Partner Ecosystem Solutions
• More Information

Agenda Hortonworks Connected Data Platforms
HDF 2.0 Platform
Data Ingestion into Hadoop made EASY
HDF 2.0 Platform Use Cases
HDF 2.0 Product Integration Certification
HDF Partner Ecosystem Solutions
More Information

Actionable Intelligence from
Connected Data Platforms
 Capturing perishable
insights from data in motion
 Ensuring rich, historical insights
on data at rest
 Necessary for modern
data applications
DATA AT REST
DATA IN
MOTION
ACTIONABLE
INTELLIGENCE
Modern Data Applications
Hortonworks
DataFlow
Hortonworks
Data Platform

Modern Data Apps
Custom or Off the Shelf
Real-Time Cyber Security
protects systems with superior threat
detection
Smart Manufacturing
dramatically improves yields by managing
more variables in greater detail
Connected, Autonomous Cars
drive themselves and improve road safety
Future Farming
optimizing soil, seeds and equipment to
measured conditions on each square foot
Automatic Recommendation Engines
match products to preferences in milliseconds
DATA AT
REST
DATA IN
MOTION
ACTIONABLE
INTELLIGENCE
Modern Data Applications
Hortonworks
DataFlow
Hortonworks
Data Platform

HDF 2.0 Platform
More Info

 Constrained
 High-latency
 Localized context
 Hybrid – cloud / on-premises
 Low-latency
 Global context
Core
Infrastructure
Hortonworks DataFlow Manages Data in Motion
Regional
InfrastructureSources

Easy, Real-Time, Coding Free Data Movement
Dynamic data pipeline as not all data is equal
AWS
Azure
Google Cloud
Hadoop

Expand To The Very Edge With MiNiFi
AWS
Azure
Google Cloud
Hadoop
Capture new sources of data

Edge Intelligence with Apache MiNiFi
 Guaranteed delivery
 Data buffering
‒ Backpressure
‒ Pressure release
 Prioritized queuing
 Flow specific QoS
‒ Latency vs. throughput
‒ Loss tolerance
 Data provenance
 Recovery / recording a rolling log
of fine-grained history
 Designed for extension
Different from Apache NiFi
 Design and Deploy
 Warm re-deploys
Key Features

Immediate Insights At Massive Scale
AWS
Azure
Google Cloud
Hadoop
Adapt to differing rates of data creation & delivery
(Kafka) with real-time streaming analytics (Storm)

Hortonworks DataFlow Management and Stream Processing
Core
InfrastructureSources
 Constrained
 High-latency
 Localized context
 Hybrid – cloud / on-premises
 Low-latency
 Global context
Regional
Infrastructure

Flow management
D A T A I N M O T I O N D A T A A T R E S T
IoT Data Sources AWS
Azure
Google Cloud
Hadoop
NiFi
Kafka
Storm
Others…
NiFi
NiFi NiFi
MiNiFi
MiNiFi
MiNiFi
MiNiFi
MiNiFi
MiNiFi
MiNiFi
NiFi
HDF 2.0: Data-in-Motion Platform
Enterprise Services
Ambari Ranger Other services
Flow management + Stream Processing

HDF 2.0 Platform
More Information

Problems Today: Timely Access to Data and Decisions
HDF helps us to streamline the flow of data and build
models and visualisations quickly, so that my team can
work iteratively with business colleagues on building
solutions that work for the business. -Royal Mail
http://diginomica.com/2016/04/22/royal-mail-starts-to-deliver-on-hortonworks-data-in-motion-promise/

Store Data
Process and Analyze
Data
Acquire Data
Simplistic View of DataFlows: Easy, Definitive
Dataflow

Realistic View of Dataflows: Complex, Convoluted
Store Data
Process and Analyze
Data
Acquire Data
Store DataStore Data
Store Data
Store Data
Acquire Data
Acquire Data
Acquire Data
Dataflow

HDP
HORTONWORKS
DATA PLATFORM
Powered by Apache Hadoop
HDF Makes Big Data Ingest Fast, Easy
HDP
HORTONWORKS
DATA PLATFORM
HDP
HORTONWORKS
DATA PLATFORM
Powered by Apache Hadoop
Complicated, messy, and takes weeks to
months to move the right data into Hadoop
Streamlined, Efficient, Easy

HDF 2.0 Platform
More Information

Use Cases for Data in Motion
Use Cases for Data-in-Motion Using DataFlow Mgmt
• Data Ingestion
• Edge Intelligence
• First Mile Problem
• Physical Data Movement
• Simple event processing such as Route, Filter, Enrich,
Transform, etc.
When Only DataFlow
Management is
Required
Use Cases for Data-in-Motion Using DataFlow Mgmt and
Steam Processing
• Flow Management to deliver data for Stream Processing
• PLUS: Complex pattern matching on unbounded streams of
data.
When Both DataFlow
Management and
Stream Processing

Data Ingestion: Optimize Log Analytics with Content Based Routing

Example: Company X provides alerting services when users’ resting heart rate higher
than a threshold
Real-Time Insights
Acquire
Data
Company X Cloud
Instance 1
Acquire
Data
Company X Cloud
Instance 2
Acquire
Data
Company X Cloud
Instance 3
Acquire Data
Across Cloud
Instances
Parse, Filter,
Validate, Enrich
and Route
Core Data Center
Analytics/Pattern
Match
Data
Store
Alerts
Dashboards/Visualization
Flow Management Stream ProcessingLegend:

Data in Motion Needs Dataflow Management and Stream Processing
 Acquire data from various Wearable Device’s Cloud Instances
 Move Data from Customer Cloud Instances to on-premise instance
 Perform Intelligent Routing & Filtering of data. The routing and filtering rules will be often
changed at run-time.
 Deliver the data data to various downstream systems. New downstream apps should will always
appear and the data should be fed to it when it comes online.
 Parse the device data to standardized format that downstream sysem can understand
 Enrich the data with contextual information including patient/customer info (age, sex, etc..)
 Recognize the Pattern when the resting heart rate exceeds a certain threshold (the insight),
and then create an alert/notification.
 Run a Outlier detection model on streaming heart rate that comes in. If the score is above
certain threshold, alert on the heart rate.
Flow
Management
(NiFi, MiNiFi
and
Partner
Integration)
Stream
Processing
(Storm, Kafka
and
Partner
Integration)

Data in
Motion
(Cloud)
Data in
Motion
(on-premises)
Data at
Rest
(on-premises)
Edge
Data
Data in
Motion
Edge
Analytics
Data at
Rest
(Cloud)
Edge
Data
Data at
Rest
(on-premises)
Closed Loop
Analytics
Machine
Learning
Deep
Historical
Analysis
The Future of Data
Architectural Transformation Enabled By Connected Data Platforms
On PremCloud

HDF 2.0 Platform
More Information

http://hortonworks.com/partners/product-integration-certification/
• Announced August 10th, 2016
• As the adoption of HDF expands,
enterprises are looking for proven
integrations that mitigate deployment risk,
pre-tested and certified. The HDF Certified
badge is earned by partners with certified
integrations with HDF
• Email partnerdesk@hortonworks.com to
get started
Product Integration Certification for HDP 2.0

Product Certification
3 SIMPLE STEPS
Step 1 : Join Partnerworks
Step 2 : Complete HDF 2.0 Certification Kit
 Certification Report
 Reference Architecture
 Solutions Overview
Step 3: Joint Review w Partner Engineering

HDF 2.0 Platform
More Information

Hortonworks DataFlow: Connecting Data Between Ecosystems
Hash
Extract
Merge
Duplicate
Scan
GeoEnrich
Replace
ConvertSplit
Translate
Route Content
Route Context
Route Text
Control Rate
Distribute Load
Generate Table Fetch
Jolt Transform JSON
Prioritized Delivery
Encrypt
Tail
Evaluate
Execute
HL7
FTP
UDP
XML
SFTP
HTTP
Syslog
Email
HTML
Image
AMQP
MQTT
All Apache project logos are trademarks of the ASF and the respective projects All other trademarks are the property of their respective owners.
Fetch

Connected Vehicle Case – Big Data Primary Data Flow
Sensitive
structured and
unstructured data
Hadoop Edge
Nodes
HPE SecureData
Hadoop Tools
Hadoop Cluster Teradata EDW
Sensitive
structured
sources
Cognos
Analytics &
Data Science
HPE SecureData
Key Servers &
WS API’s
~2 Billion real time
transactions/day
Other real-time data
feeds – customer
data from
dealerships,
manufacturers
Sqoop
Hive
UDFs
Map Reduce
“Landing
zone”
“Integration
Controls”
Real time
ingest
Existing data sets
and 3rd party data,
e.g.. accident data
UDFs
IBM DataStage
Driver
Blood
Pressure
Sensor
Data
Exadata
TDE
Server or
laptop log
files
Public data
sources such
as NHTSA
Storm
Kafka

Analytics and Hortonworks
STORAGE
STORAGE
GROUP 2GROUP 1
GROUP 4GROUP 3
D A T A A T R E S T
INTERNET
OF
ANYTHING
C L O U D
O N P R E M I S E
STORAGE
STORAGE
GROUP 2GROUP 1
GROUP 4GROUP 3
D A T A A T R E S T
ESP
ESP
ESP
HDP
HDP
USAGE CASE EXAMPLES
Cyber Security; Fraud; Predictive Maintenance; Customer
Experience; Stream Data Management
Deep
Historical
Analysis
Machine
Learning
Edge Analytics, MLearning,
Historical Analytics

CDC
MSG
n 2 1
MSG MSG
Data Streaming
Transaction
logs
In memory optimized metadata
management and data transport
Bulk
Load
MSG
n 2 1
MSG MSG
Data Streaming
Message
broker
Message
broker
Data Streaming into Kafka  HDF  HDP

Filtered data flow
from edge to
central site
Perform basic
operations like
ingest, alert, filter,
transform, etc.
SITE 1
• Data is read from
multiple sensors
• The ‘Nifi’ installation
on the edge node
ingests the data
from the sensors
SITE 2
CENTRAL NODES
SITE 3

D A T A I N M O T I O N D A T A A T R E S T
Data Sources Polling & Protocol Translation Real Time Database Long Term Storage
DCS
PLC
MiNiFi
Meters
Vehicles
Analyzers
RTU
Data Access
Polling
Engine
Protocol
Proxy Service
Time Series
Storage
Archive
Files
Varied
Support
Governance
&Integration
Security
Operations
Data Access
Data
Management

Edge
Gateway
Server
MiNiFi
Mobile
Client
Libraries
IoT Devices
Client
Libraries
Server Cluster
NiFi
Devices
MiNiFi
Regional Center
NiFi NiFi
Core Data Center
Server Cluster
NiFi NiFi NiFi
AWS
Azure
Google Cloud
DB
Data WHIoT Devices
Client
Libraries
eCompute eStorageeFabric® Data Plane
Software Defined Fabric
eNetwork
eFabric®
Control Plane
Zone based micro-segmentation for data security
AppHub – Application gallery for building, deploying and managing data pipelines
Seamless connectivity to public cloud services
www.midfinsystems.com

HDF 2.0 Platform
More Information

More Information, Resources
Hortonworks Community Connection:
Data Ingestion and Streaming
https://community.hortonworks.com
 Partnerworks: http://hortonworks.com/partners/
 HDF Certification:
http://hortonworks.com/partners/product-integration-certification/
 Webinars: http://hortonworks.com/events-webcasts/
 Sandbox: http://hortonworks.com/events-webcasts/
 HDF: http://hortonworks.com/hdf/
 HDP: http://hortonworks.com/hdp/

Thank You

Hortonworks Data In Motion Webinar Series Pt. 2

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Hortonworks Data In Motion Webinar Series Pt. 2

Similar to Hortonworks Data In Motion Webinar Series Pt. 2 (20)

More from Hortonworks

More from Hortonworks (20)

Recently uploaded

Recently uploaded (20)

Hortonworks Data In Motion Webinar Series Pt. 2

Editor's Notes