Data on the Move - DataCon DC

Data on the Move
Applications of Streaming Data
Joseph Witt
VP Engineering, Hortonworks
Member, Apache NiFi PMC
Member, Apache Software Foundation
Copyright © 2017 The Apache Software Foundation, Licensed under the Apache License, Version 2.0.
Apache, the Apache feather logo, Apache NiFi, Apache Storm, Apache Kafka, Apache Superset (Incubating),
Apache Apex, Apache Flink and the project logos are trademarks of The Apache Software Foundation.

2 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Agenda
Powering the Information Supply Chain

The Data Driven Enterprise - Lifecycle
Collect Process Analyze
Event Time Processing Time

Different organizations/business units across different geographic locations…

Conducting business in different legal and network domains…

Operating on very different infrastructure (power, space, cooling)…

Capable of different volume, velocity, bandwidth, and latency…

Interacting with different business partners and customers

Data driven organization or leader…
Observe: Wants all the data – now
Orient: To understand it all and how it relates
Decide: To decide what has happened or predict what will
Act: To outpace competitors and maximize value to customers
https://en.wikipedia.org/wiki/OODA_loop

Challenges
Wants all the data – now
• Bandwidth: Constrained, Error prone, Expensive
• Compliance requirements (legal, policy, geographic restrictions)
• Competing priorities between organizations
• Perishable vs delay tolerant data
• Storage is cheap but rate of data sensing is skyrocketing
• Controlling the edge systems (ex sampled/split rollout)
To understand it all and how it relates
To decide what has happened or predict what will
To outpace competitors and maximize value to customers
http://www.eugdpr.org/eugdpr.org.html
http://www.zdnet.com/article/report-iot-devices-to-dominate-connected-device-landscape-by-2021/

Challenges
Wants all the data – now
• Formats/schemas evolve
• Semantics of data evolve
• Processing bandwidth limits
• Signal/noise results in inefficient processing
• Compliance implications of joining data
http://www.eugdpr.org/eugdpr.org.html

Challenges
Wants all the data – now.
• How to distribute accumulated context
• How to track origin of most/least valuable sources
• Develop, distribute, retrain effective models
• Where to execute/manage the decision processes
• Compliance implications of sharing observations/insights
• Data Quality is not a system running in your cluster!!!
https://en.wikipedia.org/wiki/Privacy_by_design
http://jeffjonas.typepad.com/jeff_jonas/2006/08/accumulating_co.html

Challenges
Wants all the data – now.
• How to maximize value of distributed enterprise
• Ability to redo decisions (what if I took a different path)
• Ability to interact with live data and systems to effect change
• Trace actions to decisions to inputs…’cite your sources’
• Can you trust/prove your assertions?

The Information Supply Chain – Bidirectional/closed loop system
§ Constrained
§ High-latency
§ Localized context
§ Hybrid – cloud/on-premises
§ Low-latency
§ Global context
SOURCES
REGIONAL
INFRASTRUCTURE
CORE
INFRASTRUCTURE

It’s not just about minimizing
latency between event time and
access time – it’s about how
quickly you can change behavior
and seize new opportunities

Agenda
Data in Motion Use Cases

IoT/Edge Use Cases for Data In Motion
Ã Insurance (Behavior based risk management)
– Insured asset -> Data Aggregator -> Insurance Company -> Consumer
– Who owns the data (consumer, insurance company, auto company)?
– Incentivize safe driving
– Legal question: If the data suggested you were drunk who has liability if this is detectable/known
and someone gets hurt?
Ã Energy (Efficient power delivery to home and consumption)
– Smart Meters help the consumer track/tune their usage patterns
– ..and power companies need greater insight to actual usage, better ability to reroute/switch power

IoT/Edge Use Cases for Data In Motion
Ã Planes (Better fuel efficiency and customer experience)
– Ownership of data: Pilot’s union? Airplane manufacturer? Airline? Passenger?
Ã Trains (Keep trains in service and tracks operational)
– Sensors on the trains, tracks, overpasses, drones…
– Assessing health and lifespan of critical components
Ã Automobiles (Connected, autonomous, recall management)
– The number of sensors and accessible data points is staggering
– Notify the driver they need to head to the dealership. Notify dealership to have a specific part on
hand.
– Multiple business units at an auto manufacture will have conflicting priorities on the data they
need. Amount of data available dramatically exceeds bandwidth to transmit…

‘Traditional’ Use Cases for Data in Motion
Ã Technical Operations
– Log collection from applications and infrastructure to monitor health of the enterprise
– Edge collection
Ã Cybersecurity
– Collection from routers/switches/firewalls, applications, probes, packet capture
– Enrichment, filtering, aggregation at massive scale / Multi network domain
Ã Common themes/Challenges
– Edge collection/prioritization/tracking
– Systems at the edge are harder to replace and constantly revving – supporting many versions,
formats, schemas at once
– Many edge systems are disconnected or access is periodic. Asynchronous/survivable as a default

Data in Motion – Terminology & Trends
Ã Ingest, ETL, CDC, Messaging -> Flow Management
Ã CDC is still a big deal but organizations are increasingly sourcing from the origin!
Ã Batch vs Streaming misses the point. What matters is knowing as quickly as necessary
and with highest confidence
– Something interesting has happened
– What interesting things are happening
– What interesting things will happen
Ã Open source technology is driving innovation and giving greater access to powerful tools
to solve these problems!

Agenda
Flow Management

Flow Management: The big lessons
Ã Data in motion tends to stay in motion until acted upon by a system that
doesn’t know how to work with it
Ã The farther apart any two systems are the less likely they are to be able to
communicate
Ã Data quality is an end to end problem – not an analytic/system in your
datacenter
Ã Data prioritization is essential yet organizations remain unable to define the
priorities!

Command and Control
§ Constrained
§ High-latency
§ Low-latency
§ Global context
SOURCES
REGIONAL
INFRASTRUCTURE
CORE
INFRASTRUCTURE
Distributed agent
Central control
Multi-tenant
Interactive command and controlStrong SDLC
• Versioned Flows
• Versioned Components
• Versioned Assets

Provenance
§ Constrained
§ High-latency
§ Low-latency
§ Global context
SOURCES
REGIONAL
INFRASTRUCTURE
CORE
INFRASTRUCTURE
Origin – attribution
Replay – recovery
Evolution of topologies
Long retention
Types of Lineage
• Event (runtime)
• Configuration (design time)

The need for data provenance
For Operators
• Traceability, lineage
• Recovery and replay
For Compliance
• Audit trail
• Remediation
For Business / Mission
• Value sources
• Value IT investment
BEGIN
END
LINEAGE

Security
§ Constrained
§ High-latency
§ Low-latency
§ Global context
SOURCES
REGIONAL
INFRASTRUCTURE
CORE
INFRASTRUCTURE
Sign, encrypt, static
(data and control)
TLS, obfuscation, dynamic entitlements
Kerberos, PKI, AD/DS, etc.

The need for fine-grained security and compliance
It’s not enough to say you have
encrypted communications
• Enterprise authorization
services –entitlements
change often
• People and systems with
different roles require
difference access levels
• Tagged/classified data

Apache NiFi – Key Features
• Guaranteed delivery
• Data buffering
- Backpressure
- Pressure release
• Prioritized queuing
• Flow specific QoS
- Latency vs. throughput
- Loss tolerance
• Data provenance
• Supports push and
pull models
• Visual/Interactive C2
• Templates
• Pluggable/Multi-Tentant
Auth
• Designed for extension
• Clustering
• Record Oriented
Processing
- Enrichment
- SQL Query based
routing/filtering
- Schema Aware
- Partitioning

Architecture

NiFi Architecture – Repositories – Pass by reference
FlowFile Content Provenance
F1à C1 C1
Excerpt of demo flow… What’s happening inside the repositories…
BEFORE
AFTER
F2à C1 C1 P3à F2 – Clone (F1)
F1à C1 P2à F1 – Route
P1à F1 – Create
P1à F1 – Create
https://nifi.apache.org/docs.html

NiFi Architecture – Repositories – Copy on Write
FlowFile Content Provenance
F1à C1 C1 P1à F1 - CREATE
Excerpt of demo flow… What’s happening inside the repositories…
BEFORE
AFTER
F1à C1
F1.1à C2 C2 (encrypted)
C1 (plaintext) *
P2à F1.1 - MODIFY
P1à F1 - CREATE
* C1 (plaintext) is now eligible to be removed. But if we keep it around as long as possible what cool things can we do?
https://nifi.apache.org/docs.html

Agenda
Flow Management
Stream Processing

Stream Processing – Key Components
Ã High scale, durable, multi-tenant & replayable stream access
– Apache Kafka
Ã Stream Processing Engine
– Apache Storm
– Apache Spark Streaming
– Apache Flink
– Apache Apex

Stream Processing – Key Components
Ã Stream Process API/Definition
– Apache Beam
Ã Stream Application Design, Deployment, Monitoring, Data Interaction
– Streaming Analtyics Manager
– Apache Superset (Incubating)
Ã Strong Integration with Enterprise Services

Agenda
Flow Management
Stream Processing
Enterprise Services

Enterprise Services – nexus of Data in Motion and Data at Rest
Ã Centralized Registries
– Schemas
– Models
– Components / Processing Functions
Ã Governance
– Apache Atlas
Ã Security/Policy
– Apache Knox
– Apache Ranger
Ã Deployment
– Apache Ambari

Closing Thoughts
• Data in Motion is an end-to-end problem
• Edge Collection
• Flow Management
• Stream Processing
• Enterprise Services
• Data Science depends on healthy data pipelines
• A powerful data in motion platform and
enterprise strategy is required

Data on the Move - DataCon DC

Recommended

Recommended

More Related Content

What's hot

What's hot (17)

Similar to Data on the Move - DataCon DC

Similar to Data on the Move - DataCon DC (20)

Recently uploaded

Recently uploaded (20)

Data on the Move - DataCon DC