Future of Data New Jersey - HDF 3.0 Deep Dive

© Hortonworks Inc. 2011 – 2016. All Rights Reserved2
Agenda
Quick Overview of HDF & Apache NiFi
What’s new in HDF 3.0?
Latest Efforts in the Apache NiFi Community

© Hortonworks Inc. 2011 – 2016. All Rights Reserved
HDF Data-In-Motion Platform
HDF provides the flow management, stream processing, and enterprise services
needed to collect, curate, analyze and act on data-in-motion across the data center and
cloud.
Flow Management
Data acquisi*on and delivery
Simple transforma*on and data rou*ng
Simple event processing
End to end provenance
Edge intelligence & bi-direc*onal communica*on
Stream Processing
Scalable data broker for streaming apps
Scale out complex transforma6on
Stream Analy2cs
Pa8ern Matching
Prescrip6ve & Predic6ve Stream Analy6cs
Complex Event Processing
Con6nuous Insight
Enterprise Services
Provisioning, Management, Monitoring, Security, Audit, Compliance, Governance, Mul: -tenancy
Java
Agent
C++
Agent

Empower users to manage the
collection and flow of data

The Problem at Hand
Producers A.K.A Things
Anything
AND
Everything
Internet!
Consumers
• User
• Storage
• System
• …More Things

Moving data effectively is hard
Standards: http://xkcd.com/927/

Apache NiFi: A Primer
Key Features and Principles
• Guaranteed delivery
• Data buffering
- Backpressure
- Pressure release
• Prioritized queuing
• Flow specific QoS
- Latency vs. throughput
- Loss tolerance
• Data provenance
• Recovery/recording
a rolling log of fine-grained
history
• Visual command and
control
• Flow templates
• Pluggable/multi-role
security
• Designed for extension
• Clustering

NiFi is based on Flow Based Programming (FBP)
FBP Term NiFi Term Description
Information
Packet
FlowFile Each object moving through the system.
Black Box FlowFile
Processor
Performs the work, doing some combination of data routing, transformation,
or mediation between systems.
Bounded
Buffer
Connection The linkage between processors, acting as queues and allowing various
processes to interact at differing rates.
Scheduler Flow
Controller
Maintains the knowledge of how processes are connected, and manages the
threads and allocations thereof which all processes use.
Subnet Process
Group
A set of processes and their connections, which can receive and send data via
ports. A process group allows creation of entirely new component simply by
composition of its components.

NiFi & Data Agnosticism
 NiFi is data agnostic!
 But, NiFi was designed understanding that users
can care about specifics and provides tooling
to interact with specific formats, protocols, etc.
ISO 8601 - http://xkcd.com/1179/
Robustness principle
Be conservative in what you do,
be liberal in what you accept from others“

Apache NiFi - MiNiFi
 Let me get the key parts of NiFi close to where data begins
 Bidirectional data transfer
 Greater illuminate journey with provenance
 NiFi lives in the data center. Give it an enterprise server or a cluster of
them.
 MiNiFi lives as close to where data is born and is a guest on that device or
system

Connecting the Drops
SOURCES
REGIONAL
INFRASTRUCTURE
CORE
INFRASTRUCTURE

Managing data flow for a courier service
Physical Store
Gateway
Server
Mobile Devices
Registers
Server Cluster
Distribution Center
Kafka
Core Data Center at HQ
Server Cluster
Others
Storm / Spark /
Flink / Apex
Kafka
Storm / Spark / Flink / Apex
On Delivery Routes
Trucks Deliverers
Delivery Truck: Creative Stall, https://thenounproject.com/creativestall/
Deliverer: Rigo Peter, https://thenounproject.com/rigo/
Cash Register: Sergey Patutin, https://thenounproject.com/bdesign.by/
Hand Scanner: Eric Pearson, https://thenounproject.com/epearson001/
Client
Libraries
Client
Libraries
MiNiFi
MiNiFi
NiFi NiFi NiFi NiFi NiFi NiFi
Client
Libraries

What’s new in HDF 3.0?

Record Based Processing Mechanism
 Why?
– Improve operational efficiency
– Intuitive and flexible filtering/routing strategies powered by ‘QueryRecord’
– Simpler dataflow design
 What?
– Introduce ‘record’ based operation model
– ‘RecordReader’ and ‘RecordWriter’ controller services
– A series of processors supporting the reader/writer processing mechanism
• Plugin record reader to de-serialize bytes to record objects
• Plugin record writer to serialize record objects to bytes
• Enable operations against in-memory record objects
 How?
Record Processing

Record Reader CS
Record Processing

Record Reader/Writer Schema Access Strategy
Record Processing

NiFi Record Based Processing
Data
Source
Centralized Schema
Repository
Lookup Schema

What is Schema Registry? What Value Does it Provide?
 What is Schema Registry?
• A shared repository of schemas that allows applications to flexibly interact with each other - in order to save or
retrieve schemas for the data they need to access
 What Value does Schema Registry Provide?
1. Data Governance
• Provide reusable schema (centralized registry)
• Define relationship between schemas (version management)
• Enable generic format conversion, and generic routing (schema validation)
2. Operational Efficiency
 To avoid attaching schema to every piece of data (centralized registry)
 Consumers and producers can evolve at different rates (version management)
 Data quality (schema validation)

Schema Registry Concepts
Schema Group
• Group Name
• Schema Group - A logical
grouping/container for similar
type of schemas or based any
criteria that the customer has
from managing the schemas
Schema Metadata 1
• Schema Name
• Schema Type
• Description
• Compatibility Policy
• Serializers
• Deserializers
• Schema Meta - Metadata
associated with a named
schema.
Schema Version 1
• Schema Version
• Schema Text
• Schema Version- The
actual versioned schema
associated a schema meta
definition

Schema Registry
Schema Registry Component Architecture
SR Web Server
Schema Registry
Web App
REST API
Pluggable Storage
Schema Metadata
Storage
Serializer/Deserializer
Jar Storage
Supported Meta
Stores
mySQL In-Memory
Supported Jar
Storage
Local File
System
HDFS
Schema Registry Client
Java Client
Integrations
NiFi Processors Kafka Se/DeSerializers
SAM
Processors

Integration of Schema Registry with HDF
 NiFi Processors for Schema Registry (HDF 2.2+)
 Fetch Schema
 Serialize/Deserialize with Schema
 SAM processors for Schema Registry (HDF 2.2, Available in Early Access Bundle)
 Lookup Schema of a Kafka Topic
 Atlas integration with Schema (Upcoming)
 Just like Atlas pull schema info from Hive MetaStore, Atlas can now capture schema,
format and semantic metadata from events in HDF via the registry. Coming in HDF
2.3

‘QueryRecord’ Processor
Record Processing

‘QueryRecord’ Use Case
Record Processing
 Use Case Background
– Oil & Gas: log data collection from remote drilling station
– expensive BGAN satellite network connectivity (64KB)
 Problem Statement
– 1000 sensors, costly to send ALL data ALL THE TIME
– Collect all sensor data only if: upstream pump is off, BUT you have an elevated pipe
pressure
 Solution
– Define routing strategy using ’QueryRecord’

Demo

Component versioning
Component Version
 Why?
– Foundational work to enable extension registry
– Foundational work to enhance flow migration experience
 What?
– Support multiple versions of the same NAR in a single NIFI instance
– E.g. Hadoop NAR version A: Apache Hadoop client lib; Hadoop NAR version B: proprietary Hadoop
client lib
 How?

Component versioning
Component Version

Latest Efforts in the Apache NiFi Community

Addressing common asks
How can I … How do I ... What about ...
 Version my flows?
 Drive CI/CD processes?
 Migrate flows between environments?
 Provision distributions of NiFi with a set of components?
 Make reference datasets/extensions available to the entirety of my data
flow?
 Certify / Audit / Sign-off on flows as compliant per regulations?

Capturing the essence of a flow in your organization
 The n-dimensions of data flow
 Consider a flowfile to be a singular event at a given juncture in its processing
 A flow is the directed graph of processing at a given point in time
 With each component’s:
 Configuration
 Version
 Referenced Assets

Introducing

Registry is an enabler
 SDLC
 Manage variables, sensitive properties for environments
 Extension Registry
 Association/tagging of data with the flow that created it
 Enhanced Command and Control of MiNiFi instances

Apache NiFi - MiNiFi: Centralized Command & Control (C2)
 Provide flow updates, information and assets to instances where they live
 Act as a gateway to/from network enclaves
 Provide a user interface/experience for design & deploy and monitoring
Extend the reach of user experience and operations

Docker Containerization
 First Apache community released version in 1.2.0
 https://hub.docker.com/r/apache/nifi
 Provide Docker images for all components of the NiFi ecosystem
Further supplement operations

The Evolution of Apache NiFi
 Our core substrate for data flow is NiFi & MiNiFi
 Command and Control facilitates operations and management of components
 Registry for common tasks with disparate resources across the NiFi ecosystem

Why the Apache NiFi Ecosystem?
 Moving data is multifaceted in its challenges and these are present in different contexts
at varying scopes
 Provide components and a platform with common tooling and extensions that are
commonly needed but be flexible for extension in all aspects
– Allow organizations to integrate with their existing infrastructure
 Empower folks managing your infrastructure to make changes and reason about issues
that are occurring
– Data Provenance to show context and data’s journey
– User Interface/Experience a key component

Thank You

Future of Data New Jersey - HDF 3.0 Deep Dive

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Future of Data New Jersey - HDF 3.0 Deep Dive

Similar to Future of Data New Jersey - HDF 3.0 Deep Dive (20)

Recently uploaded

Recently uploaded (20)

Future of Data New Jersey - HDF 3.0 Deep Dive