Who changed my data? Need for data governance and provenance in a streaming world

Who changed my data?
Need for data governance and
provenance in a streaming world
Digital capability requires granular control of all data assets.
Dinesh Chandrasekhar
Director, Product Marketing
Paige Bartley
Senior Analyst, Data and
Enterprise Intelligence

Business challenges in achieving digital capability include:
 Reproducibility of analytics results
 Debugging of models and algorithms
 Ensuring correct access rights to data
 Consistent application of data policies
 Meeting regulatory compliance requirements
 Unifying data across repositories and silos
 Finding the right data at the right time
Digital Capability Depends on Full Control of Data
Addressing these
challenges requires
understanding how
data changes over time.

Governance and Transparency of
Data Assets is More Important
than Ever

More Data:
 Economics of storage have made keeping data cheap.
 New data types – sensor data, etc. – need to be combined with historical data.
More Users:
 Self-service era means more data consumers and more frequent data access.
 Varying users have varying access rights and privileges.
 More users means more proliferation of data versions.
More Complexity:
 Data repositories have become more distributed, and data sources more varied.
 Data resides in more locations than ever before, in the cloud and on-prem.
Factors Within the Enterprise

More Regulatory Pressure
Regulations such as GDPR have indirect requirements for tracking lineage.
 Article 30 requirements for record keeping necessitate knowledge of provenance.
More Competitive Pressure
 Leverage of data is increasingly a competitive differentiator.
 Pace of change is accelerating, and comprehensive understanding of data is critical.
 Disruptors are emerging from unlikely industries, using data to their advantage.
Factors External to the Enterprise

 Article 4: Definition of Personal Data
A person can be identified indirectly or directly
Data sources can be combined to make personal data
 Article 9: Processing of Special Categories of Personal Data
Processing of biometric data is highly restricted
Many types of sensors produce biometric data
 Article 30: Records of Processing Activities
“Who, what, when, where, and why” of processing
Need deep understanding of metadata and data lineage.
GDPR doesn’t differentiate between data-in-motion and data-at-rest! “Who changed what” is critical.
Lineage and provenance, while not directly required by GDPR, are critical to meeting requirements.
GDPR’s Specific Requirements for Data

<<
<<
From an analytics standpoint, reaping the benefits of
big data means investing in data management and
governance. Without the correct people, processes, and
infrastructure, more casual business users will likely
struggle to see the benefits of big data technologies.
Laurent-Olivier Lioté
Analyst, Data and Enterprise Intelligence, Ovum

A Holistic View of Data Requires Both Data-in-Motion and Data-at-Rest
Data at Rest Data in Motion
Contextual
Understanding
of Data

Having a common enterprise metadata framework allows data of different types and from different sources
to be managed consistently.
A common metadata framework allows for:
 Common search and lineage for datasets
 Lifecycle management from ingestion to disposition
 Metadata exchange with other metadata tools
 Analysis of data usage and access trends
 Consistent application of access rights
 Analysis of behavior and anomalies
How Do We Do This? Metadata Management is Necessary for Governance
Metadata
Creation
Metadata
Enrichment
Metadata
Analysis

The data lake, if properly managed, can support a common metadata framework which underpins enterprise data.
 Data-in-motion
 Data-at-rest
 Structured data
 Unstructured data
Common management of metadata allows for streamlined control and
visibility into data. Better control of data results in better business outcomes.
The Managed Data Lake Can Support a Common Metadata Framework
All metadata, managed together.

The enterprise increasingly wants to analyze all data, both in-motion and at-rest, in context with each other.
Governance and lineage for data-in-motion allows for:
 Audit and regulatory compliance
 Insight into data history and provenance
 Comprehensive lifecycle management
 Security and access controls
 Better quality data = better analytics
Governance standards for data-in-motion need to match those for data-at-rest.
Governance Standards Need to be Equal
Common Metadata Framework
Data-in-Motion Data-at-Rest
Data Management Platform

13 © Hortonworks Inc. 2011–2018. All rights reserved
Changing face of data
Challenges and Solutions

The New Way of Business Is Fueled By Connected Data
• Connected Customers,
Vehicles, Devices
• Socially crowd-sourced
requirements
• Digital design and
analysis
• Digital prototypes and
tests (simulations)
• Connected Factories,
Sensors, Devices
• Human-robotic
interaction
• 3D-printing on
demand
• Connected Trucks,
Inventory
• Location, traffic,
weather-aware
distribution
• Real-time inventory
visibility
• Dynamic rerouting
• Connected Customers,
Devices
• Omni- channel
demand sensing
• Real-Time
Recommendations
• Connected Assets
• Remote service
monitoring & delivery
• Predictive
maintenance
• OTA Updates
DEVELOPMENT MANUFACTURING DISTRIBUTION MARKETING/SALES SERVICE

Today’s Digital Enterprises
RFID TRACKERS AND
NANO-DEVICES
to give you visibility into
movement of your goods
MOBILE NOTIFICATIONS
to inform you of shipment
delay from a supplier
BLOCKCHAINS
to give complete trust and
provenance in your supply
chain
VIRTUAL ASSISTANTS
to enhance your customer
experience
AI-POWERED CHATBOTS
to improve your customer
support functions
ELECTRONIC B2B
EXCHANGES
to streamline order processing
with partners

16
© Hortonworks, Inc. 2011-2018. All rights reserved. | Hortonworks confidential and proprietary information.
Modern Data Architecture
DATA CENTER
Machine
Learning/
Artificial
Intelligence
Telemetry –
Connected
Devices
Time Series
Databases
Stream Analytics
Deep Historical
Analysis
Exception
Monitoring
Legacy/
Operational
Data
Sensors,
Control
Systems
Cyber
Security
Edge
Analytics
Social Mobile
IoT
IoT
CLOUD
Geo Location

17
Data Challenges
Cannot get a
360 VIEW of
your customer? DROWNING in
data lakes?
TOO MUCH DATA
coming in from
TOO MANY
SOURCES and
devices?
New business
initiatives leading
to EXCESSIVE IT
COSTS?`
MOST IMPORTANTLY…
Don’t have the right data at the right time to make the right decision?

18
G L O B A L D ATA M A N A G E M E N T
DATA
SOURCES
DATA CENTER CLOUD EDGE
Exception
Monitoring
360 View of
Operations
Cyber
Security
Telemetry –
Connected
Devices
Time Series
Sensors,
Control
Systems
Telemetry –
Connected
Devices
Sensors,
Control
Systems
Time Series
Exception
Monitoring
Cyber
Security
Legacy/
Operational
Data
Global Data Management Enables Modern Data Architecture

19
Data Management Challenges
• Dealing with multi-clouds
• Avoiding cloud/ vendor lock-in
• Future proofing your architecture
• Common view of security, governance
• Manage all data, regardless of type or location
• Maximize data re-use for multiple workloads
DATA
SOURCES
Exception
Monitoring
360 View of
Operations
Cyber
Security
Telemetry –
Connected
Devices
Time Series
Sensors,
Control
Systems

20
Global Data Management Platform
DATA
SOURCES
Exception
Monitoring
360 View of
Operations
Cyber
Security
Telemetry –
Connected
Devices
Time Series
Sensors,
Control
Systems
DATA-IN-MOTION DATA-AT-REST
MANAGE, SECURE, GOVERN, CONSUME

21
Global Data Management - Powering Innovation
MODERN DATA USE CASES
EDW
OPTIMIZATION
CYBERSECURITY DATA SCIENCE
ADVANCED
ANALYTICS
IOT/ STREAMING
ANALYTICS
DATA
SOURCES
Exception
Monitoring
360 View of
Operations
Cyber
Security
Telemetry –
Connected
Devices
Time Series
Sensors,
Control
Systems
DATA-IN-MOTION DATA-AT-REST
MANAGE, SECURE, GOVERN, CONSUME

22
Apache NiFi Overview
• Guaranteed delivery
• Data buffering
- Backpressure
- Pressure release
• Prioritized queuing
• Flow specific QoS
- Latency vs. throughput
- Loss tolerance
• Data provenance
• Supports push and pull
models
• Recovery/recording
a rolling log of fine-
grained history
• Visual command and
control
• Flow templates
• Pluggable/multi-role
security
• Designed for extension
• Clustering

Watch real time flow of data: Data Provenance in Apache NiFi
Select Data Provenance

Easily access and trace changes to dataflow in Apache NiFi

Apache Atlas
• Enterprise data
governance
• Integration with
Apache NiFi
• Integration with
Apache Ranger
Apache Atlas
Knowledge Store
Audit Store
ModelsType-System
Policy RulesTaxonomies
Tag Based
Policies
Data Lifecycle
Management
Real Time Tag BasedAccess Control
REST API
Services
Search Lineage Exchange
Healthcare
HIPAA
HL7
Financial
SOX
Dodd-Frank
Energy
PPDM
Retail
PCI
PII
Other
CWM SERVICE: DATA STEWARD STUDIODSS
Discover&
Fingerprint
Data
Smart
Enterprise
Search
Data & Metadata
Security
Data Lineage &
Impact Analysis
Enterprise
Data
Catalog
Organize&
CurateData

Thank you

Who changed my data? Need for data governance and provenance in a streaming world

More Related Content

What's hot

Similar to Who changed my data? Need for data governance and provenance in a streaming world

More from DataWorks Summit

Recently uploaded

Who changed my data? Need for data governance and provenance in a streaming world

Editor's Notes