Enterprises have dealt with data governance over the years, but it has been mostly around master data. With the advent of IoT/web/app streams everywhere in the ecosystem surrounding an enterprise, data-in-motion has become a strong force to reckon. Data-in-motion passes through several levels of transformations and augmentation before it becomes data-at-rest. Through this, it is pertinent to preserve the sanctity of such data or at least track the provenance through the various changes. This is very important for a lot of verticals where there are strong regulatory and compliance laws that exist around "who changed what." This session will go into detail around some specific use cases of how data gets changed, how it can be tracked seamlessly and why this is important for certain verticals. This will be presented in two parts. The first part will cover the industry angle to this and its importance weighed in by several regulatory bodies. The second part will address the technology aspect of it and discuss how companies can leverage Apache Atlas and Ranger in conjunction with NiFi and Kafka to embrace data governance and provenance of their data streams. Speakers Dinesh Chandrasekhar, Director, Hortonworks Paige Bartley, Senior Analyst - Data and Enterprise Intelligence, Ovum

1. Who changed my data?
Need for data governance and
provenance in a streaming world
Digital capability requires granular control of all data assets.
Dinesh Chandrasekhar
Director, Product Marketing
Paige Bartley
Senior Analyst, Data and
Enterprise Intelligence

2. Ovum | TMT intelligence | informa2 Copyright © Informa PLC

3. Ovum | TMT intelligence | informa3 Copyright © Informa PLC
Business challenges in achieving digital capability include:
 Reproducibility of analytics results
 Debugging of models and algorithms
 Ensuring correct access rights to data
 Consistent application of data policies
 Meeting regulatory compliance requirements
 Unifying data across repositories and silos
 Finding the right data at the right time
Digital Capability Depends on Full Control of Data
Addressing these
challenges requires
understanding how
data changes over time.

4. Ovum | TMT intelligence | informa4 Copyright © Informa PLC
Governance and Transparency of
Data Assets is More Important
than Ever

5. Ovum | TMT intelligence | informa5 Copyright © Informa PLC
More Data:
 Economics of storage have made keeping data cheap.
 New data types – sensor data, etc. – need to be combined with historical data.
More Users:
 Self-service era means more data consumers and more frequent data access.
 Varying users have varying access rights and privileges.
 More users means more proliferation of data versions.
More Complexity:
 Data repositories have become more distributed, and data sources more varied.
 Data resides in more locations than ever before, in the cloud and on-prem.
Factors Within the Enterprise

6. Ovum | TMT intelligence | informa6 Copyright © Informa PLC
More Regulatory Pressure
Regulations such as GDPR have indirect requirements for tracking lineage.
 Article 30 requirements for record keeping necessitate knowledge of provenance.
More Competitive Pressure
 Leverage of data is increasingly a competitive differentiator.
 Pace of change is accelerating, and comprehensive understanding of data is critical.
 Disruptors are emerging from unlikely industries, using data to their advantage.
Factors External to the Enterprise

7. Ovum | TMT intelligence | informa7 Copyright © Informa PLC
 Article 4: Definition of Personal Data
A person can be identified indirectly or directly
Data sources can be combined to make personal data
 Article 9: Processing of Special Categories of Personal Data
Processing of biometric data is highly restricted
Many types of sensors produce biometric data
 Article 30: Records of Processing Activities
“Who, what, when, where, and why” of processing
Need deep understanding of metadata and data lineage.
GDPR doesn’t differentiate between data-in-motion and data-at-rest! “Who changed what” is critical.
Lineage and provenance, while not directly required by GDPR, are critical to meeting requirements.
GDPR’s Specific Requirements for Data

8. Ovum | TMT intelligence | informa8 Copyright © Informa PLC
<<
<<
From an analytics standpoint, reaping the benefits of
big data means investing in data management and
governance. Without the correct people, processes, and
infrastructure, more casual business users will likely
struggle to see the benefits of big data technologies.
Laurent-Olivier Lioté
Analyst, Data and Enterprise Intelligence, Ovum

9. Ovum | TMT intelligence | informa9 Copyright © Informa PLC
A Holistic View of Data Requires Both Data-in-Motion and Data-at-Rest
Data at Rest Data in Motion
Contextual
Understanding
of Data

10. Ovum | TMT intelligence | informa10 Copyright © Informa PLC
Having a common enterprise metadata framework allows data of different types and from different sources
to be managed consistently.
A common metadata framework allows for:
 Common search and lineage for datasets
 Lifecycle management from ingestion to disposition
 Metadata exchange with other metadata tools
 Analysis of data usage and access trends
 Consistent application of access rights
 Analysis of behavior and anomalies
How Do We Do This? Metadata Management is Necessary for Governance
Metadata
Creation
Metadata
Enrichment
Metadata
Analysis

11. Ovum | TMT intelligence | informa11 Copyright © Informa PLC
The data lake, if properly managed, can support a common metadata framework which underpins enterprise data.
 Data-in-motion
 Data-at-rest
 Structured data
 Unstructured data
Common management of metadata allows for streamlined control and
visibility into data. Better control of data results in better business outcomes.
The Managed Data Lake Can Support a Common Metadata Framework
All metadata, managed together.

12. Ovum | TMT intelligence | informa12 Copyright © Informa PLC
The enterprise increasingly wants to analyze all data, both in-motion and at-rest, in context with each other.
Governance and lineage for data-in-motion allows for:
 Audit and regulatory compliance
 Insight into data history and provenance
 Comprehensive lifecycle management
 Security and access controls
 Better quality data = better analytics
Governance standards for data-in-motion need to match those for data-at-rest.
Governance Standards Need to be Equal
Common Metadata Framework
Data-in-Motion Data-at-Rest
Data Management Platform

13. 13 © Hortonworks Inc. 2011–2018. All rights reserved
Changing face of data
Challenges and Solutions

14. 14 © Hortonworks Inc. 2011–2018. All rights reserved
The New Way of Business Is Fueled By Connected Data
• Connected Customers,
Vehicles, Devices
• Socially crowd-sourced
requirements
• Digital design and
analysis
• Digital prototypes and
tests (simulations)
• Connected Factories,
Sensors, Devices
• Human-robotic
interaction
• 3D-printing on
demand
• Connected Trucks,
Inventory
• Location, traffic,
weather-aware
distribution
• Real-time inventory
visibility
• Dynamic rerouting
• Connected Customers,
Devices
• Omni- channel
demand sensing
• Real-Time
Recommendations
• Connected Assets
• Remote service
monitoring & delivery
• Predictive
maintenance
• OTA Updates
DEVELOPMENT MANUFACTURING DISTRIBUTION MARKETING/SALES SERVICE

15. 15 © Hortonworks Inc. 2011–2018. All rights reserved
Today’s Digital Enterprises
RFID TRACKERS AND
NANO-DEVICES
to give you visibility into
movement of your goods
MOBILE NOTIFICATIONS
to inform you of shipment
delay from a supplier
BLOCKCHAINS
to give complete trust and
provenance in your supply
chain
VIRTUAL ASSISTANTS
to enhance your customer
experience
AI-POWERED CHATBOTS
to improve your customer
support functions
ELECTRONIC B2B
EXCHANGES
to streamline order processing
with partners

16. 16
© Hortonworks, Inc. 2011-2018. All rights reserved. | Hortonworks confidential and proprietary information.
Modern Data Architecture
DATA CENTER
Machine
Learning/
Artificial
Intelligence
Telemetry –
Connected
Devices
Time Series
Databases
Stream Analytics
Deep Historical
Analysis
Exception
Monitoring
Legacy/
Operational
Data
Sensors,
Control
Systems
Cyber
Security
Edge
Analytics
Social Mobile
IoT
IoT
CLOUD
Geo Location

17. 17
© Hortonworks, Inc. 2011-2018. All rights reserved. | Hortonworks confidential and proprietary information.
Data Challenges
Cannot get a
360 VIEW of
your customer? DROWNING in
data lakes?
TOO MUCH DATA
coming in from
TOO MANY
SOURCES and
devices?
New business
initiatives leading
to EXCESSIVE IT
COSTS?`
MOST IMPORTANTLY…
Don’t have the right data at the right time to make the right decision?

18. 18
© Hortonworks, Inc. 2011-2018. All rights reserved. | Hortonworks confidential and proprietary information.
G L O B A L D ATA M A N A G E M E N T
DATA
SOURCES
DATA CENTER CLOUD EDGE
Exception
Monitoring
360 View of
Operations
Cyber
Security
Telemetry –
Connected
Devices
Time Series
Sensors,
Control
Systems
Telemetry –
Connected
Devices
Sensors,
Control
Systems
Time Series
Exception
Monitoring
Cyber
Security
Legacy/
Operational
Data
Global Data Management Enables Modern Data Architecture

19. 19
© Hortonworks, Inc. 2011-2018. All rights reserved. | Hortonworks confidential and proprietary information.
Data Management Challenges
• Dealing with multi-clouds
• Avoiding cloud/ vendor lock-in
• Future proofing your architecture
• Common view of security, governance
• Manage all data, regardless of type or location
• Maximize data re-use for multiple workloads
DATA
SOURCES
DATA CENTER CLOUD EDGE
Exception
Monitoring
360 View of
Operations
Cyber
Security
Telemetry –
Connected
Devices
Time Series
Sensors,
Control
Systems

20. 20
© Hortonworks, Inc. 2011-2018. All rights reserved. | Hortonworks confidential and proprietary information.
Global Data Management Platform
DATA
SOURCES
DATA CENTER CLOUD EDGE
Exception
Monitoring
360 View of
Operations
Cyber
Security
Telemetry –
Connected
Devices
Time Series
Sensors,
Control
Systems
DATA-IN-MOTION DATA-AT-REST
MANAGE, SECURE, GOVERN, CONSUME

21. 21
© Hortonworks, Inc. 2011-2018. All rights reserved. | Hortonworks confidential and proprietary information.
Global Data Management - Powering Innovation
MODERN DATA USE CASES
EDW
OPTIMIZATION
CYBERSECURITY DATA SCIENCE
ADVANCED
ANALYTICS
IOT/ STREAMING
ANALYTICS
DATA
SOURCES
DATA CENTER CLOUD EDGE
Exception
Monitoring
360 View of
Operations
Cyber
Security
Telemetry –
Connected
Devices
Time Series
Sensors,
Control
Systems
DATA-IN-MOTION DATA-AT-REST
MANAGE, SECURE, GOVERN, CONSUME

22. 22
© Hortonworks, Inc. 2011-2018. All rights reserved. | Hortonworks confidential and proprietary information.
Apache NiFi Overview
• Guaranteed delivery
• Data buffering
- Backpressure
- Pressure release
• Prioritized queuing
• Flow specific QoS
- Latency vs. throughput
- Loss tolerance
• Data provenance
• Supports push and pull
models
• Recovery/recording
a rolling log of fine-
grained history
• Visual command and
control
• Flow templates
• Pluggable/multi-role
security
• Designed for extension
• Clustering

23. 23 © Hortonworks Inc. 2011–2018. All rights reserved
Watch real time flow of data: Data Provenance in Apache NiFi
Select Data Provenance

24. 24 © Hortonworks Inc. 2011–2018. All rights reserved
Easily access and trace changes to dataflow in Apache NiFi

25. 25 © Hortonworks Inc. 2011–2018. All rights reserved
Apache Atlas
• Enterprise data
governance
• Integration with
Apache NiFi
• Integration with
Apache Ranger
Apache Atlas
Knowledge Store
Audit Store
ModelsType-System
Policy RulesTaxonomies
Tag Based
Policies
Data Lifecycle
Management
Real Time Tag BasedAccess Control
REST API
Services
Search Lineage Exchange
Healthcare
HIPAA
HL7
Financial
SOX
Dodd-Frank
Energy
PPDM
Retail
PCI
PII
Other
CWM SERVICE: DATA STEWARD STUDIODSS
Discover&
Fingerprint
Data
Smart
Enterprise
Search
Data & Metadata
Security
Data Lineage &
Impact Analysis
Enterprise
Data
Catalog
Organize&
CurateData

26. 26 © Hortonworks Inc. 2011–2018. All rights reserved
Thank you