The document discusses the need for integrated provenance in data lakes to ensure effective management and traceability of varied data types. It highlights challenges posed by the flexibility of data lakes, such as increased difficulty in manageability and the concept of 'data swamps.' The authors propose a system for capturing provenance across different analytics frameworks to enhance data traceability and ensure reliable data processing within the data lake environment.

1. Crossing Analytics Systems: Case for
Integrated Provenance in Data Lakes
Isuru Suriarachchi and Beth Plale
School of Informatics and Computing
Indiana University
IEEE E-science 2016 : Hot Topics

2. The Data Lake has arisen within
last couple of years as
conceptualization of data
management framework with
flexibility to support multiple
data processing tools needed for
truly Big Data analytics.

3. Data Warehouse
• Supports multidimensional analytical processing
– Online Analytical Processing (OLAP) or
Multidimensional OLAP
• Numeric facts (measures) categorized by
dimensions creating vector space (OLAP cube).
• Interface is matrix interface like Pivot tables
• Schema is star schema, snowflake schema
• Storage is largely relational database

4. Credit: https://www.linkedin.com/topic/data-warehouse-architecture
Data Warehouse Architecture
• ETL: Extraction, Transformation, Load

5. Challenging the Warehouse: Big Data
• From numerous sources
– social media, sensor data, IoT devices, server logs,
clickstream etc.
• Not all numeric (quantitative) thus differently
structured
– Structured, semi-structured, unstructured
• Continuously generated or archived

6. Suitability of Data Warehouse for
Today’s Big Data
• ETL imposes burden
– Schema on write
– Inflexibility/inefficiency at ingest time
– Information loss upon schema translation
• Weak fit for popular Big Data analytical tools
(e.g., Spark, Hadoop) and data serving
platforms (e.g., HDFS, S3)

7. Data Lake
• A scalable storage infrastructure with no schema
enforcement at ingest
• Data ingested in raw form: no loss
• Schema-on-read
• Integrated Transformations
– With e.g., Hadoop, Spark
IngestAPI
Data
Data Lake
Clickstream
Sensor data
IoT Devices
Social Media
Could Platforms
Server Logs Metadata Lineage
Transform Transform Transform
Data Data Data
Analysis
Big Data Processing Frameworks
Ex: Hadoop, Spark, Storm

8. Data Lake Challenges
• Increased flexibility leads to harder
manageability
– Differently typed data can be easily dumped into
the Data Lake
– Data products can be in different stages of their
lifecycle: raw, half processed, processed etc.
– Can easily turn into “data swamps”
• Requires traceability!!..
– Provenance can help

9. Data Provenance
• Information about activities, entities and people
who involved in producing a data product
• Standards
– OPM
– PROV
• If a Data Lake ensures that every data product’s
provenance is in place starting from data
product’s origin, critical traceability can be had

10. What provenance perspective could
bring to a Data Lake?
• Track origins of data, chained transformations
• Contribute to reuse determinations of trust
and quality
• React!! Minimally constrain what enters a
Lake?

11. Challenges in Provenance Capturing
• Chains of Transformations
– Different analytics systems: Hadoop, Spark etc.
• Need is end to end integrated provenance across
transformations
• System specific provenance
collection methods are less
useful
– Integration/stitching
problems
– E.g.: RAMP, HadoopProv
for Hadoop

12. Solution to minimal lake governance
• All components in lake stream provenance to
central provenance subsystem
– Stores provenance for long term queries
– Monitors provenance stream in real time
• Event in stream represented by edge in
provenance graph
• Global lake wide policy: Uniform Persistent ID
(PID) (Handle, UUIDs, DOIs) attached to all
data objects in Data Lake
– required to guarantee integrated provenance

13. Model
• PID assigned to all data objects
– granularity
• Transformations T1, T2, and T3
– Distributed
– May use different frameworks
d1 T1
d2
d3
d4
d5
d6
d7
d8T2 T3
d1d3
d4
d6
d7
d8
Chain of
transformations
sharing Ids
Backward
provenance
from central
provenance store

14. Provenance traces integrate across systems of
Data Lake

15. Reference Architecture
IngestAPI
Batch Processing
Ex: Hadoop, Spark
Lineage
Raw Data from
various sources
Transformations
Workflow Engines
Ex: Kepler
Legacy Scripts
Stream Processing
Ex: Storm, Spark
Monitoring
Debugging Reproducing Data Quality
QueriesVisualization
Data Data Data
Data
Import
Lineage
Data
Export
Data Lake
Messaging System
Ingest API
Query API
ProvenanceSubsystem
Prov Stream
Processing
Prov
Storage
Prov
Stream
• Real-time provenance stream processing
• Stored provenance for long term usage

16. Prototype Use Case
• Different frameworks used
– Flume: Captures tweets and write into HDFS
– Hadoop Job: Computes hashtag counts
– Spark Job: Computes category counts

17. Central provenance store
• Uses Komadu
– A distributed
provenance
collection tool
– Visualization,
Custom Queries
I. Suriarachchi, Q. Zhou and B. Plale (2015). Komadu: A Capture and Visualization System for Scientific Data
Provenance. Journal of Open Research Software 3(1):e4

18. Client Library
• Log4j like API for provenance capture
• Dedicated thread pool in provenance layer
• Batching to minimize network overhead
Application Layer API
Komadu Client Layer
RabbitMQ Client Layer
client.addGeneration(A, E)
batching
prov thread
pool
RabbitMQ
Server
Komadu
Client Library

19. Use case evaluation
• Flume, Hadoop and Spark jobs instrumented
using Komadu client libraries
• Jobs stream provenance events into central
provenance store (Komadu)
• Persistent IDs (UUID) assigned for each data
object at entry to data lake; PID persists
thereafter with data object

20. Use case evaluation: experimental
environment
• 5 small VM instances, 2 2.5GhZ cores, 4 GB
RAM, 50 GB local storage
• 4 VM instances used for HDFS cluster
• 3.23 GB Twitter data collected over 5 days
running Flume on master node
• Hadoop and Spark set up on top of HDFS
cluster
• Separate instance for RabbitMQ and Komadu

21. Use case evaluation: Metrics
• Batch size:
– impact of batch size on provenance capture efficiency.
Measured by total execution time for Hadoop using
provenance event batching mechanism in Komadu
library
• Overhead of provenance capture:
– Measured against total tool-specific execution time
– measure overhead of customized value field (in key
value pair)
– Measure overhead of provenance capture for Hadoop
and Spark

22. Batch Size Test
• Hadoop job execution times with varying
batch sizes
• Optimal batch size: ~5000 KB

23. Overhead: Hadoop
• custom val: emits PID with key value pair
as (#nba, <2, id>) instead of (#nba, 2)
• data prov HDFS: writes provenance into HDFS,
used by HadoopProv and RAMP

24. Overhead: Spark
• Higher provenance capture overhead
compared to Hadoop

25. Future Work
• Performance overhead is prohibitively high
– decouple PID assignment from execution?
Examine granularity
• Live provenance stream processing for real
time monitoring/reaction
• Explore minimal provenance at on-line rates
and more comprehensive provenance at off-
line rates

26. Work funded in part by National Science
Foundation OCI-0940824
IEEE E-science 2016 : Hot Topics