Data Con LA, profile picture
Uploaded byData Con LA
884 views

Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Data Provenance Support in Spark, Matteo Interlandi, PostDoc, UCLA

The document outlines challenges and solutions for debugging and developing interactive programs in Apache Spark, focusing on data provenance techniques like lineage tracking and selective replay of computations to improve visibility and interactivity in data-intensive scalable computing systems. It discusses the Titan programming interface and various examples of data processing workflows to demonstrate these debugging techniques. Additionally, it addresses the overhead of previous systems and emphasizes the need for optimized solutions in large-scale job contexts.

Technology◦
Related topics:
Apache Spark•

Embed presentation

1 / 126
2 / 126
3 / 126
4 / 126
5 / 126
6 / 126
7 / 126
8 / 126
9 / 126
10 / 126
11 / 126
12 / 126
13 / 126
14 / 126
15 / 126
16 / 126
17 / 126
18 / 126
19 / 126
20 / 126
21 / 126
22 / 126
23 / 126
24 / 126
25 / 126
26 / 126
27 / 126
28 / 126
29 / 126
30 / 126
31 / 126
32 / 126
33 / 126
34 / 126
35 / 126
36 / 126
37 / 126
38 / 126
39 / 126
40 / 126
41 / 126
42 / 126
43 / 126
44 / 126
45 / 126
46 / 126
47 / 126
48 / 126
49 / 126
50 / 126
51 / 126
52 / 126
53 / 126
54 / 126
55 / 126
56 / 126
57 / 126
58 / 126
59 / 126
60 / 126
61 / 126
62 / 126
63 / 126
64 / 126
65 / 126
66 / 126
67 / 126
68 / 126
69 / 126
70 / 126
71 / 126
72 / 126
73 / 126
74 / 126
75 / 126
76 / 126
77 / 126
78 / 126
79 / 126
80 / 126
81 / 126
82 / 126
83 / 126
84 / 126
85 / 126
86 / 126
87 / 126
88 / 126
89 / 126
90 / 126
91 / 126
92 / 126
93 / 126
94 / 126
95 / 126
96 / 126
97 / 126
98 / 126
99 / 126
100 / 126
101 / 126
102 / 126
103 / 126
104 / 126
105 / 126
106 / 126
107 / 126
108 / 126
109 / 126
110 / 126
111 / 126
112 / 126
113 / 126
114 / 126
115 / 126
116 / 126
117 / 126
118 / 126
119 / 126
120 / 126
121 / 126
122 / 126
123 / 126
124 / 126
125 / 126
126 / 126
Interactive Programs Debugging and Development in Apache Spark
Outline ‣ Motivating Scenario ‣ Titian Programming Interface ‣ Internals ‣ Vega ‣ Conclusions
Outline ‣ Motivating Scenario ‣ Titian Programming Interface ‣ Internals ‣ Vega ‣ Conclusions
๏ Debugging data processing logic in Data-Intensive Scalable Computing (DISC) system is difficult ๏ Analysis tools are still in their “infancy” ๏ Today’s large-scale jobs are black boxes: • Job submitted to a cluster • Results come back minutes to hours later • No visibility into running algorithm Big Data Debugging
Big Data Debugging - State of the Art
Big Data Debugging - State of the Art
Big Data Debugging - State of the Art
Big Data Debugging - State of the Art
Big Data Debugging - State of the Art
๏ Easy to use GDB-like debugger [ICSE 16] (not covered in this talk) ๏ Visibility of data into running workflow • E.g., what (input) data led to this (outlier) result? ๏ Selectively replaying a portion of the data processing steps on subsets of intermediate data leading to outliers results ๏ Interactive program analysis Big Data Debugging - Desiderata
๏ Visibility of data -> Tracking the dependencies between the individual inputs and outputs records ๏ Selective replay -> Storage of intermediate results: • Dataset shared among running job and analysis tool ๏ Interactivity -> Implementation Constraints: • Latency constraint - In memory computation • Programming interface constraint - Integration with Spark DSL Big Data Debugging - Challenges
๏ Well known technique in databases ๏ Two granularities of provenance • Transformation (coarse-grained) provenance – Records the complete workflow of the derivation of a dataset – Spark RDD lineage is an example of this form of provenance • Data (fine-grained) provenance – Records data dependencies between input and output records – The type of provenance Titian focuses on Data Provenance (Lineage)
Tuple-ID Time Sendor-ID Temperature T1 11AM 1 34 T2 11AM 2 35 T3 11AM 3 35 T4 12PM 1 35 T5 12PM 2 35 T6 12PM 3 100 T7 1PM 1 35 T8 1PM 2 35 T9 1PM 3 80 SELECT AVG(temp),time
 FROM sensors
 GROUP BY time Sensors Result-ID Time AVG(temp) ID-1 11AM 34.6 ID-2 12PM 56.6 ID-3 1PM 50 Data Provenance - Example
Tuple-ID Time Sendor-ID Temperature T1 11AM 1 34 T2 11AM 2 35 T3 11AM 3 35 T4 12PM 1 35 T5 12PM 2 35 T6 12PM 3 100 T7 1PM 1 35 T8 1PM 2 35 T9 1PM 3 80 SELECT AVG(temp),time
 FROM sensors
 GROUP BY time Sensors Result-ID Time AVG(temp) ID-1 11AM 34.6 ID-2 12PM 56.6 ID-3 1PM 50 Outlier Outlier Why ID-2 and ID-3 have those high Data Provenance - Example
Tuple-ID Time Sendor-ID Temperature T1 11AM 1 34 T2 11AM 2 35 T3 11AM 3 35 T4 12PM 1 35 T5 12PM 2 35 T6 12PM 3 100 T7 1PM 1 35 T8 1PM 2 35 T9 1PM 3 80 SELECT AVG(temp),time
 FROM sensors
 GROUP BY time Sensors Result-ID Time AVG(temp) ID-1 11AM 34.6 ID-2 12PM 56.6 ID-3 1PM 50 Outlier Outlier Why ID-2 and ID-3 have those high Data Provenance - Example
๏ They use external storage systems (HDFS in RAMP [CIDR-11], DBMS in Newt [SOCC-13]) to retain lineage data ๏ Data provenance queries are supported in a separate programming interface Previous Data Provenance DISC Systems
๏ They use external storage systems (HDFS in RAMP [CIDR-11], DBMS in Newt [SOCC-13]) to retain lineage data ๏ Data provenance queries are supported in a separate programming interface High overhead Previous Data Provenance DISC Systems
๏ They use external storage systems (HDFS in RAMP [CIDR-11], DBMS in Newt [SOCC-13]) to retain lineage data ๏ Data provenance queries are supported in a separate programming interface High overhead Low interactivity Previous Data Provenance DISC Systems
๏ Word Count job ๏ RAMP is up to 4X Spark ๏ Newt up to 86X Experience with Newt and RAMP 100 1000 1 10 100 Time(s) Dataset Size (GB) Spark Newt RAMP
Outline ‣ Motivating Scenario ‣ Titian Programming Interface ‣ Internals ‣ Vega ‣ Conclusions
Loads error messages from a log, counts the number of errors occurrences and returns a report containing the description of each error lc = new LineageContext(sc) lines = lc.textFile(“hdfs://...”) errors = lines.filter(_.startswith(“error”)) codes = errors.map(_.split(“t”)(1)) pairs = codes.map(word =>(word, 1)) counts = pairs.reduceByKey(word =>(_ + _)) reports = counts.map(kv => (dscr(kv._1), kv._2)) reports.collect.foreach(println) Example: Log Analysis
Given the result of the previous example, select the most frequent error and trace back to the input lines containing them Example: Backward Tracing
Given the result of the previous example, select the most frequent error and trace back to the input lines containing them frequentPair = reports.sortBy(_._2, false).take(1) frequent = reports.filter(_ == frequentPair) lineage = frequent.getLineage() input = lineage.goBackAll() input.collect().foreach(println) Example: Backward Tracing
Return the error codes generated from the network sub-system (indicated in the log by a “NETWORK” tag) Example: Forward Tracing
Return the error codes generated from the network sub-system (indicated in the log by a “NETWORK” tag) network = errors.filter(_.contains(“NETWORK”)) lineage = network.getLineage() output = lineage.goNextAll() output.collect().foreach(println) Example: Forward Tracing
Return the error distribution without the ones cause by the Guest user Example: Selective Replay
Return the error distribution without the ones cause by the Guest user lineage = reports.getLineage() inputLines = lineage.goBackAll() noGuest = inputLines.filter(!_.contains(“Guest”) && _.startswith(“error”)) newCodes = noGuest.map(_.split(“t”)(1)) newPairs = newCodes.map(word =>(word, 1)) newCounts = newPairs.reduceByKey(word =>(_ + _)) newRep = newCounts.map(kv => (dscr(kv._1), kv._2)) newRep.collect Example: Selective Replay
Outline ‣ Motivating Scenario ‣ Titian Programming Interface ‣ Internals ‣ Vega ‣ Conclusions
๏ LineageContext wrap SparkContext • Providing visibility into the submitted job ๏ Instrument LineageRDD at stage boundaries • Wrap native RDDs • Specific LineageRDD implementation based on instrument transformation ๏ Provenance data is buffered inside LineageRDDs • Saved into Spark BlockManager for querying Provenance Capturing
countspairscodeserrorslines Stage 1 Stage 2 reports lines = sc.textFile(“hdfs://...”) errors = lines.filter(_.startswith(“error”)) codes = errors.map(_.split(“t”)(1)) pairs = codes.map(word =>(word, 1)) counts = pairs.reduceByKey(word =>(_ + _)) reports = counts.map(kv => (dscr(kv._1), kv._2)) Spark Stage DAG
Instrumented Spark Stage DAG Combiner LineageRDD Reducer LineageRDD Hadoop LineageRDD counts pairscodeserrorslines Stage 1 Stage 2 reports Stage LineageRDD
Instrumented Workflow Combiner LineageRDD Reducer LineageRDD Hadoop LineageRDD counts pairscodeserrorslines Stage 1 Stage 2 reports Stage LineageRDD Input ID Output ID { id1, id 3} 400 { id2 } 4 Input ID Output ID offset1 id1 offset2 id2 offset3 id3 Input ID Output ID [p1, p2] 400 [ p1 ] 4 Input ID Output ID 400 id1 4 id2
Lineage Capture Runtime Overheads 100 1000 1 10 100 Time(s) Dataset Size (GB) Spark Titian Newt RAMP ๏ Same Word Count job ๏ Titian is in average 1.3X slower than Spark
Input ID Output ID offset1 id1 offset2 id2 offset3 id3 Input ID Output ID { id1, id 3} 400 { id2 } 4 Input ID Output ID [p1, p2] 400 [ p1 ] 4 Input ID Output ID 400 id1 4 id2 Hadoop Combiner Reducer Stage Example: Captured Data Lineage
Input ID Output ID offset1 id1 offset2 id2 offset3 id3 Input ID Output ID { id1, id 3} 400 { id2 } 4 Input ID Output ID [p1, p2] 400 [ p1 ] 4 Input ID Output ID 400 id1 4 id2 Hadoop Combiner Reducer Stage Example: Trace Back
Input ID Output ID offset1 id1 offset2 id2 offset3 id3 Input ID Output ID { id1, id 3} 400 { id2 } 4 Input ID Output ID [p1, p2] 400 [ p1 ] 4 Input ID Output ID 400 id1 4 id2 Hadoop Combiner Reducer Stage Example: Trace Back Stage.Input IDReducer.Output ID
Reducer.Output IDCombiner.Output ID Example: Trace Back Input ID Output ID offset1 id1 offset2 id2 offset3 id3 Input ID Output ID { id1, id 3} 400 { id2 } 4 Input ID Output ID [p1, p2] 400 [ p1 ] 4 Input ID Output ID 400 id1 4 id2 Hadoop Combiner Reducer Stage
Example: Trace Back Input ID Output ID offset1 id1 offset2 id2 offset3 id3 Input ID Output ID { id1, id 3} 400 { id2 } 4 Input ID Output ID [p1, p2] 400 [ p1 ] 4 Input ID Output ID 400 id1 4 id2 Hadoop Combiner Reducer Stage Combiner.Input IDHadoop.Output ID Now let’s do it for real!
Worker1 Worker2 Worker3 Input ID Output ID [p1, p2] 400 [ p1 ] 4 Input ID Output ID 400 id1 4 id2 Reducer Stage Input ID Output ID offset1 id1 offset2 id2 offset3 id3 Input ID Output ID { id1, id 3} 400 { id2 } 4 Hadoop Combiner Input ID Output ID offset1 id1 … … Input ID Output ID { id1, …} 400 Hadoop Combiner Example: Trace Back
Example: Trace Back Worker1 Worker2 Worker3 Input ID Output ID [p1, p2] 400 [ p1 ] 4 Input ID Output ID 400 id1 4 id2 Reducer Stage Input ID Output ID offset1 id1 offset2 id2 offset3 id3 Input ID Output ID { id1, id 3} 400 { id2 } 4 Hadoop Combiner Input ID Output ID offset1 id1 … … Input ID Output ID { id1, …} 400 Hadoop Combiner
Example: Trace Back Worker3 Input ID Output ID [p1, p2] 400 [ p1 ] 4 Input ID Output ID 400 id1 4 id2 Reducer Stage
Example: Trace Back Worker3 Input ID Output ID [p1, p2] 400 [ p1 ] 4 Input ID Output ID 400 id1 4 id2 Reducer Stage Worker3 Input ID Output ID [p1, p2] 400 [ p1 ] 4 Input ID Output ID 400 id1 4 id2 Reducer Stage Stage.Input IDReducer.Output ID
Example: Trace Back Worker1 Worker2 Worker3 Input ID Output ID [p1, p2] 400 [ p1 ] 4 Input ID Output ID 400 id1 4 id2 Reducer Stage Input ID Output ID offset1 id1 offset2 id2 offset3 id3 Input ID Output ID { id1, id 3} 400 { id2 } 4 Hadoop Combiner Input ID Output ID offset1 id1 … … Input ID Output ID { id1, …} 400 Hadoop Combiner
Example: Trace Back Worker1 Worker2 Worker3 Input ID Output ID [p1, p2] 400 [ p1 ] 4 Input ID Output ID 400 id1 4 id2 Reducer Stage Input ID Output ID offset1 id1 offset2 id2 offset3 id3 Input ID Output ID { id1, id 3} 400 { id2 } 4 Hadoop Combiner Input ID Output ID offset1 id1 … … Input ID Output ID { id1, …} 400 Hadoop Combiner Worker1 Worker2 Worker3 Input ID Output ID [p1, p2] 400 [ p1 ] 4 Input ID Output ID 400 id1 4 id2 Reducer Stage Input ID Output ID offset1 id1 offset2 id2 offset3 id3 Input ID Output ID { id1, id 3} 400 { id2 } 4 Hadoop Combiner Input ID Output ID offset1 id1 … … Input ID Output ID { id1, …} 400 Hadoop Combiner Input ID Output ID p1 400 Input ID Output ID p1 400 Targeted Shuffle
Example: Trace Back Worker1 Worker2 Worker3 Input ID Output ID 400 id1 4 id2 Stage Input ID Output ID offset1 id1 offset2 id2 offset3 id3 Input ID Output ID { id1, id 3} 400 { id2 } 4 Hadoop Combiner Input ID Output ID offset1 id1 … … Input ID Output ID { id1, …} 400 Hadoop Combiner Input ID Output ID p1 400 Input ID Output ID p1 400
Example: Trace Back Worker1 Worker2 Input ID Output ID offset1 id1 offset2 id2 offset3 id3 Input ID Output ID { id1, id 3} 400 { id2 } 4 Hadoop Combiner Input ID Output ID offset1 id1 … … Input ID Output ID { id1, …} 400 Hadoop Combiner Input ID Output ID p1 400 Input ID Output ID p1 400 Combiner.Output ID Reducer.Output ID Combiner.Output ID Reducer.Output ID
Example: Trace Back Hadoop Combiner Worker1 Worker2 Input ID Output ID offset1 id1 offset2 id2 offset3 id3 Input ID Output ID { id1, id 3} 400 { id2 } 4 Hadoop Combiner Input ID Output ID offset1 id1 … … Input ID Output ID { id1, …} 400 Hadoop Combiner
Example: Trace Back Hadoop Combiner Worker1 Worker2 Input ID Output ID offset1 id1 offset2 id2 offset3 id3 Input ID Output ID { id1, id 3} 400 { id2 } 4 Hadoop Combiner Input ID Output ID offset1 id1 … … Input ID Output ID { id1, …} 400 Hadoop Combiner Worker1 Worker2 Input ID Output ID offset1 id1 offset2 id2 offset3 id3 Input ID Output ID { id1, id 3} 400 { id2 } 4 Hadoop Combiner Input ID Output ID offset1 id1 … … Input ID Output ID { id1, …} 400 Hadoop Combiner Combiner.Input IDHadoop.Output ID Combiner.Input IDHadoop.Output ID
Tracing Performance ๏ Word Count job ๏ Tracing one record backward in < 1 sec for dataset < 100GB ๏ 18 sec for 500GB dataset
Vega: Optimizations for Selective Replay Matteo Interlandi, Sai Deep Tetali, Muhammad Ali Gulzar, Joseph Noor Miryung Kim, Todd Millstein, Tyson Condie Under Submission
Debugging workflow ๏ Run program ๏ Understand the cause for bugs / outliers: • Lineage • Breakpoints/watchpoints • Crash culprit ๏ Fix bug • Fast selective replay } } Titian [VLDB 2016] BigDebug [ICSE 2016]
First Strategy Convert changes in code to changes in data
Incremental Plan Input aa b c aa c Map (aa, 1) (b, 1) (c, 1) (aa, 1) (c, 1) Shuffle (aa, [1, 1]) (b, 1) (c, [1, 1]) Reduce (aa, 2) (b, 1) (c, 2) countspairslines Stage 1 Stage 2 shuffle input .map(x=>(x,1)) .reduceByKey(_+_)
Incremental Plan Inject a filter in the workflow countspairslines Stage 1 Stage 2 shufflefilter input .filter(x=>x!=‘c’).map(x=>(x,1)) .reduceByKey(_+_)
Input aa b c aa c Map (aa, 1) (b, 1) (aa, 1) Shuffle (aa, [1, 1]) (b, 1) Reduce (aa, 2) (b, 1) Filter aa b aa countspairslines Stage 1 Stage 2 shufflefilter input .filter(x=>x!=‘c’).map(x=>(x,1)) .reduceByKey(_+_) Incremental Plan
Incremental Plan Input aa b c aa c Map (aa, 1) (b, 1) (c, 1) (aa, 1) (c, 1) Shuffle (aa, [1, 1]) (b, 1) (c, [1, 1]) Reduce (aa, 2) (b, 1) (c, 2) Filter aa b c aa c
Input aa b c aa c Map (aa, 1) (b, 1) (c, 1) (aa, 1) (c, 1) Shuffle (aa, [1, 1]) (b, 1) (c, [1, 1]) Reduce (aa, 2) (b, 1) (c, 2) Filter aa b c aa c δFilter —c —c Incremental Plan
Incremental Plan Input aa b c aa c Map (aa, 1) (b, 1) (c, 1) (aa, 1) (c, 1) Shuffle (aa, [1, 1]) (b, 1) (c, [1, 1]) Reduce (aa, 2) (b, 1) (c, 2) Filter aa b c aa c δFilter —c —c ∆Map —(c, 1) —(c, 1)
Incremental Plan Input aa b c aa c Map (aa, 1) (b, 1) (c, 1) (aa, 1) (c, 1) Shuffle (aa, [1, 1]) (b, 1) (c, [1, 1]) Reduce (aa, 2) (b, 1) (c, 2) Filter aa b c aa c δFilter —c —c ∆Map —(c, 1) —(c, 1) ∆Shuffle c, [—1, —1])
Incremental Plan Input aa b c aa c Map (aa, 1) (b, 1) (c, 1) (aa, 1) (c, 1) Shuffle (aa, [1, 1]) (b, 1) (c, [1, 1]) Reduce (aa, 2) (b, 1) (c, 2) Filter aa b c aa c δFilter —c —c ∆Map —(c, 1) —(c, 1) ∆Shuffle c, [—1, —1]) ∆Reduce —(c, 2)
Performance Input data size (GB) Time (s) About 10X faster
Performance ๏ Good up to a certain point ๏ Two factors dominate: • Space utilization • Time to shuffle deltas ๏ Insight: • The more downstream the filter is placed, the better the incremental performance • Especially beneficial if we can place it past the shuffle
Second Strategy Push code changes downstream
Commutative Rewrite Input aa b c aa c Map (aa, 1) (b, 1) (c, 1) (aa, 1) (c, 1) Shuffle (aa, [1, 1]) (b, 1) (c, [1, 1]) Reduce (aa, 2) (b, 1) (c, 2) Filter aa b c aa c filter(x=>x!=‘c’)
Commutative Rewrite Input aa b c aa c Map (aa, 1) (b, 1) (c, 1) (aa, 1) (c, 1) Shuffle (aa, [1, 1]) (b, 1) (c, [1, 1]) Reduce (aa, 2) (b, 1) (c, 2) Filter aa b c aa c filter(x=>x!=‘c’) But the input to the filter is (word, 1) We cannot use the filter anymore
Commutative Rewrite Input aa b c aa c Map (aa, 1) (b, 1) (c, 1) (aa, 1) (c, 1) Shuffle (aa, [1, 1]) (b, 1) (c, [1, 1]) Reduce (aa, 2) (b, 1) (c, 2) Filter aa b c aa c filter(x=>x!=‘c’) Observe that the map is invertible We can use the old filter by using the inverse of the map
Commutative Rewrite Input aa b c aa c Map (aa, 1) (b, 1) (c, 1) (aa, 1) (c, 1) Shuffle (aa, [1, 1]) (b, 1) (c, [1, 1]) Reduce (aa, 2) (b, 1) (c, 2) Filter’ aa b c aa c filter’((x, o)=>x!=‘c’) Rewritten filter
Commutative Rewrite Input aa b c aa c Map (aa, 1) (b, 1) (c, 1) (aa, 1) (c, 1) Shuffle (aa, [1, 1]) (b, 1) (c, [1, 1]) Reduce (aa, 2) (b, 1) (c, 2) Filter’ (aa, 2) (b, 1) filter((x, o)=>x!=‘c’) Shuffle and Reduce operations preserve keys
Performance Input data size (GB) Time About 1000X faster
Why does it scale so well? ๏ Runtime in the order of output ๏ Output depends on the number of unique words ๏ Unique words << total words
Combining Strategies ๏ Push the changed transform past as many shuffles as possible with rewrites • The new transform can be placed only after materialization points • By default we materialize shuffle output • Efficient because Spark already save shuffle output for fault tolerance ๏ Use delta computation for the remaining workflow
Vega ๏ Built on Spark and Spark SQL (only filter rewrite) ๏ Spark SQL API is unchanged ๏ Spark API includes: • Functions with inverses (for maps) • Inverse values (for incremental reduce) ๏ Automatically rewrites workflows using commutativity and incremental evaluation
๏ Titian provides to Spark users the ability of tracing through program execution ๏ Features: • Intermediate results are shared in memory • Tight integration with the Spark API (LineageRDD) • Low job overhead • Efficient lineage query ๏ Vega provides 1–3 orders magnitude performance gains over rerunning the computation from scratch ๏ Both provide results in a few seconds for many workflows allowing interactive usage } Transformation provenance Conclusions
Thank you
Outline ‣ Motivating Scenario ‣ Titian Programming Interface ‣ Internals ‣ Performance ‣ Conclusions
Configuration ๏ Two set of experiments: • Unstructured - grep and word count • Structured - PigMix queries ๏ Datasets: • Unstructured: from 500MB to 500GB files contains words generated using a Zipf distribution from a dictionary of 8000 words • Structured: we used the PigMix generator to create dataset of sizes ranging from 1GB to 1TB ๏ Configuration: • 16 4 cores (2 hyper threads per core) machines, 32GB of RAM, 1TB disk • Spark 1.2.1
Lineage Capture Runtime Overheads
Tracing Performance
๏ Titian provides to Spark users the ability of tracing through program execution at interactive speed ๏ Features: • Intermediate results are shared in memory • Tight integration with the Spark API (LineageRDD) • Low job overhead • Efficient lineage query ๏ We believe Titian will open the door to program logic debugging, iterative data (and program) cleaning, and exploratory analysis }Transformation provenance Titian: Data Provenance in Spark
Combiner LineageR DD Reducer LineageR DD Instrumented Workflow Hadoop LineageR DD count s pairscodes error s lines Stage 1 Stage 2 repor ts Stage LineageR DD
Capturing: HadoopLineageRDD Hadoop LineageRDD linesInput records Output records Input ID Output ID TaskCont ext
Capturing: HadoopLineageRDD Input records Output records Input ID Output ID Hadoop LineageRDD linesoffset1, “error 400 …” TaskCont ext
Capturing: HadoopLineageRDD Input records Output records Input ID Output ID Get input Id Hadoop LineageRDD linesoffset1, “error 400 …” TaskCont ext
Capturing: HadoopLineageRDD Input records Output records Input ID Output ID offset1 Get input Id Hadoop LineageRDD linesoffset1, “error 400 …” TaskCont ext
Capturing: HadoopLineageRDD Input records Output records Input ID Output ID offset1 “error 400 …” Hadoop LineageRDD lines TaskCont ext
Capturing: HadoopLineageRDD Input records Output records Input ID Output ID offset1 Get output Id Hadoop LineageRDD lines “error 400 …” TaskCont ext
Capturing: HadoopLineageRDD Input records Output records Input ID Output ID offset1 id1 Get output Id Hadoop LineageRDD lines “error 400 …” TaskCont ext
Capturing: HadoopLineageRDD Input records Output records Input ID Output ID offset1 id1 Save Hadoop LineageRDD lines “error 400 …” TaskCont ext
Capturing: HadoopLineageRDD Input records Output records Input ID Output ID offset1 id1 Save Hadoop LineageRDD lines “error 400 …” TaskCont ext id1
Capturing: HadoopLineageRDD Input records Output records Hadoop LineageRDD lines Input ID Output ID offset1 id1 offset2 offset2, “error 4 …” TaskCont ext id1
Capturing: HadoopLineageRDD Input records Output records Hadoop LineageRDD lines Input ID Output ID offset1 id1 offset2 id2 “error 4 …” TaskCont ext id2
Capturing: HadoopLineageRDD Input records Output records Hadoop LineageRDD lines Input ID Output ID offset1 id1 offset2 id2 offset3 offset3, “error 400 …” TaskCont ext id2
Capturing: HadoopLineageRDD Input records Output records Hadoop LineageRDD lines Input ID Output ID offset1 id1 offset2 id2 offset3 id3 “error 400 …” TaskCont ext id3
Combiner LineageR DD Reducer LineageR DD Instrumented Workflow Hadoop LineageR DD count s pairscodes error s lines Stage 1 Stage 2 repor ts Stage LineageR DD
Combiner LineageR DD Combiner Build Phase Hadoop LineageR DD pairscodes error s lines Stage 1 offset1, “error 400 …” Ke y Input IDs Ke y Agg Value
Combiner LineageR DD Combiner Build Phase Hadoop LineageR DD pairscodes error s lines Stage 1Input ID Output ID offset1 id1 “error 400 …” Ke y Agg Value Ke y Input IDs TaskCont ext id1
Combiner LineageR DD Combiner Build Phase Hadoop LineageR DD pairscodes error s lines Stage 1Input ID Output ID offset1 id1 “error 400 …” Ke y Agg Value Ke y Input IDs TaskCont ext id1
Combiner LineageR DD Combiner Build Phase Hadoop LineageR DD pairscodes error s lines Stage 1Input ID Output ID offset1 id1 400 Ke y Agg Value Ke y Input IDs TaskCont ext id1
Combiner LineageR DD Combiner Build Phase Hadoop LineageR DD pairscodes error s lines Stage 1Input ID Output ID offset1 id1 (400, 1) Ke y Agg Value Ke y Input IDs TaskCont ext id1
Combiner LineageR DD Combiner Build Phase Hadoop LineageR DD pairscodes error s lines Stage 1Input ID Output ID offset1 id1 Ke y Agg Value 40 0 1 Ke y Input IDs 400 TaskCont ext id1
Combiner LineageR DD Combiner Build Phase Hadoop LineageR DD pairscodes error s lines Stage 1Input ID Output ID offset1 id1 Ke y Agg Value 40 0 1 Ke y Input IDs 400 TaskCont ext id1
Combiner LineageR DD Combiner Build Phase Hadoop LineageR DD pairscodes error s lines Stage 1Input ID Output ID offset1 id1 Ke y Agg Value 40 0 1 Ke y Input IDs 400 { id1 } TaskCont ext id1
Combiner LineageR DD Combiner Build Phase Hadoop LineageR DD pairscodes error s lines Stage 1 Ke y Agg Value 40 0 1 Ke y Input IDs 400 { id1 } offset2, “error 4 …” Input ID Output ID offset1 id1 TaskCont ext id1
Combiner LineageR DD Combiner Build Phase Hadoop LineageR DD pairscodes error s lines Stage 1Input ID Output ID offset1 id1 offset2 id2 Ke y Agg Value 40 0 1 4 1 Ke y Input IDs 400 { id1 } 4 { id2 } TaskCont ext id2
Combiner LineageR DD Combiner Build Phase Hadoop LineageR DD pairscodes error s lines Stage 1Input ID Output ID offset1 id1 offset2 id2 Ke y Agg Value 40 0 1 4 1 Ke y Input IDs 400 { id1 } 4 { id2 } TaskCont ext id2 offset3, “error 400 …”
Combiner LineageR DD Combiner Build Phase Hadoop LineageR DD pairscodes error s lines Stage 1Input ID Output ID offset1 id1 offset2 id2 offset3 id3 Ke y Agg Value 40 0 2 4 1 Ke y Input IDs 400 { id1, id3} 4 { id2 } TaskCont ext id3
Combiner Probe Phase Input records Output records Input ID Output ID Combiner LineageRDD pairs TaskCont ext id3 Key Input IDs 400 { id1, id3 } 4 { id 2 } Ke y Agg Value 40 0 2 4 1
Combiner Probe Phase Input records Output records Input ID Output ID Combiner LineageRDD pairs TaskCont ext id3 Key Input IDs 400 { id1, id3 } 4 { id 2 } Ke y Agg Value 40 0 2 4 1 (400, 2)
Combiner Probe Phase Input records Output records Input ID Output ID Combiner LineageRDD pairs TaskCont ext id3 Key Input IDs 400 { id1, id3 } 4 { id 2 } Ke y Agg Value 40 0 2 4 1 (400, 2) Get output Id
Combiner Probe Phase Input records Output records Input ID Output ID {id1, id 3} 400 Combiner LineageRDD pairs TaskCont ext id3 Key Input IDs 400 { id1, id3 } 4 { id 2 } Ke y Agg Value 40 0 2 4 1 (400, 2) Get output Id
Combiner Probe Phase Input records Output records Input ID Output ID {id1, id 3} 400 { id2 } 4 Combiner LineageRDD pairs TaskCont ext id3 Key Input IDs 400 { id1, id3 } 4 { id 2 } Ke y Agg Value 40 0 2 4 1 (4, 1)
Combiner LineageR DD Reducer LineageR DD Instrumented Workflow Hadoop LineageR DD count s pairscodes error s lines Stage 1 Stage 2 repor ts Stage LineageR DD Input ID Output ID offset1 id1 TaskConte xt Id1 Input ID Output ID { id1, id 3} 400 { id2 } 4 (400, 2) (4, 1)
Combiner LineageR DD Reducer LineageR DD Instrumented Workflow Hadoop LineageR DD count s pairscodes error s lines Stage 1 Stage 2 repor ts Stage LineageR DD Input ID Output ID offset1 id1 TaskConte xt Id1 Input ID Output ID { id1, id 3} 400 { id2 } 4 (400, (2, p1))(4, (1, p1))
Combiner LineageR DD Reducer LineageR DD Instrumented Workflow Hadoop LineageR DD count s pairscodes error s lines Stage 1 Stage 2 repor ts Stage LineageR DD Input ID Output ID offset1 id1 TaskConte xt Id1 Input ID Output ID { id1, id 3} 400 { id2 } 4 (400, (2, p1))(4, (1, p1))(400, (5, p2)) …
Combiner LineageR DD Reducer LineageR DD Instrumented Workflow Hadoop LineageR DD count s pairscodes error s lines Stage 1 Stage 2 repor ts Stage LineageR DD Input ID Output ID offset1 id1 TaskConte xt Id1 Input ID Output ID { id1, id 3} 400 { id2 } 4 (400, (2, p1))(4, (1, p1))(400, (5, p2)) …
Combiner LineageR DD Reducer LineageR DD Instrumented Workflow Hadoop LineageR DD count s pairscodes error s lines Stage 1 Stage 2 repor ts Stage LineageR DD Input ID Output ID offset1 id1 TaskConte xt Id1 Input ID Output ID { id1, id 3} 400 { id2 } 4 (400, (2, p1))(4, (1, p1))(400, (5, p2)) … TaskConte xt 400 Input ID Output ID [ p1, p2 ] 400
Capturing: StageLineageRDD Stage LineageR DDInput records Output records Input ID Output ID TaskCont ext 400
Capturing: StageLineageRDD Stage LineageR DDInput records Output records Input ID Output ID TaskCont ext 400 (Bad request, 7)
Capturing: StageLineageRDD Stage LineageR DDInput records Output records Input ID Output ID TaskCont ext 400 Get input Id (Bad request, 7)
Capturing: StageLineageRDD Stage LineageR DDInput records Output records Input ID Output ID TaskCont ext 400 Get input Id (Bad request, 7)
Capturing: StageLineageRDD Stage LineageR DDInput records Output records Input ID Output ID 400 TaskCont ext 400 Get input Id (Bad request, 7)
Capturing: StageLineageRDD Stage LineageR DDInput records Output records Input ID Output ID 400 TaskCont ext 400 (Bad request, 7)
Capturing: StageLineageRDD Stage LineageR DDInput records Output records Input ID Output ID 400 TaskCont ext 400 Get output Id (Bad request, 7)
Get output Id Capturing: StageLineageRDD Stage LineageR DDInput records Output records Input ID Output ID 400 id1 TaskCont ext 400 (Bad request, 7)
Capturing: StageLineageRDD Stage LineageR DDInput records Output records Input ID Output ID 400 id1 4 TaskCont ext 4 (Failure, 1)
Capturing: StageLineageRDD Stage LineageR DDInput records Output records Input ID Output ID 400 id1 4 id2 TaskCont ext 4 (Failure, 7)

More Related Content

PDF
Sparkling Water Meetup
bySri Ambati
 
PPTX
Apache Spark
byMajid Hajibaba
 
PPTX
Advanced goldengate training ⅰ
byoggers
 
PPTX
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Iterative Spark Developmen...
byData Con LA
 
PDF
Spark workshop
byWojciech Pituła
 
PPTX
Apache spark Intro
byTudor Lapusan
 
PDF
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Real-time Aggregations, Ap...
byData Con LA
 
PDF
Joining the Club: Using Spark to Accelerate Big Data at Dollar Shave Club
byData Con LA
 
Sparkling Water Meetup
bySri Ambati
 
Apache Spark
byMajid Hajibaba
 
Advanced goldengate training ⅰ
byoggers
 
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Iterative Spark Developmen...
byData Con LA
 
Spark workshop
byWojciech Pituła
 
Apache spark Intro
byTudor Lapusan
 
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Real-time Aggregations, Ap...
byData Con LA
 
Joining the Club: Using Spark to Accelerate Big Data at Dollar Shave Club
byData Con LA
 

What's hot

PDF
Interactive Session on Sparkling Water
bySri Ambati
 
PDF
User Defined Aggregation in Apache Spark: A Love Story
byDatabricks
 
PDF
Why you should be using structured logs
byStefan Krawczyk
 
PDF
[QE 2018] Łukasz Gawron – Testing Batch and Streaming Spark Applications
byFuture Processing
 
PDF
Distributed computing with spark
byJavier Santos Paniego
 
PPTX
Writing Hadoop Jobs in Scala using Scalding
byToni Cebrián
 
ODP
Stratosphere Intro (Java and Scala Interface)
byRobert Metzger
 
PDF
Engineering Fast Indexes for Big-Data Applications: Spark Summit East talk by...
bySpark Summit
 
PDF
The Mechanics of Testing Large Data Pipelines (QCon London 2016)
byMathieu Bastian
 
PPT
Distributed Queries in IDS: New features.
byKeshav Murthy
 
PDF
Distributed Real-Time Stream Processing: Why and How 2.0
byPetr Zapletal
 
PDF
Introduction to Spark Datasets - Functional and relational together at last
byHolden Karau
 
PDF
Apache Spark & Streaming
byFernando Rodriguez
 
PDF
Scala introduction
byvito jeng
 
PDF
Beyond Shuffling, Tips and Tricks for Scaling Apache Spark updated for Spark ...
byHolden Karau
 
PDF
2014 09 30_sparkling_water_hands_on
bySri Ambati
 
PDF
Scaling up data science applications
byKexin Xie
 
PDF
Beyond Shuffling - Effective Tips and Tricks for Scaling Spark (Vancouver Sp...
byHolden Karau
 
PDF
Scalding - the not-so-basics @ ScalaDays 2014
byKonrad Malawski
 
PDF
Apache Spark - Key-Value RDD | Big Data Hadoop Spark Tutorial | CloudxLab
byCloudxLab
 
Interactive Session on Sparkling Water
bySri Ambati
 
User Defined Aggregation in Apache Spark: A Love Story
byDatabricks
 
Why you should be using structured logs
byStefan Krawczyk
 
[QE 2018] Łukasz Gawron – Testing Batch and Streaming Spark Applications
byFuture Processing
 
Distributed computing with spark
byJavier Santos Paniego
 
Writing Hadoop Jobs in Scala using Scalding
byToni Cebrián
 
Stratosphere Intro (Java and Scala Interface)
byRobert Metzger
 
Engineering Fast Indexes for Big-Data Applications: Spark Summit East talk by...
bySpark Summit
 
The Mechanics of Testing Large Data Pipelines (QCon London 2016)
byMathieu Bastian
 
Distributed Queries in IDS: New features.
byKeshav Murthy
 
Distributed Real-Time Stream Processing: Why and How 2.0
byPetr Zapletal
 
Introduction to Spark Datasets - Functional and relational together at last
byHolden Karau
 
Apache Spark & Streaming
byFernando Rodriguez
 
Scala introduction
byvito jeng
 
Beyond Shuffling, Tips and Tricks for Scaling Apache Spark updated for Spark ...
byHolden Karau
 
2014 09 30_sparkling_water_hands_on
bySri Ambati
 
Scaling up data science applications
byKexin Xie
 
Beyond Shuffling - Effective Tips and Tricks for Scaling Spark (Vancouver Sp...
byHolden Karau
 
Scalding - the not-so-basics @ ScalaDays 2014
byKonrad Malawski
 
Apache Spark - Key-Value RDD | Big Data Hadoop Spark Tutorial | CloudxLab
byCloudxLab
 

Viewers also liked

PDF
Big Data Day LA 2016/ Use Case Driven track - Hydrator: Open Source, Code-Fre...
byData Con LA
 
PPTX
Stream your Operational Data with Apache Spark & Kafka into Hadoop using Couc...
byData Con LA
 
PDF
Big Data Day LA 2016/ Data Science Track - The Right Tool for the Job: Guidel...
byData Con LA
 
PPTX
Explore big data at speed of thought with Spark 2.0 and Snappydata
byData Con LA
 
PDF
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Deep Learning at Scale - A...
byData Con LA
 
PPTX
Big Data Day LA 2016/ NoSQL track - Analytics at the Speed of Light with Redi...
byData Con LA
 
PPTX
Big Data Day LA 2016/ Big Data Track - Apply R in Enterprise Applications, Lo...
byData Con LA
 
PDF
Big Data Day LA 2016/ NoSQL track - Apache Kudu: Fast Analytics on Fast Data,...
byData Con LA
 
PPTX
Big Data Day LA 2016/ NoSQL track - MongoDB 3.2 Goodness!!!, Mark Helmstetter...
byData Con LA
 
PPTX
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Why is my Hadoop cluster s...
byData Con LA
 
PDF
Big Data Day LA 2016/ NoSQL track - Architecting Real Life IoT Architecture, ...
byData Con LA
 
PPTX
Big Data Day LA 2016/ Use Case Driven track - From Clusters to Clouds, Hardwa...
byData Con LA
 
PPTX
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Panel - Interactive Applic...
byData Con LA
 
PDF
Big Data Day LA 2016/ Use Case Driven track - How to Use Design Thinking to J...
byData Con LA
 
PDF
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Introduction to Kafka - Je...
byData Con LA
 
PPTX
Big Data Day LA 2016/ Data Science Track - Intuit's Payments Risk Platform, D...
byData Con LA
 
PDF
Big Data Day LA 2016/ Data Science Track - Backstage to a Data Driven Culture...
byData Con LA
 
PDF
Big Data Day LA 2016/ Use Case Driven track - Data and Hollywood: "Je t'Aime ...
byData Con LA
 
PDF
Big Data Day LA 2016/ NoSQL track - Introduction to Graph Databases, Oren Gol...
byData Con LA
 
PDF
Big Data Day LA 2016/ Use Case Driven track - The Encyclopedia of World Probl...
byData Con LA
 
Big Data Day LA 2016/ Use Case Driven track - Hydrator: Open Source, Code-Fre...
byData Con LA
 
Stream your Operational Data with Apache Spark & Kafka into Hadoop using Couc...
byData Con LA
 
Big Data Day LA 2016/ Data Science Track - The Right Tool for the Job: Guidel...
byData Con LA
 
Explore big data at speed of thought with Spark 2.0 and Snappydata
byData Con LA
 
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Deep Learning at Scale - A...
byData Con LA
 
Big Data Day LA 2016/ NoSQL track - Analytics at the Speed of Light with Redi...
byData Con LA
 
Big Data Day LA 2016/ Big Data Track - Apply R in Enterprise Applications, Lo...
byData Con LA
 
Big Data Day LA 2016/ NoSQL track - Apache Kudu: Fast Analytics on Fast Data,...
byData Con LA
 
Big Data Day LA 2016/ NoSQL track - MongoDB 3.2 Goodness!!!, Mark Helmstetter...
byData Con LA
 
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Why is my Hadoop cluster s...
byData Con LA
 
Big Data Day LA 2016/ NoSQL track - Architecting Real Life IoT Architecture, ...
byData Con LA
 
Big Data Day LA 2016/ Use Case Driven track - From Clusters to Clouds, Hardwa...
byData Con LA
 
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Panel - Interactive Applic...
byData Con LA
 
Big Data Day LA 2016/ Use Case Driven track - How to Use Design Thinking to J...
byData Con LA
 
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Introduction to Kafka - Je...
byData Con LA
 
Big Data Day LA 2016/ Data Science Track - Intuit's Payments Risk Platform, D...
byData Con LA
 
Big Data Day LA 2016/ Data Science Track - Backstage to a Data Driven Culture...
byData Con LA
 
Big Data Day LA 2016/ Use Case Driven track - Data and Hollywood: "Je t'Aime ...
byData Con LA
 
Big Data Day LA 2016/ NoSQL track - Introduction to Graph Databases, Oren Gol...
byData Con LA
 
Big Data Day LA 2016/ Use Case Driven track - The Encyclopedia of World Probl...
byData Con LA
 

Similar to Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Data Provenance Support in Spark, Matteo Interlandi, PostDoc, UCLA

PDF
Python business intelligence (PyData 2012 talk)
byStefan Urbanek
 
PPTX
Provenance for Data Munging Environments
byPaul Groth
 
PDF
Extracting Insights from Data at Twitter
byPrasad Wagle
 
PPTX
Spline: Data Lineage For Spark Structured Streaming
byVaclav Kosar
 
PPTX
The lifecycle of reproducible science data and what provenance has got to do ...
byPaolo Missier
 
PPTX
Zaharia spark-scala-days-2012
bySkills Matter Talks
 
PPTX
Provenance Week 2023 talk on DP4DS (Data Provenance for Data Science)
byPaolo Missier
 
PDF
Apache Spark Overview @ ferret
byAndrii Gakhov
 
PDF
WhereHows: Taming Metadata for 150K Datasets Over 9 Data Platforms
byMars Lan
 
PPT
Paper talk (presented by Prof. Ludaescher), WORKS workshop, 2010
byPaolo Missier
 
PPTX
"Data Provenance: Principles and Why it matters for BioMedical Applications"
byPinar Alper
 
PDF
New Analytics Toolbox
byRobbie Strickland
 
PPT
Enabling Precise Identification and Citability of Dynamic Data: Recommendatio...
byResearch Data Alliance
 
PPTX
Capturing and querying fine-grained provenance of preprocessing pipelines in ...
byPaolo Missier
 
PDF
Sharing massive data analysis: from provenance to linked experiment reports
byGaignard Alban
 
PDF
Workflow Provenance: From Modelling to Reporting
byRayhan Ferdous
 
PDF
Enabling Precise Identification and Citability of Dynamic Data: Recommendatio...
byLEARN Project
 
PDF
Crossing Analytics Systems: Case for Integrated Provenance in Data Lakes
byIsuru Suriarachchi
 
PDF
ETC & Authors in the Driver's Seat
byBertram Ludäscher
 
PPT
Provinance in scientific workflows in e science
bybdemchak
 
Python business intelligence (PyData 2012 talk)
byStefan Urbanek
 
Provenance for Data Munging Environments
byPaul Groth
 
Extracting Insights from Data at Twitter
byPrasad Wagle
 
Spline: Data Lineage For Spark Structured Streaming
byVaclav Kosar
 
The lifecycle of reproducible science data and what provenance has got to do ...
byPaolo Missier
 
Zaharia spark-scala-days-2012
bySkills Matter Talks
 
Provenance Week 2023 talk on DP4DS (Data Provenance for Data Science)
byPaolo Missier
 
Apache Spark Overview @ ferret
byAndrii Gakhov
 
WhereHows: Taming Metadata for 150K Datasets Over 9 Data Platforms
byMars Lan
 
Paper talk (presented by Prof. Ludaescher), WORKS workshop, 2010
byPaolo Missier
 
"Data Provenance: Principles and Why it matters for BioMedical Applications"
byPinar Alper
 
New Analytics Toolbox
byRobbie Strickland
 
Enabling Precise Identification and Citability of Dynamic Data: Recommendatio...
byResearch Data Alliance
 
Capturing and querying fine-grained provenance of preprocessing pipelines in ...
byPaolo Missier
 
Sharing massive data analysis: from provenance to linked experiment reports
byGaignard Alban
 
Workflow Provenance: From Modelling to Reporting
byRayhan Ferdous
 
Enabling Precise Identification and Citability of Dynamic Data: Recommendatio...
byLEARN Project
 
Crossing Analytics Systems: Case for Integrated Provenance in Data Lakes
byIsuru Suriarachchi
 
ETC & Authors in the Driver's Seat
byBertram Ludäscher
 
Provinance in scientific workflows in e science
bybdemchak
 

More from Data Con LA

PDF
Data Con LA 2022 - What's new with MongoDB 6.0 and Atlas
byData Con LA
 
PPTX
Data Con LA 2022 - Modernizing Analytics & AI for today's needs: Intuit Turbo...
byData Con LA
 
PPTX
Data Con LA 2022 - Why Data Quality vigilance requires an End-to-End, Automat...
byData Con LA
 
PDF
Data Con LA 2022 - Improving disaster response with machine learning
byData Con LA
 
PDF
Data Con LA 2022 - Intro to Data Science
byData Con LA
 
PDF
Data Con LA 2022 - Real world consumer segmentation
byData Con LA
 
PPTX
Data Con LA 2022- Embedding medical journeys with machine learning to improve...
byData Con LA
 
PPTX
Data Con LA 2022-Perfect Viral Ad prediction of Superbowl 2022 using Tease, T...
byData Con LA
 
PPTX
Data Con LA 2022 Keynote
byData Con LA
 
PPTX
Data Con LA 2022 - Data Streaming with Kafka
byData Con LA
 
PDF
Data Con LA 2022 Keynote
byData Con LA
 
PDF
Data Con LA 2022 - Using Google trends data to build product recommendations
byData Con LA
 
PDF
Data Con LA 2022 - Collaborative Data Exploration using Conversational AI
byData Con LA
 
PPTX
Data Con LA 2022 - Moving Data at Scale to AWS
byData Con LA
 
PPTX
Data Con LA 2022 - AI Ethics
byData Con LA
 
PDF
Data Con LA 2022 - Why Database Modernization Makes Your Data Decisions More ...
byData Con LA
 
PPTX
Data Con LA 2022 Keynotes
byData Con LA
 
PDF
Data Con LA 2022 - How are NFTs and DeFi Changing Entertainment
byData Con LA
 
PPTX
Data Con LA 2022 - Startup Showcase
byData Con LA
 
PPTX
Data Con LA 2022 Keynotes
byData Con LA
 
Data Con LA 2022 - What's new with MongoDB 6.0 and Atlas
byData Con LA
 
Data Con LA 2022 - Modernizing Analytics & AI for today's needs: Intuit Turbo...
byData Con LA
 
Data Con LA 2022 - Why Data Quality vigilance requires an End-to-End, Automat...
byData Con LA
 
Data Con LA 2022 - Improving disaster response with machine learning
byData Con LA
 
Data Con LA 2022 - Intro to Data Science
byData Con LA
 
Data Con LA 2022 - Real world consumer segmentation
byData Con LA
 
Data Con LA 2022- Embedding medical journeys with machine learning to improve...
byData Con LA
 
Data Con LA 2022-Perfect Viral Ad prediction of Superbowl 2022 using Tease, T...
byData Con LA
 
Data Con LA 2022 Keynote
byData Con LA
 
Data Con LA 2022 - Data Streaming with Kafka
byData Con LA
 
Data Con LA 2022 Keynote
byData Con LA
 
Data Con LA 2022 - Using Google trends data to build product recommendations
byData Con LA
 
Data Con LA 2022 - Collaborative Data Exploration using Conversational AI
byData Con LA
 
Data Con LA 2022 - Moving Data at Scale to AWS
byData Con LA
 
Data Con LA 2022 - AI Ethics
byData Con LA
 
Data Con LA 2022 - Why Database Modernization Makes Your Data Decisions More ...
byData Con LA
 
Data Con LA 2022 Keynotes
byData Con LA
 
Data Con LA 2022 - How are NFTs and DeFi Changing Entertainment
byData Con LA
 
Data Con LA 2022 - Startup Showcase
byData Con LA
 
Data Con LA 2022 Keynotes
byData Con LA
 

Recently uploaded

PDF
[BDD 2025 - Full-Stack Development] Agentic AI Architecture: Redefining Syste...
byWintari Yasmin
 
PPTX
"Feelings versus facts: why metrics are more important than intuition", Igor ...
byFwdays
 
PDF
[BDD 2025 - Mobile Development] Exploring Apple’s On-Device FoundationModels
byWintari Yasmin
 
PDF
Agentic Intro and Hands-on: Build your first Coded Agent
byUiPathCommunity
 
PDF
[BDD 2025 - Full-Stack Development] Digital Accessibility: Why Developers nee...
bywintari3
 
PDF
Cheryl Hung, Vibe Coding Auth Without Melting Down! isaqb Software Architectu...
byCheryl Hung
 
PDF
PCCC25（設立25年記念PCクラスタシンポジウム）：エヌビディア合同会社 テーマ２「NVIDIA BlueField-4 DPU」
byPC Cluster Consortium
 
PDF
Accessibility & Inclusion: What Comes Next. Presentation of the Digital Acces...
byAssociazione Digital Days
 
PPTX
MuleSoft AI Series : Introduction to MCP
byMulesoftMunichMeetup
 
PDF
ODSC AI West: Agent Optimization: Beyond Context engineering
byShashikant Jagtap
 
PDF
Open Source Post-Quantum Cryptography - Matt Caswell
byAll Things Open
 
PDF
Dev Dives: Build smarter agents with UiPath Agent Builder
byUiPathCommunity
 
PDF
How Much Does It Cost to Build an eCommerce Website in 2025.pdf
byPixlogix Infotech
 
PPTX
Support, Monitoring, Continuous Improvement & Scaling Agentic Automation [3/3]
byUiPathCommunity
 
PDF
So You Want to Work at Google | DevFest Seattle 2025
byMargaret Maynard-Reid
 
PDF
DUBAI IT MODERNIZATION WITH AZURE MANAGED SERVICES.pdf
byLogicEra
 
PDF
10 Best Automation QA Testing Software Tools in 2025.pdf
byRohitBhandari66
 
PDF
"DISC as GPS for team leaders: how to lead a team from storming to performing...
byFwdays
 
PDF
The partnership effect: Libraries and publishers on collaborating and thrivin...
byBookNet Canada
 
PDF
How Much Does It Cost To Build Software
bycollinmishell2
 
[BDD 2025 - Full-Stack Development] Agentic AI Architecture: Redefining Syste...
byWintari Yasmin
 
"Feelings versus facts: why metrics are more important than intuition", Igor ...
byFwdays
 
[BDD 2025 - Mobile Development] Exploring Apple’s On-Device FoundationModels
byWintari Yasmin
 
Agentic Intro and Hands-on: Build your first Coded Agent
byUiPathCommunity
 
[BDD 2025 - Full-Stack Development] Digital Accessibility: Why Developers nee...
bywintari3
 
Cheryl Hung, Vibe Coding Auth Without Melting Down! isaqb Software Architectu...
byCheryl Hung
 
PCCC25（設立25年記念PCクラスタシンポジウム）：エヌビディア合同会社 テーマ２「NVIDIA BlueField-4 DPU」
byPC Cluster Consortium
 
Accessibility & Inclusion: What Comes Next. Presentation of the Digital Acces...
byAssociazione Digital Days
 
MuleSoft AI Series : Introduction to MCP
byMulesoftMunichMeetup
 
ODSC AI West: Agent Optimization: Beyond Context engineering
byShashikant Jagtap
 
Open Source Post-Quantum Cryptography - Matt Caswell
byAll Things Open
 
Dev Dives: Build smarter agents with UiPath Agent Builder
byUiPathCommunity
 
How Much Does It Cost to Build an eCommerce Website in 2025.pdf
byPixlogix Infotech
 
Support, Monitoring, Continuous Improvement & Scaling Agentic Automation [3/3]
byUiPathCommunity
 
So You Want to Work at Google | DevFest Seattle 2025
byMargaret Maynard-Reid
 
DUBAI IT MODERNIZATION WITH AZURE MANAGED SERVICES.pdf
byLogicEra
 
10 Best Automation QA Testing Software Tools in 2025.pdf
byRohitBhandari66
 
"DISC as GPS for team leaders: how to lead a team from storming to performing...
byFwdays
 
The partnership effect: Libraries and publishers on collaborating and thrivin...
byBookNet Canada
 
How Much Does It Cost To Build Software
bycollinmishell2
 

Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Data Provenance Support in Spark, Matteo Interlandi, PostDoc, UCLA