Download as PDF, PPTX
Uploaded byDatabricks
Building a Distributed Collaborative Data Pipeline with Apache Spark
The document discusses the challenges of data management and reproducibility in scientific research, highlighting issues exemplified by the retraction of a hydroxychloroquine study due to data inconsistencies. It advocates for a new data supply chain protocol, Open Data Fabric, to enhance reproducibility and transparency in data handling through a framework-agnostic, stream processing approach. The author calls for a paradigm shift in how data is treated, emphasizing the need for strict ownership of data validity and the adoption of temporal data models.
More Related Content
Similar to Building a Distributed Collaborative Data Pipeline with Apache Spark
Building a Distributed Collaborative Data Pipeline with Apache Spark
- 1. Building a DistributedCollaborative Data Pipeline with Apache Spark Sergii Mikhtoniuk Founder @ Kamu Data Inc.
- 2. 2 Example: Hydroxychloroquine Study Publishedin "The Lancet" journal in May Data: ⬝ > 96,000 COVID-19 patients ⬝ 671 hospitals worldwide ⬝ Proprietary database: Surgisphere Finding: Increased risk of in-hospital mortality Global trials are halted completely * https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(20)31180-6/fulltext
- 3. 3 Example: Hydroxychloroquine Study(cont.) Publication retracted in June ⬝ After numerous data inconsistencies pointed out ⬝ Data provenance could not be established ⬝ Surgisphere database goes offline A data management issue ⬝ Was ignored for years ⬝ Derailed life-saving efforts in critical time Just one of many examples…
- 4. 4 Reproducibility Crisis Reproducibility isthe foundation of the scientific method Nearly-impossible to achieve in practice Problem originates in our data management practices * https://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970
- 5. Reproducibility & Verifiability inModern Data
- 6. Two Kinds ofData Derivative DataSource Data 6 Generated by some system ⬝ Or a result of observations Cannot be reconstructed if lost Publishers ⬝ Have full authority ⬝ Responsible for validity Created from other data via some transformations ⬝ Aggregates, summaries, ML etc. Can be reconstructed if lost Validity depends on source data
- 7. 7 Source Data: Expectation Twounrelated parties at different times should be able to access the same data and validate that it comes unaltered from the trusted source Stable References Validation Mechanism
- 8. Source Data: Reality TemporalDatasetsNon-Temporal Datasets 8 Store "current state" of the domain Destructive in-place updates Constant loss of history Store events / observations In-place updates Same URL - different data
- 9. 9 Derivative Data: Expectation Repeatingall transformation steps produces same results as the original. Having data, any party can verify that it was produced by declared transformations without being accidentally or maliciously altered. Transparency Determinism
- 10. 10 Derivative Data: Reality Reproducibilityis a very manual process (workflows, compliance) Requires: ⬝ Stable references to source data ⬝ Reproducible environments ⬝ Code, libraries, transitive dependencies, OS, hardware ⬝ Self-contained projects ⬝ No external API calls ⬝ Eliminating randomness When time is pressing - it goes out of the window
- 11. 11 Examples Sharing data onKaggle or Github ⬝ Goal: Collaborating on data cleaning ⬝ Reality: Cannot be trusted Data hubs & portals ⬝ Goal: improve discoverability & break down silos ⬝ Reality: Validity cannot be established Copy + Version approach ⬝ Works only within the enterprise "bubble" ⬝ Doesn’t work between independent parties
- 12. 12 Summary We are mismanagingdata on the most fundamental level Collaboration on data is impossible - forward progress is constantly lost Reached the limit of workflows - need a technical solution
- 13. 13 Is There aBetter Way? We set out to design a new kind of data supply chain designed from ground up for Reproducibility & verifiability Low latency Complete provenance Data reuse and collaboration
- 14.
- 15. ODF is aprotocol specification for reliable data exchange and transformation The world’s first P2P data pipeline! 15 Introducing Open Data Fabric
- 16. 16 Strict Definition of"Data" There is no such thing as "current state" - only history ⬝ Data is temporal History doesn’t change ⬝ Data is immutable Future decision making relies on history ⬝ Data must have infinite retention Time is relative, information propagates with a delay ⬝ Use bitemporality to reflect that
- 17. 17 Data Model Dataset -a potentially infinite stream of events / observations Every event has ⬝ Event time - when it occured in the outside world ⬝ System time - when it was first observed by the system (monotonic) Stable Ref = (dataset_id, system_time) Validity = (stable_ref, hash)
- 18.
- 19. Dataset Manifests Derivative DatasetRootDataset 19
- 20. 20 Temporal Metadata Metadata isa series of events ⬝ Stores everything that has ever influenced the data Key enabler of: ⬝ Reproducibility ⬝ Verifiability ⬝ Dataset evolution ⬝ Efficient data sharing ⬝ Fine-grained provenance (WIP)
- 21. 21 Metadata Chain Blockchain-like Cryptographically secured Linkedto data Extensible ⬝ Semantics / Ontology ⬝ Governance / Licensing ⬝ Privacy / Security ⬝ …
- 22.
- 23. 23 Batch processing isunfit for purpose! ⬝ Relies on simplicity of non-temporal data ⬝ Ignores temporal problems ⬝ Late / out-of-order arrivals ⬝ Backfills & corrections ⬝ Arrival cadence differences in joins Using batch would be ⬝ Extremely error-prone ⬝ Impossible to audit Goodbye Batch
- 24. 24 Hello Stream Processing! Isdesigned for temporal problems ⬝ Event-time processing (bitemporality) ⬝ Watermarks, Triggers ⬝ Windowing ⬝ Projections ⬝ Stream-to-Stream Joins ⬝ Projection (Temporal-Table) Joins Useful only for near real-time processing? ⬝ A common misconception!
- 25. 25 Stream Processing asthe Primary Transformation Method Cons Agnostic of how and how often data arrives Declarative and expressive - easier to audit Better for determinism and reproducibility Minimal latency Unfamiliarity Limited framework support Pros SELECT TUMBLE_ROWTIME(order_time, INTERVAL '1' DAY), order_id, count(*) as num_shipments, sum(shipped_quantity) as shipped_total FROM order_shipments GROUP BY TUMBLE(order_time, INTERVAL '1' DAY), order_id
- 26. 26 Execution Environment ODF isframework-agnostic Strict versioning + Sandboxing ensures reproducibility
- 27. 27 Data Sharing Implications Datastorage ⬝ Root - durable and highly available (e.g. S3, GS, HDFS) ⬝ Derivative - cheap, unreliable, or none at all ⬝ A form of caching ⬝ Can be fully reconstructed from metadata and root data Metadata ⬝ A digital passport of data ⬝ Used for verifying data integrity ⬝ Validating work of another peer to spot malicious activity ⬝ Exchanged securely between peers
- 28.
- 29. ODF coordinator Single-binary app ⬝Written in Rust! Two prototype engines: ⬝ Apache Spark ⬝ Apache Flink Convenience features: ⬝ SQL Shell ⬝ Notebooks Alpha-quality but actively developed 29 kamu-cli
- 30.
- 31.
- 32.
- 33.
- 34. 34 ODF Summary Data pipelinedesigned around properties essential for collaboration ⬝ Addresses decades of stagnation in data management A much stricter model ⬝ No API calls, no "black box" transformations Requires a mindset shift ⬝ Redefining what data is and how we treat it ⬝ Embracing temporal problems
- 35. 35 The Next Levelof Data Democratization One of the pillars of the Digital Democracy and the next-generation decentralized IT Supply of factual data for blockchain contracts ⬝ With no central authority ⬝ Resistant to malicious behavior Foundation for data monetization ⬝ Publisher incentives Data provider for next generation web & apps
- 36. 36 References kamu-cli: https://github.com/kamu-data/kamu-cli/ Open DataFabric: http://opendatafabric.org/ Blog: https://www.kamu.dev/blog/ Email: smikhtoniuk@kamu.dev
- 37. 37 Call for Action Datais our modern age history book - it should be treated as such ⬝ Stop modifying and copying data ⬝ Don’t version data - use bitemporal modelling instead Data publishers have to take ownership of reproducibility ⬝ Need good standards and tools Non-temporal data is a local optima ⬝ Temporal data should be the default choice Stream processing will displace batch ⬝ Let’s collaborate on improving tools not oversimplifying problems!
- 38. Feedback Your feedback isimportant to us. Don’t forget to rate and review the sessions.