The document discusses the challenges faced by data scientists in sourcing, maintaining, and sharing datasets, highlighting that data preparation comprises roughly 80% of their work. It introduces the Splitgraph platform, which enables better management of datasets through Docker-like functionalities, including data ingestion, publication, usage, updating, and maintenance. Key features include delta compression for efficient updates and a focus on data provenance within metadata.

1. Splitgraph
"Docker for Data"
Artjoms Iškovs, Miles Richardson

2. "B.D." Building Packages Before Docker
The Dark Ages
• Sourcing packages
• Rebuilding,
reconfiguring,
rebuilding...
• Googling, rage
inducing

3. Data preparation
accounts for about
80% of the work
of data scientists.

4. Why so hard to build and maintain data sets?
• Sourcing data is not composable
• Why can’t I query multiple data sets at once?
• Wrangling and cleaning data is not maintainable
• Why can’t I keep my data sets up to date?
• Running ad-hoc queries is not reproducible
• Why can’t I share my data sets?

5. What do we mean by data?
Sources
• Open Data
• Internal Data
• Licensed Data
Types
• SQL Databases
• NoSQL Databases
• CSV Files...

6. The journey of a dataset: Scenario
• Two publishers:
• NOAA publishes climate data
• USDA publishes corn yields
• Consumer wants to merge both data sets
• Let’s follow the climate data...

7. The journey of a dataset: Introduction

8. The journey of a dataset: Introduction

9. The journey of a dataset 1: Creation

10. Ingesting data from another DB via CLI
$ sgr mount -t mongo_fdw me:pwd@my_db:27017 ’
{ "rainfall": {
"db": "observations",
"coll": "rainfall",
"schema": {
"timestamp": "timestamp",
"state": "varchar",
"rainfall": "numeric
} } }’ staging
$ sgr import staging
’SELECT timestamp, state, rainfall FROM rainfall’
noaa/climate rainfall

11. The journey of a dataset 2: Publication

12. Committing and Publishing Data via CLI
$ sgr publish noaa/climate data.splitgraph.com

13. The journey of a dataset 3: Usage

14. SGFiles: Dockerfiles for data
• Like Dockerfiles.
• Image: state of a database schema
• Layers w/ deterministic hashes and cache invalidation if:
• Previous layer changes
• Command changes
• Commands:
• FROM – base the image on something else
• IMPORT – import tables from another image
• SQL – run SQL against the image

15. Consumption: Demo
FROM usda/yields IMPORT crop_yields
FROM noaa/climate:latest IMPORT rainfall
SQL CREATE TABLE rainfall_yields AS
SELECT * FROM rainfall JOIN crop_yields ...

18. The journey of a dataset 4: Updating

19. The journey of a dataset 4: Updating
• Puerto Rico is now a US state
• NOAA wants to revise its climate data
• Can the consumer get just the changes?

20. Delta compression
• Only care about changes
• Need to efficiently:
• Create diffs (→ commit, push)
• Apply diffs (→ checkout, pull)

21. Delta compression
Docker
• Files
• Custom FS
Git
• Lines
• diff
Splitgraph
• Rows
• Audit triggers

22. Updating: Demo

29. The journey of a dataset 5: Maintenance

30. The journey of a dataset 5: Maintenance
• Can we update it?
• Where did this dataset come from?
• Build context fully encapsulated within the metadata

31. Provenance and rebasing demo

39. Q&A
twitter.com/splitgraph · splitgraph.com