Pachyderm is a platform that utilizes containers to manage big data applications, offering features like version control, data processing through batched and streaming methods, and integration with object storage services. The document discusses the role of containers and Kubernetes in enhancing data processing efficiency and collaboration. It also outlines the functionality of Pachyderm's containerized data pipelines and provides a demo for using the platform on Kubernetes.

1. Pachyderm
Big Data Applications with Containers
Joe Doliner
Founder & CEO
jd@pachyderm.io

2. About me

3. What is Pachyderm?
Big data with Containers
• Version control for data
• Uses containers for data processing
• Batched and streaming
• Data lives in object storage (S3, GCS, Ceph)
• Shares no code with Hadoop

4. Intro to Containers
• What are containers and why are they
useful for Big Data Applications?
• What is Kubernetes and why is it useful
for Big Data Application?

5. Version Control for Data
• View diffs of data
• Collaboration
• Data provenance
Commit
0
Commit
1
Commit
2
Commit
3
Commit
4
Git for huge data sets

6. Containerized Data Pipelines
• DAG of jobs
• Pipelines triggered by data
changes
• Processing efficiency
• Incrementality
Task 1
Task 2 Task 3
Task 4
Dashboard
Task 5
Task 6

7. Data Lake Use Case
Production
databases
Application
Pachyderm
Object storage
Pachyderm Pipelines
Data
Warehouse
Pachyderm File System

8. Demo:
• Start Pachyderm on K8s
• Add data
• Create pipeline
• Incrementally update steaming results
Steaming data pipelines

9. Thank You!
Questions?
pachyderm.io
jd@pachyderm.io
github.com/pachyderm/pachyderm