This document discusses using lakeFS to implement CI/CD best practices for data engineering. It describes how lakeFS can be used to: 1) Place all data assets under version control to enable building isolated test environments from previous snapshots. 2) Automatically create isolated test environments on demand from the versioned data to run integration and unit tests. 3) Integrate with testing frameworks like Great Expectations and custom tests using lakeFS hooks to automate data quality checks and enable continuous deployment by reverting changes if tests fail.

1. CI/CD for Data –
Building Dev/Test Data
Environments With lakeFS
Vino SD, Developer Advocate

2. About Me
■ I am Vino Duraisamy.
■ SWE -> Data/ML Engineer -> Developer Advocate
■ Open-source contributor @lakeFS
@vinodhini_sd @vinodhini-sd
vinodhini-sd.medium.com

3. A Day in the Life of a Data Engineer!
Expectation Reality

4. What Can Go Wrong?
■ EMR/Spark upgrade
■ Schema changes
■ Change in business logic
■ Infrastructure changes
■ Troubleshooting failed spark jobs
■ Backfilling historic data
■ Non-idempotent pipelines
■ Maintaining legacy DAGs/pipelines

6. ■ Unit Testing
■ Integration Testing
■ E2E Testing
■ Realistic Input Data
■ Mock data
■ Sampled production data
■ Copy all of Production data
Effective Testing Strategy

7. CI/CD for Data–
Apply engineering best
practices to data
engineering

8. Build
■ Place all data
assets under data
version control
Test
■ Create isolated
data environments
on-demand
■ Automate all your
tests
Deploy
■ Data quality checks
passed
■ Automatic rollbacks
are in place
CI/CD Best Practices

9. Build
■ Place all data
assets under data
version control
Test (CI)
■ Create isolated
data environments
on-demand
■ Automate all your
tests
Deploy
■ Data quality checks
passed
■ Automatic rollbacks
are in place
CI/CD Best Practices

10. Build
■ Place all data
assets under data
version control
Test (CI)
■ Create isolated
data environments
on-demand
■ Automate all your
tests
Deploy (CD)
■ Data quality checks
passed
■ Automatic rollbacks
are in place
CI/CD Best Practices

11. Build – Version
Control

12. Data Repo
Branches of the
whole repository
Data Versioning at
the object level
Place All Data Assets Under Version
Control

13. Git for Data with lakeFS
s3://data-repo/collections/foo
lakefs://data-repo/main/collections/foo
lakectl branch create
lakefs://repo/experiment-1
--source lakefs://repo/main
# output:
# created branch 'experiment-1',
# pointing to commit ID: 'd1e9adc71c10a’

14. How lakeFS Works

15. Test - CI

16. Ingestion
Experimentation
Safe CI: Create Isolated Data Environments
On-Demand

17. Automated Tests Using lakeFS Hooks

18. Demo Time

19. Deploy - CD

20. ■ Conceptually similar to Github actions
■ Hooks run as a standalone webserver - performs tests when triggered
■ Run custom tests or use test suites like Great Expectations, etc.,
Automating Data Quality Checks Using
lakeFS Hooks

21. ■ post_merge event: Failed
Hook runs can
programmatically revert
the changes to a branch.
■ Reliability of data in Prod.
$lakectl revert main^1
CD: Automatic Rollbacks in Place

22. Easily revert
corrupted
data.
Time travel
to debug a
snapshot
Safely test your
pipelines on an
isolated branch
from production
Production
identical
isolated
branch and
hooks to test
your data
Data Lakes with lakeFS

23. .io

24. Thank You