The document discusses data testing principles and their integration into production and development workflows, emphasizing the importance of automated testing to catch issues early. It outlines different types of tests, including assertions and metric monitoring, and suggests various tools for each, such as dbt and datafold. The goal is to prevent breaking existing functionality during development while ensuring data integrity and accuracy in production.

1. Data testing
Gleb Mezhanskiy
CEO & Co-founder @ Datafold
Ex-Lyft Data PM

2. Agenda
1. Principles of data testing
2. Embedding testing in production & development workflows
3. Types of tests, pros/cons & tools

3. principles of effective
data testing3
Embed testing
in existing
workflows
Automate
everything
Cut
the noise
1 2 3

4. Data testing in production
Goal: catch issues as early and upstream as possible
Run ETL batch
Run tests
Tests
pass?
YES
NO
Notify
owners
Investigate
/ Fix
Publish
new data

5. What to test in production
Assertions
Metric
monitoring
checking
hard rules
detecting
anomalies
in metrics

6. When
to use
assertions
Value-level checks
> assert `email` is of xxx@yyy.com format
> assert `user_id` is unique and not-null
> assert SUM(source.revenue) = SUM(target.revenue)
Integrity checks
Balance checks

7. Tools for running assertions
Embedded in ETL tools
> dbt for SQL
> Dagster for general ETL
Standalone
> great_expectations for SQL
> deequ for Spark

8. Hard rules don’t work for metrics because
of natural variance, trend and seasonality!
When to use metric monitoring

9. Hard rules don’t work for metrics because
of natural variance, trend and seasonality! Answer:
apply a bit of ML!
When to use metric monitoring

10. Tools for metric monitoring
Prophet
by Facebook
Datafold
Alerting

11. Data testing in development
Goal: do no harm – prevent breaking things that work
Build & Backfill
Run tests
Tests
pass?
YES
NO
Code
review
Deploy
Investigate
/ Fix

12. How to test in development
Assertions Data diff
checking
hard rules
– just like
in production!
visualizing
changes
in data

14. Data diff = git diff for data
Compares values
Production
Development

15. …and distributions

16. Remember –
automate!

17. Diff tools
> dbt-audit-helper
> BigDiffy by Spotify
> Datafold Diff

18. Bottom line
Development Production
What changes in between tests
Goal
Frequency
Trigger
Methods
Assertions
Data diff
Assertions
Metric monitoring
On every new data batchOn every commit / PR
Github/lab + CI ETL orchestrators
Prevent regressions
Source code Data
Learn about issues ASAP