"We can all agree that streaming is super cool. And for a while now, the adoption conversation has been largely led with an all-in mentality. But that’s silly. The only concerns end users have are: -The freshness of their data -Latency they require to meet their SLAs from source to consumption -All while maintaining data quality and governance. Luckily, the industry has realized this and we have seen a shift of streaming capabilities surfacing as an in-database technology, via objects as trivial to analytics engineers as views - materialized that is. With this convergence of streaming capabilities and batch level accessibility, this is when ELT tools like dbt can join in and expand out the adoption story. dbt is the T in ELT, Extract Load and Transform. In dbt, analytics engineers design models - SQL (and occasional python) statements that encapsulate business logic. At runtime, dbt will wrap that logic in a DDL statement and send it over to the data platform to execute. In this session, we’ll discuss how we see streaming at dbt Labs. We will dive into how we are extending dbt to support low-latency scenarios and the recent additions we have made to make batch and streaming allies in a DAG rather than archenemies."

1. Streaming is a Detail
Current 2023
Amy Chen & Florian Eiden
1

2. 2
Introductions
Amy Chen
Staff Partner Engineer
Fun fact: Iʼve made dbt soap 🧼
Florian Eiden
Staff Product Manager
Fun fact: I live on an 🏝

3. Pieces of the puzzle
- Transactional vs Analytical
- Analytics Engineering vs Data Engineering
- Personas
- ELT vs ETL
- Streaming in the ELT world (bring streaming to the database, or the other way around)
- Operational Analytics
- Why does it have to be Batch vs Streaming?
- Is streaming the biggest trend in analytics?
- What is the next big milestone for streaming?
- How do we solve CI/CD? Testing? Replayability?
- Is Flink a database now?
- Is pipeline the right way to bundle logic? What about DAGs?
- Logic: plumbing vs business logic

5. dbt is ELT
5

6. The dbt viewpoint:
Build data like
developers build
applications
6
dbt uses testing, version
control, reusable code, and
documentation to get to the
right answer, faster.
Work like engineers
Pairing code-based
transformation with your
favorite git provider means
flexibility without chaos.
Code Reigns

7. Write reusable & referenceable logic with SQL + Jinja

8. Infer lineage for automated dependency management.

9. 9
SQL-friendly +
version control safely
expands participants
Reusable code
speeds development
Built-in CI/CD
increases pipeline
reliability
Dependency
management speeds
troubleshooting
01
Visible lineage
increases data
understanding
Testing and
documentation
increase data trust
02 03 04 05 06
Data Engineers,
Analysts, and Data
Scientists
Collaborative Code Dashboard A =
Dashboard B
Automatic
Documentation
Analysts and
Business Users

10. Leverage your
existing cloud data
platform, with
out-of-the-box
adapters to all
major warehouses.
Benefit from
partnerships
across the Modern
Data Stack.
10
Data Quality
Orchestration
Data Ingestion
Other
Cloud Data Platform
Analysis & Visualization
Data Catalog & Active Metadata
Operational Analytics
Develop Test &
Document
Deploy

11. MVs for everyone!

12. The questions on when to use MVs
What are the costs associated with running the materialized view versus a batched incremental model?
(this will vary depending on your data platform as some will require different compute nodes)
Does your data platform support joins, aggregations, and window functions on MVs if you need them?
What are the latency needs of your development environment? In production? (If not near real time, you
can make the choice between a batch incremental model or a MV with a longer refresh schedule.)
How often do your upstream dependencies update? If your answer is not frequent, you may not need a
MV.
How large is your dataset?(It might be cheaper to use MVs for extremely large datasets)
How often do you need your query refreshed? What are your downstream dependencies and their
stakeholders? (If near real time is important, MVs might be the right choice).
Do you have real time machine learning models training or applications using your transformed dataset?