Alluxio x Tobiko - ETL Happy Hour April 16, 2024 For more Alluxio events: https://alluxio.io/events/ Speaker: Toby Mao (CTO @ Tobiko Data) Writing efficient and correct incremental pipelines is challenging. Data practitioners who take on this challenge are viewed as performing an "advanced" function, which discourages broader teams from adopting incremental loads. In this lightning talk, CTO of Tobiko Data, Toby Mao, will demystify incremental loading data at scale. 

1. Correctly Loading Incremental Data at Scale
April 16, 2024
Copyright © 2024 Tobiko Data, Inc.

2. About Me
● Co-Founder & CTO at Tobiko Data
● Creator of SQLGlot
● Previously Airbnb, Netflix
● Experience with rec sys, xp,
semantics layer

3. What kind of data is incremental?
● Facts
○ Clicks, views, etc...
● Previous events immutable
● Can be quite large

4. What is not incremental?
● Dimensions
○ Users, billing info, etc...
● Rows can change over time
● Usually smaller

5. Should I avoid incremental loading?

6. Incremental loading exists for a reason

7. How do you read data incrementally?
maximal timestamp time partitions

8. Maximal timestamp
● If the table doesn’t exist, compute the whole history in one shot
● If the table does exist, query it to find the last processed timestamp
and then use that to filter the upstream source data.

9. Maximal timestamp pros / cons
● Doesn't require extra state
● It assumes that you’re able to load the entire table in one go
● The query is more complicated to write and maintain
● Custom SQL is often needed to handle incremental models
differently in development
● You cannot detect or fix data gaps

10. Time partitions
● Scheduler tracks what time ranges need to run and passes it to the
query

11. Time partitions pros / cons
● Requires extra state
● The queries are simpler
● Backfills are more scalable and reliable
● You can easily compute just one day of data or manually fix gaps

12. Initial load maximal timestamps vs partitions

13. ● Requires a primary key
● Simple to ensure consistency and no duplicates
● Can have performance issues
Merge - incremental by unique key

14. ● Very efficient
● Doesn't handle updates or late arriving data easily
Insert overwrite

15. Merge vs Insert overwrite
● Merge is expensive without partition pruning
● Merge is more effective when only a few records need to be updated
● Insert overwrite is inefficient when only a few rows have changed
● Insert overwrite doesn't need to match rows so can be more efficient

16. Reading late arriving data

17. Writing late arriving data
● Avoiding data leakage with insert overwrite
○ Filter your results to the expected range and insert the complete
range
● Use predicate push down with merge
○ Avoid full scanning your data

18. Thank you!