We want to share our story of migration of our storage in BigQuery to a new partitioning schema, what we’ve learned on the way and what we have achieved at the end. Speaker - Derar Bakr, Senior Data Engineer @ AppsFlyer Data Group.

1. 1
The Google BigQuery Story:
Optimizing 25PB Storage

2. Software Engineer @AppsFlyer
Interests:
• Reactive Programming
• Non-blocking algorithms
• Distributed systems and DBs
Who am I
2
Derar Bakr

3. • Analytical storage with over 25PB of data
• Serves raw data reports
• Used for internal analytics
BigQuery in AppsFlyer
3

4. Highly scalable
Powerful toolset
Fully managed
Fast feeding
Fast SQL queries
Why

5. BigQuery Internals
• Dremel Implementation
• Columnar Storage
• Tree Architecture
5

6. Columnar Storage
6

7. Columnar Storage
• Less traffic
• Efficient scans (SIMD instructions)
• Highly compressible
• No updates
7

8. Tree Architecture
Massively Parallel Execution
8

9. • Legacy SQL dialect
• Table-based partitioning
• Manual data retention
Legacy Storage Overview
9

10. Legacy Storage Overview
10
• installs_ios_google_20181001
• installs_ios_google_20181002
• …
• installs_ios_google_20190101

11. • Reduce slots consumption == $
• Optimize query time
• Improve table management
The Goal
11

12. • Legacy SQL -> Standard SQL
• Modern partitioning
• Implement clustered tables
• Analyze and restructure queries
The Solution
12

13. • installs_ios_google_20181001
• installs_ios_google_20181002
• …
• installs_ios_google_20190101
• …
Change Partitioning
13
installs_ios_google
partitioned by event date

14. • Organizes data inside files
• Adjacent data next to each other
• We always have `app_id` predicate in queries
Clustered Tables
14

15. How to migrate
with zero downtime?

16. • Query time reduce
• Slots consumption reduce
• Easier tables management
Achievements
16

17. 1. Create new tables, start writing
2. Migrate legacy data (duplicate storage)
3. Test
4. Switch to new tables
5. Repeat steps 1-4 for all topics
Migration Process
17

18. • Data copying process
• Storage duplication
• GDPR
• SQL Dialect change
Challenges
18

19. • Making schema strongly typed
• Simplify queries (no regex)
• Restructuring queries
What didn’t help
19

20. • https://www.appsflyer.com/product/overview/
• https://cloud.google.com/bigquery/docs/
• https://cloud.google.com/files/BigQueryTechnicalWP.pdf
• https://cloud.google.com/bigquery/docs/clustered-tables
Links
20

21. Thanks!
derar@appsflyer.com