The document discusses strategies for conducting zero downtime migrations at scale, particularly for large datasets involving over 100 million users and significant backup data. Key considerations include dual writes architecture, intermediate state development, and validating the migration process to ensure system responses remain consistent. The process emphasizes careful scaling, incremental rollouts, and enhanced visibility into the system during the migration phase.

1. Zero Downtime Migrations
at Scale
Aysylu Greenberg ∽ April 28, 2018 ∽ Medellín
Photo by Iván Erre Jota / CC BY-SA

2. Aysylu Greenberg
@aysylu22

3. Zero Downtime Migrations
at Scale

4. Zero Downtime Migrations
at Scale

5. Data
Migration
Architecture
Migration

9. Data
Migration
Architecture
Migration

12. Zero Downtime Migrations
at Scale

13. Zero Downtime Migrations
at Scale
We will be performing scheduled site
maintenance on Saturday from 3 am to 5 am!

14. Zero Downtime Migrations
at Scale

15. New data storage
and architecture
Zero-Downtime
Migration for
Backups

16. Zero Downtime Migrations
at Scale

17. Considerations Before Migration
● Scale: O(100M) of users with 1B+ backups,
representing 1T+ objects

18. Considerations Before Migration
● Scale: O(100M) of users with 1B+ backups,
representing 1T+ objects
● No sharing, no search, no folders

19. Considerations Before Migration
● Scale: O(100M) of users with 1B+ backups,
representing 1T+ objects
● No sharing, no search, no folders
● Mainly write traffic, read traffic is rare and high
priority

20. Considerations Before Migration
● Scale: O(100M) of users with 1B+ backups,
representing 1T+ objects
● No sharing, no search, no folders
● Mainly write traffic, read traffic is rare and high
priority
● Spanner: global strong consistency, SQL

21. Considerations Before Migration
● Scale: O(100M) of users with 1B+ backups,
representing 1T+ objects
● No sharing, no search, no folders
● Mainly write traffic, read traffic is rare and high
priority
● Spanner: global strong consistency, SQL
● Backups can get large

22. Considerations Before Migration
● Scale: O(100M) of users with 1B+ backups,
representing 1T+ objects
● No sharing, no search, no folders
● Mainly write traffic, read traffic is rare and high
priority
● Spanner: global strong consistency, SQL
● Backups can get large

23. So how do you
migrate all this data?
Photo by Richard Evea / CC BY-SA 2.0

24. So how do you
migrate all this data?
copy/paste...
Photo by Richard Evea / CC BY-SA 2.0

25. Zero-Downtime
Migration for
Backups
Data migration
&
Architecture migration

26. Data migration
● Dual writes
Architecture migration
So how do you migrate all this data?

27. Data migration
● Dual writes
Architecture migration
So how do you migrate all this data?
Are we
storing data
correctly?

28. Data migration
● Dual writes
Architecture migration
● Dual writes
So how do you migrate all this data?

29. Data migration
● Dual writes
Architecture migration
● Dual writes
So how do you migrate all this data?
No effect on
latency and
error rates!

30. Data migration
● Dual writes
● Backfill data
Architecture migration
● Dual writes
So how do you migrate all this data?

31. Data migration
● Dual writes
● Backfill data
Architecture migration
● Dual writes
So how do you migrate all this data?
Do we
understand all
client behavior
and adapt the
data correctly?

32. Data migration
● Dual writes
● Backfill data
Architecture migration
● Dual writes
● Prove the stack
So how do you migrate all this data?

33. Data migration
● Dual writes
● Backfill data
Architecture migration
● Dual writes
● Prove the stack
So how do you migrate all this data?
Is the response
from the new
system same as
from the old?

34. Data migration
● Dual writes
● Backfill data
● Learn the new storage
Architecture migration
● Dual writes
● Prove the stack
So how do you migrate all this data?

35. Data migration
● Dual writes
● Backfill data
● Learn the new storage
Architecture migration
● Dual writes
● Prove the stack
So how do you migrate all this data?
New storage
mechanism or
schema?

36. Data migration
● Dual writes
● Backfill data
● Learn the new storage
Architecture migration
● Dual writes
● Prove the stack
● Harden the system
So how do you migrate all this data?

37. Data migration
● Dual writes
● Backfill data
● Learn the new storage
Architecture migration
● Dual writes
● Prove the stack
● Harden the system
So how do you migrate all this data?
How to get it to
production
readiness to
serve full load?

38. Data migration
● Dual writes
● Backfill data
● Learn the new storage
● Migrate slowly
Architecture migration
● Dual writes
● Prove the stack
● Harden the system
So how do you migrate all this data?

39. Data migration
● Dual writes
● Backfill data
● Learn the new storage
● Migrate slowly
Architecture migration
● Dual writes
● Prove the stack
● Harden the system
So how do you migrate all this data?
Validate, validate,
validate
Resource constraints?
Scale migration

40. Data migration
● Dual writes
● Backfill data
● Learn the new storage
● Migrate slowly
Architecture migration
● Dual writes
● Prove the stack
● Harden the system
● Roll out slowly
So how do you migrate all this data?

41. Data migration
● Dual writes
● Backfill data
● Learn the new storage
● Migrate slowly
Architecture migration
● Dual writes
● Prove the stack
● Harden the system
● Roll out slowly
So how do you migrate all this data?
Scale carefully
& proactively

42. Zero-Downtime Migrations at Scale
FOCUS ON INTERMEDIATE STATE

43. Zero-Downtime Migrations at Scale
FOCUS ON INTERMEDIATE STATE
● Prepare to write code for intermediate state
>>> Quality of code corresponds to the expected lifetime

44. Zero-Downtime Migrations at Scale
FOCUS ON INTERMEDIATE STATE
● Prepare to write code for intermediate state
>>> Quality of code corresponds to the expected lifetime
● Migrate backends first

45. Zero-Downtime Migrations at Scale
FOCUS ON INTERMEDIATE STATE
● Prepare to write code for intermediate state
>>> Quality of code corresponds to the expected lifetime
● Migrate backends first

46. Zero-Downtime Migrations at Scale
FOCUS ON INTERMEDIATE STATE
● Prepare to write code for intermediate state
>>> Quality of code corresponds to the expected lifetime
● Migrate backends first

47. Zero-Downtime Migrations at Scale
FOCUS ON INTERMEDIATE STATE
● Prepare to write code for intermediate state
>>> Quality of code corresponds to the expected lifetime
● Migrate backends first
● Invest into visibility into system & migration state

48. Zero-Downtime Migrations at Scale
FOCUS ON INTERMEDIATE STATE
ROLL OUT INCREMENTALLY

49. Zero-Downtime Migrations at Scale
FOCUS ON INTERMEDIATE STATE
ROLL OUT INCREMENTALLY
● Validate scalability while affecting fewest users

50. Zero-Downtime Migrations at Scale
FOCUS ON INTERMEDIATE STATE
ROLL OUT INCREMENTALLY
● Validate scalability while affecting fewest users
● Decouple launch of services

51. Zero-Downtime Migrations at Scale
FOCUS ON INTERMEDIATE STATE
ROLL OUT INCREMENTALLY
● Validate scalability while affecting fewest users
● Decouple launch of services

52. Zero-Downtime Migrations at Scale
FOCUS ON INTERMEDIATE STATE
ROLL OUT INCREMENTALLY
● Validate scalability while affecting fewest users
● Decouple launch of services

53. Zero-Downtime Migrations at Scale
FOCUS ON INTERMEDIATE STATE
ROLL OUT INCREMENTALLY
VALIDATE & PRACTICE ROLLOUT

54. Zero-Downtime Migrations at Scale

55. Zero-Downtime Migrations at Scale

56. Zero-Downtime Migrations at Scale

57. Zero-Downtime Migrations at Scale

58. Zero-Downtime Migrations at Scale

59. Zero-Downtime Migrations at Scale

60. Zero-Downtime Migrations at Scale
FOCUS ON INTERMEDIATE STATE
ROLL OUT INCREMENTALLY
VALIDATE & PRACTICE ROLLOUT

61. Gratitude
Steve Clark
Thomas Escobar
Matt Welsh
Ranjodh Mathial
Tatiana Marquez

62. Zero Downtime Migrations
at Scale
Aysylu Greenberg ∽ April 28, 2018 ∽ Medellín
Photo by Iván Erre Jota / CC BY-SA