Zero Downtime Migration

1. Confidential + Proprietary Zero Downtime Migration How to Replace a Jet Engine of Your System In-Flight Aysylu Greenberg Google, October 19, 2017

2. Aysylu Greenberg @aysylu22

3. Zero Downtime Migration

5. Data Migration Old Data Storage New Data Storage

6. Data Migration Old Data Storage New Data Storage copy Parity verification

7. Server Client Client Service A Service B Service C Architecture Migration

9. Build System Story Photo by imanka / CC BY

10. Towards Distributed Build System

16. Continuous integration system BuildRabbit Source System Google engineers & teams Release infrastructure Integration testing infrastructure Build artifact storage Blaze

17. Architecture Upgrade Photo by Magyshadow / CC BY-SA 4.0

18. Architecture Upgrade Photo by vermazeren / CC BY-SA 2.0

19. Old Architecture: Client / Server Client for User BuildRabbit Scheduler BuildRabbit Worker

20. Client for User Persistent Queue BuildRabbit Worker Build Artifacts Build Progress Info New Architecture: Build Service

21. Persistent Queue BuildRabbit Worker Build Artifacts Build Progress Info Client for User New Architecture: Build Service

22. Client for User BuildRabbit Scheduler BuildRabbit Worker

23. Persistent Queue BuildRabbit Worker Build Progress InfoBuild Artifacts Client for User Client for User BuildRabbit Scheduler BuildRabbit Worker

24. Persistent Queue BuildRabbit Worker Build Progress InfoBuild Artifacts Client for User Client for User BuildRabbit Scheduler BuildRabbit Worker RESILIENCE LOGIC RESILIENCE LOGIC

25. Continuous integration system BuildRabbit Source System Google engineers & teams Release infrastructure Integration testing infrastructure Build artifact storage Blaze

26. Migration with Zero Downtime Photo by moonjazz / CC B

27. Photo by la_bretagne_a_paris / CC BY SA FOCUS ON INTERMEDIATE STATE

28. Prepare to write code for intermediate state FOCUS ON INTERMEDIATE STATE

29. Client for User BuildRabbit Scheduler BuildRabbit Worker Persistent Queue

30. Prepare to write code for intermediate state FOCUS ON INTERMEDIATE STATE >>> Quality of code corresponds to its expected lifetime

31. Old Data Storage New Data Storage copy Parity verification

32. Migrate backends first FOCUS ON INTERMEDIATE STATE

34. Server Client Old Data Storage New Data Storage copy

35. Visibility into system state: metrics & verification FOCUS ON INTERMEDIATE STATE

36. Prepare to write code for intermediate state Migrate backends first Visibility into system state: metrics & verification FOCUS ON INTERMEDIATE STATE Photo by la_bretagne_a_paris / CC BY SA

37. Photo by brunogirin / CC BY SA ROLL OUT INCREMENTALLY

38. Validate scalability while affecting fewest users ROLL OUT INCREMENTALLY

39. Client Server Old Data Storage New Data Storage copy

40. Decouple launch of service ROLL OUT INCREMENTALLY

42. Photo by brunogirin / CC BY SA ROLL OUT INCREMENTALLY Validate scalability while affecting fewest users Decouplelaunch of service

43. PRACTICE ROLLOUT

44. Have a solid rollback plan PRACTICE ROLLOUT

45. Client for User BuildRabbit Scheduler BuildRabbit Worker Persistent Queue

46. Server Client Old Data Storage New Data Storage copy

47. Use checklists PRACTICE ROLLOUT

48. Have a solid rollback plan Use checklists PRACTICE ROLLOUT

49. PRACTICE ROLLOUT FOCUS ON INTERMEDIATE STATE ROLL OUT INCREMENTALLY

50. Confidential + Proprietary Zero Downtime Migration How to Replace a Jet Engine of Your System In-Flight Aysylu Greenberg Google, October 19, 2017