This document describes Luca Milanesio's presentation on zero downtime upgrades for Gerrit. It discusses the different types of upgrades for Gerrit, including code-only, code and database schema, and code, database schema, and repository upgrades. It then outlines an 8 stage process for performing zero downtime upgrades of Gerrit using HAProxy load balancing to minimize downtime and ensure high availability, redirecting traffic from the old to new upgraded version.

1. #GerritUserSummit
Zero Downtime Gerrit
How to upgrade your Gerrit master while keeping the balls rolling.
Luca Milanesio
luca@gerritforge.com

2. @gitenterprise @gerritreview #GerritUserSummit
• Founded in the UK
• HQ in London, Offices in the UK and Italy
• Committed to OpenSource
About GerritForge

3. @gitenterprise @gerritreview #GerritUserSummit
• Co-founder and Director of
GerritForge
• over 20 years in Agile
Development and ALM
• Contributor to many OpenSource
projects
• Mind and hands of @gitenterprise
and @gerritreview Twitter feeds
About Luca Milanesio

4. @gitenterprise @gerritreview #GerritUserSummit
Life outside Google Data-Centers
• Gerrit is quite stable
• GerritHub uptime to date: 226 days
• Extensive unit & integration tests suite

5. @gitenterprise @gerritreview #GerritUserSummit
Life outside Google Data-Centers
BUT

6. @gitenterprise @gerritreview #GerritUserSummit
Life outside Google Data-Centers
"Sometimes" we need to upgrade
– Feb 2 gerrit-2.11.6
– Feb 12 gerrit-2.11.7
– Mar 3 gerrit-2.12.1
– Mar 9 gerrit-2.11.8
– Mar 11 gerrit-2.12.2
– May 17 gerrit-2.11.9
– Jul 7 gerrit-2.12.3
– Aug 24 gerrit-2.12.4
– Aug 26 gerrit-2.11.10
– Sep 24 gerrit-2.12.5
– Oct 21 gerrit-2.13
– Oct 21 gerrit-2.13.1
– Oct 21 gerrit-2.13.2

7. @gitenterprise @gerritreview #GerritUserSummit
Life outside Google Data-Centers
How others are
managing
upgrades 

8. @gitenterprise @gerritreview #GerritUserSummit
Life outside Google Data-Centers
Three flavors up upgrades
1. Code-only : Typically patch-releases
2. Code + DB Schema upgrade: Major releases
3. Code + DB Schema upgrade + Repos upgrade: Painful Major
release

9. @gitenterprise @gerritreview #GerritUserSummit
Life outside Google Data-Centers
How Google is
managing
upgrades 

10. @gitenterprise @gerritreview #GerritUserSummit
Life outside Google Data-Centers
Level of pain in the OpenSource Gerrit implementations
1. Code-only
 5 mins of fear
2. Code + DB Schema upgrade
 1/2 days of planning + 15/20' outage
3. Code + DB Schema upgrade + Repos upgrade:
 1/2 months of planning + leap of faith

11. @gitenterprise @gerritreview #GerritUserSummit
Levels of Zero-Downtime
0. Outage
1. Server always-on and reachable
2. Gerrit UX visible and responsive
3. Git/HTTP and Git/SSH accessible
4. Most of the functions operational
5. All functions fully operational

12. @gitenterprise @gerritreview #GerritUserSummit
Levels of Zero-Downtime: where are we?
0. Outage
1. Server always-on and reachable
2. Gerrit UX visible and responsive
3. Git/HTTP and Git/SSH accessible
4. Most of the functions operational
5. All functions fully operational

13. @gitenterprise @gerritreview #GerritUserSummit
HA Proxy
Zero-Downtime Upgrade – Stage 1
Client traffic
Ver. X (empty)
RSync
DB Export / Import

14. @gitenterprise @gerritreview #GerritUserSummit
HA Proxy
Zero-Downtime Upgrade – Stage 2
Client traffic
Ver. X Ver. X
Gerrit replication

15. @gitenterprise @gerritreview #GerritUserSummit
HA Proxy
Zero-Downtime Upgrade – Stage 3 upgrade
Client traffic
Ver. X Ver. X + N
Upgrade Gerrit war to Ver. X+N
Init + Off-line reindex

16. @gitenterprise @gerritreview #GerritUserSummit
HA Proxy
Zero-Downtime Upgrade – Stage 4 – read-only
Client traffic
Ver. X Ver. X + N
Install Read-only plugin

17. @gitenterprise @gerritreview #GerritUserSummit
HA Proxy
Zero-Downtime Upgrade – Stage 5 – delta replication
Client traffic
Ver. X Ver. X + N
Install Read-only plugin
Delta replication
DB Export / Re-Import

18. @gitenterprise @gerritreview #GerritUserSummit
HA Proxy
Zero-Downtime Upgrade – Stage 6 – Reinit
Client traffic
Ver. X Ver. X + N
READ-ONLY Init + Startup

19. @gitenterprise @gerritreview #GerritUserSummit
HA Proxy
Zero-Downtime Upgrade – Stage 7 – delta reindex
Client traffic
Ver. X Ver. X + N
Delta reindexREAD-ONLY

20. @gitenterprise @gerritreview #GerritUserSummit
HA Proxy
Zero-Downtime Upgrade – Stage 8 – traffic redirect
Client traffic
Ver. X Ver. X + N
Cache warm-up

21. @gitenterprise @gerritreview #GerritUserSummit
Timing and service level
1Stages 2 3 4 5 6 7
Times: days hours 10s 5' 5' 1s
Service
Level 100% up on Ver. X Read-only Ver. X 100% up on Ver. X + N
8

22. @gitenterprise @gerritreview #GerritUserSummit
DEMO

23. @gitenterprise @gerritreview #GerritUserSummit
Wants to know more?
www.gerritforge.com/contact