Gerrit Code Review migrations step-by-step

0
Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 0
Gerrit migrations
Smoothly and Step by step
Luca Milanesio
Gerrit Code Review Maintainer
GerritForge

1
About
GerritForge
Founded in
the UK
HQ in
London
Committed to
OpenSource
+ Sunnyvale
CA

2
Gerrit migrations smoothly
step-by-step
https://www.futurelearn.com/courses/migration-facts

3
Everything changes and nothing
stands still
Heraclitus of Ephesus
Plato in Cratylus (360 BCE)

4
What if you do not want to change?
https://stream.org/our-world-is-changing-too-fast/
The world around
you
is changing
anyway

5
Example: Chrome 76 and Gerrit
v2.13

6
Migration dimensions:
distance
Who has migrated
to another home?
to another state?
to another country?

7
Migrating to another house
Get a man with a van
he moves everything
you pack your bags
One day … easy, done.

8
Migrating to another state
Get interstate moving
Plan well in advance
Pack the essential, throw the rest
Plan for storage
Change utilities companies
More days needed … more pain

9
Migrating to another country
Global movers
Get paperwork sorted: VISA etc.
Pack your bags, sell the rest
Study a foreign language
Start a fresh new life
Months needed … expensive, painful

10
Migration dimensions: people
Who is migrating with you
just your partner?
with the kids?
the whole family with relatives?

11
Small teams migrations
Easy coordination, small costs
• Plan a date
• Get a car
• Move
https://driving-tests.org/beginner-drivers/how-to-pack-a-car-for-vacation/

12
Large teams migrations
Coordinate people and high costs
• Lots of advance planning
• Get proper transport
• Move in stages
https://driving-tests.org/beginner-drivers/how-to-pack-a-car-for-vacation/

13
Migrations categories
Size
Distance
trivial complex
risky ultrahazardous

14
Common high-level change management
process
1. Staging as a copy of production
2. Migrate in staging
3. Verify success, resolve problems
4. Migrate in production

15
Trivial migrations
Small distance: +1 version (e.g. 2.14 to 2.15)
Small team: < 100 users
Small number repos: < 100
Small repos size: < 100MBytes
Small number of refs: < 10k refs

16
Trivial migrations
Stop + migrate + start, as per release notes
1. Read release-notes
2. Shutdown Gerrit
3. Backup repos, index, cache and db
4. Upgrade plugins
5. Gerrit init + reindex
6. Startup Gerrit

17
Complex migrations
Small distance: +1 version (e.g. 2.14 to 2.15)
Large team: 100s or 1000s of users
Large number repos: 100s or 1000s of repos
OR/AND
Large repos size: GBs in size
Large number of refs: 100ks of refs

18
Complex migrations:
requirements
1. High SLA (99.9% or higher) = 8h outage /year
2. Minimize risk
3. Enforce consistency
4. Manage people expectations
5. Impact analysis on other tools (Jenkins, Jira, etc)

19
Complex migrations: planning
• Resources allocation
budget, people, boxes, storage, CPUs
• High level schedule of the phases
1. Health-check of current setup
2. Development and preparation
3. Testing and bug-fix
4. Execution
5. Clean-up

20
Complex migrations: health-
check
What are the current system metrics?
Do I have the right configuration?
Are the repos and changes consistent?
How is the system performing?
What is the status of heap utilization?
Are the repos GCed?

21
Complex migrations: health-
check
Consequences of migrating an unhealthy system?
• Re-indexing errors
• Unexpected configuration behavior
• Unexpected performance issues
• Inability to manage the migrated system
• Very long migration times

22
Complex migrations:
preparation
Simulate in staging what will happen during cutover
• Measure exact timings
• Prepare scripts and configs for migration
• Prepare acceptance tests
• Test, fix, measure, iterate

23
Gerrit master
(Blue)
Gerrit master
(Green)
HAproxy HAproxy
HA
Plugin
HA
Plugin
Shared NFS
Complex migrations:
blue/green rollout
Reduced risk and
downtime
• Setup HAProxy and HA Plugin
• Upgrade one node at a time
• Test upgraded node before
opening traffic

24
Gerrit blue
(Ver. N)
Gerrit green
(Ver. N)
HAproxy HAproxy
HA
Plugin
HA
Plugin
Shared NFS
Complex migrations:
blue/green rollout
Begin of the rollout
• Both blue and green active
• RO plugin inserted
• Backup of the
Repos/ReviewDb
RO
Plugin
RO
Plugin

25
Gerrit blue
(Ver. N)
Gerrit green
(Ver. N)
HAproxy HAproxy
HA
Plugin
HA
Plugin
Shared NFS
Complex migrations:
blue/green rollout
Upgrade on green
• Shutdown Gerrit green
• Run init and upgrade plugins
• Start Gerrit green
• Online reindex + validation
RO
Plugin
RO
Plugin

26
Gerrit blue
(Ver. N)
Gerrit green
(Ver. N+1)
HAproxy HAproxy
HA
Plugin
HA
Plugin
Shared NFS
Complex migrations:
blue/green rollout
Gerrit green live on Ver. N+1
• Move traffic to Gerrit green
• Observe metrics
• Identify and address issues
• Wait for online reindex to finish
RO
Plugin

27
Gerrit blue
(Ver. N+1)
Gerrit green
(Ver. N+1)
HAproxy HAproxy
HA
Plugin
HA
Plugin
Shared NFS
Complex migrations:
blue/green rollout
Gerrit blue to Ver. N+1
• Shutdown Gerrit blue
• Upgrade war and plugins on blue
• Copy indexes from green to blue
• Startup Gerrit blue
• Delta reindex
RO
Plugin

28
Gerrit blue
(Ver. N+1)
Gerrit green
(Ver. N+1)
HAproxy HAproxy
HA
Plugin
HA
Plugin
Shared NFS
Complex migrations:
blue/green rollout
Migration completed
• Remote RO plugin on blue
• Gerrit blue + green active
• All indexes migrated

29
Risky migrations
Large distance: 2+ version (e.g. 2.11 to 2.16)
Small team: < 100 users
Small number repos: < 100
Small repos size: < 100MBytes
Small number of refs: < 10k refs

30
Risky migrations
Test migrations for all releases from A to B
1. Read release-notes of *all* releases between A and B
2. Create a staging Gerrit
3. Apply the migration of all releases, one at a time
4. Identify problems, apply fixes, restart from 2.

31
Risky migrations
Stop + migrate (all from A to B) + start
1. Shutdown Gerrit
2. Backup repos, index, cache and db
3. For every release from A to B:
• Upgrade plugins
• Gerrit init + reindex
4. Startup Gerrit

32
Ultrahazardous migrations
https://company.sbb.ch/en/media/background-information/gotthard-base-tunnel.html

33
Automating Gerrit migrations
with Ansible

34
Enabling a smooth
migration
 Migration is an issue that comes
up all of the times and is clearly
the major headache in managing
Gerrit
 Reasons for migrating:
compatibility, added
 Migration is scary because brings
DISRUPTION and OUTAGE on
Gerrit, vital in the dev life of
organizations

35
Enabling a smooth
migration
 Gerrit has evolved to a point where the
technology and tooling allow
 limited risk
 nearly zero down time migration.
 With a proper planning in conjunction with the
use of HA and well managed test / prod
separation we can migrate without lowering the
level of service we are currently providing to our
providing to our organization

36
GerritForge smooth migration
proposal
1. Pre-migration Health check
• Identify installation possible weak spots
• Identify some quick wins
• Determine level of maturity needed to
2. Provide a migration plan
• Timeline of the migration process
• Discussed and agreed with the client
3. Assist the HA setup and migration in the test
environment (*), T&C apply.
4. Three months included of Gerrit Enterprise
Support
• Allows full testing and possibly moving on
All of the above for $15,000
Offer time-limited to Dec
31st 2019
(*) First 16h included, standard rate for extra hours. Travelling expenses
excluded.

37
Poll: what are your plans for
migrating?
Image from: http://cypp.rutgers.edu/ru-voting/political-information/public-opinion-polls/

38
Wants to know more?
GerritForge.com/contact

Gerrit Code Review migrations step-by-step

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Gerrit Code Review migrations step-by-step

Similar to Gerrit Code Review migrations step-by-step (20)

More from Luca Milanesio

More from Luca Milanesio (20)

Recently uploaded

Recently uploaded (20)

Gerrit Code Review migrations step-by-step