The document outlines the discussions and developments presented at the Gerrit User Summit 2019 held in Gothenburg, Sweden, focusing on Gerrit's transition to a multi-site, open-source architecture. It details the implementation of high availability and disaster recovery strategies through master-slave replication and various plugins. Key features include split-brain prevention and a proposed next step involving location-aware DNS for improved traffic management.

1. 0Gerrit User Summit 2019 – Volvo Cars – Gothenburg - Sweden GerritForge.com 0
Gerrit goes multi-site
… and is 100% open source
Luca Milanesio
Gerrit Code Review Maintainer
GerritForge

2. 1Gerrit User Summit 2019 – Volvo Cars – Gothenburg - Sweden GerritForge.com 1
About GerritForge
Founded in the
UK
HQ in London Committed to
OpenSource

3. 2Gerrit User Summit 2019 – Volvo Cars – Gothenburg - Sweden GerritForge.com 2
The Journey to multi-site

4. 3Gerrit User Summit 2019 – Volvo Cars – Gothenburg - Sweden GerritForge.com 3
Single master
Gerrit Master
Incoming traffic

5. 4Gerrit User Summit 2019 – Volvo Cars – Gothenburg - Sweden GerritForge.com 4
Gerrit master
Replication
plugin
Incoming traffic
Gerrit master + slave

6. 5Gerrit User Summit 2019 – Volvo Cars – Gothenburg - Sweden GerritForge.com 5
Gerrit Master (+DR) + Slave
Gerrit master
Replication
plugin
Gerrit DR
Incoming traffic

7. 6Gerrit User Summit 2019 – Volvo Cars – Gothenburg - Sweden GerritForge.com 6
Gerrit master (HA+DR) + Slave
Gerrit master
1
Replication
plugin
Gerrit DR
Incoming traffic
Gerrit master
2
Replication
plugin
HAproxy HAproxy
HAplugin
HApluginShared NFS

8. 7Gerrit User Summit 2019 – Volvo Cars – Gothenburg - Sweden GerritForge.com 7
Gerrit master (HA+DR) + Slave: components
• HAProxy
• Gerrit >= v2.14 + plugins
• https://gerrit.googlesource.com/plugins/+
• replication
• high-availability (GerritForge’s fork)
github.com/GerritForge/plugins_high_availability
• websession-flatfile
• healthcheck

9. 8Gerrit User Summit 2019 – Volvo Cars – Gothenburg - Sweden GerritForge.com 8
Gerrit multi-master – two-sites
Gerrit master
(CA)
Replication
plugin
R/W Traffic
Gerrit master
(CA)
Replication
plugin
HAproxy HAproxy
Multi-site
Multi-site
Gerrit master
(GER)
Replication
plugin
Gerrit master
(GER)
Replication
plugin
HAproxy HAproxy
Multi-site
Multi-site
R Analytics Traffic
Shared NFSShared NFS

10. 9Gerrit User Summit 2019 – Volvo Cars – Gothenburg - Sweden GerritForge.com 9
(CA)(CA)
Gerrit multi-master multi-site
Gerrit
masters
(USA)
Replication
Gerrit
masters
(India)
Replication
Multi-site
Multi-site
Reliable broker
ü Sync index and caches across sites
ü Reliable across failures
ü Fully async, max performance
ü Fully distributed, max availability
ü Eventually consistent across sites

11. 10Gerrit User Summit 2019 – Volvo Cars – Gothenburg - Sweden GerritForge.com 10
Gerrit multi-master multi-site
Gerrit multi-site
IS
Highly Available
Partition tolerant

12. 11Gerrit User Summit 2019 – Volvo Cars – Gothenburg - Sweden GerritForge.com 11
Split-brain
prevention

13. 12Gerrit User Summit 2019 – Volvo Cars – Gothenburg - Sweden GerritForge.com 12
Ref-Db
Coordinator
• The Ref-Db coordinator is
used to guarantee that
two Gerrit instances do
not go into a split-brain
scenario
• Reject any push where
parent ObjectId is not
aligned with the shared
ref-db

14. 13Gerrit User Summit 2019 – Volvo Cars – Gothenburg - Sweden GerritForge.com 13
Next steps: location-aware DNS
Gerrit master
(CA)
Rest of the World
Traffic (R/W)
Gerrit master
(CA)
HAproxy HAproxy
Gerrit master
(GER)
Gerrit master
(GER)
HAproxy HAproxy
EU Traffic (R/W)
Multi-site pluginMulti-site pluginMulti-site pluginMulti-site plugin

15. 14Gerrit User Summit 2019 – Volvo Cars – Gothenburg - Sweden GerritForge.com 14
Poll: have you implemented multi-master/HA?
Image from: http://cypp.rutgers.edu/ru-voting/political-information/public-opinion-polls/

16. 15Gerrit User Summit 2019 – Volvo Cars – Gothenburg - Sweden GerritForge.com 15
Wants to know more?
GerritForge.com/contact