GerritHub evolved from a simple master-slave setup into a truly multi-master and multi-site service. See how the solution has been implemented and the next steps for making the service even better and more distributed and scalable.

1. 0Gerrit User Summit 2018 – Palo Alto CA GerritForge.com 0
Gerrit multi-master/site
An open source solution with GerritHub
Luca Milanesio
Gerrit Code Review Maintainer
GerritForge

2. 1Gerrit User Summit 2018 – Palo Alto CA GerritForge.com 1
About GerritForge
Founded in the
UK
HQ in London Committed to
OpenSource

3. 2Gerrit User Summit 2018 – Palo Alto CA GerritForge.com 2
GerritHub master – slave
Hetzner – (GER)
replication
2013
Incoming traffic

4. 3Gerrit User Summit 2018 – Palo Alto CA GerritForge.com 3
2013
Gerrit master
(GER)
Replication
plugin
Incoming traffic
GerritHub master – slave

5. 4Gerrit User Summit 2018 – Palo Alto CA GerritForge.com 4
GerritHub master (+dr) – slave
OVH – (CA)
replication
2015
Incoming traffic
Hetzner – (GER)
replication

6. 5Gerrit User Summit 2018 – Palo Alto CA GerritForge.com 5
GerritHub master (+dr) – slave 2015
Gerrit master
(CA)
Replication
plugin
Gerrit master
(GER)
Incoming traffic

7. 6Gerrit User Summit 2018 – Palo Alto CA GerritForge.com 6
GerritHub multi-master (+dr) – slave
OVH – (CA)
replication
2017
Incoming traffic
Hetzner – (GER)
replication

8. 7Gerrit User Summit 2018 – Palo Alto CA GerritForge.com 7
GerritHub master (pri/dr) – slave 2017
Gerrit master
(CA)
Replication
plugin
Gerrit master
(GER)
Incoming traffic
Gerrit master
(CA)
Replication
plugin
HAproxy HAproxy
HAplugin
HApluginShared NFS

9. 8Gerrit User Summit 2018 – Palo Alto CA GerritForge.com 8
GerritHub master (pri/dr) – slave components
• Gerrit >= v2.14 + plugins
• https://gerrit.googlesource.com/plugins/+
• replication
• github
• delete-project
• account
• high-availability (GerritForge’s fork)
github.com/GerritForge/plugins_high_availability
• websession-flatfile
• scripting/groovy-provider

10. 9Gerrit User Summit 2018 – Palo Alto CA GerritForge.com 9
GerritHub multi-master/site – slave
OVH – (CA)
replication
2018
Incoming traffic (R/W)
Hetzner – (GER)
replication
Incoming traffic (R)

11. 10Gerrit User Summit 2018 – Palo Alto CA GerritForge.com 10
GerritHub master (pri/dr) – slave 2018
Gerrit master
(CA)
Replication
plugin
R/W Traffic
Gerrit master
(CA)
Replication
plugin
HAproxy HAproxy
HAplugin
HAplugin
Gerrit master
(GER)
Replication
plugin
Gerrit master
(GER)
Replication
plugin
HAproxy HAproxy
HAplugin
HAplugin
R Analytics Traffic
Shared NFSShared NFS

12. 11Gerrit User Summit 2018 – Palo Alto CA GerritForge.com 11
GerritHub multi-master/site limitations
1.Lag in sites replication
2.Indexes aligned every 30 mins
3.Latency of disaster recovery (DNS)

13. 12Gerrit User Summit 2018 – Palo Alto CA GerritForge.com 12
Next steps: repositories sharding
Gerrit master
(CA)
Replication
plugin
R/W Traffic
Gerrit master
(CA)
Replication
plugin
HAproxy HAproxyHAplugin
HAplugin
HAproxy with repo-specific rules:
• URL-based node selection
Share-nothing between masters:
• HA plugin to be tolerant do delays
• Repo-specific replication configs
Alignment checks and metrics

14. 13Gerrit User Summit 2018 – Palo Alto CA GerritForge.com 13
Next steps: auto-reindex across sites
Gerrit master
(CA)
Replication
plugin
Gerrit master
(CA)
Replication
plugin
Multi-site
plugin
Multi-site
plugin
Reliable broker
New multi-site plugin
• Auto-reindex
• Reliable across failures
• Fully async

15. 14Gerrit User Summit 2018 – Palo Alto CA GerritForge.com 14
Next steps: location-aware DNS
Gerrit master
(CA)
Rest of the World
Traffic (R/W)
Gerrit master
(CA)
HAproxy HAproxy
Gerrit master
(GER)
Gerrit master
(GER)
HAproxy HAproxy
EU Traffic (R/W)
Multi-site pluginMulti-site pluginMulti-site pluginMulti-site plugin

16. 15Gerrit User Summit 2018 – Palo Alto CA GerritForge.com 15
Poll: have you implemented multi-master/HA?
Image from: http://cypp.rutgers.edu/ru-voting/political-information/public-opinion-polls/

17. 16Gerrit User Summit 2018 – Palo Alto CA GerritForge.com 16
Wants to know more?
GerritForge.com/contact