Successfully reported this slideshow.
Your SlideShare is downloading. ×

Gerrit Code Review migrations step-by-step

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Loading in …3
×

Check these out next

1 of 39 Ad
Advertisement

More Related Content

Slideshows for you (20)

Similar to Gerrit Code Review migrations step-by-step (20)

Advertisement

More from Luca Milanesio (20)

Recently uploaded (20)

Advertisement

Gerrit Code Review migrations step-by-step

  1. 1. 0 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 0 Gerrit migrations Smoothly and Step by step Luca Milanesio Gerrit Code Review Maintainer GerritForge
  2. 2. 1 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 1 About GerritForge Founded in the UK HQ in London Committed to OpenSource + Sunnyvale CA
  3. 3. 2 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 2 Gerrit migrations smoothly step-by-step https://www.futurelearn.com/courses/migration-facts
  4. 4. 3 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 3 Everything changes and nothing stands still Heraclitus of Ephesus Plato in Cratylus (360 BCE)
  5. 5. 4 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 4 What if you do not want to change? https://stream.org/our-world-is-changing-too-fast/ The world around you is changing anyway
  6. 6. 5 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 5 Example: Chrome 76 and Gerrit v2.13
  7. 7. 6 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 6 Migration dimensions: distance Who has migrated to another home? to another state? to another country?
  8. 8. 7 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 7 Migrating to another house Get a man with a van he moves everything you pack your bags One day … easy, done.
  9. 9. 8 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 8 Migrating to another state Get interstate moving Plan well in advance Pack the essential, throw the rest Plan for storage Change utilities companies More days needed … more pain
  10. 10. 9 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 9 Migrating to another country Global movers Get paperwork sorted: VISA etc. Pack your bags, sell the rest Study a foreign language Start a fresh new life Months needed … expensive, painful
  11. 11. 10 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 10 Migration dimensions: people Who is migrating with you just your partner? with the kids? the whole family with relatives?
  12. 12. 11 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 11 Small teams migrations Easy coordination, small costs • Plan a date • Get a car • Move https://driving-tests.org/beginner-drivers/how-to-pack-a-car-for-vacation/
  13. 13. 12 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 12 Large teams migrations Coordinate people and high costs • Lots of advance planning • Get proper transport • Move in stages https://driving-tests.org/beginner-drivers/how-to-pack-a-car-for-vacation/
  14. 14. 13 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 13 Migrations categories Size Distance trivial complex risky ultrahazardous
  15. 15. 14 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 14 Common high-level change management process 1. Staging as a copy of production 2. Migrate in staging 3. Verify success, resolve problems 4. Migrate in production
  16. 16. 15 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 15 Trivial migrations Small distance: +1 version (e.g. 2.14 to 2.15) Small team: < 100 users Small number repos: < 100 Small repos size: < 100MBytes Small number of refs: < 10k refs
  17. 17. 16 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 16 Trivial migrations Stop + migrate + start, as per release notes 1. Read release-notes 2. Shutdown Gerrit 3. Backup repos, index, cache and db 4. Upgrade plugins 5. Gerrit init + reindex 6. Startup Gerrit
  18. 18. 17 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 17 Complex migrations Small distance: +1 version (e.g. 2.14 to 2.15) Large team: 100s or 1000s of users Large number repos: 100s or 1000s of repos OR/AND Large repos size: GBs in size Large number of refs: 100ks of refs
  19. 19. 18 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 18 Complex migrations: requirements 1. High SLA (99.9% or higher) = 8h outage /year 2. Minimize risk 3. Enforce consistency 4. Manage people expectations 5. Impact analysis on other tools (Jenkins, Jira, etc)
  20. 20. 19 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 19 Complex migrations: planning • Resources allocation budget, people, boxes, storage, CPUs • High level schedule of the phases 1. Health-check of current setup 2. Development and preparation 3. Testing and bug-fix 4. Execution 5. Clean-up
  21. 21. 20 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 20 Complex migrations: health- check What are the current system metrics? Do I have the right configuration? Are the repos and changes consistent? How is the system performing? What is the status of heap utilization? Are the repos GCed?
  22. 22. 21 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 21 Complex migrations: health- check Consequences of migrating an unhealthy system? • Re-indexing errors • Unexpected configuration behavior • Unexpected performance issues • Inability to manage the migrated system • Very long migration times
  23. 23. 22 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 22 Complex migrations: preparation Simulate in staging what will happen during cutover • Measure exact timings • Prepare scripts and configs for migration • Prepare acceptance tests • Test, fix, measure, iterate
  24. 24. 23 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 23 Gerrit master (Blue) Gerrit master (Green) HAproxy HAproxy HA Plugin HA Plugin Shared NFS Complex migrations: blue/green rollout Reduced risk and downtime • Setup HAProxy and HA Plugin • Upgrade one node at a time • Test upgraded node before opening traffic
  25. 25. 24 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 24 Gerrit blue (Ver. N) Gerrit green (Ver. N) HAproxy HAproxy HA Plugin HA Plugin Shared NFS Complex migrations: blue/green rollout Begin of the rollout • Both blue and green active • RO plugin inserted • Backup of the Repos/ReviewDb RO Plugin RO Plugin
  26. 26. 25 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 25 Gerrit blue (Ver. N) Gerrit green (Ver. N) HAproxy HAproxy HA Plugin HA Plugin Shared NFS Complex migrations: blue/green rollout Upgrade on green • Shutdown Gerrit green • Run init and upgrade plugins • Start Gerrit green • Online reindex + validation RO Plugin RO Plugin
  27. 27. 26 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 26 Gerrit blue (Ver. N) Gerrit green (Ver. N+1) HAproxy HAproxy HA Plugin HA Plugin Shared NFS Complex migrations: blue/green rollout Gerrit green live on Ver. N+1 • Move traffic to Gerrit green • Observe metrics • Identify and address issues • Wait for online reindex to finish RO Plugin
  28. 28. 27 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 27 Gerrit blue (Ver. N+1) Gerrit green (Ver. N+1) HAproxy HAproxy HA Plugin HA Plugin Shared NFS Complex migrations: blue/green rollout Gerrit blue to Ver. N+1 • Shutdown Gerrit blue • Upgrade war and plugins on blue • Copy indexes from green to blue • Startup Gerrit blue • Delta reindex RO Plugin
  29. 29. 28 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 28 Gerrit blue (Ver. N+1) Gerrit green (Ver. N+1) HAproxy HAproxy HA Plugin HA Plugin Shared NFS Complex migrations: blue/green rollout Migration completed • Remote RO plugin on blue • Gerrit blue + green active • All indexes migrated
  30. 30. 29 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 29 Risky migrations Large distance: 2+ version (e.g. 2.11 to 2.16) Small team: < 100 users Small number repos: < 100 Small repos size: < 100MBytes Small number of refs: < 10k refs
  31. 31. 30 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 30 Risky migrations Test migrations for all releases from A to B 1. Read release-notes of *all* releases between A and B 2. Create a staging Gerrit 3. Apply the migration of all releases, one at a time 4. Identify problems, apply fixes, restart from 2.
  32. 32. 31 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 31 Risky migrations Stop + migrate (all from A to B) + start 1. Shutdown Gerrit 2. Backup repos, index, cache and db 3. For every release from A to B: • Upgrade plugins • Gerrit init + reindex 4. Startup Gerrit
  33. 33. 32 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 32 Ultrahazardous migrations https://company.sbb.ch/en/media/background-information/gotthard-base-tunnel.html
  34. 34. 33 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 33 Automating Gerrit migrations with Ansible
  35. 35. 34 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 34 Enabling a smooth migration  Migration is an issue that comes up all of the times and is clearly the major headache in managing Gerrit  Reasons for migrating: compatibility, added  Migration is scary because brings DISRUPTION and OUTAGE on Gerrit, vital in the dev life of organizations
  36. 36. 35 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 35 Enabling a smooth migration  Gerrit has evolved to a point where the technology and tooling allow  limited risk  nearly zero down time migration.  With a proper planning in conjunction with the use of HA and well managed test / prod separation we can migrate without lowering the level of service we are currently providing to our providing to our organization
  37. 37. 36 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 36 GerritForge smooth migration proposal 1. Pre-migration Health check • Identify installation possible weak spots • Identify some quick wins • Determine level of maturity needed to 2. Provide a migration plan • Timeline of the migration process • Discussed and agreed with the client 3. Assist the HA setup and migration in the test environment (*), T&C apply. 4. Three months included of Gerrit Enterprise Support • Allows full testing and possibly moving on All of the above for $15,000 Offer time-limited to Dec 31st 2019 (*) First 16h included, standard rate for extra hours. Travelling expenses excluded.
  38. 38. 37 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 37 Poll: what are your plans for migrating? Image from: http://cypp.rutgers.edu/ru-voting/political-information/public-opinion-polls/
  39. 39. 38 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 38 Wants to know more? GerritForge.com/contact

×