Coordinating unattended reboots using a distributed mutex at GOV.UK
•
0 likes•1,532 views
At GOV.UK, we have borrowed an idea from CoreOS' reboot mechanism and are using etcd-backed distributed mutex to orchestrate unattended reboots across our Ubuntu server estate to apply essential security updates. This talk will explain how we arrived at this approach, the checks we have in place to prevent the reboots from affecting our availability and how you might go about implementing a similar solution.
More Related Content
What's hot
Similar to Coordinating unattended reboots using a distributed mutex at GOV.UK
Similar to Coordinating unattended reboots using a distributed mutex at GOV.UK (20)
Recently uploaded
Recently uploaded (20)
Coordinating unattended reboots using a distributed mutex at GOV.UK
- 1. I work on GOV.UK GDS@mattbostock
- 2. A single domain! for the UK government GDS@mattbostock
- 3. Home Working, jobs and pensions Holidays, time off, sick leave, maternity and paternity leave UK bank holidays The next bank holiday in England and Wales is 3 April Good Friday Add bank holidays for England and Wales to your calendar (ICS, 10KB) Upcoming bank holidays in England and Wales Holiday entitlement More School term and holiday dates GOV.UK Holidays, time off, sick leave, maternity and paternity leave Elsewhere on GOV.UK Search Scotland Northern IrelandEngland and Wales GDSwww.gov.uk/bank-holidays
- 4. Home Passports, travel and living abroad Travel abroad Foreign travel advice GOV.UK Foreign travel advice Egypt Summary Current travel advice Safety and security Terrorism Local laws and customs Entry requirements Health Money Contact FCO Travel Advice Team Get updates email feed Hand luggage restrictions at UK airports Driving abroad More Travel abroad Renew or replace your adult passport More Passports, travel and living abroad Search Still current at: 7 November 2014 Updated: 7 November 2014 Latest update: Safety and security section (North Sinai) - a state of emergency and curfew declared Summary GDSwww.gov.uk/foreign-travel-advice/egypt
- 5. Home Working, jobs and pensions Holidays, time off, sick leave, maternity and paternity leave GOV.UK Calculate your maternity pay or benefits Maternity Allowance More Holidays, time off, sick leave, maternity and paternity leave Sure Start Maternity Grant Shared Parental Leave Elsewhere on GOV.UK Search Use this tool to: estimate your Statutory Maternity Pay (SMP) and work out your qualifying week estimate your Maternity Allowance if you don’t qualify for SMP check what other help and benefits you might get when you have a baby Start now GDSwww.gov.uk/calculate-your-maternity-pay
- 7. Over 300 government departments ! 12 million unique visitors every week GDS@mattbostock
- 8. Coordinating unattended server reboots 8GDS@mattbostock
- 9. Coordinating unattended server reboots ! Why automating server reboots is useful 9GDS@mattbostock
- 10. Coordinating unattended server reboots ! Why automating server reboots is useful What’s included in Ubuntu already 10GDS@mattbostock
- 11. Coordinating unattended server reboots ! Why automating server reboots is useful What’s included in Ubuntu already How CoreOS handles automatic server reboots 11GDS@mattbostock
- 12. Coordinating unattended server reboots ! Why automating server reboots is useful What’s included in Ubuntu already How CoreOS handles automatic server reboots How we implemented the same mechanism on Ubuntu 12GDS@mattbostock
- 13. Coordinating unattended server reboots ! Why automating server reboots is useful What’s included in Ubuntu already How CoreOS handles automatic server reboots How we implemented the same mechanism on Ubuntu How we iterated to make it work better for us 13GDS@mattbostock
- 14. Coordinating unattended server reboots ! Why automating server reboots is useful 14GDS@mattbostock
- 16. Some updates require a reboot 16GDS@mattbostock
- 17. GDS@mattbostock GOV.UK Second Line: 2 developers and a webops engineer
- 18. Rebooting servers manually is time consuming 18GDS@mattbostock
- 19. Coordinating unattended server reboots ! Why automating server reboots is useful What’s included in Ubuntu already 19GDS@mattbostock
- 22. Coordinating unattended server reboots ! Why automating server reboots is useful What’s included in Ubuntu already How CoreOS handles automatic server reboots 22GDS@mattbostock
- 23. CoreOS has a mechanism for automatic server reboots 23GDScoreos.com/using-coreos/updates/
- 27. 27GDS@mattbostock Locksmith uses a mutual exclusion lock stored in etcd
- 28. Atomic compare and swap 28GDS@mattbostock
- 30. Coordinating unattended server reboots ! Why automating server reboots is useful What’s included in Ubuntu already How CoreOS handles automatic server reboots How we implemented the same mechanism on Ubuntu 30GDS@mattbostock
- 33. The mechanism: ! Reboot window between 00:00-8:59 33GDS@mattbostock
- 34. The mechanism: ! Reboot window between 00:00-8:59 Does /var/run/reboot-required exist? 34GDS@mattbostock
- 37. The mechanism: ! Reboot window between 00:00-8:59 Does /var/run/reboot-required exist? Is the machine safe to reboot? 37GDS@mattbostock
- 38. The mechanism: ! Reboot window between 00:00-8:59 Does /var/run/reboot-required exist? Is the machine safe to reboot? Check for alerts in our monitoring (Icinga) 38GDS@mattbostock
- 39. The mechanism: ! Reboot window between 00:00-8:59 Does /var/run/reboot-required exist? Is the machine safe to reboot? Check for alerts in our monitoring (Icinga) Obtain the reboot lock 39GDS@mattbostock
- 40. The mechanism: ! Reboot window between 00:00-8:59 Does /var/run/reboot-required exist? Is the machine safe to reboot? Check for alerts in our monitoring (Icinga) Obtain the reboot lock Reboot! 40GDS@mattbostock
- 41. Or wait for one minute before trying again 41GDS@mattbostock
- 42. On boot, release the lock 42GDS@mattbostock
- 44. 44
- 45. Coordinating unattended server reboots ! Why automating server reboots is useful What’s included in Ubuntu already How CoreOS handles automatic server reboots How we implemented the same mechanism on Ubuntu How we iterated to make it work better for us 45GDS@mattbostock
- 46. How we made it work better for us: 46GDS@mattbostock
- 47. How we made it work better for us: ! Ran unattended-upgrades more often 47GDS@mattbostock
- 48. How we made it work better for us: ! Ran unattended-upgrades more often Added support for multiple etcd endpoints 48GDS@mattbostock
- 50. How we made it work better for us: ! Ran unattended-upgrades more often Added support for multiple etcd endpoints Increased time between reboots 50GDS@mattbostock
- 51. How we made it work better for us: ! Ran unattended-upgrades more often Added support for multiple etcd endpoints Increased time between reboots Rewrote the Bash script to query monitoring in Ruby 51GDS@mattbostock
- 52. How we made it work better for us: ! Ran unattended-upgrades more often Added support for multiple etcd endpoints Increased time between reboots Rewrote the Bash script to query monitoring in Ruby Fixed string matching bug when querying our monitoring 52GDS@mattbostock
- 53. How we made it work better for us: ! Ran unattended-upgrades more often Added support for multiple etcd endpoints Increased time between reboots Rewrote the Bash script to query monitoring in Ruby Fixed string matching bug when querying our monitoring Reduced reboot window to 00:00 - 05:55 53GDS@mattbostock
- 54. How we made it work better for us: ! Ran unattended-upgrades more often Added support for multiple etcd endpoints Increased time between reboots Rewrote the Bash script to query monitoring in Ruby Fixed string matching bug when querying our monitoring Reduced reboot window to 00:00 - 05:55 Beware kernel oops 54GDS@mattbostock
- 56. How we made it work better for us: ! Ran unattended-upgrades more often Added support for multiple etcd endpoints Increased time between reboots Rewrote the Bash script to query monitoring in Ruby Fixed string matching bug when querying our monitoring Reduced reboot window to 00:00 - 05:55 Beware kernel oops locksmithctl now supports machine groups 56GDS@mattbostock