Getting From Folsom to Grizzly - A DevOps Upgrade Pattern.pptx


Published on


  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Getting From Folsom to Grizzly - A DevOps Upgrade Pattern.pptx

  1. 1. Greg AlthausDell, Principal Engineer m
  2. 2. • The “Problem“ with Migration• Paths to Nirvana (or Roads to Perdition)• Alternatives• An Opinion• Discussion F G H
  3. 3. • OpenStack has 3 month release major/minor cycle • Major version every 6 months • Minor version (but important) 3 & 6 months after release• Lots of Changes • Bugs are fixed • Operating Systems upgrade • New technologies appear • Whole projects are split off• We expect operators to • Keep systems running • Never loose data • And… Stay up to date sockeye-salmon-predated-by-grizzly-bear-on-migration-upstream.jpg
  4. 4. • What are we upgrading? • OpenStack - Yes! • Dependent packages - Probably? • Base OS - Maybe?• What is the state during the "in-between" time? • Infrastructure downtime? • VM downtime? VM Reboot? Controlled/Informed? • Availability Windows?• What contingency plans? • Dry run? Maybe. • Recover by going backwards? Maybe.• What level of safety and trust do you need? • Assure data integrity? • Assure Infrastructure Integrity? • Maintain Security?• How long can the migration take? • Big bang move or gradual migrate? • How will my API consumers/ecosystem cope? • Can Keystone Grizzly work with Folsom Nova??? • What about futures? G.1 to G.2? H to I? • Can I skip versions? Jump from G to I? Steep Steps by Peter Griffin
  5. 5. • Beginning Answers • Distros will manage dependencies and packaging • We can’t lose data or compromise security • Infrastructure state and integrity will vary by solution• Assumption of Staging • Some managed environment (not a manual deploy) • Staging/test environment to get "familiar" with the problem. • Maintenance window for production - limits scope of change • Step-wise changes are OK (big bang is not required) • We can make trade-offs to defray expensive requirements• Beyond Assumptions… Paradigm Shifts • There are shared best practices • Upgrades can be automated in a sharable way
  6. 6. All the nodes update to the latest codein a short time window• Details: 1. Cookbooks include update (instead of install) directives. 2. Control upstream package point (e.g. apt-update when appropriate) 3. Force chef-client run 4. Now at new level• Considerations • Pros: Potentially fast, continuous operation • Cons: Dont mess up, it is your production environment • Scope: Security updates • Code Assumptions: • System can function through service restarts. • Underlying data models dont change or migrate appropriately.
  7. 7. Nodes migrate in staged groups• Details: 1. Choose subset of machines and quiesce them. 2. Update set 3. Freeze state (by tenant) 4. Migrate service/tenant content 5. Repurpose after complete.• Considerations • Pros: Safer, more controlled, and can move tenants as needed • Cons: Takes longer, still has cut-over point, but less open
  8. 8. Nodes changed individually by a system-wideorchestration that supports components of multipleversions• Details 1. Components must be able to straddle versions 2. Orchestration updates core components to new version 3. System as a whole queiseces and is validated (requires self test) 4. Orchestration individually migrates components (return to step 3)• Considerations • Pros: Creates a highly resilient system that handles higher rate of
  9. 9. • Orchestration (not just deployment automation)• Awareness of physical layout is required • Must respect fault zones to sustain HA • Proximity of resources matters for migration • Networking transitions are essential• Collaboration with development teams is essential • Components must support current and previous • Upgrade plan must be baked into configuration and tested • Upgrade dependencies must be 1) clear and 2) minimized• HA complicates upgrades • Upgrade can be detected as a failure • HA system must be able to bridge versions
  10. 10. • Deployment Upstream Cookbooks/Modules• Best Practice Discussions• Code for Upgradeability• Crowbar Collaboration • Upgrade is a FEATURE! • Orchestration + Chef • Pull from Source Deployments • System Discovery • Networking Configuration • Operating System Install