Getting From Folsom to Grizzly - A DevOps Upgrade Pattern.pptx
Upcoming SlideShare
Loading in...5
×
 

Getting From Folsom to Grizzly - A DevOps Upgrade Pattern.pptx

on

  • 1,251 views

true

true

Statistics

Views

Total Views
1,251
Views on SlideShare
1,251
Embed Views
0

Actions

Likes
0
Downloads
37
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Getting From Folsom to Grizzly - A DevOps Upgrade Pattern.pptx Getting From Folsom to Grizzly - A DevOps Upgrade Pattern.pptx Presentation Transcript

  • Greg AlthausDell, Principal Engineer http://lifeatthebar.co m
  • • The “Problem“ with Migration• Paths to Nirvana (or Roads to Perdition)• Alternatives• An Opinion• Discussion F G H http://learn.genetics.utah.edu/content/begin/cells/organelles/
  • • OpenStack has 3 month release major/minor cycle • Major version every 6 months • Minor version (but important) 3 & 6 months after release• Lots of Changes • Bugs are fixed • Operating Systems upgrade • New technologies appear • Whole projects are split off• We expect operators to • Keep systems running • Never loose data • And… Stay up to date http://cdn2.arkive.org sockeye-salmon-predated-by-grizzly-bear-on-migration-upstream.jpg
  • • What are we upgrading? • OpenStack - Yes! • Dependent packages - Probably? • Base OS - Maybe?• What is the state during the "in-between" time? • Infrastructure downtime? • VM downtime? VM Reboot? Controlled/Informed? • Availability Windows?• What contingency plans? • Dry run? Maybe. • Recover by going backwards? Maybe.• What level of safety and trust do you need? • Assure data integrity? • Assure Infrastructure Integrity? • Maintain Security?• How long can the migration take? • Big bang move or gradual migrate? • How will my API consumers/ecosystem cope? • Can Keystone Grizzly work with Folsom Nova??? • What about futures? G.1 to G.2? H to I? • Can I skip versions? Jump from G to I? http://www.publicdomainpictures.net Steep Steps by Peter Griffin
  • • Beginning Answers • Distros will manage dependencies and packaging • We can’t lose data or compromise security • Infrastructure state and integrity will vary by solution• Assumption of Staging • Some managed environment (not a manual deploy) • Staging/test environment to get "familiar" with the problem. • Maintenance window for production - limits scope of change • Step-wise changes are OK (big bang is not required) • We can make trade-offs to defray expensive requirements• Beyond Assumptions… Paradigm Shifts • There are shared best practices • Upgrades can be automated in a sharable wayhttp://www.theemailadmin.com/wp-content/uploads/2012/09/GFI229-hot-water-migration.jpg
  • All the nodes update to the latest codein a short time window• Details: 1. Cookbooks include update (instead of install) directives. 2. Control upstream package point (e.g. apt-update when appropriate) 3. Force chef-client run 4. Now at new level• Considerations • Pros: Potentially fast, continuous operation • Cons: Dont mess up, it is your production environment • Scope: Security updates • Code Assumptions: • System can function through service restarts. • Underlying data models dont change or migrate appropriately.
  • Nodes migrate in staged groups• Details: 1. Choose subset of machines and quiesce them. 2. Update set 3. Freeze state (by tenant) 4. Migrate service/tenant content 5. Repurpose after complete.• Considerations • Pros: Safer, more controlled, and can move tenants as needed • Cons: Takes longer, still has cut-over point, but less open http://allgodscrittersgotrhythm.blogspot.com/2010_08_01_archive.html
  • Nodes changed individually by a system-wideorchestration that supports components of multipleversions• Details 1. Components must be able to straddle versions 2. Orchestration updates core components to new version 3. System as a whole queiseces and is validated (requires self test) 4. Orchestration individually migrates components (return to step 3)• Considerations • Pros: Creates a highly resilient system that handles higher rate ofhttp://www.grizzlycentral.com/forum/grizzly-tire-wheel-combos/1204-upgrade-tires-grizzly.html
  • • Orchestration (not just deployment automation)• Awareness of physical layout is required • Must respect fault zones to sustain HA • Proximity of resources matters for migration • Networking transitions are essential• Collaboration with development teams is essential • Components must support current and previous • Upgrade plan must be baked into configuration and tested • Upgrade dependencies must be 1) clear and 2) minimized• HA complicates upgrades • Upgrade can be detected as a failure • HA system must be able to bridge versions
  • • Deployment Upstream Cookbooks/Modules• Best Practice Discussions• Code for Upgradeability• Crowbar Collaboration • Upgrade is a FEATURE! • Orchestration + Chef • Pull from Source Deployments • System Discovery • Networking Configuration • Operating System Install http://farm3.static.flickr.com/2561/3891653055_262410bc31.jpg