Dev ops for software architects
Upcoming SlideShare
Loading in...5
×
 

Dev ops for software architects

on

  • 716 views

 

Statistics

Views

Total Views
716
Views on SlideShare
715
Embed Views
1

Actions

Likes
0
Downloads
31
Comments
0

1 Embed 1

https://twitter.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Dev ops for software architects Dev ops for software architects Presentation Transcript

  • DevOps for Software Architects Len Bass NICTA Copyright 2012 From imagination to impact
  • Outline • What is DevOps? • What are the implications of DevOps practices on system structure? – Team practices. – Deployment practices. NICTA Copyright 2012 From imagination to impact 2
  • What is DevOps? • “DevOps is a software development method that stresses communication, collaboration, and integration between software developers and IT professionals” – Wikipedia • “DevOps is a new term describing what has also been called “agile system administration” or “agile operations” joined together with the values of agile collaboration between development and operations staff. Effectively, you can define DevOps as system administrators participating in an agile development process alongside developers and using a many of the same agile techniques for their systems work. “ - http://theagileadmin.com/what-is-devops/ NICTA Copyright 2012 From imagination to impact 3
  • What is DevOps - 2 • DevOps is accompanied by a certain amount of mysticism. – “Be Self-Aware – Be aware of a project’s maturity – Be aware of others” http://architects.dzone.com/articles/zen-and-art-collaborative • Similar to the early days of agile. NICTA Copyright 2012 From imagination to impact 4
  • What problem is DevOps trying to solve? • Poor communication between developers and operations personnel • Slow release schedule • Limited capacity of operations staff • Limited organizational insight into operations NICTA Copyright 2012 From imagination to impact 5
  • My Take on DevOps • DevOps is a set of practices intended to – – – – – Reduce management overhead Speed up deployment Move some (formerly) IT responsibilities to developers Increase communication between developers and operations Reduce operations costs • These practices have implications on – Team size, communication, and responsibilities. – Deployment • In turn, there are implications for – System structure NICTA Copyright 2012 From imagination to impact 6
  • Outline • What is DevOps? • What are the implications of DevOps practices on system structure? – Team practices. – Deployment practices. NICTA Copyright 2012 From imagination to impact 7
  • Teams • • • • Size/coordination Responsibilities Processes/tooling Structural implications NICTA Copyright 2012 From imagination to impact 8
  • Team Size • Teams are small. Amazon has a “two pizza” rule. • It is easy for small teams to have good internal coordination. • Small teams mean – – – – A lot of teams Small scope for each team Short delivery times Coordination among teams becomes an issue NICTA Copyright 2012 From imagination to impact 9
  • Coordination among teams • Asynchronous rather than synchronous – Allows team members to respond when it is convenient for them – Avoids time zone coordination • Persistent and visible – E-mail is not generally visible to all of the team – Chat boards, Wikis, issue trackers, comments in code are all persistent and visible – Connect a message to something – issue, feature, person. NICTA Copyright 2012 From imagination to impact 10
  • Team Responsibilities with respect to Services • Requirements are sliced thinly both horizontally (breadth of requirement) and vertically (decomposition of service into utilities) • Each service has an owner (a developer) • Service owner decides when to deploy service to production. Deployment done with tooling. • Deployment may involve use of canaries (discussed with deployment) • When a service is deployed, service owner examines monitoring data and decides when/if to roll back. • Service owner is called if there is a problem. NICTA Copyright 2012 From imagination to impact 11
  • Structural Implications of Team Practices • Conway’s Law (1968) – The structure of a system reflects the structure of the organization that constructed the system. • DevOps advocates – Small teams – Mostly independent teams • Conway’s Law & many small, mostly independent teams => Service Oriented Architecture with – Many services with small scope of each service – Loose coupling between services NICTA Copyright 2012 From imagination to impact 12
  • Outline • What is DevOps? • What are the implications of DevOps practices on system structure? – Team practices. – Deployment practices. NICTA Copyright 2012 From imagination to impact 13
  • Deployment Practices • Deployment can be either an initial deployment or an upgrade of an existing system. • We will discuss – Upgrade – Continuous deployment – Roll back NICTA Copyright 2012 From imagination to impact 14
  • Deployment tools • Have “recipes” for standard configurations • Moving outside standard configurations may introduce errors • Recipes managed by DevOps group • Configuration specification is version controlled • Leads to “scripts are code too” mentality – Development – Staging – Deployment • Goal is to support developer’s ability to automatically deploy NICTA Copyright 2012 From imagination to impact 15
  • Upgrade • How many at once? – One at a time (rolling upgrade) – Groups at a time (staged upgrade, e.g. canaries) – All at once (big flip) • Big flip requires double the number of resources. Infeasible in environment that uses large numbers of resources. • Standard practice is rolling upgrade, possibly with canaries. NICTA Copyright 2012 From imagination to impact 16
  • Rolling Upgrade Process • Suppose there are 100s or 1000s of instances of an application running in the cloud. • Then it is too expensive to make a copy of a new version while leaving the old version running with all of its instances. • The solution is to install the new version on one server at a time – called a rolling upgrade • Figure on the right is an example of a process for a rolling upgrade. • This process is implemented by a deployment tool. Update Auto Scaling Group Sort Instances Confirm Upgrade Spec Remove & Deregister Old Instance from ELB Terminate Old Instance Wait for ASG to Start New Instance Register New Instance with ELB NICTA Copyright 2012 From imagination to impact 17
  • Upgrading a service within the service hierarchy Suppose we are doing a rolling upgrade at Service level N+1 Version B assumes new features from Service level N+2 Service level N Service level N+1 (A) Service level N+2 NICTA Copyright 2012 Service level N+2 From imagination to impact Service level N+1 (B) Service level N+1 (B) Service level N+2 Service level N+2 18
  • Staging Upgrades • Service level N+2 must be activated before activating service level N+1. • Distinction between upgrading and activating. Upgrades can occur at any time as long as they are not activated. • Structural implication – Upgrades can be activated through software switches. Could use Zookeeper for coordinating active versions. – Activates all of the instances at (essentially) same time. NICTA Copyright 2012 From imagination to impact 19
  • Upgrades can fail • Functionally incorrect • Incompatibility with other portions of the application or infrastructure • Resource limitations • Configuration inconsistency NICTA Copyright 2012 From imagination to impact 20
  • After failure is detected • Turn off new features in level N+1 and its clients (level N). • May need to propagate to top of hierarchy. • Structural implications • Features are software switchable • Require all versions to be backward compatible with previous versions. If new version on level N+1 is switched off, do not need to worry about level N+2. NICTA Copyright 2012 From imagination to impact 21
  • Canaries • Canaries are a small number of new versions in order to perform live testing in a production environment. • A/B testing is similar to canaries in that A/B testing compares two proposed new interfaces. Different in that canaries test new functionality. NICTA Copyright 2012 From imagination to impact 22
  • Canary Issues • Canaries are a form of live testing. Put a new version into limited production to test its correctness. • Issues – How long are new versions tested to determine correctness? • Period based – for some period of time • Load based – under some utilization assumptions • Result based – until some criteria is met – How are clients of new version chosen and how is this choice enforced? NICTA Copyright 2012 From imagination to impact 23
  • Continuous Deployment • Puts deployment decisions into developers hands. • May mean simultaneous deployment from independent teams. Some organizations report dozens of deployments a day. • Deployment tool must set configuration information e.g., in Zookeeper, so that services know what features are currently active. NICTA Copyright 2012 From imagination to impact 24
  • Structural Implications of Continuous Deployment • Packaging • Maintaining Backward Compatibility NICTA Copyright 2012 From imagination to impact 25
  • Packaging • Two dimensions – Flat vs deep service hierarchy – One service per virtual machine vs many services per virtual machine NICTA Copyright 2012 From imagination to impact 26
  • Flat vs Deep Service Hierarchy • Trading off independence of teams and possibilities for reuse. • Flat Service Hierarchy – Limited dependence among services & limited coordination needed among teams – Difficult to reuse services • Deep Service Hierarchy – Provides possibility for reusing services – Requires coordination among teams to discover reuse possibilities. NICTA Copyright 2012 From imagination to impact 27
  • Services per VM Image One service per VM VM image Service Develop Embed Multiple services per VM Develop Service 1 Embed VM image Develop NICTA Copyright 2012 Service 2 From imagination to impact Embed 28
  • One Possible Race Condition with Multiple Services per VM Initial State: VM image with Version N of Service 1 and Version N of Service 2 Developer 1 Developer 2 TIME Build new image with VN+1|VN Begin provisioning process with new image Build new image with VN|VN+1 Begin provisioning process with new image without new version of Service 1 Results in Version N+1 of Service 1 not being updated until next build of VM image Could be prevented by VM image build tool NICTA Copyright 2012 From imagination to impact 29
  • Another Possible Race Condition with Multiple Services per VM Initial State: VM image with Version N of Service 1 and Version N of Service 2 Developer 1 Developer 2 TIME Build new image with VN+1|VN Build new image with VN+1|VN+1 Begin provisioning process with new image overwrites image created by developer 2 Begin provisioning process with new image Results in Version N+1 of Service 2 not being updated until next build of VM image Could be prevented by provisioning tool NICTA Copyright 2012 From imagination to impact 30
  • Trade offs • One service per VM – Message from one service to another must go through inter VM communication mechanism – adds latency – No possibility of race condition • Multiple Services per VM – Inter VM communication requirements reduced – reduces latency – Adds possibility of race condition caused by simultaneous deployment NICTA Copyright 2012 From imagination to impact 31
  • Motivating Backward Compatibility • New version of a service may be introduced at any time • Existing clients of that service should not have to be changed • Require APIs and DB schemas to be backward compatible. NICTA Copyright 2012 From imagination to impact 32
  • Achieving Backwards Compatibility • APIs and DB schemas can be extended but must always be backward compatible. • Leads to a translation layer Client Client External APIs (unchanging but with ability to extend or add new ones) Translation to internal APIs Internal APIs (changes require changes to translation layer but do not propagate further) NICTA Copyright 2012 From imagination to impact
  • Summary • DevOps is a collection of practices that have implications on system structure. – These practices can be categorized into • Team practices • Deployment practices • Some structural implications are – Loosely coupled systems with deep hierarchy of services – Version aware – Backward compatible – Packaging services per VM NICTA Copyright 2012 From imagination to impact 34