Continuous Delivery Distilled
Upcoming SlideShare
Loading in...5
×
 

Continuous Delivery Distilled

on

  • 575 views

Matt Callanan takes the 15 chapters of the famous "Continuous Delivery" book by Jez Humble & Dave Farey and distills it down into 1 hour of convincing arguments, walking through the pieces involved to ...

Matt Callanan takes the 15 chapters of the famous "Continuous Delivery" book by Jez Humble & Dave Farey and distills it down into 1 hour of convincing arguments, walking through the pieces involved to make it happen including cultural challenges, automated testing, automated deployment & deployment pipelines. Not sure how to get started with DevOps? Finding it hard to convince colleagues & managers that CD is the way forward? Matt has used this presentation to help facilitate enterprise-wide adoption of Continuous Delivery. Slides from a presentation given at DevOps Brisbane March 2014.

Statistics

Views

Total Views
575
Views on SlideShare
574
Embed Views
1

Actions

Likes
4
Downloads
35
Comments
1

1 Embed 1

http://www.linkedin.com 1

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Continuous Delivery Distilled Continuous Delivery Distilled Presentation Transcript

  • Con$nuous   Delivery   Dis$lled   Ma#  Callanan   @mcallana  
  • Why  Con/nuous  Delivery?   •  “The  most  important  problem  that  we  face  as   so?ware  professionals  is  this:     –  If  somebody  thinks  of  a  good  idea,  how  do  we  deliver  it  to   users  as  quickly  as  possible?”  –  CD  Book   •  “Our  highest  priority  is:   –  to  sa/sfy  the  customer  through  early  and  con$nuous   delivery  of  valuable  so?ware.”  –  Agile  Manifesto  
  • Why  Con/nuous  Delivery?   •  Cycle  /me  should  be  measured  in  hours  –  not  months   –  How  long  does  a  single  change  take  to  release?   •  Quality  should  be  built-­‐in  to  the  process   –  Not  an  a?erthought  of  manual  inspec/on   •  Feedback  should  be  close  as  possible  to  point  of  failure   –  Late  feedback  is  expensive  
  • What  is  Con/nuous  Delivery?   •  A  set  of  prac/ces  and  principles   –  aimed  at,  building,  tes/ng,  and  releasing  so?ware  faster   and  more  frequently   Principles  of  So>ware  Delivery   Create  a  Repeatable,  Reliable  Process  for  Releasing  So?ware   Automate  Almost  Everything     Version  Control  Everything   If  It  Hurts,  Do  It  More  O?en  –  “Bring  the  Pain  Forward”   Build  Quality  In     “Done”  Means  Released     Everybody  Is  Responsible  for  the  Delivery  Process   Con/nuous  Improvement  
  • What  is  Con/nuous  Delivery?   •  A  way  of  reducing  cycle  /me   –  How  long  does  a  single  change  take  to  release?   –  I.e.  how  long  it  takes  a  simple  code  change  to  get  to   produc/on?  
  • What  is  Con/nuous  Delivery?   •  A  way  of  joining  development,  deployment,  tes/ng  &   release  ac/vi/es   –  coordina/ng  them  to  make  the  process  as  efficient  and   reliable  as  possible   –  reducing  feedback  delays  by  automa/ng  manual  processes   Develop   Deploy   Test   Release  
  • Central  Concept:  Deployment  Pipeline   •  Make  process  visible   •  Enable  automated  tes/ng   –  Ensure  quality  is  built  into  process   •  Enable  automated  deploys  to  any  environment   •  Improve  feedback  cycles   Commit   stage   Acceptance   stage   Exploratory   tes$ng   UAT   Capacity   Tes$ng   Produc$on  
  • What  is  not  Con/nuous  Delivery:     An/-­‐pa#erns   •  Deploying  so?ware  manually   •  Deploying  to  prod-­‐like  env  only  a?er  dev  is  complete   •  Manual  configura/on  management  of  produc/on  
  • Deployment  Pipeline           Agile  /  Lean  /  DevOps  /  Culture   Configura$on   Management   Tes$ng   Deployment   Automa$on   Disciplined   Development   Risk  Management   Con$nuous  Integra$on   Building  Blocks  
  • Deployment  Pipeline           Agile  /  Lean  /  DevOps  /  Culture   Configura$on   Management   Tes$ng   Deployment   Automa$on   Disciplined   Development   Risk  Management   Con$nuous  Integra$on   Building  Blocks  
  • Requirements   • Analyst  &   Customer   Design   • Architect   Development   • Developer   QA   • Tester   Release   • Opera/ons   Months  /  Years!   Waterfall  
  • Agile  
  • Agile  -­‐>  Con/nuous  Delivery  
  • Agile  Culture   Analyst/ Customer   Testers  Devs   Acceptance   Criteria   Automa/on  
  • DevOps  Culture   Analyst/ Customer   Testers   Ops   Devs   Acceptance   Criteria   Automa/on  
  • Silos   •  Ul/mately,  we  succeed  or  fail  as  a  team     –  not  as  individuals   •  Symptoms  of  Silos     –  Dev  throws  work  over  wall  to  Test  throws  over  wall  to  Ops   –  End  up  spending  as  much  /me   blaming  each  other  as  fixing   defects  that  arise  from  siloed   approach   h#p://www.gesilos.com.au/images/pellet/3%20x%2026%20Tonne%20Pellet%20Silos.jpg  
  • Repairing  Silos   •  How  to  break  down  silos   –  Get  everybody  involved  together  from  start  of  project   –  Ensure  they  can  communicate  on  a  frequent,  regular  basis   –  Increase  visibility  and  self-­‐serviceablilty    
  • Lean   •  Example  Value  Stream  Map   Require-­‐   ments   Develop-­‐ ment   Tes$ng   Staging   Capacity   Tes$ng   Release   Value-­‐   Added   Time   Elapsed   Non-­‐Value  Added     Time   Business   Customer  
  • Batch  Size  vs  Risk   h#p://www.slideshare.net/jallspaw/ops-­‐metametrics-­‐the-­‐currency-­‐you-­‐pay-­‐for-­‐change   •  Reducing  Batch  Size  Reduces  Risk  
  • Batch  Size  Science  
  • High  Release  Cost  =  High  Batch  Size                                                                                                                         Batch  Size   Cost   Release  Cost   Total  Cost  
  • Low  Release  Cost  =  Low  Batch  Size   Batch  Size   Cost   Release  Cost  
  • Ac/ng  on  Feedback   •  The  delivery  team  must  receive  feedback   and  then  act  on  it   •  Involve  everybody  in  feedback  process   –  Work  in  cross-­‐func$onal  teams  or  meet  o?en   –  Retrospec$ve:  discuss  how  to  improve   delivery  process  next  itera/on   •  Broadcast  the  informa/on   –  Big,  visible  dashboards  ensures  feedback  gets   into  someone’s  head   •  Feedback  must  be  acted  upon   –  Requires  discipline  and  planning   –  Stop  and  decide  on  a  course  of  ac$on   –  Only  once  this  is  done  should  the  team  carry  on   with  their  work  
  • Deployment  Pipeline           Agile  /  Lean  /  DevOps  /  Culture   Configura$on   Management   Tes$ng   Deployment   Automa$on   Disciplined   Development   Risk  Management   Building  Blocks   Con$nuous  Integra$on  
  • Con/nuous  Integra/on   Developer   • Commit  to  Version   Control   Build  Server   • Compile   Tes/ng   • Unit  Tests   • Component  Tests   Repor/ng   • Test  Results   • Coverage   • Sta/c  Analysis  
  • Disciplined   Development   Configura$on   Management   Tes$ng   Deployment   Automa$on   Deployment  Pipeline           Deployment  Pipeline           Configura$on   Management   Tes$ng   Deployment   Automa$on   Disciplined   Development   Risk  Management   Con$nuous  Integra$on   Building  Blocks   Agile  /  Lean  /  DevOps  /  Culture  
  • Deployment  Pipeline   Commit   stage   Acceptance   stage   Exploratory   tes$ng   UAT   Capacity   Tes$ng   Produc$on   Increasing  confidence  in  build’s  produc/on  readiness   Environments  become  more  produc/on-­‐like   Faster  feedback   •  Tension:     – Environmental  Confidence  vs  Feedback  Speed  
  • Ar/fact  Repository  (e.g.  Yum)   Version  Control  (e.g.  Git)   Deployment  Pipeline   Source   Code   Commit  stage     Compile   Sta/c  analysis   Unit  Tests   Coverage   Build  binary  ar/facts   Acceptance  stage       Configure  Env   Deploy   Smoke  Tests   Acceptance  Tests     ar6facts  ar6facts   UAT     Deploy   Compliance  Tests   Smoke  Tests     Capacity  Stage     Deploy   Compliance  Tests   Smoke  Tests   Load  Tests     Produc$on     Deploy   Compliance  Tests   Smoke  Tests       ar6facts   CM  code/data,   Orchestra6on,  Puppet,   Chef,  Fabric,  Ansible,   etc   Testers   Self-­‐service     deploys   Ops   Push-­‐bu4on     releases   Env  &   App   Config   Java,  Ruby,  Groovy,   Scala,  config,  Maven,   Gradle,  etc  
  • Deployment  Pipeline   •  Every  commit  is  considered  a  release  candidate   •  Not  every  release  candidate  is  released   •  Candidates  must  first  pass:   –  Compile,  Unit  Test,  Sta/c  Analysis,  Create/Upload  RPMs   –  Deploy  RPM,  Start  server   –  Smoke  Tests   –  Acceptance  Tests   –  Performance  Tests   –  UAT  /  Manual  Exploratory   –  Business  decision  to  go-­‐live   –  Approval  from  gatekeepers  
  • Deployment  Pipeline   h#p://con/nuousdelivery.com/2010/02/con/nuous-­‐delivery/  
  • Deployment  Pipeline  -­‐  Scripts   •  Deployment  pipelines  are  powered  by  scripts   •  Script  Ideals   –  Obey  the  Unix  Philosophy  –  do  one  thing  well   –  Environment-­‐agnos/c   –  Sensible  exit  code   •  Script  Architecture   –  Build  Scripts   •  Convert  build  output  into  deployment  input   –  Deploy  Scripts   •  Environment-­‐independent  instruc/ons  to  deploy  so?ware   remotely   •  Can  run  on  central  command-­‐and-­‐control  server   –  Test  Scripts   •  Repeatable  instruc/ons  to  invoke  automated  tes/ng  
  • Configura$on   Management   Tes$ng   Deployment   Automa$on   Deployment  Pipeline           Deployment  Pipeline           Configura$on   Management   Tes$ng   Deployment   Automa$on   Disciplined   Development   Risk  Management   Con$nuous  Integra$on   Building  Blocks   Disciplined   Development   Agile  /  Lean  /  DevOps  /  Culture  
  • Disciplines   Incremental  development   Feature  switches   Mainline  development   •  Can’t  even  do  CI  with  mul/ple  branches   Small  releases   Keep  code  releasable   Keep  automated  tests  green   •  TDD:  Red  before  Green   Disciplined  Development  
  • Configura$on   Management   Deployment   Automa$on   Disciplined   Development   Deployment  Pipeline           Deployment  Pipeline           Configura$on   Management   Tes$ng   Deployment   Automa$on   Disciplined   Development   Risk  Management   Con$nuous  Integra$on   Building  Blocks   Tes$ng   Agile  /  Lean  /  DevOps  /  Culture  
  • Tes/ng   h#p://lisacrispin.com/2011/11/08/using-­‐the-­‐agile-­‐tes/ng-­‐quadrants/  
  • Tes/ng   You  can  not  save  money  if  you  are  more  worried  about   money,  than  you  are  about  quality.   W.  Edwards  Deming:   Costs  ↓  Focus  on  Quality  ↑   Costs  ↑  +  Quality  ↓  Focus  on  Costs  ↓  
  • Tes/ng  Triangle   Unit   Manual   Automated   Other   Exploratory   System   Acceptance   Unit  
  • Tes/ng   •  “Eliminate  the  need  for  mass  inspec/ons,  as  the  way  of  life  to   achieve  quality,  by  building  quality  into  the  product  in  the  first   place.”   •  “Quality  doesn’t  come  from  inspec/on,  but  from  improvement  of   the  process.  Improve  the  process  so  that  defects  aren’t  produced  in   the  first  place.  This  eliminates  the  need  for  inspec/on  on  a  mass   basis.”   •  “Rou/ne  inspec/on  is  the  same  as  planning  for  defects,   acknowledging  that  the  process  isn’t  correct,  or  that  the   specifica/ons  made  no  sense  in  the  first  place.”   •  “Inspec/on  is  too  late  as  well  as  ineffec$ve  and  costly.”     h#p://www.signsculpt.com.au/wp-­‐content/uploads/2013/03/focus-­‐on-­‐quality.jpg   h#p://leanandkanban.wordpress.com/2011/07/15/demings-­‐14-­‐points/  
  • Prod-­‐Like  Environments   •  Every  diff  with  Prod  is  a  risk   •  Need  to  take  calculated  risks     –  Trade-­‐offs  for  efficiency   –  Need  to  understand  the  cost   •  E.g.   –  DBs   –  Configura/on  Management   –  Load  Balancers   –  Deployment  Process   •  Spot  the  Difference  =  waste   h#p://www.nairaland.com/a#achments/1500785_spot_jpge3cbd8220a9ec91ea49adz6c79aeb2  
  • Configura$on   Management   Tes$ng   Disciplined   Development   Deployment  Pipeline           Deployment  Pipeline           Configura$on   Management   Tes$ng   Deployment   Automa$on   Disciplined   Development   Risk  Management   Con$nuous  Integra$on   Building  Blocks   Deployment   Automa$on   Agile  /  Lean  /  DevOps  /  Culture  
  • Deployment  Automa/on   •  Releases  should  be  low-­‐ceremony,  stress-­‐free  events   –  We’ve  already  prac/ced  deployment  100s  of  /mes  using   the  exact  same  process  in  prod-­‐like  environments   –  Don’t  have  to  remember  complex  manual  steps  or  rely  on   wri#en  instruc/ons   –  Only  tes/ng  required  is  to  verify  the  environment     •  (not  func/onality  –  already  tested  with  every  commit)   •  Tes/ng  is  automated   –  Releases  take  minutes  (not  hours)  
  • An/pa#ern:     Deploying  So?ware  Manually   •  An/pa#ern  Signs:   –  Extensive,  detailed  release  documenta$on   –  Reliance  on  manual  tes$ng  to  confirm  app  is  correct   –  Explaining  why  deployment  is  going  wrong  on  release  day   –  Frequent  correc$ons  to  release  process  during  release   –  Environments  that  differ  in  their  configura$on   –  Releases  that  take  more  than  a  few  minutes  to  perform   –  Releases  that  are  unpredictable  in  their  outcome  
  • Deployment  Automa/on   •  Deployments  should  tend  towards  being  fully  automated   –  Human:  1)  pick  version  &  env,  2)  press  “deploy”  bu#on   •  Automated  deployment  scripts  =  up-­‐to-­‐date  doco     –  Encourages  collabora$on  –  everything  is  explicit  in  script   •  Don’t  depend  on  the  deployment  expert   –  Boring,  repe$$ve  manual  tasks  requiring  significant  exper$se  ==  most   certain  way  of  ensuring  human  error   •  Automated  deployment  process  is  cheap  and  easy  to  test   –  Deployment  smoke  tests   •  Automated  processes  are  fully  auditable   •  Automated  process  must  be  used  by  everybody   –  Should  be  the  only  way  in  which  the  so?ware  is  ever  deployed  
  • Con/nuous  Deployment?   •  Idea:  Take  pipeline  and  make  final  “deploy  to  prod”  step  automa/c   •  Tests   –  Automated  tests  have  to  be  fantas/c  -­‐  automated  UT/CT/FAT/NFAT  covering  en/re  app   –  Have  to  write  all  tests  first,  so  only  when  story  is  complete  will  check-­‐ins  pass  ATs   •  Intui/ve  objec/on:  too  risky   –  But,  more  frequent  releases  !  lower  risk  in  any  par/cular  release   –  Amount  of  change  between  releases  goes  down   –  Releasing  every  change  !  risk  is  limited  to  just  risk  inherent  in  one  change   –  Con/nuous  deployment  is  a  great  way  to  reduce  the  risk  of  release   –  Forces  you  to  do  the  right  thing   •  Pre-­‐requisites:   –  You  can’t  do  it  without  automa/ng  your  en/re  build,  deploy,  test,  &  release  process   –  You  can’t  do  it  without  a  comprehensive,  reliable  set  of  automated  tests   –  You  can’t  do  it  without  wri/ng  system  tests  that  run  against  a  prod-­‐like  env  
  • Con/nuous  Deployment?   •  If  you  can’t  actually  release  every  set  of  changes  that  passes  all  tests,  aim   to  create  process  that  lets  you  do  so   •  Pipelines  support  this  process  by:   –  Crea/ng  repeatable,  reliable,  automated  system  for  ge{ng  changes  into  prod  ASAP   –  Crea/ng  highest  quality  so?ware  using  highest  quality  process  -­‐  massively  reducing  risks   •  Con/nuous  deployment  takes  this  approach  to  its  logical  conclusion   •  It  should  be  taken  seriously,  because  it  represents  a  paradigm  shi>  in  the   way  so?ware  is  delivered   •  Even  if  you  have  good  reasons  for  not  releasing  every  change  you  make— and  there  are  less  such  reasons  than  you  might  think—you  should  behave   as  if  you  were  going  to  do  so  
  • Database  Migra/on   DB   v10   App  v200   compa/ble  with     DB  v10  &  v11   App  v210   compa/ble  with     DB  v11   App  v220   compa/ble  with     DB  v11  &  v12   Time   App  v230   compa/ble  with     DB  v12   …   DB   v11   DB   v12   …   …   •  Decouple  database  changes  from  applica/on  deployment  
  • Tes$ng   Deployment   Automa$on   Disciplined   Development   Deployment  Pipeline           Deployment  Pipeline           Configura$on   Management   Tes$ng   Deployment   Automa$on   Disciplined   Development   Risk  Management   Con$nuous  Integra$on   Building  Blocks   Configura$on   Management   Agile  /  Lean  /  DevOps  /  Culture  
  • Configura/on  Management   •  Provisioning  &  management  should  be   automated   •  Keep  everything  you  need  to  create/maintain   infra  under  version  control   –  E.g.  Kickstart,  Puppet,  DNS  zone  files,  DHCP,  SMTP,   firewall,  scripts   –  Inputs  to  deployment  pipeline  (same  as  source  code)  
  • Deployment  Toolchain   CI  -­‐>  Repo  -­‐>  Orchestra/on  -­‐>  CM     •  Examples   – TeamCity  -­‐>  Yum  -­‐>  Fabric  -­‐>  Puppet/Hiera   – Bamboo  -­‐>  Yum  -­‐>  Rundeck  -­‐>  Chef   – Jenkins  -­‐>  Nexus  -­‐>  Ansible    
  • DevOps  Toolchain   h#p://www.slideshare.net/AnthonyShortland/dto-­‐chefconf2012  
  • Example  DevOps  Toolchain   h#p://www.slideshare.net/AnthonyShortland/dto-­‐chefconf2012  
  • Deployment  Pipeline           Configura$on   Management   Tes$ng   Deployment   Automa$on   Disciplined   Development   Con$nuous  Integra$on   Building  Blocks   Risk  Management   Agile  /  Lean  /  DevOps  /  Culture  
  • Risk  Management   •  Repeatability  =  confidence   h#p://www.slideshare.net/jallspaw/ops-­‐metametrics-­‐the-­‐currency-­‐you-­‐pay-­‐for-­‐change  
  • Risk  Management   •  What  about  PCI?   –  Lock  down  who  is  able  to  access  “privileged”  envs   –  Change  mgmt  process  for  making  changes  to  privileged  envs   –  Approvals  from  mgmt  before  deployments  can  be  performed   –  Protect  against  poten/al  malicious  interven/ons   –  Audit  all  deployments     •  Deployment  pipeline  makes  it  possible  to  enforce  these   strategies  while  enabling  efficient  delivery  process   –  Automa$on  over  Documenta/on       –  Enforce  Traceability  
  • Risk  Management   •  Process  for  managing  approvals   –  Automated  CR  mgmt  system     –  Adding  access  control  to  deployment  pipeline  is  a  trivial  exercise   •  How  does  CAB  decide  whether  change  should  be  executed?     –  If  risks  outweigh  benefits,  change  should  not  be  made   •  Principles:     –  Keep  metrics  on  the  system  &  make  visible:     •  How  long,  how  many  wai$ng,  propor$on  denied?   •  MTBF/MTTR   •  Cycle  $me   –  Regular  retrospec$ves  &  improve  system  based  on  feedback  
  • CD  Maturity  Model   Figure  15.1  Maturity  model    
  • CD  Maturity  Model   h#ps://flic.kr/p/afDTK9   Below  this  line,  we’re  slipping  backwards   Above  this  line,  we’re  climbing  the  ladder  of  maturity  
  • In  One  Slide   Quality   Cycle  Time  
  • Beyond  the  Book   •  How  to  manage  interconnected  pipelines?   – Independent  Releases:  release  one  service  at  a   /me   – Ensure  all  releases  are  backwards  compa/ble  with   current  consumers   – Discipline  Required   •  Short  cycle  /me  &  small  batch  size   •  Feature  Switching   •  System  tes/ng/monitoring   –  E.g.  synthe/c  transac/ons  
  • Beyond  the  Book   •  Consumer  Driven  Contracts   – Allows  independent  development  and  release  of   dependent  services   h#p://mar/nfowler.com/ar/cles/consumerDrivenContracts.html   C   B   B   A   A   C   Code   Tests   B  dev/tester   Maintains   Calls   As  ‘C’  changes,     its  tests  for   consumers  ‘A’  and   ‘B’  are  executed