Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Making Easy = Right (DevOps @ Wotif)

3,827 views

Published on

"DevOps @ Wotif: Making Easy = Right"
Videos: https://vimeo.com/102211796 / https://www.youtube.com/watch?v=yPy44B9h820
Related IEEE Article: http://online.qmags.com/ISW0516#pg56&mode2
Abstract:
We know the “right thing” to do is to work together, to remove bottlenecks, to automate, automate, automate. But when the status quo is finger-pointing, mistrust, blame, and over-exhaustion from increasingly complex manual releases, it seems impossible to find a way to climb out of the downward spiral. How did an IT department turn this around to become a harmonious department with common goals and over 10x faster cycle times?

Winning the hearts and minds needed for lasting DevOps change requires something more than just great automation. This is the story of how local Brisbane success story, Wotif Group, found a way to incentivise a DevOps transformation across an entire IT department, resulting in cycle times measured in hours instead of weeks/months. While this journey involved tools such as Puppet, Hiera, Fabric & ZooKeeper, Matt (dev) and Alexandra (ops) extract principles that they hope can be applied to any organisation at grassroots and leadership levels using existing toolchains to support not only the best ideas of the present but also the legacy of the past and the unknown innovation of the future.

Speakers:

Alexandra Spillane (@ajbw) is a systems administrator with more than ten years experience administering Linux and Unix systems. She has been a member of Wotif Group's Continuous Delivery team since its inception in 2013. She loves using tools like Puppet to solve thorny problems in creative ways and will be speaking at PuppetConf 2014.

Matt Callanan (@mcallana) is a senior software developer in the Continuous Delivery team at Wotif Group. He has over 14 years development experience including DevOps implementation and agile mentoring in small teams and large enterprises. He is most passionate about helping companies realise their potential through automation and reducing feedback cycles to quickly deliver value to customers. Matt has spoken internationally on DevOps, is a well-received presenter to IT departments and executive teams, and co-organises the DevOps Brisbane Meetup.

http://devopsdays.org/events/2014-brisbane/proposals/devops_at_wotif_making_easy_equals_right/

Published in: Technology
  • Be the first to comment

Making Easy = Right (DevOps @ Wotif)

  1. 1. DevOps @ Wotif Making Easy = Right Alexandra Spillane • @ajbw Matt Callanan • @mcallana Right Easy
  2. 2. Outline • What is “The Right Thing”? • How do we Make it Easy? • Lessons Learned Making Easy = Right 3
  3. 3. What is “The Right Thing”? DevOps @ Wotif: Making Easy = Right Making Easy = Right 4
  4. 4. “The Right Thing” Making Easy = Right 5 Collaborate! Automate!
  5. 5. But… The Dev<->Ops Chasm Making Easy = Right 6
  6. 6. What Were we Doing Wrong? Making Easy = Right 7
  7. 7. Downward Spiral of Manual Activity Making Easy = Right 8
  8. 8. Entrenched Silos Making Easy = Right 9
  9. 9. High Release Overheads Making Easy = Right 10
  10. 10. Huge Batch Sizes Making Easy = Right 11
  11. 11. Poor Visibility Making Easy = Right 12
  12. 12. Calendar of Doom Making Easy = Right 13
  13. 13. Unpredictable Prioritisation Making Easy = Right 14
  14. 14. Superstitious Gatekeeping Making Easy = Right 15 just  in  case
  15. 15. Stagnating Applications Making Easy = Right 16
  16. 16. Manual Deployment Anti-Patterns Making Easy = Right 17 Extensive, detailed release documentation Reliance on manual testing to confirm app is correct Explaining why deployment is going wrong on release day Frequent corrections to release process during release Environments that differ in their configuration Releases that take more than a few minutes to perform Releases that are unpredictable in their outcome
  17. 17. Cycle Time measured in weeks/months Application Pipelines Ops Days Weeks! Days Staging Load Test Production Days Dev QA Dev QA Dev QA Dev QA Dev QA Dev QA Dev QA Dev QA Dev QA !? PrioritisedReleaseQueue
  18. 18. Let’s Migrate to Microservices! • Explosion of manual effort Making Easy = Right 19 A B C D E F A B1 C1 G C2 H I B2 D1 J D2 K L1 M L2 N E F1 O P F2 Q R S T
  19. 19. HeavyWeight to LightWeight Glassfish • Feature overhead • Encouraged manual config • Complex/Slowdeployment DropWizard • Trimmed down to basics • Standard config files • Simple/Fast deployment Making Easy = Right 20
  20. 20. But… Combinatorial Explosion Making Easy = Right 21
  21. 21. Unpredictability Slows You Down Making Easy = Right 22
  22. 22. Moving from wrong to right Making Easy = Right 23 Wrong Right
  23. 23. Reduce Cycle Time Making Easy = Right
  24. 24. Flexibility vs Predictability • Freedom is Great for Innovation • Consistency is Essential For Speed • How do we promote a culture of freedom and responsibility but still provide predictability? Flexible Predictable Making Easy = Right 25
  25. 25. Standardisation Making Easy = Right 26 Standardised Freedom • Standardise on how services are deployed and how they communicate. • Be flexible about their contents.
  26. 26. Defining Standards • Discussions – One-on-one – IM – Emails – Meetings • Incentivise input • Confluence – Application Deployment Standards 1.0 Making Easy = Right 27
  27. 27. Example Standards Making Easy = Right 28 Logging • Log file locations, filenames Log rotate Directories/Files • Ownership, permissions Puppet Supervisord config.yaml • Filename, location Metrics JVM settings Network • Ports • Firewall rules Cron Endpoints • Healthchecks, status, metrics SSL • Certs, keystores RPM • Versioning • Packaging init.d scripts • Sub-commands Versioning Migration Notes
  28. 28. Example Standards Making Easy = Right 29
  29. 29. Future Considerations Making Easy = Right 30
  30. 30. Standards Lifecycle Making Easy = Right 31
  31. 31. Common Ground Making Easy = Right 32
  32. 32. How do we Make it Easy? DevOps @ Wotif: Making Easy = Right Making Easy = Right 33
  33. 33. Automated Verification Making Easy = Right Compliance Test Suite Bring Operations into Development Fast Operational Feedback Test Driven Operational Compatibility Backwards Compatibility
  34. 34. Ops Testing with Fabric Making Easy = Right 36 •Does rpm have correct metadata linking to git repo? Check existence/absence of filesrpm •Is correct version actually installed?yum •Check existence/absence of files. Check file permissions/ownershipls •Is correct version actually running?lsof •Check correct ports exposednetstat •Is correct log format in use?tail •Check cron configgrep •Check contents of standalone jar file – e.g. metadata, library versionsunzip •Check standard endpoints for e.g. content, status code, response timecurl •Check JVM optionsjps •Check 1 and only 1 process runningps
  35. 35. Compliance Test Example Making Easy = Right 37
  36. 36. Reference Implementation • “helloworld-service” • Deployed to production Making Easy = Right 38
  37. 37. Rolling Upgrade • Safely orchestrate cluster upgrades – automated deployment & testing • Initially interactive Making Easy = Right 39 • Check Load Balancer Exit Pool • Orchestrate Puppet Upgrade • Compliance Test • Smoke Test • Feature Test Test • Check Load Balancer Enter Pool • exitpool • stop • yum remove • yum install • Wait for manual testing! • enterpool Manual Release
  38. 38. SLIPway Making Easy = Right 40 • Big review of old release processes – Focus on reducing cycle time • Simple Lightweight Independent Path Way • Simple rules for release process • Keep changes independent
  39. 39. Simple Rules for SLIPWay • Independent • One application per release • Backwards-compatible • No DB/Network/OS changes • No manual testing • Cannot book specific time • Feature switch for time • Ops will service queue within 24 hours • Compliance with latest standards
  40. 40. Simple Rules for SLIPWay 42Making Easy = Right
  41. 41. Simple Rules for SLIPWay 43Making Easy = Right
  42. 42. Application Pipelines Ops Days Weeks! Days Staging Load Test Production Days Dev QA Dev QA Dev QA Dev QA Dev QA Dev QA Dev QA Dev QA Dev QA !? PrioritisedReleaseQueue
  43. 43. 45 Application Pipelines Ops PrioritisedReleaseQueue Dev QA Dev QA Dev QA Dev QA Dev QA Dev QA Dev QA Dev QA Dev QA Staging Load Test Production Staging Load Test Production Staging Load Test Production Staging Load Test Production Staging Load Test Production Staging Load Test Production Staging Load Test Production Staging Load Test Production Staging Load Test Production Hours Hours SimpleRules
  44. 44. Making Easy = Right 46
  45. 45. Making Easy = Right 47
  46. 46. SLIPway Chat Room Making Easy = Right 48
  47. 47. SLIPway Example Release Making Easy = Right 49
  48. 48. Freeing up 3 full time employees 0 5 10 15 20 25 Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Man-hoursperrelease* man-hourstoshiparelease 95% reduction in man-hours GRIPway SLIPway SLIPway introduction -18.5hrs Hours Spent Shipping Release
  49. 49. 0 5 10 15 20 Cycletimeinbusinessdays businessdays-8 -2 86%cycle time reduction SLIPway GRIPway average 2013 GRIPway today Jan JanJulApr Oct Apr 20142013 Jul Oct Time-to-market Reduction Making Easy = Right 2014          
  50. 50. 0 20 40 60 80 100 120 Numberofreleasespermonth Throughputx2.9 190% releases GRIPway average 2013 SLIPway introduction GRIPwaySLIPway SLIPway average GRIPway Jan JanJulApr Oct Apr 20142013 Jul 2.9x Releases Per Month Making Easy = Right
  51. 51. Release Cycle Time Comparison Making Easy = Right 7x lower Avg. Cycle Time 0 1 2 3 4 5 6 7 8 9 10 GRIPway Avg Cycle Time Days SLIPway Avg Cycle Time Days
  52. 52. Commit stage Acceptance stage Capacity Testing Production Continuous  Delivery Continuous  Deployment Automatic  Trigger Manual  Trigger Commit stage Acceptance stage Capacity Testing Production
  53. 53. Release Cycle Time Comparison Making Easy = Right 7x lower Avg. Cycle Time 30x lower vs SLIPway 200x lower vs GRIPway 21mins 0 1 2 3 4 5 6 7 8 9 10 GRIPway Avg Cycle Time Days SLIPway Avg Cycle Time Days Continuous DEPLOYMENT
  54. 54. Upward Spiral Making Easy = Right 56
  55. 55. Lessons Learned DevOps @ Wotif: Making Easy = Right 57
  56. 56. Lessons Learned Making Easy = Right 58 Get the Balance Right Chip Away At The Iceberg Migration Notes The Carrot, Not The Stick Strongly Type Your Discussions Use Semantic Versioning Be Opinionated – Not Arrogant Warranty Period Embrace Legacy
  57. 57. Get the Balance Right - Optimise for Speed No Rules Too Many Rules Making Easy = Right 59
  58. 58. Chip Away at the Iceberg Making Easy = Right 60
  59. 59. Migration Notes • Annotate changes since previous standards – E.g. “(since v2.3)” • Each standard version has migration notes from previous version • Can upgrade between several versions by following migration notes nav bar Making Easy = Right 61 1.0 2.0 2.1 2.2 2.3
  60. 60. The Carrot, Not The Stick • We tried the stick approach for years – You’re doing it wrong! – (But we won’t tell you how to do it right) • Blame games, anger • No business impetus to improve things • Then we tried the carrot approach – Offer a great new alternative – It’s optional! – You know you want it… • Business invested in the process Making Easy = Right 62
  61. 61. Strongly Type Your Discussions • Choose catchy names – Helps make decisions – Helps climb out of muck • Use State Transition tables – State -> Action -> New State – Visualise problems – Decide what to support Making Easy = Right 63
  62. 62. State Transitions – Current Making Easy = Right 64
  63. 63. State Transitions – Future Making Easy = Right 65
  64. 64. Use Semantic Versioning • Especially for shared libraries Making Easy = Right 66 1 . 2 . 3 Major Minor Patch semver.org
  65. 65. Be Opinionated – Not Arrogant • Strong decisions are important • But you’ll never get it 100% up front Making Easy = Right 67 I’m kind of a big deal
  66. 66. Warranty Period • Can’t get standards 100% right • But need to lock them down • Two-week warranty period • Trial in range of different applications Making Easy = Right 68
  67. 67. Small Batch Size for Standards • Minor updates every 1-2 months • Fast feedback • Enable innovation • Easier to update Making Easy = Right 69
  68. 68. Embrace Legacy • You’re creating legacy • Make migration easy • Incentivise catch-up Making Easy = Right 70
  69. 69. Building Blocks SLIPway • Simple Rules, Independence Rolling Upgrade • Automated Deploy & Test Compliance Tests • Test-driven Ops Compatibility Standards • The Agreed “Right Thing” Making Easy = Right 71 The Right Thing Making It Easy
  70. 70. Key Takeaways • Make the interface between teams (dev/ops) consistent and predictable • Cultural choices (agreement) > tool choices • Prioritise operational feedback to developers • Faster releases leads to smaller batch size leads to lower risk leads to happier customers • Start with interactive automation • Independent releases with no dependencies • Incentivise legacy catch-up through operational compliance to latest agreed standards Making Easy = Right 72 Consistent  and  predictable  interface between  teams  (dev/ops) Cultural  choices  (agreement)  >  tool  choices Prioritise fast  operational  feedback  to  developers Faster  releases  à smaller  batch  size  à lower  risk  à happier   customers Build  trust  with  interactive  automation Decoupled  releases  – independent&  dependency-­‐free à speed Incentivise legacy  catch-­‐up
  71. 71. Making Easy = Right 73 Right Easy
  72. 72. Making Easy = Right 74 Right Easy
  73. 73. Thanks for listening! Any questions? Alexandra Spillane • @ajbw Matt Callanan • @mcallana slides, etc • mattcallanan.net IEEE Article • http://bit.ly/29htOn8 Making Easy = Right
  74. 74. Image Attribution Image “Downward  Spiral”  (http://flic.kr/p/67Giiz)  by  Chad  K is  licensed  under  CC  BY  2.0  (https://creativecommons.org/licenses/by/2.0/) “Angels  Landing  from  the  Deertrap  Mountain  Trail”  (Chasm  -­‐ https://flic.kr/p/abuWHt)  by  Zion  National  Park is  licensed  under  CC  BY  2.0  (https://creativecommons.org/licenses/by/2.0/) “WRONG  WAY”  (https://flic.kr/p/cRjc6q)   by  David  Goehring is  licensed  under  CC  BY  2.0  (https://creativecommons.org/licenses/by/2.0/) “Old  Gravel  Silos,  Buffalo  Bayou,  East  of  Jensen,  Houston,  Texas  0906091555BW” (https://flic.kr/p/6WueVm)   by  Patrick  Feller is  licensed  under  CC  BY  2.0   (https://creativecommons.org/licenses/by/2.0/) “Fisher  men  lifting  a  boat  in  Bangladesh.  Photo  by  Finn  Thilsted”  (https://flic.kr/p/ddUf4S)  by  WorldFish is  licensed  under  CC  BY  NC ND  2.0 (https://creativecommons.org/licenses/by-­‐nc-­‐nd/2.0) “Poor  visibility  from  Kelud  eruption  14  February  2014,  Yogyakarta”  (http://bit.ly/1N3Lf7A)  by  Aldnonymous is  licensed  under  BY  SA  3.0  (https://creativecommons.org/licenses/by-­‐sa/3.0/deed.en) “Changed  priorities ahead”  (https://flic.kr/p/debvm)  by  Peter  Reed is  licensed  under  CC  BY NC  2.0 (https://creativecommons.org/licenses/by-­‐nc/2.0) “Stop Sign” (https://flic.kr/p/aUEW1D) by DonkeyHotey is licensed under CC BY 2.0 (https://creativecommons.org/licenses/by/2.0/) “green  scum”  (https://flic.kr/p/3MvvsH)  by  M&R  Glasgow is  licensed  under  CC  BY  ND  2.0 (https://creativecommons.org/licenses/by-­‐nd/2.0) “Thinking…  please  wait “  (https://flic.kr/p/6wdLat)  by  Karola  Riegler is  licensed  under  CC  BY  ND  2.0  (https://creativecommons.org/licenses/by-­‐nd/2.0) “Elastic“ (https://flic.kr/p/a2XZDB)  by  Chris  Stevenson is  licensed  under  CC  BY  NC  2.0 (https://creativecommons.org/licenses/by-­‐nc/2.0) “A  Garden  Of  Climbs”  (https://flic.kr/p/4U2BB1)   by  Jasen  Miller is  licensed  under  CC  BY  2.0  (https://creativecommons.org/licenses/by/2.0/) “Alexander  Vinokourov  competing  in  the  London  2012  Men's  Olympic  Time  Trial”  (http://bit.ly/1QndiBd)  by  Diliff is  licensed  under  BY  SA  3.0 (https://creativecommons.org/licenses/by-­‐sa/3.0/) “Upward  Spiral”  (https://flic.kr/p/noFia)  by  Clint  Vigil by  is  licensed  under  CC  BY  NC  2.0 (https://creativecommons.org/licenses/by-­‐nc/2.0) “Rivers  of  Humanity”  (Indian  Traffic  -­‐ https://flic.kr/p/3yMSMW)  by  pangalactic  gargleblaster  and  the  heart  of  gold is  licensed  under  CC  BY  NC ND  2.0  (https://creativecommons.org/licenses/by-­‐nc-­‐ nd/2.0) “Red  Tape“ (https://flic.kr/p/b7T3JD) by  Free  Press/  Free  Press  Action  Fund is  licensed  under  BY  NC  SA  2.0 (https://creativecommons.org/licenses/by-­‐nc-­‐sa/2.0) “To  the  Right  A  Bit”  (Slipway  -­‐ https://flic.kr/p/9oQVh8) by  tiffany  terry is  licensed  under  CC  BY  NC ND  2.0  (https://creativecommons.org/licenses/by-­‐nc-­‐nd/2.0) “Iconic  iceberg  sail-­‐by” (https://flic.kr/p/hV1vwH) by  Visit  Greenland is  licensed  under  CC  BY  NC ND  2.0 (https://creativecommons.org/licenses/by-­‐nc-­‐nd/2.0) “Luckiamute  Falls”  (https://flic.kr/p/7EMs4T) by  Ian  Sane is  licensed  under  CC  BY  2.0  (https://creativecommons.org/licenses/by/2.0/) “Three Keys” (https://openclipart.org/detail/219938/water)  by  maryjean.jones@fennwoode.com is  licensed  under  unlimited-­‐commercial-­‐use (https://openclipart.org/unlimited-­‐commercial-­‐use-­‐ clipart) Presentation  includes  stock  images  used  under license  from  Shutterstock.com and  iStockPhoto.com.  Authors  include:  Stepanek Photography/Shutterstock.com,  skvoor/Shutterstock.com,  Michael  D   Brown/Shutterstock.com,  argus/Shutterstock.com,  Sergey  Nivens/Shutterstock.com,  Anze Mulec/Shutterstock.com,  Jaz_czc/Shutterstock.com,  DenisNata/Shutterstock.com,  Arnaud   Weisser/Shutterstock.com,  iStock.com/gehringj Standardisation Theory  Diagram  inspired  by  Sam  Newman:  http://www.slideshare.net/spnewman/practical-­‐microservices-­‐yow-­‐2013/56

×