• Save
DevOps @ Wotif
Upcoming SlideShare
Loading in...5
×
 

DevOps @ Wotif

on

  • 465 views

"DevOps @ Wotif.com" - Alexandra Spillane & Matt Callanan present the journey Wotif has been on implementing a DevOps and Continuous Delivery transformation to reduce cycle times measured in months ...

"DevOps @ Wotif.com" - Alexandra Spillane & Matt Callanan present the journey Wotif has been on implementing a DevOps and Continuous Delivery transformation to reduce cycle times measured in months down to hours, sometimes even minutes. Importantly, not just about the tools (such as Puppet, Hiera, Fabric, TeamCity, ZooKeeper, etc) which featured heavily in the transformation but also about the cultural changes and how they were implemented, the lessons learned, and the key takeaways that attendees can take back to their workplaces. Alexandra & Matt work in the "Continuous Delivery" team at Wotif, charged with rolling out an enterprise-wide adoption of DevOps & Continuous Delivery.

DevOps Brisbane: http://www.meetup.com/Devops-Brisbane/events/184682022/

Video: https://www.youtube.com/watch?v=L8y5L3YQFk8

Statistics

Views

Total Views
465
Views on SlideShare
463
Embed Views
2

Actions

Likes
1
Downloads
0
Comments
0

1 Embed 2

http://www.slideee.com 2

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

DevOps @ Wotif DevOps @ Wotif Presentation Transcript

  • DevOps @ Wotif DevOps @ Wotif Matt Callanan @mcallana Alexandra Spillane @ajbw
  • DevOps @ Wotif Outline •  History & Background •  Solutions •  Principles Learned 2
  • HISTORY & BACKGROUND 3
  • DevOps @ Wotif 2009 Manage DeploySupport Hosting Provider OperationsDevelopment 4History & Background
  • DevOps @ Wotif 2010 Manage DeploySupport…? OperationsDevelopment 5History & Background
  • DevOps @ Wotif 2010-2012 Manage Deploy, Support “Throw it over the wall” OperationsDevelopment 6History & Background
  • DevOps @ Wotif DOWNWARD SPIRAL 7History & Background
  • DevOps @ Wotif 2012 - Downward Spiral •  Entrenched silos •  High release overheads •  Huge batch sizes •  Calendar of Doom •  Unpredictable prioritisation •  Superstitious gatekeeping •  Stagnating applications •  CD anti-patterns list read like someone had recorded our processes! h"p://flic.kr/p/67Giiz   8History & Background
  • DevOps @ Wotif Release Anti-Patterns 9 Extensive, detailed release documentation   Reliance on manual testing to confirm app is correct   Explaining why deployment is going wrong on release day   Frequent corrections to release process during release   Environments that differ in their configuration   Releases that take more than a few minutes to perform   Releases that are unpredictable in their outcome   History & Background
  • DevOps @ Wotif Plus… •  Migrating to MSA = explosion of manual effort 10History & Background
  • DevOps @ Wotif Manual Effort – Impact on Roles 11 Dev Manual tag/release process Disparate Build Servers Deploys done by different teams to divergent environments QA Manual deploys Environment drift Different Puppet repos to production Manual testing No time to fully regression test Ops Manual deploys Cheat Sheets Had to learn idiosyncrasies of each application through experience History & Background
  • DevOps @ Wotif OLD RELEASE SCENARIO 12History & Background
  • Application Pipelines Ops Days Weeks! Days Staging Load Test Production Days Dev QA Dev QA Dev QA Dev QA Dev QA Dev QA Dev QA Dev QA Dev QA PrioritisedReleaseQueue
  • DevOps @ Wotif One application’s deployment... Copy and paste line-by-line… /etc/init.d/wcs exitpool securepayment # watch /logs/network/lb/*pool.log on log1 for appropriate port number exit /etc/init.d/wcs stop securepayment yum clean all yum remove securepayment-domain # run migrate db script prior to deployment: https://svn/OracleProject/app/script1.sql yum install securepayment-domain-1.30-9 yum install securepayment-application-2011.10.31-78 /apps/system/scripts/sync_content.sh -d /apps/wotifapps/securepayment/content -s app5 -s app6 -u contentsync -p /var/www/vhosts/images.wotif.com/webroot/booking/ # run migrate db script following deployment: https://svn/OracleProject/app/script2.sql # Execute test plan /etc/init.d/wcs enterpool securepayment on 3x servers… followed by a second app on 2x servers… followed by a third app on 1 server… followed by a fourth app on 2x servers… followed by a fifth app on 2x servers… followed by followed by a point’n’click ASP release… followed by 3 more manual database scripts… with pauses after every instance for manual testing! … 5 hours later ... 14History & Background
  • SOLUTIONS 15
  • DevOps @ Wotif Solutions •  Lightweight Standardisation •  Technical Solutions •  Puppet •  Rolling Upgrade •  Team Structure •  SLIPWay onewaystock.com   16Solutions
  • DevOps @ Wotif LIGHTWEIGHT STANDARDISATION 17Solutions
  • DevOps @ Wotif HeavyWeight to LightWeight 18Solutions ! Lightweight Standardisation
  • DevOps @ Wotif DropWizard •  Opinionated Java framework for 
 developing ops-friendly, 
 high-performance, RESTful 
 web services 19Solutions ! Lightweight Standardisation
  • DevOps @ Wotif Glassfish •  Feature overhead •  Encouraged manual config •  Complex/Slow deployment •  We’d built standard scripts –  Scripts were clunky –  Scripts wouldn’t get updated DropWizard •  Trimmed down to basics •  Standard config files •  Simple/Fast deployment •  No standard scripts –  Combinatorial explosion of interaction styles HeavyWeight to LightWeight 20Solutions ! Lightweight Standardisation
  • DevOps @ Wotif Combinatorial Explosion 21Solutions ! Lightweight Standardisation
  • DevOps @ Wotif Flexibility vs Predictability •  Freedom is Good –  Adopt new technologies and adapt to market •  Opinions are Good –  Be opinionated to reduce combinatorial complexity for production support •  How do we promote a culture of freedom and responsibility but still provide predictability? ? 22 Flexible Predictable Solutions ! Lightweight Standardisation
  • DevOps @ Wotif Standards •  “Standards” doesn’t have to be a dirty word •  “Any standard is better than no standard” •  People want to do the right thing •  People will do the easiest thing •  Make it easy to do the right thing 23Solutions ! Lightweight Standardisation
  • DevOps @ Wotif Standardisation: Theory 24Solutions ! Lightweight Standardisation
  • DevOps @ Wotif Defining •  Confluence Page –  “Application Deployment Standards 1.0 (DRAFT)” •  Extract input from people: –  Ops: This will make life easier at 2am –  Dev: This will enable faster releases •  Discussions: Ops & Dev involvement –  Meetings: Devs, Ops, Leads, Architects, Managers –  Emails –  IM –  One-on-one 25Solutions ! Lightweight Standardisation
  • DevOps @ Wotif Standards Lifecycle 26Solutions ! Lightweight Standardisation
  • DevOps @ Wotif Example Standards 27 Logging •  Log file locations, filenames Log rotate Directories/Files •  Ownership, permissions Puppet Supervisord config.yaml •  Filename, location Metrics JVM settings Network •  Ports •  Firewall rules Cron Endpoints •  Healthchecks, status, metrics SSL •  Certs, keystores RPM •  Versioning •  Packaging init.d scripts •  Sub-commands Versioning Migration Notes Solutions ! Lightweight Standardisation
  • DevOps @ Wotif Example  Standards   28Solutions ! Lightweight Standardisation
  • DevOps @ Wotif Example  Standards   29Solutions ! Lightweight Standardisation
  • DevOps @ Wotif Future  ConsideraBons   30Solutions ! Lightweight Standardisation
  • DevOps @ Wotif Reference Implementation •  “helloworld-service” •  Deployed to production 31Solutions ! Lightweight Standardisation
  • DevOps @ Wotif Standardisation: Testing •  Standards are pointless without auto- verification •  Test-driven ops compatibility •  Backwards-compatible –  Test cases are versioned alongside standards •  Reference implementation 32Solutions ! Lightweight Standardisation
  • DevOps @ Wotif Standardisation: Testing 33 • Does rpm have correct metadata linking to git repo? Check existence/absence of filesrpm • Is correct version actually installed?yum • Check existence/absence of files. Check file permissions/ownershipls • Is correct version actually running?lsof • Check correct ports exposednetstat • Is correct log format in use?tail • Check cron configgrep • Check contents of standalone jar file – e.g. metadata, library versionsunzip • Check standard endpoints for e.g. content, status code, response timecurl • Check JVM optionsjps • Check 1 and only 1 process runningps Solutions ! Lightweight Standardisation
  • DevOps @ Wotif Standards Compliance Tests 34Solutions ! Lightweight Standardisation
  • DevOps @ Wotif Migration Notes •  Annotate changes since previous standards –  E.g. "(since v2.3)” •  Each standard version has migration notes from previous version •  Can upgrade between several versions by following migration notes nav bar 35 1.0 2.0 2.1 2.2 2.3 Solutions ! Lightweight Standardisation
  • DevOps @ Wotif MigraBon  Notes   36Solutions ! Lightweight Standardisation
  • DevOps @ Wotif Standardisation Progress 37 Standardised Packaging Standardised Deployment Puppet Smoke Tests Solutions ! Lightweight Standardisation
  • DevOps @ Wotif TECHNICAL SOLUTIONS 38Solutions
  • DevOps @ Wotif Puppet •  Yay, configuration management! But… •  Prod(-like) environment != test environment •  Prod puppet != test puppet •  Merge! –  post-merge: “if env == ‘test’”, “if env == ‘prod’” conditionals everywhere •  Difficult to test changes •  Difficult to promote changes through environments 39Solutions ! Technical Solutions
  • DevOps @ Wotif Puppet Branches •  Puppet repo locked down, just like prod! –  Ops gate-keep all master commits •  How can devs contribute? •  “develop” Branch? Branch per environment? –  Never merged to master –  Never merged from master 40Solutions ! Technical Solutions
  • DevOps @ Wotif Puppet Virtual Branches! •  /etc/puppet… •  Nightly rebuild / rebuild at will •  master + all opted-in branches –  Branch list maintained in Hiera –  Skip conflicting branches •  Remove “old” branches –  Encourages people to merge to master! 41Solutions ! Technical Solutions
  • DevOps @ Wotif Promoting changes •  “if env == ‘test’” / “if env == ‘prod’” –  Gets old fast! •  Promoting changes to production means untested merge requests mid-release! •  Unmaintainable, untidy, error-prone 42Solutions ! Technical Solutions
  • DevOps @ Wotif Puppet Feature Switches •  Eradicate environment conditionals •  Encouraged to be temporary – “Gimmicks” •  gimmick() is a convenience wrapper for hiera() if env == ‘test’ test.yaml: gimmick.some_feature: true prod.yaml: gimmick.some_feature: false if gimmick(‘feature’) 43Solutions ! Technical Solutions
  • DevOps @ Wotif Rolling Upgrade •  Safely automate upgrades of HA clustered applications 44 •  Check Load Balancer Exit Pool •  Orchestrate Puppet Upgrade •  Compliance Test •  Smoke Test •  Feature Test Test •  Check Load Balancer Enter Pool •  exitpool •  stop •  yum remove •  yum install •  Wait for manual testing •  enterpool Manual Release Solutions ! Technical Solutions
  • DevOps @ Wotif Rolling Upgrade 45Solutions ! Technical Solutions
  • DevOps @ Wotif Rolling Upgrade 46Solutions ! Technical Solutions
  • DevOps @ Wotif Rolling  Upgrade:  Summary   47Solutions ! Technical Solutions
  • DevOps @ Wotif Rolling  Upgrade:  Smoke  Tests   48Solutions ! Technical Solutions
  • DevOps @ Wotif TEAM STRUCTURE 49Solutions
  • DevOps @ Wotif Team Structure •  ITSSU re-org late 2013 –  “Houses” led by senior devs (not BAs or PMs) •  Allowed a focus on tech debt & service delivery –  Continuous Delivery team •  Tech innovation •  Third level support •  Communication! 50Solutions ! Team Structure
  • DevOps @ Wotif Continuous Delivery Team •  Communication facilitators –  Not know-it-all / do-it-all heroes! –  Emphasis on collaboration –  chatrooms galore –  “peer-to-peer help” •  Weekly DevOps standups –  Any interested devs / ops / architects –  Raise anything interesting to a wider audience •  DevOps champions 51Solutions ! Team Structure
  • DevOps @ Wotif SLIPWAY 52Solutions
  • DevOps @ Wotif “SLIPWay” •  Big review of old release processes •  Simple Lightweight Independent Path Way •  Simple rules for release process •  Keep changes independent •  Focus on reducing cycle time 53 h"p://goo.gl/vjaN9F   Solutions ! SLIPWay
  • Application Pipelines Ops Days Weeks! Days Staging Load Test Production Days Dev QA Dev QA Dev QA Dev QA Dev QA Dev QA Dev QA Dev QA Dev QA PrioritisedReleaseQueue
  • Application Pipelines Ops PrioritisedReleaseQueue Dev QA Dev QA Dev QA Dev QA Dev QA Dev QA Dev QA Dev QA Dev QA Staging Load Test Production Staging Load Test Production Staging Load Test Production Staging Load Test Production Staging Load Test Production Staging Load Test Production Staging Load Test Production Staging Load Test Production Staging Load Test Production Hours Hours LightweightRules
  • DevOps @ Wotif SLIPWay Conditions 56Solutions ! SLIPWay
  • DevOps @ Wotif SLIPWay Conditions 57Solutions ! SLIPWay
  • DevOps @ Wotif SLIPWay Chat Room 60Solutions ! SLIPWay
  • DevOps @ Wotif SLIPWay  Example  Release   61Solutions ! SLIPWay
  • DevOps @ Wotif Total  releases  in  last  8  months   62Solutions ! SLIPWay # Releases
  • DevOps @ Wotif Manual Effort – Impact on Roles 63 Dev Manual tag/release process Disparate Build Servers Deploys done by different teams to divergent environments QA Manual deploys Environment drift Different Puppet repos to production Manual testing No time to fully regression test Ops Manual deploys Cheat Sheets Had to learn idiosyncrasies of each application through experience Solutions ! SLIPWay
  • DevOps @ Wotif Automation – Impact on Roles 64 Dev Deploy to prod-like environment on every commit Compliance test & smoke test on every commit Tweak Maven to allow any commit to be released without manual intervention Consolidate build servers for consistency and visibility QA Automated self-service deploys to exploratory testing environments Test environment as close to production as possible via shared Puppet repository Regression testing covered by automated Acceptance Tests Could spend time on exploratory testing Ops Have confidence that application has been through compliance testing Have confidence in rejecting an application if it fails compliance tests or smoke tests Have less idiosyncrasies and less rules to keep in their head when supporting applications in production Solutions ! SLIPWay
  • PRINCIPLES 65
  • DevOps @ Wotif Principles 66 You Can't CHANGE Culture Strongly Type Your Discussions Make it Easy To Do The Right Thing Reduce Transaction Cost Before Batch Size The Carrot, Not The Stick Principles
  • DevOps @ Wotif You Can’t CHANGE Culture •  … but you can change it. •  Start with what you have –  Incremental, palatable changes –  Involve everyone 67Principles ! You Can’t CHANGE Culture
  • DevOps @ Wotif Strongly Type Your Discussions •  Choose catchy/easy-to-pronounce names –  Helps to make decisions –  Helps climb out of the muck –  If someone thinks of a good name - go with it •  Use state tables and transition diagrams –  State -> Action -> New State –  Visualise current problems –  Decide which state transitions to support 68Principles ! Strongly Type Your Discussions
  • DevOps @ Wotif E.g. Glassfish Service States 69Principles ! Strongly Type Your Discussions
  • DevOps @ Wotif E.g. Lightweight Service States 70Principles ! Strongly Type Your Discussions
  • DevOps @ Wotif Be Opinionated – Not Arrogant •  Strong decisions are important •  But never think you’ll get it all correct up front 71 h"p://goo.gl/YN1WJz   Principles ! Be Opinionated
  • DevOps @ Wotif Make it Easy To Do The Right Thing 72Principles ! Make Right Easy Easy Right
  • DevOps @ Wotif Make it Easy To Do The Right Thing 73Principles ! Make Right Easy Easy Right
  • DevOps @ Wotif Reduce Transaction Cost Before Batch Size flic.kr/p/bEVLmG   74Principles ! Reduce Transaction Cost
  • DevOps @ Wotif Batch Size vs Risk h"p://www.slideshare.net/jallspaw/ops-­‐metametrics-­‐the-­‐currency-­‐you-­‐pay-­‐for-­‐change   •  “reduce risk by reducing batch size” 75Principles ! Reduce Transaction Cost
  • DevOps @ Wotif Batch Size Science •  But we couldn’t reduce batch size until we reduced transaction (release) cost 76Principles ! Reduce Transaction Cost
  • DevOps @ Wotif High Release Cost = High Batch Size                                                                                                                       Batch  Size   Cost   Release  Cost   Total  Cost   77Principles ! Reduce Transaction Cost
  • DevOps @ Wotif Low Release Cost = Low Batch Size Batch  Size   Cost   Release  Cost   78Principles ! Reduce Transaction Cost
  • DevOps @ Wotif The Carrot, Not The Stick h"p://www.creditcards.com/credit-­‐card-­‐news/images/carrot-­‐no-­‐sBck2.jpg   79Principles ! The Carrot, Not The Stick
  • DevOps @ Wotif Stick vs Carrot •  We tried the stick approach for years ­ You’re doing it wrong! ­ (But we won’t tell you how to do it right) •  Blame games, anger •  No business impetus to improve things 80 •  Now we’re trying the carrot approach ­ Offer a great new alternative – It’s optional! – You know you want it… •  Business invested in the process Principles ! The Carrot, Not The Stick
  • RECAP 81
  • DevOps @ Wotif 2010-2012 Manage Deploy, Support “Throw it over the wall” OperationsDevelopment 82Recap
  • DevOps @ Wotif 2014 Manage Deploy (fast & painless) ((mostly)) OperationsDevelopment 83 COMMUNICATE! Recap
  • DevOps @ Wotif Recap •  History & Background •  Solutions –  Team Structure –  Puppet tweaks –  Rolling Upgrade deployment mechanism –  SLIPWay rapid release process •  Principles –  You Can’t CHANGE Culture –  Strongly Type Your Discussions –  Make it Easy to do the Right Thing –  Reduce Transaction Cost Before Batch Size –  The Carrot, Not The Stick 84Recap
  • Thanks for listening! Any questions?
  • Yes, we’re hiring! www.wotifgroup.com/working-at-wotifgroup/vacancies