DevOps @ Wotif
DevOps @ Wotif
Matt Callanan
@mcallana
Alexandra Spillane
@ajbw
DevOps @ Wotif
Outline
•  History & Background
•  Solutions
•  Principles Learned
2
HISTORY & BACKGROUND
3
DevOps @ Wotif
2009
Manage
DeploySupport
Hosting Provider
OperationsDevelopment
4History & Background
DevOps @ Wotif
2010
Manage
DeploySupport…?
OperationsDevelopment
5History & Background
DevOps @ Wotif
2010-2012
Manage
Deploy,
Support
“Throw it over the wall”
OperationsDevelopment
6History & Background
DevOps @ Wotif
DOWNWARD SPIRAL
7History & Background
DevOps @ Wotif
2012 - Downward Spiral
•  Entrenched silos
•  High release overheads
•  Huge batch sizes
•  Calendar of Doo...
DevOps @ Wotif
Release Anti-Patterns
9
Extensive, detailed release documentation	
  
Reliance on manual testing to confirm...
DevOps @ Wotif
Plus…
•  Migrating to MSA = explosion of manual effort
10History & Background
DevOps @ Wotif
Manual Effort – Impact on Roles
11
Dev
Manual tag/release
process
Disparate Build Servers
Deploys done by
d...
DevOps @ Wotif
OLD RELEASE SCENARIO
12History & Background
Application Pipelines Ops
Days
Weeks!
Days
Staging
Load
Test
Production
Days
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev...
DevOps @ Wotif
One application’s deployment...
Copy and paste line-by-line…
/etc/init.d/wcs exitpool securepayment
# watch...
SOLUTIONS
15
DevOps @ Wotif
Solutions
•  Lightweight Standardisation
•  Technical Solutions
•  Puppet
•  Rolling Upgrade
•  Team Struct...
DevOps @ Wotif
LIGHTWEIGHT STANDARDISATION
17Solutions
DevOps @ Wotif
HeavyWeight to LightWeight
18Solutions ! Lightweight Standardisation
DevOps @ Wotif
DropWizard
•  Opinionated Java framework for 

developing ops-friendly, 

high-performance, RESTful 

web s...
DevOps @ Wotif
Glassfish
•  Feature overhead
•  Encouraged manual config
•  Complex/Slow deployment
•  We’d built standard s...
DevOps @ Wotif
Combinatorial Explosion
21Solutions ! Lightweight Standardisation
DevOps @ Wotif
Flexibility vs Predictability
•  Freedom is Good
–  Adopt new technologies and adapt to market
•  Opinions ...
DevOps @ Wotif
Standards
•  “Standards” doesn’t have to be a dirty word
•  “Any standard is better than no standard”
•  Pe...
DevOps @ Wotif
Standardisation: Theory
24Solutions ! Lightweight Standardisation
DevOps @ Wotif
Defining
•  Confluence Page
–  “Application Deployment Standards 1.0 (DRAFT)”
•  Extract input from people:
–...
DevOps @ Wotif
Standards Lifecycle
26Solutions ! Lightweight Standardisation
DevOps @ Wotif
Example Standards
27
Logging
•  Log file locations,
filenames
Log rotate
Directories/Files
•  Ownership,
pe...
DevOps @ Wotif
Example	
  Standards	
  
28Solutions ! Lightweight Standardisation
DevOps @ Wotif
Example	
  Standards	
  
29Solutions ! Lightweight Standardisation
DevOps @ Wotif
Future	
  ConsideraBons	
  
30Solutions ! Lightweight Standardisation
DevOps @ Wotif
Reference Implementation
•  “helloworld-service”
•  Deployed to production
31Solutions ! Lightweight Standa...
DevOps @ Wotif
Standardisation: Testing
•  Standards are pointless without auto-
verification
•  Test-driven ops compatibil...
DevOps @ Wotif
Standardisation: Testing
33
• Does rpm have correct metadata linking to git repo? Check existence/absence o...
DevOps @ Wotif
Standards Compliance Tests
34Solutions ! Lightweight Standardisation
DevOps @ Wotif
Migration Notes
•  Annotate changes since previous standards 
–  E.g. "(since v2.3)”
•  Each standard versi...
DevOps @ Wotif
MigraBon	
  Notes	
  
36Solutions ! Lightweight Standardisation
DevOps @ Wotif
Standardisation Progress
37
Standardised
Packaging
Standardised
Deployment
Puppet
Smoke Tests
Solutions ! L...
DevOps @ Wotif
TECHNICAL SOLUTIONS
38Solutions
DevOps @ Wotif
Puppet
•  Yay, configuration management! But…
•  Prod(-like) environment != test environment
•  Prod puppet ...
DevOps @ Wotif
Puppet Branches
•  Puppet repo locked down, just like prod!
–  Ops gate-keep all master commits
•  How can ...
DevOps @ Wotif
Puppet Virtual Branches!
•  /etc/puppet…
•  Nightly rebuild / rebuild at will
•  master + all opted-in bran...
DevOps @ Wotif
Promoting changes
•  “if env == ‘test’” / “if env == ‘prod’”
–  Gets old fast!
•  Promoting changes to prod...
DevOps @ Wotif
Puppet Feature Switches
•  Eradicate environment conditionals
•  Encouraged to be temporary – “Gimmicks”
• ...
DevOps @ Wotif
Rolling Upgrade
•  Safely automate upgrades of HA clustered
applications
44
•  Check Load
Balancer
Exit Poo...
DevOps @ Wotif
Rolling Upgrade
45Solutions ! Technical Solutions
DevOps @ Wotif
Rolling Upgrade
46Solutions ! Technical Solutions
DevOps @ Wotif
Rolling	
  Upgrade:	
  Summary	
  
47Solutions ! Technical Solutions
DevOps @ Wotif
Rolling	
  Upgrade:	
  Smoke	
  Tests	
  
48Solutions ! Technical Solutions
DevOps @ Wotif
TEAM STRUCTURE
49Solutions
DevOps @ Wotif
Team Structure
•  ITSSU re-org late 2013
–  “Houses” led by senior devs (not BAs or PMs)
•  Allowed a focus...
DevOps @ Wotif
Continuous Delivery Team
•  Communication facilitators
–  Not know-it-all / do-it-all heroes!
–  Emphasis o...
DevOps @ Wotif
SLIPWAY
52Solutions
DevOps @ Wotif
“SLIPWay”
•  Big review of old
release processes
•  Simple Lightweight
Independent Path Way
•  Simple rules...
Application Pipelines Ops
Days
Weeks!
Days
Staging
Load
Test
Production
Days
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev...
Application Pipelines Ops
PrioritisedReleaseQueue
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Staging L...
DevOps @ Wotif
SLIPWay Conditions
56Solutions ! SLIPWay
DevOps @ Wotif
SLIPWay Conditions
57Solutions ! SLIPWay
DevOps @ Wotif
SLIPWay Chat Room
60Solutions ! SLIPWay
DevOps @ Wotif
SLIPWay	
  Example	
  Release	
  
61Solutions ! SLIPWay
DevOps @ Wotif
Total	
  releases	
  in	
  last	
  8	
  months	
  
62Solutions ! SLIPWay
# Releases
DevOps @ Wotif
Manual Effort – Impact on Roles
63
Dev
Manual tag/release
process
Disparate Build Servers
Deploys done by
d...
DevOps @ Wotif
Automation – Impact on Roles
64
Dev
Deploy to prod-like
environment on every
commit
Compliance test & smoke...
PRINCIPLES
65
DevOps @ Wotif
Principles
66
You Can't CHANGE Culture
Strongly Type Your Discussions
Make it Easy To Do The Right Thing
Re...
DevOps @ Wotif
You Can’t CHANGE Culture
•  … but you can change it.
•  Start with what you have
–  Incremental, palatable ...
DevOps @ Wotif
Strongly Type Your Discussions
•  Choose catchy/easy-to-pronounce names
–  Helps to make decisions
–  Helps...
DevOps @ Wotif
E.g. Glassfish Service States
69Principles ! Strongly Type Your Discussions
DevOps @ Wotif
E.g. Lightweight Service States
70Principles ! Strongly Type Your Discussions
DevOps @ Wotif
Be Opinionated – Not Arrogant
•  Strong decisions are important
•  But never think you’ll get it all correc...
DevOps @ Wotif
Make it Easy To Do The Right Thing
72Principles ! Make Right Easy
Easy
Right
DevOps @ Wotif
Make it Easy To Do The Right Thing
73Principles ! Make Right Easy
Easy
Right
DevOps @ Wotif
Reduce Transaction Cost Before
Batch Size
flic.kr/p/bEVLmG	
  
74Principles ! Reduce Transaction Cost
DevOps @ Wotif
Batch Size vs Risk
h"p://www.slideshare.net/jallspaw/ops-­‐metametrics-­‐the-­‐currency-­‐you-­‐pay-­‐for-­...
DevOps @ Wotif
Batch Size Science
•  But we couldn’t reduce batch size until we
reduced transaction (release) cost
76Princ...
DevOps @ Wotif
High Release Cost = High Batch Size
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  ...
DevOps @ Wotif
Low Release Cost = Low Batch Size
Batch	
  Size	
  
Cost	
  
Release	
  Cost	
  
78Principles ! Reduce Tran...
DevOps @ Wotif
The Carrot, Not The Stick
h"p://www.creditcards.com/credit-­‐card-­‐news/images/carrot-­‐no-­‐sBck2.jpg	
  ...
DevOps @ Wotif
Stick vs Carrot
•  We tried the stick approach for years
­ You’re doing it wrong!
­ (But we won’t tell you ...
RECAP
81
DevOps @ Wotif
2010-2012
Manage
Deploy,
Support
“Throw it over the wall”
OperationsDevelopment
82Recap
DevOps @ Wotif
2014
Manage
Deploy
(fast & painless)
((mostly))
OperationsDevelopment
83
COMMUNICATE!
Recap
DevOps @ Wotif
Recap
•  History & Background
•  Solutions
–  Team Structure
–  Puppet tweaks
–  Rolling Upgrade deployment...
Thanks for listening!
Any questions?
Yes, we’re hiring!
www.wotifgroup.com/working-at-wotifgroup/vacancies
DevOps @ Wotif
DevOps @ Wotif
Upcoming SlideShare
Loading in...5
×

DevOps @ Wotif

845

Published on

"DevOps @ Wotif.com" - Alexandra Spillane & Matt Callanan present the journey Wotif has been on implementing a DevOps and Continuous Delivery transformation to reduce cycle times measured in months down to hours, sometimes even minutes. Importantly, not just about the tools (such as Puppet, Hiera, Fabric, TeamCity, ZooKeeper, etc) which featured heavily in the transformation but also about the cultural changes and how they were implemented, the lessons learned, and the key takeaways that attendees can take back to their workplaces. Alexandra & Matt work in the "Continuous Delivery" team at Wotif, charged with rolling out an enterprise-wide adoption of DevOps & Continuous Delivery.

DevOps Brisbane: http://www.meetup.com/Devops-Brisbane/events/184682022/

Video: https://www.youtube.com/watch?v=L8y5L3YQFk8

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
845
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "DevOps @ Wotif"

  1. 1. DevOps @ Wotif DevOps @ Wotif Matt Callanan @mcallana Alexandra Spillane @ajbw
  2. 2. DevOps @ Wotif Outline •  History & Background •  Solutions •  Principles Learned 2
  3. 3. HISTORY & BACKGROUND 3
  4. 4. DevOps @ Wotif 2009 Manage DeploySupport Hosting Provider OperationsDevelopment 4History & Background
  5. 5. DevOps @ Wotif 2010 Manage DeploySupport…? OperationsDevelopment 5History & Background
  6. 6. DevOps @ Wotif 2010-2012 Manage Deploy, Support “Throw it over the wall” OperationsDevelopment 6History & Background
  7. 7. DevOps @ Wotif DOWNWARD SPIRAL 7History & Background
  8. 8. DevOps @ Wotif 2012 - Downward Spiral •  Entrenched silos •  High release overheads •  Huge batch sizes •  Calendar of Doom •  Unpredictable prioritisation •  Superstitious gatekeeping •  Stagnating applications •  CD anti-patterns list read like someone had recorded our processes! h"p://flic.kr/p/67Giiz   8History & Background
  9. 9. DevOps @ Wotif Release Anti-Patterns 9 Extensive, detailed release documentation   Reliance on manual testing to confirm app is correct   Explaining why deployment is going wrong on release day   Frequent corrections to release process during release   Environments that differ in their configuration   Releases that take more than a few minutes to perform   Releases that are unpredictable in their outcome   History & Background
  10. 10. DevOps @ Wotif Plus… •  Migrating to MSA = explosion of manual effort 10History & Background
  11. 11. DevOps @ Wotif Manual Effort – Impact on Roles 11 Dev Manual tag/release process Disparate Build Servers Deploys done by different teams to divergent environments QA Manual deploys Environment drift Different Puppet repos to production Manual testing No time to fully regression test Ops Manual deploys Cheat Sheets Had to learn idiosyncrasies of each application through experience History & Background
  12. 12. DevOps @ Wotif OLD RELEASE SCENARIO 12History & Background
  13. 13. Application Pipelines Ops Days Weeks! Days Staging Load Test Production Days Dev QA Dev QA Dev QA Dev QA Dev QA Dev QA Dev QA Dev QA Dev QA PrioritisedReleaseQueue
  14. 14. DevOps @ Wotif One application’s deployment... Copy and paste line-by-line… /etc/init.d/wcs exitpool securepayment # watch /logs/network/lb/*pool.log on log1 for appropriate port number exit /etc/init.d/wcs stop securepayment yum clean all yum remove securepayment-domain # run migrate db script prior to deployment: https://svn/OracleProject/app/script1.sql yum install securepayment-domain-1.30-9 yum install securepayment-application-2011.10.31-78 /apps/system/scripts/sync_content.sh -d /apps/wotifapps/securepayment/content -s app5 -s app6 -u contentsync -p /var/www/vhosts/images.wotif.com/webroot/booking/ # run migrate db script following deployment: https://svn/OracleProject/app/script2.sql # Execute test plan /etc/init.d/wcs enterpool securepayment on 3x servers… followed by a second app on 2x servers… followed by a third app on 1 server… followed by a fourth app on 2x servers… followed by a fifth app on 2x servers… followed by followed by a point’n’click ASP release… followed by 3 more manual database scripts… with pauses after every instance for manual testing! … 5 hours later ... 14History & Background
  15. 15. SOLUTIONS 15
  16. 16. DevOps @ Wotif Solutions •  Lightweight Standardisation •  Technical Solutions •  Puppet •  Rolling Upgrade •  Team Structure •  SLIPWay onewaystock.com   16Solutions
  17. 17. DevOps @ Wotif LIGHTWEIGHT STANDARDISATION 17Solutions
  18. 18. DevOps @ Wotif HeavyWeight to LightWeight 18Solutions ! Lightweight Standardisation
  19. 19. DevOps @ Wotif DropWizard •  Opinionated Java framework for 
 developing ops-friendly, 
 high-performance, RESTful 
 web services 19Solutions ! Lightweight Standardisation
  20. 20. DevOps @ Wotif Glassfish •  Feature overhead •  Encouraged manual config •  Complex/Slow deployment •  We’d built standard scripts –  Scripts were clunky –  Scripts wouldn’t get updated DropWizard •  Trimmed down to basics •  Standard config files •  Simple/Fast deployment •  No standard scripts –  Combinatorial explosion of interaction styles HeavyWeight to LightWeight 20Solutions ! Lightweight Standardisation
  21. 21. DevOps @ Wotif Combinatorial Explosion 21Solutions ! Lightweight Standardisation
  22. 22. DevOps @ Wotif Flexibility vs Predictability •  Freedom is Good –  Adopt new technologies and adapt to market •  Opinions are Good –  Be opinionated to reduce combinatorial complexity for production support •  How do we promote a culture of freedom and responsibility but still provide predictability? ? 22 Flexible Predictable Solutions ! Lightweight Standardisation
  23. 23. DevOps @ Wotif Standards •  “Standards” doesn’t have to be a dirty word •  “Any standard is better than no standard” •  People want to do the right thing •  People will do the easiest thing •  Make it easy to do the right thing 23Solutions ! Lightweight Standardisation
  24. 24. DevOps @ Wotif Standardisation: Theory 24Solutions ! Lightweight Standardisation
  25. 25. DevOps @ Wotif Defining •  Confluence Page –  “Application Deployment Standards 1.0 (DRAFT)” •  Extract input from people: –  Ops: This will make life easier at 2am –  Dev: This will enable faster releases •  Discussions: Ops & Dev involvement –  Meetings: Devs, Ops, Leads, Architects, Managers –  Emails –  IM –  One-on-one 25Solutions ! Lightweight Standardisation
  26. 26. DevOps @ Wotif Standards Lifecycle 26Solutions ! Lightweight Standardisation
  27. 27. DevOps @ Wotif Example Standards 27 Logging •  Log file locations, filenames Log rotate Directories/Files •  Ownership, permissions Puppet Supervisord config.yaml •  Filename, location Metrics JVM settings Network •  Ports •  Firewall rules Cron Endpoints •  Healthchecks, status, metrics SSL •  Certs, keystores RPM •  Versioning •  Packaging init.d scripts •  Sub-commands Versioning Migration Notes Solutions ! Lightweight Standardisation
  28. 28. DevOps @ Wotif Example  Standards   28Solutions ! Lightweight Standardisation
  29. 29. DevOps @ Wotif Example  Standards   29Solutions ! Lightweight Standardisation
  30. 30. DevOps @ Wotif Future  ConsideraBons   30Solutions ! Lightweight Standardisation
  31. 31. DevOps @ Wotif Reference Implementation •  “helloworld-service” •  Deployed to production 31Solutions ! Lightweight Standardisation
  32. 32. DevOps @ Wotif Standardisation: Testing •  Standards are pointless without auto- verification •  Test-driven ops compatibility •  Backwards-compatible –  Test cases are versioned alongside standards •  Reference implementation 32Solutions ! Lightweight Standardisation
  33. 33. DevOps @ Wotif Standardisation: Testing 33 • Does rpm have correct metadata linking to git repo? Check existence/absence of filesrpm • Is correct version actually installed?yum • Check existence/absence of files. Check file permissions/ownershipls • Is correct version actually running?lsof • Check correct ports exposednetstat • Is correct log format in use?tail • Check cron configgrep • Check contents of standalone jar file – e.g. metadata, library versionsunzip • Check standard endpoints for e.g. content, status code, response timecurl • Check JVM optionsjps • Check 1 and only 1 process runningps Solutions ! Lightweight Standardisation
  34. 34. DevOps @ Wotif Standards Compliance Tests 34Solutions ! Lightweight Standardisation
  35. 35. DevOps @ Wotif Migration Notes •  Annotate changes since previous standards –  E.g. "(since v2.3)” •  Each standard version has migration notes from previous version •  Can upgrade between several versions by following migration notes nav bar 35 1.0 2.0 2.1 2.2 2.3 Solutions ! Lightweight Standardisation
  36. 36. DevOps @ Wotif MigraBon  Notes   36Solutions ! Lightweight Standardisation
  37. 37. DevOps @ Wotif Standardisation Progress 37 Standardised Packaging Standardised Deployment Puppet Smoke Tests Solutions ! Lightweight Standardisation
  38. 38. DevOps @ Wotif TECHNICAL SOLUTIONS 38Solutions
  39. 39. DevOps @ Wotif Puppet •  Yay, configuration management! But… •  Prod(-like) environment != test environment •  Prod puppet != test puppet •  Merge! –  post-merge: “if env == ‘test’”, “if env == ‘prod’” conditionals everywhere •  Difficult to test changes •  Difficult to promote changes through environments 39Solutions ! Technical Solutions
  40. 40. DevOps @ Wotif Puppet Branches •  Puppet repo locked down, just like prod! –  Ops gate-keep all master commits •  How can devs contribute? •  “develop” Branch? Branch per environment? –  Never merged to master –  Never merged from master 40Solutions ! Technical Solutions
  41. 41. DevOps @ Wotif Puppet Virtual Branches! •  /etc/puppet… •  Nightly rebuild / rebuild at will •  master + all opted-in branches –  Branch list maintained in Hiera –  Skip conflicting branches •  Remove “old” branches –  Encourages people to merge to master! 41Solutions ! Technical Solutions
  42. 42. DevOps @ Wotif Promoting changes •  “if env == ‘test’” / “if env == ‘prod’” –  Gets old fast! •  Promoting changes to production means untested merge requests mid-release! •  Unmaintainable, untidy, error-prone 42Solutions ! Technical Solutions
  43. 43. DevOps @ Wotif Puppet Feature Switches •  Eradicate environment conditionals •  Encouraged to be temporary – “Gimmicks” •  gimmick() is a convenience wrapper for hiera() if env == ‘test’ test.yaml: gimmick.some_feature: true prod.yaml: gimmick.some_feature: false if gimmick(‘feature’) 43Solutions ! Technical Solutions
  44. 44. DevOps @ Wotif Rolling Upgrade •  Safely automate upgrades of HA clustered applications 44 •  Check Load Balancer Exit Pool •  Orchestrate Puppet Upgrade •  Compliance Test •  Smoke Test •  Feature Test Test •  Check Load Balancer Enter Pool •  exitpool •  stop •  yum remove •  yum install •  Wait for manual testing •  enterpool Manual Release Solutions ! Technical Solutions
  45. 45. DevOps @ Wotif Rolling Upgrade 45Solutions ! Technical Solutions
  46. 46. DevOps @ Wotif Rolling Upgrade 46Solutions ! Technical Solutions
  47. 47. DevOps @ Wotif Rolling  Upgrade:  Summary   47Solutions ! Technical Solutions
  48. 48. DevOps @ Wotif Rolling  Upgrade:  Smoke  Tests   48Solutions ! Technical Solutions
  49. 49. DevOps @ Wotif TEAM STRUCTURE 49Solutions
  50. 50. DevOps @ Wotif Team Structure •  ITSSU re-org late 2013 –  “Houses” led by senior devs (not BAs or PMs) •  Allowed a focus on tech debt & service delivery –  Continuous Delivery team •  Tech innovation •  Third level support •  Communication! 50Solutions ! Team Structure
  51. 51. DevOps @ Wotif Continuous Delivery Team •  Communication facilitators –  Not know-it-all / do-it-all heroes! –  Emphasis on collaboration –  chatrooms galore –  “peer-to-peer help” •  Weekly DevOps standups –  Any interested devs / ops / architects –  Raise anything interesting to a wider audience •  DevOps champions 51Solutions ! Team Structure
  52. 52. DevOps @ Wotif SLIPWAY 52Solutions
  53. 53. DevOps @ Wotif “SLIPWay” •  Big review of old release processes •  Simple Lightweight Independent Path Way •  Simple rules for release process •  Keep changes independent •  Focus on reducing cycle time 53 h"p://goo.gl/vjaN9F   Solutions ! SLIPWay
  54. 54. Application Pipelines Ops Days Weeks! Days Staging Load Test Production Days Dev QA Dev QA Dev QA Dev QA Dev QA Dev QA Dev QA Dev QA Dev QA PrioritisedReleaseQueue
  55. 55. Application Pipelines Ops PrioritisedReleaseQueue Dev QA Dev QA Dev QA Dev QA Dev QA Dev QA Dev QA Dev QA Dev QA Staging Load Test Production Staging Load Test Production Staging Load Test Production Staging Load Test Production Staging Load Test Production Staging Load Test Production Staging Load Test Production Staging Load Test Production Staging Load Test Production Hours Hours LightweightRules
  56. 56. DevOps @ Wotif SLIPWay Conditions 56Solutions ! SLIPWay
  57. 57. DevOps @ Wotif SLIPWay Conditions 57Solutions ! SLIPWay
  58. 58. DevOps @ Wotif SLIPWay Chat Room 60Solutions ! SLIPWay
  59. 59. DevOps @ Wotif SLIPWay  Example  Release   61Solutions ! SLIPWay
  60. 60. DevOps @ Wotif Total  releases  in  last  8  months   62Solutions ! SLIPWay # Releases
  61. 61. DevOps @ Wotif Manual Effort – Impact on Roles 63 Dev Manual tag/release process Disparate Build Servers Deploys done by different teams to divergent environments QA Manual deploys Environment drift Different Puppet repos to production Manual testing No time to fully regression test Ops Manual deploys Cheat Sheets Had to learn idiosyncrasies of each application through experience Solutions ! SLIPWay
  62. 62. DevOps @ Wotif Automation – Impact on Roles 64 Dev Deploy to prod-like environment on every commit Compliance test & smoke test on every commit Tweak Maven to allow any commit to be released without manual intervention Consolidate build servers for consistency and visibility QA Automated self-service deploys to exploratory testing environments Test environment as close to production as possible via shared Puppet repository Regression testing covered by automated Acceptance Tests Could spend time on exploratory testing Ops Have confidence that application has been through compliance testing Have confidence in rejecting an application if it fails compliance tests or smoke tests Have less idiosyncrasies and less rules to keep in their head when supporting applications in production Solutions ! SLIPWay
  63. 63. PRINCIPLES 65
  64. 64. DevOps @ Wotif Principles 66 You Can't CHANGE Culture Strongly Type Your Discussions Make it Easy To Do The Right Thing Reduce Transaction Cost Before Batch Size The Carrot, Not The Stick Principles
  65. 65. DevOps @ Wotif You Can’t CHANGE Culture •  … but you can change it. •  Start with what you have –  Incremental, palatable changes –  Involve everyone 67Principles ! You Can’t CHANGE Culture
  66. 66. DevOps @ Wotif Strongly Type Your Discussions •  Choose catchy/easy-to-pronounce names –  Helps to make decisions –  Helps climb out of the muck –  If someone thinks of a good name - go with it •  Use state tables and transition diagrams –  State -> Action -> New State –  Visualise current problems –  Decide which state transitions to support 68Principles ! Strongly Type Your Discussions
  67. 67. DevOps @ Wotif E.g. Glassfish Service States 69Principles ! Strongly Type Your Discussions
  68. 68. DevOps @ Wotif E.g. Lightweight Service States 70Principles ! Strongly Type Your Discussions
  69. 69. DevOps @ Wotif Be Opinionated – Not Arrogant •  Strong decisions are important •  But never think you’ll get it all correct up front 71 h"p://goo.gl/YN1WJz   Principles ! Be Opinionated
  70. 70. DevOps @ Wotif Make it Easy To Do The Right Thing 72Principles ! Make Right Easy Easy Right
  71. 71. DevOps @ Wotif Make it Easy To Do The Right Thing 73Principles ! Make Right Easy Easy Right
  72. 72. DevOps @ Wotif Reduce Transaction Cost Before Batch Size flic.kr/p/bEVLmG   74Principles ! Reduce Transaction Cost
  73. 73. DevOps @ Wotif Batch Size vs Risk h"p://www.slideshare.net/jallspaw/ops-­‐metametrics-­‐the-­‐currency-­‐you-­‐pay-­‐for-­‐change   •  “reduce risk by reducing batch size” 75Principles ! Reduce Transaction Cost
  74. 74. DevOps @ Wotif Batch Size Science •  But we couldn’t reduce batch size until we reduced transaction (release) cost 76Principles ! Reduce Transaction Cost
  75. 75. DevOps @ Wotif High Release Cost = High Batch Size                                                                                                                       Batch  Size   Cost   Release  Cost   Total  Cost   77Principles ! Reduce Transaction Cost
  76. 76. DevOps @ Wotif Low Release Cost = Low Batch Size Batch  Size   Cost   Release  Cost   78Principles ! Reduce Transaction Cost
  77. 77. DevOps @ Wotif The Carrot, Not The Stick h"p://www.creditcards.com/credit-­‐card-­‐news/images/carrot-­‐no-­‐sBck2.jpg   79Principles ! The Carrot, Not The Stick
  78. 78. DevOps @ Wotif Stick vs Carrot •  We tried the stick approach for years ­ You’re doing it wrong! ­ (But we won’t tell you how to do it right) •  Blame games, anger •  No business impetus to improve things 80 •  Now we’re trying the carrot approach ­ Offer a great new alternative – It’s optional! – You know you want it… •  Business invested in the process Principles ! The Carrot, Not The Stick
  79. 79. RECAP 81
  80. 80. DevOps @ Wotif 2010-2012 Manage Deploy, Support “Throw it over the wall” OperationsDevelopment 82Recap
  81. 81. DevOps @ Wotif 2014 Manage Deploy (fast & painless) ((mostly)) OperationsDevelopment 83 COMMUNICATE! Recap
  82. 82. DevOps @ Wotif Recap •  History & Background •  Solutions –  Team Structure –  Puppet tweaks –  Rolling Upgrade deployment mechanism –  SLIPWay rapid release process •  Principles –  You Can’t CHANGE Culture –  Strongly Type Your Discussions –  Make it Easy to do the Right Thing –  Reduce Transaction Cost Before Batch Size –  The Carrot, Not The Stick 84Recap
  83. 83. Thanks for listening! Any questions?
  84. 84. Yes, we’re hiring! www.wotifgroup.com/working-at-wotifgroup/vacancies

×