The Devops Wonder

1. The Devops Wonder How to accelerate your IT flow 8/11/2013 1 THEODO

2. The Devops Wonder How to accelerate your IT flow A preach in the Symfony Live Berlin church on finding the path to fast IT 8/11/2013 2 THEODO

3. The need for speed in IT The Devops wonder… and facts that prove it How to start practising Devops, step-by-step 8/11/2013 3 THEODO

4. Business needs velocity The need for agility 8/11/2013 4 THEODO

5. V-Cycle is slow and risky The problems of the alternative ? Source: http://www.ucl.ac.uk/syseng/images/vDiag 8/11/2013 5 THEODO

6. Much slower and riskier than what can be estimated The natural mistake A study by two Oxford professors on 1 471 large IT projects shows : • 1 project out of 6 has cost on average 3x more than expected! • large-scale computer spendings were 20 times more likely to spiral out of control than expected! Large-scale example: • FoxMeyer Drugs’ bankruptcy after switching brutally to SAP Oxford study: http://users.ox.ac.uk/~mast2876/WP_2011_08_15.pdf Fox-Meyer bankruptcy: http://fr.slideshare.net/jmramireza/the-foxmeyer-drugs-bankruptcy-was-it-a-failureof-erp-2332065 8/11/2013 6 THEODO

7. Our brain sees the world through Bell curves Black swan blindness Average scenario Worst case scenario Good case scenario Probability density of possible outcomes 8/11/2013 7 THEODO

8. Complex systems do not act like Bell curves Black swan blindness Average scenario Worst case scenario Good case scenario Probability density of possible outcomes 8/11/2013 8 THEODO

9. This is why small iterations are so important! Lean IT Machiavel: Divide et impera! (Divide and conquer) 8/11/2013 9 THEODO

10. Agile development is about small FAKE iterations Agile development Source: http://www.mypmhome.com/scrum-methodology/ 8/11/2013 10 THEODO

11. Because features will just pile up in front of IT operations The dev VS ops silos 8/11/2013 11 THEODO

12. The Scrum prophets showed the light, and the first steps to it Scrum is not the end 8/11/2013 12 THEODO

14. Devops is about solving the silos problem The definition of Devops Devops is • making the whole IT organisation work as a unique team • with a common goal: business-oriented IT performance 8/11/2013 14 THEODO

15. The Devops trinity: tools, process, people The three layers of Devops PROCES S TOOL S DEVOP S PEOPLE Devops is • Tools: how to do it • Process: when to do it • People: why to do it 8/11/2013 15 THEODO

16. The people part is important: devs and ops are different! Devops is a lot about culture DHH, the epitom of the hipster developer Stallman, the epitom of the paranoid GNU/Linux guru 8/11/2013 16 THEODO

17. Devs want speed and think short-term Devops is a lot about culture I just coded this really cool feature using CouchDB/PouchDB/Node.js/S 3 Let me deploy it now, business wants it online today! The typical dev The typical ops 8/11/2013 17 THEODO

18. Ops want stability and think long-term Devops is a lot about culture CouchDB/PouchD B/Node.js/S3??? The typical dev The typical ops 8/11/2013 18 THEODO

19. In the worst case, you get neither short nor long-term results Devops is a lot about culture What about Backups? Documentation? Monitoring? Business is not going to be happy… Scalability? Performances? Security? The typical dev The typical ops 8/11/2013 19 THEODO

20. Devops efficiency is highly documented on the web Devops studies • • • IT Ops & DevOps productivity report 2013 by RebelLabs surveyed 620 engineers 2013 State of DevOps Report by PuppetLabs & IT Revolution Press surveyed 4000 IT ops and devs And all the testimonials from web leaders like Amazon, Netflix, Etsy, Flickr, etc. (at Velocity Conference, DevopsDays, etc.) 8/11/2013 20 THEODO

21. Devops ship code 30x more frequently! 2013 State of Devops report by PuppetLabs and IT Revolution Press Mean time between deploys 8/11/2013 21 THEODO

22. Devops ship code 8000x faster! 2013 State of Devops report by PuppetLabs and IT Revolution Press Mean time spent deploying 8/11/2013 22 THEODO

23. Devops have 50% less failures! 2013 State of Devops report by PuppetLabs and IT Revolution Press Failed deployments rate 8/11/2013 23 THEODO

24. Devops restore service 12x faster! 2013 State of Devops report by PuppetLabs and IT Revolution Press Mean time to resolution 8/11/2013 24 THEODO

25. Amazon deploys in average more than 300 times an hour! Devops success stories • • • • Founded in 1994, 61 billion$ revenue in 2012 Mean time between deployments : 11.6s (310 times per hour) Max number of deployments in an hour : 1,079 Mean number of hosts receiving a deployment : 10,000 http://assets.en.oreilly.com/1/event/60/Velocity%20Culture%20Presentation.pdf http://en.wikipedia.org/wiki/Amazon.com 8/11/2013 25 THEODO

26. At Etsy every engineer deploys, and starts on the first day! Devops success stories • • • • Founded in 2005, 1 billion$ transactions in 2013 250+ committers, everyone deploys 30+ deploys a day Every engineer at Etsy deploys on the first day http://codeascraft.com/2012/03/13/making-it-virtually-easy-to-deploy-on-day-one/ https://speakerdeck.com/astanway/bring-the-noise-continuously-deploying-under-a-hailstorm-of-metrics http://gigaom.com/2013/08/23/meet-the-man-behind-new-yorks-other-billion-dollar-internet-companythis-one-makes-money/ 8/11/2013 26 THEODO

27. The English government did a radical adoption of Devops in less than a year! Devops success stories What is gov.uk? “The websites of all government departments and many other agencies and public bodies” • Alpha launched in 2011 with 4 devs. Site was live in october 2012 • 15-20 deploys a day • Main selling point: faster than other suppliers. • • “this is what we’d like. Can you work out if it’s possible? let’s meet again in a month” -> built, shipped in 3 days http://vimeo.com/album/2384821/video/66622266 https://github.com/philandstuff/devopsdaysparis#kushal-pisavadia-kushalp-how-we-ship-software-at-govuk https://www.gov.uk/service-manual 8/11/2013 27 THEODO

29. Creating a culture of teamwork between devs and ops The three (or four) maturity levels of Devops adoption 1. Bring devs closer to the constraints of operations 2. Bring ops to empower devs 3. Further embed devs and ops in each other’s teams 8/11/2013 29 THEODO

30. Let us start by teaching junior devs basic ops concepts The common developer background The typical junior developer • learnt Java at University • uses Windows… because of the games • has installed Linux because he is curious • maintained a website for one of his clubs at University… • … « deployed » changes using FileZilla Source: http://www.essentialbaby.com.au/ 8/11/2013 30 THEODO

31. Step 0: make the baby dev a respectable dev Basic dev skills • • • • • Linux development environment Git versioning Correct git branching strategy Scrum methodology Unit and functional tests Source: http://www.aceshowbiz.com/ 8/11/2013 31 THEODO

32. Step 1: give the devs a server to play with One dedicated server per developer «I improved a lot when I started renting my own server» • • • Give a tiny instance to everyone Free to do whatever you want, whenever you want …as long as it is legal… 8/11/2013 32 THEODO

33. Step 2: make them set up a simple server… at least twice Basic system administration skills • • Amazing how many junior devs have never set up a server… Twice: to get the taste for automated provisioning What we do: • First a simple setup with Apache + mod_php • Then a more subtle setup with Nginx + fpm 8/11/2013 33 THEODO

34. Step 3: bring devs to deploy… and understand how it works Automated deployment • • • Shell scripting: forget it. Capistrano: too magic? Fabric: good compromise? Our experience: every project has a deploy.py script in a «devops» folder (or a Capfile…): • Everybody is now able to deploy automatically • Juniors seem to check if there is an experienced guy around before deploying… 8/11/2013 34 THEODO

35. Fabric makes deployment automatic and not too abstract Fabric sample @roles('prod') def deploy(): tag = "%s/%s" % (_getrole(), strftime("%Y/%m-%d-%H-%M-%S")) local('git tag -a %s -m "%s"' % (tag, _getrole())) local('git push --tags') run('git fetch’) run('git fetch origin --tags') run('git checkout ' + tag) 8/11/2013 35 THEODO

36. Step 4: make monitoring an obvious step Monitoring • • • Monitoring has become very easy: think NewRelic and AppDynamics By making it an early step, you positively change the dev culture Easy access to: performance issues and hidden 500 errors 8/11/2013 36 THEODO

37. Step 5: make continuous integration a dev responsibility Continuous integration • • • Setting up Jenkins is easy: all devs should be expert at doing it To avoid conflicts between jobs: use a separate slave node for each projet Why not Travis-ci? Good solution, but Jenkins proficiency is necessary https://wiki.jenkins-ci.org/display/JENKINS/Distributed+builds 8/11/2013 37 THEODO

38. Step 6: normalised environment with Vagrant Development VM • • • • • Macbook, Ubuntu, 32 bits, memory_limit=2048M, ulimit… Development environments are different from production in subtle ways Vagrant: wrap a VirtualBox VM that launches the same OS flavour Bonus: « vagrant package » and you can share your box with a new developer, makes him operational in a few minutes Important performance tips: http://www.whitewashing.de/2013/08/19/speedup_symfony2_on_vagrant_boxes.html 8/11/2013 38 THEODO

39. It is time for ops to collaborate and empower devs Production feedback into development 8/11/2013 39 THEODO

40. Step 7: use Vagrant with Puppet/Chef Automated provisioning ? • • • Puppet (or Chef) is painful for a dev. But by now he should find manual provisioning painful too! Make an experienced ops create a Puppet/Chef template Make an experienced dev challenge everything and simplify things! 8/11/2013 40 THEODO

41. Step 8: enable self-service Infrastructure as a Service • • Public cloud or private cloud OpenStack is now mature (enough) 8/11/2013 41 THEODO

42. Step 9: make backuping easy Backups • • S3 / Swift to push backups to Or solutions like Idera ServerBackup 8/11/2013 42 THEODO

43. Step 10: create a self-service load testing platform Performance • • • Create an easy to access (VNC) server with a Jmeter and large bandwidth Let the ops teach the devs what a good lead testing scenario is Test the load regularly (if necessary) with devs+ops together 8/11/2013 43 THEODO

44. Step 11: enforce pair-devopsing Pair devopsing • • • The dev team has an issue with Puppet/Chef? The load tests indicate a subtle performance issue? Ops see cryptic errors in the logs? Pair devopsing! • • Scrum compatible Works with curious people hire curious people Source: http://www.turnbacktogod.com 8/11/2013 44 THEODO

45. Step 12: use visual management as a common incentive Visual management techniques Example: how do you make devs focus on performance? First make it part of the DONE definition Include performance solutions in the standard provisioning AND do visible performance graphs 8/11/2013 45 THEODO

46. How do you bring devs to feel responsible for production? Sharing responsibility Why do ops feel that responsible, and not devs? Dev: «It is not my job» … it is up to the business to decide if it is worth putting devs on call. 8/11/2013 46 THEODO

47. Next steps: learn from the web leaders Sources • • • • Velocity conferences • http://lanyrd.com/2013/velocity/coverage/ • http://lanyrd.com/2012/velocity/coverage/ • Etc. Gene Kim’s “Top 11 things you need to know” • http://www.thinkhdi.com/~/media/HDICorp/Files/WhitePapers/whtppr-1112-devops-kim.pdf Etsy’s blog: http://codeascraft.com/ Netflix’s blog: http://techblog.netflix.com/ 8/11/2013 47 THEODO

48. Iterate to improve continuously Continuous improvement Scrum, Devops, Lean Startup: it is all about continuous improvement! • • • Measure everything Identify problems Try countermeasures And iterate 8/11/2013 48 THEODO

49. And I hope that you now believe in the Devops wonder The miraculous performances of Amazon Werner Vogels, CTO at Amazon, 300+ deploys per hour 8/11/2013 49 THEODO

50. Questions ? fabriceb@theodo.fr @theodo www.theodo.fr 8/11/2013 50 THEODO

The Devops Wonder

You are reading a preview.

Check these out next

The Devops Wonder

More Related Content

More from Fabrice Bernhard (7)

Recently uploaded (20)