Successfully reported this slideshow.

Deploying software at Scale

3

Share

Loading in …3
×
1 of 44
1 of 44

More Related Content

Related Audiobooks

Free with a 14 day trial from Scribd

See all

Deploying software at Scale

  1. 1. Deploying Software at Scale Kris Buytaert @krisbuytaert
  2. 2. Kris Buytaert ● I used to be a Dev, ● Then Became an Op ● Chief Trolling Officer and Open Source Consultant @inuits.eu ● Everything is an effing DNS Problem ● Building Clouds since before the bookstore ● Some books, some papers, some blogs ● Evangelizing devops
  3. 3. Todays Goals ● A reproducable way to deploy and upgrade software ● Automatically ● Fast ● Consistent
  4. 4. What's this devops thing anyhow ?
  5. 5. What's the problem ? The community of developers whose work you see on the Web, who probably don’t know what ADO or UML or JPA even stand for, deploy better systems at less cost in less time at lower risk than we see in the Enterprise. This is true even when you factor in the greater flexibility and velocity of startups. Tim Bray , on his blog January 2010
  6. 6. The Old Days ● “Put this Code Live, here's a tarball” NOW! ● What dependencies ? ● No machines available ? ● What database ? ● Security ? ● High Availability ? ● Scalability ? ● My computer can't install this ?
  7. 7. devops ● Culture ● (Lean) ● Automation ● Measurement ● Sharing Damon Edwards and John Willis Gene Kim
  8. 8. devops (<)> continuous delilvery
  9. 9. Nirvana An “ecosystem” that supports continuous delivery, from infrastructure, data and configuration management to business. Through automation of the build, deployment, and testing process, and improved collaboration between developers, testers, and operations, delivery teams can get changes released in a matter of hours — sometimes even minutes–no matter what the size of a project or the complexity of its code base. Continuous Delivery , Jez Humble
  10. 10. How many times a day ? ● 10 @ Flickr ● Deployments used to be pain ● Nobody dared to deploy a site ● Practice makes perfect ● Knowing you can vs constantly doing it
  11. 11. " Our job as engineers (and ops, dev-ops, QA, support, everyone in the company actually) is to enable the business goals. We strongly feel that in order to do that you must have the ability to deploy code quickly and safely. Even if the business goals are to deploy strongly QA’d code once a month at 3am (it’s not for us, we push all the time), having a reliable and easy deployment should be non-negotiable." Etsy Blog upon releasing Deployinator http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/
  12. 12. How do we get there ?
  13. 13. OS Baseline ● Automated Deployments ● Reproducable ● Kickstart, FAI, Preseeding, ● JeOS
  14. 14. Infrastructure as Code ● Treat configuration automation as code ● Development best practices • Model your infrastructure • Version your cookbooks / manifests • Test your cookbooks/ manifests • Dev/ test /uat / prod for your infra ● Model your infrastructure ● A working service = automated ( Application Code + Infrastructure Code + Security + Monitoring ) ● Think Puppet, Chef, Cfengine, Ansible , ....
  15. 15. Version Control ● Git ! ● Version ALL the things: • Source code Application • Source code Infrastructure • Builds • Tests • Pipelines • Scripts • Documentation • Monitoring scripts
  16. 16. Continuous Integration ● Builds ● Nightly Builds ● Builds with tests ● Nightly Builds with tests ● Frequent integration ● Continuous Integration
  17. 17. Jenkins ● Open Source Continuous Integration Server ● A zillion plugins (400) ● Have developers build stable and deployable code ● Test Infra code
  18. 18. Jenkins Pipeline
  19. 19. What's in your Pipeline ?
  20. 20. A pipeline ● Checkout code ● Syntax ● Style ● Code Coverage ● Tests ● Build ● More Tests ● Package
  21. 21. App Requirements ● Testable ● Configuration isolated ● Automated Deployments ● “If my computer can't install it , the installer is borken” Luke Kanies at Fosdem (2007) ● Bulk provisioning of data ● http://www.krisbuytaert.be/blog/how-i-my-java
  22. 22. Why ops like to package ● Packages give you features •Consistency, security, dependencies ● Uniquely identify where files come from •Package or cfg-mgmt ● Source repo not always available •Firewall / Cloud etc .. ● Weird deployment locations , no easy access ● Little overhead when you automate ● CONFIG does not belong in a package
  23. 23. #packagingsucks • Really .. ● Missing upstream ● Ancient upstream ● Unneeded dependencies ● Broken upstream ● Distro Policies ● “Maximum RPM” ? ● Packaging is needed ● Dependencies, tooling, repositories ● Anger driven development
  24. 24. Not all packages are equal
  25. 25. #packaginlove
  26. 26. fpm fpm -t rpm -s dir -n hornetq -v 2.2.5 hornetq Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.nNkVwh + umask 022 + cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD + exit 0 Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.yUd4MV + umask 022 + cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD + cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD + tar -zxf /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/data.tar.gz + exit 0 Executing(%install): /bin/sh -e /var/tmp/rpm-tmp.jkpqeA + umask 022 + cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD + /usr/lib/rpm/brp-compress + /usr/lib/rpm/brp-strip + /usr/lib/rpm/brp-strip-static-archive + /usr/lib/rpm/brp-strip-comment-note Processing files: hornetq-2.2.5-1.x86_64 Checking for unpackaged file(s): /usr/lib/rpm/check-files /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD Wrote: /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/SRPMS/hornetq-2.2.5-1.src.rpm Wrote: /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/RPMS/x86_64/hornetq-2.2.5-1.x86_64.rpm Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.z2UL3B + umask 022 + cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD + rm -rf /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD + exit 0 Created /usr/local/hornetq-2.2.5.x86_64.rpm
  27. 27. fpm in action ● https://github.com/Inuits/build-gems ● Fork, pull ● Jenkins pulls , builds , pushes to repo ● (variants for Nagios Plugins / Jenkins Plugins available)
  28. 28. A pipeline ● Checkout code ● Upload to Repo ● Syntax ● Style ● Code Coverage ● Tests ● Build ● More Tests ● Package
  29. 29. Pulp Pulp is a Python application for managing software repositories and their associated content, such as packages, errata, and distributions. It can replicate software repositories from a variety of supported sources, such as http/https, file system, ISO, and RHN, to a local on-site repository. It provides mechanisms for systems to gain access to these repositories, providing centralized software installation.
  30. 30. Pulp ● Redhat Community ● Redhat Emerging Technology ● Part of Katello
  31. 31. Pulp ● “manages” its own apache instance ● Symlinks , no copies ● Queues •Syncing in the background •No more screens ;) ● Actions are not instantly •e.g. Add / sync / delete ● Hello mongodb :( ● v1 vs v2 ● Only use repo functionality , cfgmgmt is in charge of packages
  32. 32. Repository Management
  33. 33. Version vs Latest ● Version your repos ? ensure => latests ● Latest your environments ? ● Strict versioning in config ? ensure => '0.98.4'
  34. 34. A pipeline ● Checkout code ● Upload to Repo ● Syntax ● Deploy on Test ● Style ● Code Coverage ● Tests ● Build ● More Tests ● Package
  35. 35. The Marionette Collective ● Distributed ssh ++ ● What version of ssh do I have installed on my servers ? ● On what servers is XYZ running ? ● Clean all my ssl certs ? ● Restart apache on all servers with fact X
  36. 36. mc-package mc-package -W /dev/ status jdk * [ ============================================================> ] 33 / 33 servicesdb01.dev.com version = -absent services.dev.google.com version = jdk-1.6.0_13-fcs drbdtest02.dev.google.com version = -absent services3.dev.google.com version = jdk-1.6.0_20-fcs um.dev.google.com version = jdk-1.5.0_19-fcs devtools03.uat.com version = jdk-1.6.0_29-fcs alexandria02.dev.google.com version = -absent weblink01.dev.com version = -absent wikitest.dev.google.com version = jdk-1.6.0_24-fcs payment.dev.google.com version = jdk-1.5.0_17-fcs tiff2pdf01.dev.com version = -absent devdoos.dev.com version = jdk-1.6.0_30-fcs wiki.dev.google.com version = jdk-1.6.0_24-fcs reporting01.dev.com version = -absent devtools01-dev.uat.com version = jdk-1.6.0_23-fcs devtools02.uat.com version = jdk-1.6.0_29-fcs drbdtest01.dev.google.com version = -absent ---- package agent summary ---- Nodes: 33/33 Versions: 1 * 1.5.0_17-fcs, 1 * 1.5.0_19-fcs, 1 * 1.6.0_13-fcs, 1 * 1.6.0_20-fcs, 1 * 1.6.0_23-fcs, 2 * 1.6.0_24-fcs, 2 * 1.6.0 Elapsed Time: 1.73 s
  37. 37. What to Trigger ? ● Update Package • Only updates package ● Trigger Puppet Run • Updates config + package
  38. 38. A pipeline ● Checkout code ● Upload to Repo ● Syntax ● Deploy on Test ● Style ● More Tests ● Code Coverage ● Promote ● Tests ● Deploy on UAT ● Build ● More Tests ● More Tests ● Promote ● Package ● Deploy on Prod
  39. 39. Done ? ● Close the feedback loop, ● Send metric on deployment echo "deployed.$package_name 1 `date + %s`" > /dev/tcp/<%= graphite_host %>/2003
  40. 40. Done ? A Software project is not done until your last enduser is in his grave !
  41. 41. But remember Everything is a Fscking DNS Problem No really, Everything is a Fscking DNS Problem If it's not a fucking DNS Problem .. It's an arp problem If it's not an arp problem... It's a Full Filesystem Problem If your filesystem isn't full It's a Spanning Tree problem If it's not a spanning Tree problem... It's a USB problem If it's not a USB Problem It might be an ntp problem If it's not an ntp problem It's a sharing IRQ Problem If it's not a sharing IRQ Problem But most often .. its a Freaking Dns Problem ! Or someone playing tricks on you Jan 2006
  42. 42. Contact Kris Buytaert Kris.Buytaert@inuits.be Further Reading @krisbuytaert http://www.krisbuytaert.be/b log/ http://www.inuits.be/ Inuits Duboistraat 50 2060 Antwerpen Belgium 891.514.231 +32 475 961221

×