Deploying software at Scale


Published on

Given at the Inaugural CentOS Dojo on April 8 , Antwerp, Belgium

  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Deploying software at Scale

  1. 1. Deploying Software at Scale Kris Buytaert @krisbuytaert
  2. 2. Kris Buytaert● I used to be a Dev,● Then Became an Op● Chief Trolling Officer and Open Source Consultant● Everything is an effing DNS Problem● Building Clouds since before the bookstore● Some books, some papers, some blogs● Evangelizing devops
  3. 3. Todays Goals● A reproducable way to deploy and upgrade software● Automatically● Fast● Consistent
  4. 4. Whats this devops thing anyhow ?
  5. 5. Whats the problem ?The community of developers whose work yousee on the Web, who probably don’t know whatADO or UML or JPA even stand for, deploy bettersystems at less cost in less time at lower riskthan we see in the Enterprise. This is true evenwhen you factor in the greater flexibility andvelocity of startups.Tim Bray , on his blog January 2010
  6. 6. The Old Days● “Put this Code Live, heres a tarball” NOW!● What dependencies ?● No machines available ?● What database ?● Security ?● High Availability ?● Scalability ?● My computer cant install this ?
  7. 7. devops● Culture● (Lean)● Automation● Measurement● Sharing Damon Edwards and John Willis Gene Kim
  8. 8. devops (<)> continuous delilvery
  9. 9. NirvanaAn “ecosystem” that supports continuous delivery, frominfrastructure, data and configuration management tobusiness.Through automation of the build, deployment, and testingprocess, and improved collaboration between developers,testers, and operations, delivery teams can get changesreleased in a matter of hours — sometimes even minutes–nomatter what the size of a project or the complexity of its codebase. Continuous Delivery , Jez Humble
  10. 10. How many times a day ?● 10 @ Flickr● Deployments used to be pain● Nobody dared to deploy a site● Practice makes perfect● Knowing you can vs constantly doing it
  11. 11. " Our job as engineers (and ops, dev-ops, QA,support, everyone in the company actually) is toenable the business goals. We strongly feel thatin order to do that you must have the ability todeploy code quickly and safely. Even if thebusiness goals are to deploy strongly QA’d codeonce a month at 3am (it’s not for us, we push allthe time), having a reliable and easydeployment should be non-negotiable."Etsy Blog upon releasing Deployinator
  12. 12. How do we get there ?
  13. 13. OS Baseline● Automated Deployments● Reproducable● Kickstart, FAI, Preseeding,● JeOS
  14. 14. Infrastructure as Code● Treat configuration automation as code● Development best practices • Model your infrastructure • Version your cookbooks / manifests • Test your cookbooks/ manifests • Dev/ test /uat / prod for your infra● Model your infrastructure● A working service = automated ( Application Code + Infrastructure Code + Security + Monitoring )● Think Puppet, Chef, Cfengine, Ansible , ....
  15. 15. Version Control● Git !● Version ALL the things: • Source code Application • Source code Infrastructure • Builds • Tests • Pipelines • Scripts • Documentation • Monitoring scripts
  16. 16. Continuous Integration● Builds● Nightly Builds● Builds with tests● Nightly Builds with tests● Frequent integration● Continuous Integration
  17. 17. Jenkins● Open Source Continuous Integration Server● A zillion plugins (400)● Have developers build stable and deployable code● Test Infra code
  18. 18. Jenkins Pipeline
  19. 19. Whats in your Pipeline ?
  20. 20. A pipeline● Checkout code● Syntax● Style● Code Coverage● Tests● Build● More Tests● Package
  21. 21. App Requirements● Testable● Configuration isolated● Automated Deployments● “If my computer cant install it , the installer is borken” Luke Kanies at Fosdem (2007)● Bulk provisioning of data●
  22. 22. Why ops like to package● Packages give you features•Consistency, security, dependencies● Uniquely identify where files come from•Package or cfg-mgmt● Source repo not always available•Firewall / Cloud etc ..● Weird deployment locations , no easy access● Little overhead when you automate● CONFIG does not belong in a package
  23. 23. #packagingsucks• Really .. ● Missing upstream ● Ancient upstream ● Unneeded dependencies ● Broken upstream ● Distro Policies● “Maximum RPM” ?● Packaging is needed ● Dependencies, tooling, repositories● Anger driven development
  24. 24. Not all packages are equal
  25. 25. #packaginlove
  26. 26. fpmfpm -t rpm -s dir -n hornetq -v 2.2.5 hornetqExecuting(%prep): /bin/sh -e /var/tmp/rpm-tmp.nNkVwh+ umask 022+ cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD+ exit 0Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.yUd4MV+ umask 022+ cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD+ cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD+ tar -zxf /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/data.tar.gz+ exit 0Executing(%install): /bin/sh -e /var/tmp/rpm-tmp.jkpqeA+ umask 022+ cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD+ /usr/lib/rpm/brp-compress+ /usr/lib/rpm/brp-strip+ /usr/lib/rpm/brp-strip-static-archive+ /usr/lib/rpm/brp-strip-comment-noteProcessing files: hornetq-2.2.5-1.x86_64Checking for unpackaged file(s): /usr/lib/rpm/check-files /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILDWrote: /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/SRPMS/hornetq-2.2.5-1.src.rpmWrote: /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/RPMS/x86_64/hornetq-2.2.5-1.x86_64.rpmExecuting(%clean): /bin/sh -e /var/tmp/rpm-tmp.z2UL3B+ umask 022+ cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD+ rm -rf /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD+ exit 0Created /usr/local/hornetq-2.2.5.x86_64.rpm
  27. 27. fpm in action●● Fork, pull● Jenkins pulls , builds , pushes to repo● (variants for Nagios Plugins / Jenkins Plugins available)
  28. 28. A pipeline● Checkout code ● Upload to Repo● Syntax● Style● Code Coverage● Tests● Build● More Tests● Package
  29. 29. PulpPulp is a Python application for managingsoftware repositories and their associatedcontent, such as packages, errata, anddistributions. It can replicate softwarerepositories from a variety of supported sources,such as http/https, file system, ISO, and RHN, toa local on-site repository. It provides mechanismsfor systems to gain access to these repositories,providing centralized software installation.
  30. 30. Pulp● Redhat Community● Redhat Emerging Technology● Part of Katello
  31. 31. Pulp● “manages” its own apache instance● Symlinks , no copies● Queues•Syncing in the background•No more screens ;)● Actions are not instantly•e.g. Add / sync / delete● Hello mongodb :(● v1 vs v2● Only use repo functionality , cfgmgmt is in charge of packages
  32. 32. Repository Management
  33. 33. Version vs Latest● Version your repos ? ensure => latests● Latest your environments ?● Strict versioning in config ? ensure => 0.98.4
  34. 34. A pipeline● Checkout code ● Upload to Repo● Syntax ● Deploy on Test● Style● Code Coverage● Tests● Build● More Tests● Package
  35. 35. The Marionette Collective● Distributed ssh ++● What version of ssh do I have installed on my servers ?● On what servers is XYZ running ?● Clean all my ssl certs ?● Restart apache on all servers with fact X
  36. 36. mc-packagemc-package -W /dev/ status jdk * [ ============================================================> ] 33 / version = version = version = version = version = version = version = version = version = version = version = version = version = version = version = version = version = -absent---- package agent summary ---- Nodes: 33/33 Versions: 1 * 1.5.0_17-fcs, 1 * 1.5.0_19-fcs, 1 * 1.6.0_13-fcs, 1 * 1.6.0_20-fcs, 1 * 1.6.0_23-fcs, 2 * 1.6.0_24-fcs, 2 * 1.6.0 Elapsed Time: 1.73 s
  37. 37. What to Trigger ?● Update Package • Only updates package● Trigger Puppet Run • Updates config + package
  38. 38. A pipeline● Checkout code ● Upload to Repo● Syntax ● Deploy on Test● Style ● More Tests● Code Coverage ● Promote● Tests ● Deploy on UAT● Build ● More Tests● More Tests ● Promote● Package ● Deploy on Prod
  39. 39. Done ?● Close the feedback loop,● Send metric on deployment echo "deployed.$package_name 1 `date + %s`" > /dev/tcp/<%= graphite_host %>/2003
  40. 40. Done ?A Software project is not done until your lastenduser is in his grave !
  41. 41. But remember Everything is a Fscking DNS Problem No really, Everything is a Fscking DNS Problem If its not a fucking DNS Problem .. Its an arp problem If its not an arp problem... Its a Full Filesystem Problem If your filesystem isnt full Its a Spanning Tree problem If its not a spanning Tree problem... Its a USB problem If its not a USB Problem It might be an ntp problem If its not an ntp problem Its a sharing IRQ Problem If its not a sharing IRQ Problem But most often .. its a Freaking Dns Problem ! Or someone playing tricks on you Jan 2006
  42. 42. ContactKris BuytaertKris.Buytaert@inuits.beFurther Reading@krisbuytaert Inuits Duboistraat 50 2060 Antwerpen Belgium 891.514.231 +32 475 961221