Continuous
                             Integration for
                                the world




                    ...
Patrick
                                                    Debois




Wednesday 25 November 2009

My name is Patrick Debo...
Wednesday 25 November 2009

Iʼm a freelancer always on the look out for new interesting things
Wednesday 25 November 2009

I have a sysadmin background
Wednesday 25 November 2009

Now for the past few years, Iʼve been looking together with other
sysadmins
 with great intere...
Wednesday 25 November 2009

And this year I wanted to give you my own version of Continuous
Integration
Continuous Integration is the
                             practice of integrating early and
                             ...
Traditional CI Overview




                             http://www.javaworld.com/javaworld/jw-12-2008/images/CIOverview.j...
Traditional CI Flow




                                      Courtesy Chris Read / ThoughtWorks
Wednesday 25 November 200...
CI Maturity Model




                http://www.anthillpro.com/blogs/anthillpro-blog/2009/05/05/1241542860000.html
Wednes...
Build Pipeline Flow




                                          Courtesy Chris Read / ThoughtWorks
Wednesday 25 November...
Definition of Done




Wednesday 25 November 2009

We need to go beyond the traditional testing and extend it to the
live s...
Agility in Production
  • redefine key concept
  • « the software works »
  • « the software works in production »
  • = be...
Who is part of the team
 • developers
 • testers
 • analysts
 • project managers
 • usability experts
 • architects

Wedne...
But usually we forget

     • the sysadmins
     • the guys from operations




Wednesday 25 November 2009
non-functional
                             requirements



                              “Equal rights amendment”
       ...
Iron Triangle




Wednesday 25 November 2009




“Enterprise” Triangle
 Usually only the left triangle is discussed, but i...
Iron Triangle




                       Project


Wednesday 25 November 2009




“Enterprise” Triangle
 Usually only the ...
Iron Triangle




                       Project


Wednesday 25 November 2009




“Enterprise” Triangle
 Usually only the ...
Iron Triangle




                       Project     Operations


Wednesday 25 November 2009




“Enterprise” Triangle
 Us...
Iron Triangle




                       Project     Operations


Wednesday 25 November 2009




“Enterprise” Triangle
 Us...
Iron Triangle




                       Project     Operations


Wednesday 25 November 2009




“Enterprise” Triangle
 Us...
Traditional CI limits
             • Resistance to change
             • « if it ain't broken don't fix it »
             •...
Why such a resistance


     • development = create value
     • operations = protect value
     • => operations must limi...
If it’s hard
  do it more often




Wednesday 25 November 2009

We need to deploy more often. First times will fail, but e...
Minimize Risks :
           integration of production
             • Add to traditional CI
             • deployment of VM...
CI for systems




Wednesday 25 November 2009

now managing a system is becoming more like writing software. So we will
dr...
CI for systems (2)




Wednesday 25 November 2009

packaging a system, deploying a system
Remember there is no
                   silver bullet




Wednesday 25 November 2009
Typical System
                             Installation in CI
• Download and Install a distribution
• Download and Add so...
Problems?
• It’s a one shot operation
• Systems tend to become application
     friendly
• Is this the same as production?...
Life of System
                                Birth / Adoption
                                      Life
               ...
Examples of systems
  integrations in CI software




       Hudson - PXE                           Hudson - Vmware
      ...
Evolution of operations




         In the beginning there were the dinosaurs, slow
         and with large delays
Wednes...
Operations
       evolution (2)

             • Then came the monkeys with automated
                   installations and ...
Clone system

                               • there is no such things a golden image
                               • ima...
Test in a clone of the
           production environment
• Large effort
 • similar to maintaining a ‘fork’




Wednesday 2...
Wednesday 25 November 2009

So let’s build it from the same source!

If we want to integrate let’s define what makes up the...
Virtual Machines API
 • Systems virtualization :
      Vserver, Zones, OpenVZ/
      Virtuozzo, chroots
 • Hardware Emulat...
Language Bindings

           • java libvirt / virsh
           • ruby-libvirt
           • virtualbox SOAP Interface
    ...
Example Virtualbox
  require 'soap/wsdlDriver'
                                          SOAP
  require 'pp'

  WSDL_URL="...
Example libvirt
              require 'libvirt'

                 conn = Libvirt::open("qemu:///system")
                 ...
Control the machine
 • Physical machine
  • IPMI, Light Out Module, Wake On LAN
 • Virtual machine
  • use of an API


Wed...
Network provisioning

      • DNS
      • DHCP
      • PXE
      • TFTP Boot


Wednesday 25 November 2009
Example Cobbler API
 # Set the distribution of the machine
 # The distribution was previously imported by importing an
 in...
Unattended OS Installs


                             Jumpstart
     Preseed                                     AutoYast ...
Example Kickstart
            # install_type MUST be first
            install_type      initial_install
            # sta...
It’s there
                 under your nose

                             • After your manual install (system DEV)
       ...
OS minimal



                             Just Enough Operating System




Wednesday 25 November 2009

Instead of a full ...
configuration
                             management
        • chef
        • puppet
        • cfengine 3
        • bcfg2
...
class httpd {
                                          package { httpd:

 Sample Recipe                            }
    ...
What a config mgt
                system can do for you
             • auto documentation
             • confidence
        ...
Managing Config Files API

• augeas - RedHat/Fedora
 set /files/etc/hosts/01/ipaddr 192.168.0.1
 set /files/etc/hosts/01/ca...
Scriptable Appserver

   • glassfish, Weblogic Scripting
   • Oracle silent installs




Wednesday 25 November 2009

Furthe...
Example AppServer API
   # create an cluster
      asadmin create-cluster --user admin --host hostname -port 4848
   clust...
Executing scripts
      • dsh, func, clusterssh, pssh, fabric
      • or system calls in ruby/java/..
      • exit code = ...
Make your build self-testing

Wednesday 25 November 2009
Test Driven
                                            Automation

                                   • Checks vs Tests
 ...
TDD Virtual Machines




Wednesday 25 November 2009
Testing if CPU, Memory , Disk are there by booting up a rescueCD and e...
Example DNS Testing
          Feature: Mass DNS Resolution
            In order to present a consistent brand image
      ...
Apache Recipe testing




Wednesday 25 November 2009
Thanks to the build doctor! http://www.build-doctor.com/
Testing Deployment
                    Scripts Capistrano
                                                                ...
Testing / Monitoring




                                                   Cucumber-Nagios
http://www.slideshare.net/auxe...
Lots of
             non
         functional
            tests


Wednesday 25 November 2009
http://www.acutest.co.uk/acute...
Maintain a code
                     system repository
  • Virtual Machine definitions + automation scripts
  • Network pro...
vs.


                  Snapshots




Wednesday 25 November 2009
Git doesn’t handle well large files / rsync works
Rsync sn...
Similarities
    • html/js/java - scripts & recipes
    • RPMS - jar files
    • VM’s - war files
    • local repositories

...
Every commit should
                    be built


Wednesday 25 November 2009
Automate the Build
         (of the System)




Wednesday 25 November 2009

Good practice. Not only 1 part of it , but all...
Builders
             • maven, ant,
             • buildr
             • rake, raven
             • make
             • sc...
Funny Tests


   “Truck Factor”
   “10th floor test”




Wednesday 25 November 2009

The 10th floor test , is a term first us...
triggers
     • changes in VM definition
     • changes in IP, DNS, ..
     • changes in OS, packages
     • changes in rec...
build ENV = PROD
    •     Development

    •     Test

    •     Quality Assurance

    •     Pre-production

    •     P...
Everyone commits
                     every day




Wednesday 25 November 2009
Keep the build fast




Wednesday 25 November 2009
Virtual Machine
                                Snapshots

                                     Filesystem
               ...
Continuous vs Nightly

  • During the day, recipes only
  • Nightly the whole system rebuild




Wednesday 25 November 2009
Package

  • koan : creates virtual images
  • rpath / rbuilder
  • ec2
  • SUN forklift
  • jumpbox
Wednesday 25 November...
Deployment tools


             • carpet
             • ec2 tools AMI
             • Vmware Appliance


Wednesday 25 Novem...
Everyone can see the
      results of the latest build




Wednesday 25 November 2009
Make it easy to get the
                latest deliverables
               deploy on the latest
                    instal...
Hot topics

   • Cloud Computing
   • Production virtualization
   • Continuous deployment
   • Required fully automated p...
Concept of Pipelines




Wednesday 25 November 2009
Cloud perspective




Wednesday 25 November 2009

There is actually very much of similarity with the cloud model
Key integration points
          • What environments to combine
          • Stable dev vs. Stable prod?
          • Stable...
Problems/Pitfalls
  • Infrastructure and software need to be
       adapted for easy put into production
  • update with d...
State Transition Process




                                                          http://digiorgio.com/blog/?p=239


...
Parallel Streams




                                         http://www.agilejournal.com/content/view/865/195/

Wednesday...
Development   Sysadmins




Wednesday 25 November 2009
Development    Sysadmins


                             DEVOPS




Wednesday 25 November 2009
More?
         http://www.devopsdays.org
         agile-system-administration@googlegroups.com

         IEEE-paper:
     ...
Wednesday 25 November 2009

So the next time you’re building a fun project
Wednesday 25 November 2009

Remember the sysadmins and guys from operations
We reboot with pride




Wednesday 25 November 2009

And remember to reboot with pride
Questions?




Wednesday 25 November 2009
Wednesday 25 November 2009

Thanks for listening.
Upcoming SlideShare
Loading in...5
×

Continuous Integration For The World - Xpdays Benelux 2009

11,123

Published on

Infrastructure is becoming code. In this presentation we draw the parallels between the automation of systems provisioning and development. Continuous integration is a good candidate to have your developers and sysadmins give each other faster feedback on changes and their impact on both sides.

This was presented by Patrick Debois @xpdays Benelux 2009.

With many thanks to Gildas Le Nadan who worked together on the first version @xpdays France 2009

Published in: Technology
0 Comments
12 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
11,123
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
266
Comments
0
Likes
12
Embeds 0
No embeds

No notes for slide

Transcript of "Continuous Integration For The World - Xpdays Benelux 2009"

  1. 1. Continuous Integration for the world Patrick Debois Wednesday 25 November 2009
  2. 2. Patrick Debois Wednesday 25 November 2009 My name is Patrick Debois and I come from Belgium
  3. 3. Wednesday 25 November 2009 Iʼm a freelancer always on the look out for new interesting things
  4. 4. Wednesday 25 November 2009 I have a sysadmin background
  5. 5. Wednesday 25 November 2009 Now for the past few years, Iʼve been looking together with other sysadmins with great interest to the Agile Developers community at what we could learn from Agile development
  6. 6. Wednesday 25 November 2009 And this year I wanted to give you my own version of Continuous Integration
  7. 7. Continuous Integration is the practice of integrating early and often, so as to avoid the pitfalls of "integration hell". The ultimate goal is to reduce timely rework and thus reduce cost and time. When done well, continuous integration has been shown to achieve these goals. Wednesday 25 November 2009 Let’s start with the definition on wikipedia. http://www.martinfowler.com/articles/ continuousIntegration.html http://en.wikipedia.org/wiki/Continuous_Integration
  8. 8. Traditional CI Overview http://www.javaworld.com/javaworld/jw-12-2008/images/CIOverview.jpg Wednesday 25 November 2009 Different parts: Source Control, Build, Test, Package, Deploy/ Publish Important to remember
  9. 9. Traditional CI Flow Courtesy Chris Read / ThoughtWorks Wednesday 25 November 2009 Courtesy Chris Read / ThoughtWorks http://www.slideshare.net/ChristopherRead/continuous- integration-build-pipelines-and-continuous-deployment
  10. 10. CI Maturity Model http://www.anthillpro.com/blogs/anthillpro-blog/2009/05/05/1241542860000.html Wednesday 25 November 2009 Yes some call it insane.
  11. 11. Build Pipeline Flow Courtesy Chris Read / ThoughtWorks Wednesday 25 November 2009 http://www.slideshare.net/ChristopherRead/continuous- integration-build-pipelines-and-continuous-deployment the flow - it it passes from Integration to UAT then to the next environment Huge problem is managing all these environments and keeping them aligned
  12. 12. Definition of Done Wednesday 25 November 2009 We need to go beyond the traditional testing and extend it to the live system
  13. 13. Agility in Production • redefine key concept • « the software works » • « the software works in production » • = beyond silos Wednesday 25 November 2009 http://www.lifeinamatrix.com/wp-content/uploads/2009/01/life-in-a-matrix-9- breaking-the-silos.jpg
  14. 14. Who is part of the team • developers • testers • analysts • project managers • usability experts • architects Wednesday 25 November 2009
  15. 15. But usually we forget • the sysadmins • the guys from operations Wednesday 25 November 2009
  16. 16. non-functional requirements “Equal rights amendment” Create value vs Protect value Wednesday 25 November 2009
  17. 17. Iron Triangle Wednesday 25 November 2009 “Enterprise” Triangle Usually only the left triangle is discussed, but it complements the second
  18. 18. Iron Triangle Project Wednesday 25 November 2009 “Enterprise” Triangle Usually only the left triangle is discussed, but it complements the second
  19. 19. Iron Triangle Project Wednesday 25 November 2009 “Enterprise” Triangle Usually only the left triangle is discussed, but it complements the second
  20. 20. Iron Triangle Project Operations Wednesday 25 November 2009 “Enterprise” Triangle Usually only the left triangle is discussed, but it complements the second
  21. 21. Iron Triangle Project Operations Wednesday 25 November 2009 “Enterprise” Triangle Usually only the left triangle is discussed, but it complements the second
  22. 22. Iron Triangle Project Operations Wednesday 25 November 2009 “Enterprise” Triangle Usually only the left triangle is discussed, but it complements the second
  23. 23. Traditional CI limits • Resistance to change • « if it ain't broken don't fix it » • Mostly hear as • « if it works don’t touch it » • Some even consider it best practice ! • « don’t put things into prod on a friday » Wednesday 25 November 2009
  24. 24. Why such a resistance • development = create value • operations = protect value • => operations must limit risks • change = risk • so we limit the changes • QED! Wednesday 25 November 2009
  25. 25. If it’s hard do it more often Wednesday 25 November 2009 We need to deploy more often. First times will fail, but eventually we will get better at this. The same for systems rebuilding.
  26. 26. Minimize Risks : integration of production • Add to traditional CI • deployment of VM, OS,packages, middleware • Production related tests • infrastructure • *ilities... • performance Wednesday 25 November 2009
  27. 27. CI for systems Wednesday 25 November 2009 now managing a system is becoming more like writing software. So we will draw the parallel in the next part of the presentation for Source , Build, Test
  28. 28. CI for systems (2) Wednesday 25 November 2009 packaging a system, deploying a system
  29. 29. Remember there is no silver bullet Wednesday 25 November 2009
  30. 30. Typical System Installation in CI • Download and Install a distribution • Download and Add some packages • Run some configuration scripts • Start software deployment testing Wednesday 25 November 2009
  31. 31. Problems? • It’s a one shot operation • Systems tend to become application friendly • Is this the same as production? Patches, Versions, ... • Usually allow all permissions Wednesday 25 November 2009
  32. 32. Life of System Birth / Adoption Life Palliative Death Wednesday 25 November 2009 Projects tend to think most of the birth of the software. But what about the system? We need to care about the running in production, in phase out and shutdown of a system
  33. 33. Examples of systems integrations in CI software Hudson - PXE Hudson - Vmware Not interested in the birth Wednesday 25 November 2009 Current integration of virtual machines in CI software
  34. 34. Evolution of operations In the beginning there were the dinosaurs, slow and with large delays Wednesday 25 November 2009 Often IT is viewed as this
  35. 35. Operations evolution (2) • Then came the monkeys with automated installations and automated deployment Wednesday 25 November 2009 Then they automated the installation
  36. 36. Clone system • there is no such things a golden image • image sprawl • Cloning = Adoption of a System Wednesday 25 November 2009 And then tried to clone things
  37. 37. Test in a clone of the production environment • Large effort • similar to maintaining a ‘fork’ Wednesday 25 November 2009
  38. 38. Wednesday 25 November 2009 So let’s build it from the same source! If we want to integrate let’s define what makes up the ‘source’ of our systems http://jserv.sayya.org/misc/use_source_luke.png
  39. 39. Virtual Machines API • Systems virtualization : Vserver, Zones, OpenVZ/ Virtuozzo, chroots • Hardware Emulation: VMWare, Xen, Qemu, VirtualBox, KVM Wednesday 25 November 2009 First part of the source
  40. 40. Language Bindings • java libvirt / virsh • ruby-libvirt • virtualbox SOAP Interface • vixie • vmware API Wednesday 25 November 2009
  41. 41. Example Virtualbox require 'soap/wsdlDriver' SOAP require 'pp' WSDL_URL="vboxwebService.wsdl" soap = SOAP::WSDLDriverFactory.new(WSDL_URL).create_rpc_driver soap.wiredump_dev=STDERR #soap = SOAP::WSDLDriverFactory.new(WSDL_URL).create_rpc_driver("vboxService", "vboxServicePort") #pp soap.methods vbox=soap.IWebsessionManager_logon({:username => '', :password => ''}) puts "Sessions"+vbox.returnval version=soap.IVirtualBox_getVersion({:_this => vbox.returnval}) puts version.returnval disks=soap.IVirtualBox_getHardDisks({:_this => vbox.returnval}) diskids=disks.returnval diskids.each do |diskid| type=soap.IHardDisk_getType({:_this => diskid }) size=soap.IHardDisk_getLogicalSize({:_this => diskid }) location=soap.IMedium_getLocation({:_this => diskid }) puts diskid+"-"+type.returnval+"-"+size.returnval+location.returnval end Wednesday 25 November 2009
  42. 42. Example libvirt require 'libvirt' conn = Libvirt::open("qemu:///system") puts conn.capabilities File::open("domain.xml") do |f| conn.createDomainLinux(f.readlines, nil) end dom = conn.lookupDomainByName("mydomain") dom.suspend dom.resume puts dom.xmlDesc Wednesday 25 November 2009 libvirt abstracts a lot of these types Xen, Qemu , Virtualbox, Vmware
  43. 43. Control the machine • Physical machine • IPMI, Light Out Module, Wake On LAN • Virtual machine • use of an API Wednesday 25 November 2009
  44. 44. Network provisioning • DNS • DHCP • PXE • TFTP Boot Wednesday 25 November 2009
  45. 45. Example Cobbler API # Set the distribution of the machine # The distribution was previously imported by importing an installation DVD # Cobbler will detect the possible kernels to boot # And this will also link the TFTP and PXE file necessary to Boot cobbler profile add --name=#{profile} --distro=centos53-i386 # Add the new machine with an IP and Mac address # By the IP Address it knows in which reverse DNS cobbler system add --name=#{name} --ip=#{ip} --mac=#{mac_address}" # Set the DNS domain of the machine (this determines in which zone file it is create) cobbler system edit --name=#{name} --profile=#{profile} --dns- name=puppet1.#{domainname}" # For linux machines, the kernels are provided with an option # ks=kickstart so that it will start the kickstart installation cobbler system edit --name=#{name} --kickstart=/var/lib/cobbler/ kickstarts/#{kickstart} cobbler system edit --name=#{name} --name-servers- search='#{domainname}' # This finally commits all the changes cobbler sync Wednesday 25 November 2009
  46. 46. Unattended OS Installs Jumpstart Preseed AutoYast Kickstart JET ..... Wednesday 25 November 2009
  47. 47. Example Kickstart # install_type MUST be first install_type initial_install # start with the minimal required number of packages cluster SUNWCXall cluster SUNWCapache delete cluster SUNWCpcmc delete cluster SUNWCpcmcx delete cluster SUNWCthai delete cluster SUNWClp delete cluster SUNWCnis delete cluster SUNWCppp delete # format the entire disk for Solaris fdisk all solaris all # define how the disk is partitioned partitioning explicit filesys rootdisk.s0 6144 / filesys rootdisk.s1 1024 swap filesys rootdisk.s7 free /state/partition1 # install systems as standalone system_type standalone # specify patches to install patch 119281-06 nfs 172.16.64.194:/export/patches # specify packages to install package SPROcc add nfs 172.16.64.194:/export/packages Wednesday 25 November 2009
  48. 48. It’s there under your nose • After your manual install (system DEV) • There is often a left over of silent install scripts • anaconda • silent install of oracle • answer files Wednesday 25 November 2009
  49. 49. OS minimal Just Enough Operating System Wednesday 25 November 2009 Instead of a full installation with the kickstart template. Use a minimal install/base install and then continue with config mgt.
  50. 50. configuration management • chef • puppet • cfengine 3 • bcfg2 • lcfg • smartfrog HP Wednesday 25 November 2009 Handles also live of the system.
  51. 51. class httpd { package { httpd: Sample Recipe } ensure => latest configfile { "/etc/httpd/conf/httpd.conf": source => "/httpd/httpd.conf", mode => 644, require => package["httpd"] } group { apache: gid => 48 } user { apache: comment => "Apache", uid => 48, gid => 48, home => "/var/www", shell => "/sbin/nologin" } service { httpd: running => true, subscribe => [ file["/etc/httpd/conf/httpd.conf"], package["httpd"] ] } } Wednesday 25 November 2009 Shareable recipes Formal description of what needs to happen instead of custom scripts
  52. 52. What a config mgt system can do for you • auto documentation • confidence • idempotent! sed ‘s/the/then/g’ Wednesday 25 November 2009
  53. 53. Managing Config Files API • augeas - RedHat/Fedora set /files/etc/hosts/01/ipaddr 192.168.0.1 set /files/etc/hosts/01/canonical pigiron.example.com set /files/etc/hosts/01/alias[1] pigiron set /files/etc/hosts/01/alias[2] piggy save Wednesday 25 November 2009 An extension for managing config files
  54. 54. Scriptable Appserver • glassfish, Weblogic Scripting • Oracle silent installs Wednesday 25 November 2009 Further automation
  55. 55. Example AppServer API # create an cluster asadmin create-cluster --user admin --host hostname -port 4848 cluster1 # create instance 1 asadmin create-instance --user admin --host hostname -port 4848 --cluster cluster1 --nodeagent nodeagent1 --systemproperties "JMX_SYSTEM_CONNECTOR_PORT=8687:IIOP_LISTENER_PORT=3330:IIOP_SSL_LIS TENER_PORT=4440:IIOP_SSL_MUTUALAUTH_PORT=5550:HTTP_LISTENER_PORT=111 0:HTTP_SSL_LISTENER_PORT=2220" instance1 # create instance 2 asadmin create-instance --user admin --host hostname -port 4848 --cluster cluster1 --nodeagent nodeagent1 --systemproperties “JMX_SYSTEM_CONNECTOR_PORT=8688:IIOP_LISTENER_PORT=3331:IIOP_SSL_LIS TENER_PORT=4441:IIOP_SSL_ MUTUALAUTH_PORT=5551:HTTP_LISTENER_PORT=1111:HTTP_SSL_LISTENER_PORT= 2221" instance2 # start the cluster asadmin start-cluster --user admin --host hostname --port 4848 cluster1 Wednesday 25 November 2009
  56. 56. Executing scripts • dsh, func, clusterssh, pssh, fabric • or system calls in ruby/java/.. • exit code = important! Wednesday 25 November 2009 For the final tweeks, we run shell scripts to make
  57. 57. Make your build self-testing Wednesday 25 November 2009
  58. 58. Test Driven Automation • Checks vs Tests • Monitoring vs Testing • Administration vs Automation Wednesday 25 November 2009 checks = automation of things to verify - similar to monitoring tests goes beyond check (human interpretation) Red green refactor (change of config, change of patch, ....)
  59. 59. TDD Virtual Machines Wednesday 25 November 2009 Testing if CPU, Memory , Disk are there by booting up a rescueCD and executing a script.
  60. 60. Example DNS Testing Feature: Mass DNS Resolution In order to present a consistent brand image As a System Administrator I want to ensure no domains point away from our main IP Scenario Outline: Resolve a name to a number Given a hostname of <hostname> Then I should see the IP address 266.266.266.266 Examples: | hostname | | example.org | | www.example.org | Wednesday 25 November 2009 http://blog.unixdaemon.net/cgi-bin/blosxom.pl/2009/04/06#ruby-dns-testing
  61. 61. Apache Recipe testing Wednesday 25 November 2009 Thanks to the build doctor! http://www.build-doctor.com/
  62. 62. Testing Deployment Scripts Capistrano • SMF m Wednesday 25 November 2009 http://pivotallabs.com/users/jdean/blog/articles/763-testing-capistrano-recipes-with-cucumber
  63. 63. Testing / Monitoring Cucumber-Nagios http://www.slideshare.net/auxesis/monitoring-web-application-behaviour-with-cucumbernagios Wednesday 25 November 2009 http://holmwood.id.au/~lindsay/2009/02/23/web-app-integration-testing-for-sysadmins-with- cucumber-nagios/
  64. 64. Lots of non functional tests Wednesday 25 November 2009 http://www.acutest.co.uk/acutest/load-testing-services
  65. 65. Maintain a code system repository • Virtual Machine definitions + automation scripts • Network provisioning (IP, DNS, DHCP) + scripts • Install Media (DVD, RPM’s) + • Kickstart Definitions + postinstall scripts • Config mgt recipes • Deployment scripts Wednesday 25 November 2009
  66. 66. vs. Snapshots Wednesday 25 November 2009 Git doesn’t handle well large files / rsync works Rsync snapshots (used by timemachine) File system repositories
  67. 67. Similarities • html/js/java - scripts & recipes • RPMS - jar files • VM’s - war files • local repositories Wednesday 25 November 2009
  68. 68. Every commit should be built Wednesday 25 November 2009
  69. 69. Automate the Build (of the System) Wednesday 25 November 2009 Good practice. Not only 1 part of it , but all pieces together.
  70. 70. Builders • maven, ant, • buildr • rake, raven • make • schmant • lancet • sbt, kundo, gant Wednesday 25 November 2009 There are a lot of ways to combine all this code using one of the builder tools. They know how to handle dependencies well
  71. 71. Funny Tests “Truck Factor” “10th floor test” Wednesday 25 November 2009 The 10th floor test , is a term first used by Steve Traugott from www.infrastructures.org It refers to the ability to take a random machine in your infrastructure , drop it from the 10th floor, and be able to restore your infrastructure to working within 5-10 minutes. Truck Factor - Project Truck Factor (definition): "The number of people on your team who have to be hit with a truck before the project is in serious trouble"
  72. 72. triggers • changes in VM definition • changes in IP, DNS, .. • changes in OS, packages • changes in recipes • changes in scripts Wednesday 25 November 2009
  73. 73. build ENV = PROD • Development • Test • Quality Assurance • Pre-production • Production • Differences in hardware • Differences in software • Differences in configuration Wednesday 25 November 2009 Templates/ Config should be configurable with the environment
  74. 74. Everyone commits every day Wednesday 25 November 2009
  75. 75. Keep the build fast Wednesday 25 November 2009
  76. 76. Virtual Machine Snapshots Filesystem Snapshots LVM, ZFS, ... Wednesday 25 November 2009 To speed up builds , intermedia results are stored using snapshots (system + disks)
  77. 77. Continuous vs Nightly • During the day, recipes only • Nightly the whole system rebuild Wednesday 25 November 2009
  78. 78. Package • koan : creates virtual images • rpath / rbuilder • ec2 • SUN forklift • jumpbox Wednesday 25 November 2009
  79. 79. Deployment tools • carpet • ec2 tools AMI • Vmware Appliance Wednesday 25 November 2009 http://www.agileweboperations.com/an-in-depth-look-at-a- carpet-appliance-the-apache-load-balancer/ Carpet= Mixture of puppet with capistrano
  80. 80. Everyone can see the results of the latest build Wednesday 25 November 2009
  81. 81. Make it easy to get the latest deliverables deploy on the latest installations Wednesday 25 November 2009 do you checkin wars for archiving
  82. 82. Hot topics • Cloud Computing • Production virtualization • Continuous deployment • Required fully automated provisioning Wednesday 25 November 2009 http://dev2ops.blogspot.com/2009/05/clouds-virtualization-and-continuous.html
  83. 83. Concept of Pipelines Wednesday 25 November 2009
  84. 84. Cloud perspective Wednesday 25 November 2009 There is actually very much of similarity with the cloud model
  85. 85. Key integration points • What environments to combine • Stable dev vs. Stable prod? • Stable dev vs. Test prod? • Test dev vs. Test prod? • => explosion combinations! • select the most sensible combination • be sure to be able to pass easily to the next environment Wednesday 25 November 2009
  86. 86. Problems/Pitfalls • Infrastructure and software need to be adapted for easy put into production • update with downtime (load-balancer, redundant servers, clusters, • data migrations Wednesday 25 November 2009
  87. 87. State Transition Process http://digiorgio.com/blog/?p=239 Wednesday 25 November 2009 If N+1 is in development then N is in integration , and N-1 in staging
  88. 88. Parallel Streams http://www.agilejournal.com/content/view/865/195/ Wednesday 25 November 2009 Example on how to integrate between different teams. It’s all about flow here!
  89. 89. Development Sysadmins Wednesday 25 November 2009
  90. 90. Development Sysadmins DEVOPS Wednesday 25 November 2009
  91. 91. More? http://www.devopsdays.org agile-system-administration@googlegroups.com IEEE-paper: http://www.jedi.be/presentations/IEEE-Agile-Infrastructure.pdf http://www.jedi.be/blog Email: patrick.debois@jedi.be Twitter: @patrickdebois Wednesday 25 November 2009
  92. 92. Wednesday 25 November 2009 So the next time you’re building a fun project
  93. 93. Wednesday 25 November 2009 Remember the sysadmins and guys from operations
  94. 94. We reboot with pride Wednesday 25 November 2009 And remember to reboot with pride
  95. 95. Questions? Wednesday 25 November 2009
  96. 96. Wednesday 25 November 2009 Thanks for listening.

×