DevOps for Developers

7,243 views

Published on

Published in: Technology
0 Comments
18 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
7,243
On SlideShare
0
From Embeds
0
Number of Embeds
1,061
Actions
Shares
0
Downloads
175
Comments
0
Likes
18
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • I am johann. a few people already know me. a warm welcome to you, too! \nI am founder and CTO of mayflower, sorry for not having a coffee lounge here this year. i am responsible for 65 php developers, so it‘s my job to improve the processes, and that‘s why i am talking about devops.\n\n
  • Because you are more interesting.\n
  • Who is a developer? \nWho is a administrator? Should you wear a beard?\nWho is neither a developer nor a system administrator? \nHey, nice, what are you? And why are you here? \n\n
  • Let‘s see what the development background is. How many of you are php developers? You can tell the truth, i am a PHP developer, too. sounds a bit like the alcoholics anonymous - „Hi, i am johann, and i am a PHP developer“.\nA java developer? Are there still jobs around for java developers? \n
  • \n
  • \n
  • Obviously You should do it because it‘s cool. DevOps is much of a hype right now, and you can be part of it! It‘s like the Google Wave developer hype without the disappointment later\n
  • The truth is a lot more boring - it‘s because we need it. Let me tell you a story about the dark age.\n
  • Do You remember the dark ages of development? How did development happen that days? \n(by the way: he does look a bit like benjamin eberlei, doesn‘t he?)\n
  • We used an basic vmware image. it was downloadable at some local fileserver, several gigabytes big and everything you needed for development was already installed. \n
  • This golden image always looked good in the beginning, but your application started to change. stuff was added, some kind of improvements were made. a default database was supplied, too. there was a default user used by everybody. \n
  • - but changes were needed - bash scripts to change the configuration\n- database update scripts - versioned database update scripts\n- a lot of bugs were solved by „you need to run the update script“\n- and the same amount of bugs were created by running the update script. \n- from time to time a new golden image was needed and some of the devs used it.\n
  • - after a while every developer had his own improved version of the image\n- incompatible, different versions, only the local version management sandbox was up to date\n\n
  • \n
  • \n
  • And in the good old days our application infrastructure was simple 3-tier\nweb server and database server were happening on one host.\n
  • Suddenly we had to add stuff. Like a hip NoSQL Server\n
  • And a memcache server, for Caching.\n
  • memcache became unhip, so it was replaced by redis\nan asynchronous messagequeue like gearman was introduced\n
  • Gearman wasn‘t so enterprisey in the end, so it was replaced by ActiveMQ. \nAn eJabberD was introduced for browser-side pubsub.\n
  • And actually it was 4 Servers now. \n
  • Ending up in 4 different bash scripted setup routines based on a set of 3 golden images.\n
  • On the other hand side, there wasn‘t just development, there was continious integration and production as well. sometimes with a different deployment mechanism.\n
  • And there were different Versions deployed, anyway. \n
  • With different tools, and software versions to work with different tools. Your application version happens in your version management system, your configuration in some adminstrators bash script. both are not in sync. \n
  • This wasn‘t any fun anymore. the number of wtf/minute was constantly increasing. We did not like it a lot.\n
  • 10th floor test: throw a random computer out of the windows and wait how long it takes everything is up & running again. We did not actually do it, since we are in a 5 store building. If your building is higher, try it out, it‘s a good benchmark.\n
  • That‘s our collection of fails. No simplicity, no failsafety - if a configuration is screwed it‘s screwed. \n
  • But how do we get there?\n
  • DevOps for the win!\n
  • (Danger: Code ahead)\n
  • (Danger: Code ahead). It works good on any linux, bsd etc. including Mac. Windows, especially with 64 bits is a bit hard to do, you have to use jruby. \n
  • With a cool logo!\n
  • First install vagrant and veewee. this is done using the default ruby gem install. \nlist baseboxes, choose yours and use it as your default box.\n\n
  • \n
  • \n
  • \n
  • First thing to know: configuration is code. it‘s not a setup anymore. \n
  • That are the two main players. like linux and freebsd, like gnome and kde everything opensource gets better when there are two of a kind. Does anyone still know cfengine?\n
  • On first sight chef and puppet look like twins. (Those are my sons, btw, sorry to show you, you know how proud parents are :-) )\n\n
  • \n
  • \n
  • \n
  • There are several tests and comparisons available online. half of the time puppet wins, half of the time chef does. there is no winner. have a look at it and take the tool you like. if you are an experenced ruby developer, chef is the better choice, if not, puppet can be. \n
  • That‘s how the puppet DSL looks like. You‘ll see some more examples later.\n
  • And that‘s how chef syntax looks like. The difference is:\nThis is ruby code. You have the full flexibility of the language available.\n
  • \n
  • That is the first Vagrantfile generated by vagrant init\n
  • Here we are talking vagrant. \n- puppet as a machine provisioner, with a link to the puppet directory and the default manifest for this machine - and more machines are possible\n- name of the server, network configuration, port forwarding and mount points.\n\n
  • This is the puppet configuration file for my nodes ( servers). i can include directores, and i can include other classes in my classes. \n
  • That‘s the included definition of the web class. see, there is inheritance.\nThe apache-include is a puppet module and provides for example the vhost configuration\nThe package is a resource wrapper for apt here, since this is an ubuntu natty setup.\n
  • This is an example for a custom package provider for pear packages.\n
  • \n
  • This is an example based on the zend framework default directory layout. Two parts are going to change - the configs will contain the server setup as well, and there is a new vms folder within the scripts directory, containing a Vagrantfile. let‘s cd into scripts/configuration and start to work\n
  • \n
  • That‘s our collection of fails. No simplicity, no failsafety - if a configuration is screwed it‘s screwed. \n
  • \n
  • \n
  • And all the developer has to do is a vagrant up to get his vms from the source\nif there have been configuration changes just do a vagrant provision\nno more. \nNO NEED TO SAVE YOUR VM ANYMORE!\n
  • \n
  • this is an additional module for vagrant to give you a chance to screw your vms.\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Right now libvirt-based, in future ec2-support is going to happen\n
  • \n
  • That‘s how you create a new machine. Or your developers do in self service. \nDo you remember how long this took before? \n\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Danke an Jimdo für das Beispiel.. \n
  • \n
  • Wer setzt Jenkins ein? (sonst erklären)\n
  • \n
  • Sebastian wird hierüber noch mehr erzählen. \n
  • \n
  • \n
  • - gemeinsame Standups\n- gegenseitige Teilnahme an den Sprint Plannings & Retros\n- gleiche Räume, wenn möglich\n
  • Der Code gehört auch den Admins, die Konfiguration und die Verlässlichkeit auch den Developern.\n
  • Wie bekommt man Respekt hin?\n- Soziale Interaktion, Feiern, Teambuilding\nWenn ich jemand persönliche kenne nehme ich auf seine Interessen Rücksicht\n
  • Die langfristige Planung wird gemeinsam gemacht. Es werden gemeinsame Ziele definiert, und die Lösungsstrategien gemeinsam erstellt.\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • DevOps for Developers

    1. 1. Johann-Peter Hartmann / Mayflower GmbHPractical DevOps forDevelopers
    2. 2. Hi!
    3. 3. Nice to meet You!
    4. 4. I am Johann.
    5. 5. Who are You?
    6. 6. Developers? Ops?Anything else?
    7. 7. Developers: PHP? Python? Ruby? Java?Anything else?
    8. 8. What do you expect to hear today? (if you got a good explanationi‘ll leave out the motivation slides)
    9. 9. Motivation orWhy it‘s more fun to do DevOps as a Dev right now.
    10. 10. Hype!Googles CIO Ben Fried 11 days ago: „Cult Of DevOps“
    11. 11. Actually:Because we need it.
    12. 12. Development in the dark ages ...
    13. 13. Golden ImageVMware /KVM
    14. 14. It wasn‘t reliable anymore.
    15. 15. And our applications changed
    16. 16. BrowserApache with PHP MySQL
    17. 17. Browser HipApache with PHP MySQL NoSQL Server
    18. 18. BrowserApache with PHP MySQL Hip Memcache NoSQL Performance FTW! Server
    19. 19. Browser HipApache with PHP MySQL NoSQL Server Memcache, oops, Gearman Redis now
    20. 20. Browser HipApache with PHP MySQL NoSQL ServerMemcache, oops, Gearman, ahum, eJabberD Redis now ActiveMQ
    21. 21. Browser HipApache with Apache with MySQL NoSQL PHP PHP ServerMemcache, Memcache, Gearman, oops, oops, ahum, eJabberDRedis now Redis now ActiveMQ
    22. 22. Browser HipApache with Apache with MySQL NoSQL PHP PHP ServerMemcache, Memcache, Gearman, oops, oops, ahum, eJabberDRedis now Redis now ActiveMQ
    23. 23. Apa Apa MyS Hip Apa Apa MyS HipMe Me Gea eJa Me Me Gea eJa Development Continuous Integration Apa Apa Apa Apa MyS MyS Hip Hip Me Me Me Me Gea Gea eJa eJa Production
    24. 24. Version X+2 Version X+1 Development Continuous Integration Version X Production
    25. 25. New! With Redis! Not so new with MemcachedVersion X+2 Version X+1 Development Continuous Integration Still without proper Caching :-( Version X Production
    26. 26. 10th Floor Test http://www.flickr.com/photos/75905404@N00/
    27. 27. Collection of Fails Failsafety Simplicity Fast Setup TimeRepeatability Self-ServiceConsistency Version Management http://www.flickr.com/photos/turtlemom_nancy/
    28. 28. ?
    29. 29. DevOps for Devs in3 (actually not so easy) steps
    30. 30. 1. Manage your Dev-Setup with Vagrant and VeeWee (Install VirtualBox and Ruby first)
    31. 31. Vagrant Virtualbox based automatic creation and management of VMs based on Puppet / Chef
    32. 32. Better use rvm, see http://www.jedi.be/blog/2011/03/28/using-vagrant-as-a-team/~# gem install vagrant~# gem install veewee~# vagrant basebox templatesvagrant basebox define <boxname> archlinux-i386... (40 baseboxes)vagrant basebox define <boxname> windows-2008R2-amd64~# vagrant basebox define natty ubuntu-11.04-server-amd64~# vagrant basebox build natty~# vagrant basebox export natty~# vagrant box add natty natty.box
    33. 33. http://vagrantbox.es
    34. 34. ~# vagrant init natty~# vagrant up~# vagrant sshWelcome to Ubuntu 11.04 (GNU/Linux 2.6.38-8-server x86_64)* Documentation: http://www.ubuntu.com/server/docvagrant@natty:~$
    35. 35. ~# vagrantTasks: vagrant basebox # Commands to manage baseboxes vagrant box # Commands to manage system boxes vagrant destroy # Destroy the environment, deleting the created virtualmachines vagrant halt # Halt the running VMs in the environment vagrant help [TASK] # Describe available tasks or one specific task vagrant init [box_name] [box_url] # Initializes the current folder forVagrant usage vagrant package # Package a Vagrant environment for distribution vagrant provision # Rerun the provisioning scripts on a running VM vagrant reload # Reload the environment, halting it then restartingit. vagrant resume # Resume a suspended Vagrant environment. vagrant ssh # SSH into the currently running Vagrant environment. vagrant ssh_config # outputs .ssh/config valid syntax for connecting tothis environment via ssh vagrant status # Shows the status of the current Vagrant environment. vagrant suspend # Suspend a running Vagrant environment. vagrant up # Creates the Vagrant environment vagrant version # Prints the Vagrant version information
    36. 36. 2. Manage your configuration
    37. 37. Similarities
    38. 38. • Configuration as (Ruby-)Code• Client-only or Client-server Setup• there is a series-b funded company in the background• both are officially supported by amazon• there are a lot of BIG customers using the tool• good documentation• good, vibrant communities
    39. 39. • both know the current configuration (ohai and facter)• You define your nodes (Servers)• using a lot of pre-existing resources• and a lot of default community-built cookbooks / modules• it‘s easy to extend using ruby• use configuration file templates• use providers as platform abstractions (for packaging, ...)
    40. 40. • Chef is actually Ruby, Puppet provides a DSL• puppet has the bigger community• puppet has more documentation• but chef is growing fast in both regards• puppet = europe, chef = usa• chef is more flexible because of native ruby• chef is more flexible because of clever data structures
    41. 41. There is no „better“ tool.
    42. 42. There is no „better“ tool.But we prefer Puppet. Less Ruby :-)
    43. 43. user { johann: ensure => present, uid => 507, gid => admin, shell => /bin/bash, home => /home/johann, managehome => true,}
    44. 44. user "johann" do username "johann" password "$1$P$WXmqrQEVj88fVTHevErxq." shell "/bin/bash" system true supports :manage_home => trueend
    45. 45. Back to your setup ...
    46. 46. Vagrant::Config.run do |config| config.vm.box = "natty"end
    47. 47. ~# cat VagrantfileVagrant::Config.run do |config| config.vm.provision :puppet, :module_path => "modules" do|puppet| puppet.manifests_path = "manifests" puppet.manifest_file = "development.pp" end config.vm.define :web do |web_config| web_config.vm.box = "natty" web_config.vm.host_name = "webserver01" # web_config.vm.boot_mode = :gui web_config.vm.network "33.33.33.10" web_config.vm.forward_port "http", 80, 8080 web_config.vm.forward_port "ssh", 22, 20022 web_config.vm.share_folder "v-data", "/srv/www", "../silex-demo" endend
    48. 48. ~# cat manifests/development.ppimport "classes/*"node "webserver01" { include web}node "dbserver01" { include db}node "ciserver01" { include ci}
    49. 49. ~# cat manifests/classes/web.ppclass web inherits basenode { include apache include apache::php apache::vhost { silex-demo.local: port => 80, docroot => /srv/www/docroot, } package { ["mysql-client", "php5-cli", "phpunit", "php5-curl", "php5-dev", "php5-gd", "php5-imagick", "php5-mcrypt","php5-mysql", "php5-xdebug","php5-suhosin", "php-pear","php-codesniffer" ]: ensure => present, }}
    50. 50. ~# cat manifests/classes/ci.ppclass ci inherits basenode { include apache include apache::php exec { "pear_autodiscover": command => "/usr/bin/pear config-set auto_discover 1", } package { ["pear.phpunit.de/PHP_CodeBrowser","pear.phpunit.de/PHPUnit_MockObject", "pear.phpunit.de/PHPUnit_Selenium", "pear.phpunit.de/PHP_CodeCoverage","pear.phpunit.de/PHP_Timer", "pear.phpunit.de/phpcpd","pear.phpunit.de/phploc"]: ensure => latest, provider => "pear", require => Exec["pear_autodiscover"] }}
    51. 51. 3. Make the configuration part of your sourcecode
    52. 52. • application• data• docs• library• public• scripts • jobs • build • configuration • VagrantFile • manifests • modules• temp
    53. 53. Ok, that was a lot of work. Why did i do that again?
    54. 54. Collection of Wins Failsafety Simplicity Fast Setup TimeRepeatability Self-ServiceConsistency Version Management http://www.flickr.com/photos/turtlemom_nancy/
    55. 55. There is no golden Image anymore
    56. 56. There is just one directory in your source ...
    57. 57. „vagrant up“„vagrant provision“
    58. 58. Even more vagrant fun ...
    59. 59. Fail Safety with Vagrant-Snapvagrant snap take -d “snap1“vagrant snap listvagrant snap go “snap1“
    60. 60. What if i need tosimulate 20
    61. 61. McCloudWrapper likeVagrant around FogTransparent local & cloud usageSupports EC2, OpenStack, KVM, etc
    62. 62. What if i need tosimulate 2000machines?
    63. 63. mCollectivedssh/ssh-for-loop on steroidsfast management for loads of serversuses puppet/facter or chef/ohai, MQ- based
    64. 64. $ mc-package -W "architecture=x86" status apache * [ ============================================================> ] 10 / 10host01.example.com version = apache-2.2.9-7host02.example.com version = apache-2.2.9-7host03.example.com version = apache-2.2.9-7host04.example.com version = apache-2.2.9-7host05.example.com version = apache-2.2.9-7host06.example.com version = apache-2.2.9-7host07.example.com version = apache-2.2.9-7host08.example.com version = apache-2.2.9-7host09.example.com version = apache-2.2.9-7host10.example.com version = apache-2.2.9-7---- package agent summary ---- Nodes: 10 / 10 Versions: 10 * 0.25.5-1.el5 Elapsed Time: 1.03 s
    65. 65. The Foreman - Machine Life CycleManagementActs as a web front end for PuppetShows You the system inventoryCreates new machines and takes care of provisioning
    66. 66. Knowing everything about yourserver configuration is great!
    67. 67. Create new servers
    68. 68. Your (possible) Todolist
    69. 69. 1. Install Vagrant, Veewee etc
    70. 70. 2. Add anconfigurationfolder to your
    71. 71. 3. Configure Your Vagrant images using chef or puppet
    72. 72. 4. Setup a chef-or puppet-server using this configuration
    73. 73. 5. Move your CI, Staging andProduction-Setup to chef/puppet, too
    74. 74. 6. Create a self-service-plattform for thedevelopment team
    75. 75. More cool stuff you can do:
    76. 76. Test-Driven infrastructure using chef/puppet-cucumber
    77. 77. Automatedmonitoring usingnagios or munin
    78. 78. Automatedreporting using graphite
    79. 79. Thanks!Johann-Peter Hartmannjohann__ @ freenode I hope you enjoyed it!Mail / Jabber:johann-peter.hartmann@mayflower.deFurther reading:http://www.planetdevops.nethttp://dzone.comhttp://twitter.com/#!/DEVOPS_BORAThttp://github.com/johannhartmann/
    80. 80. Vegetable Test DrivenInfrastructure Kontinuierliche Entiwcklung - und dann? I Mayflower GmbH I 28. Oktober 2010 I 87
    81. 81. actually a behavior driven design toolused for test driven infrastructurecucumber-puppetchef-cucumber
    82. 82. Feature: Manualsearch In order to find an article As an developer I want to use the search function Scenario: Search for bdd and check resulting page Given I go to "http://it-republik.de/php/" When I fill in "search_itr" with "bdd" And I click "search2" Then I should see "Suche"
    83. 83. Given /^I go to "([^"]*)"$/ do |url| visit urlendWhen /^I fill in "([^"]*)" with "([^"]*)"$/ do |field, value| fill_in field, :with => valueendWhen /^I click "([^"]*)"$/ do |button| click_button(button)endThen /^I should see "([^"]*)"$/ do |text| response_body.should include(text)end
    84. 84. johann$ cucumberFeature: Manualsearch In order to find an article As an developer I want to use the search function Scenario: Search for bdd and check resulting page # features/search.feature:5 Given I go to "http://it-republik.de/php/" # features/step_definitions/search_steps.rb:1 When I fill in "search_itr" with "bdd" # features/step_definitions/search_steps.rb:5 And I click "search2" # features/step_definitions/search_steps.rb:9 Then I should see "Suche" # features/step_definitions/search_steps.rb:131 scenario (1 passed)4 steps (4 passed)0m1.615s
    85. 85. Feature: Install inetd In order to serve the web the httpd service must be installed Scenario: Setup httpd Given a node of class “web“ When I compile the catalog Then package “httpd“ should be “installed“ Then there should be a resource “Service[httpd]“ And the service should have “enable“ set to “true“ And State should be “running“ And the service should require “Package[httpd]“ Kontinuierliche Entiwcklung - und dann? I Mayflower GmbH I 28. Oktober 2010 I 92
    86. 86. Hey, i understandOperations - and i‘m working in marketing
    87. 87. Unit TestsAcceptance TestsMetricsCoding Style... you get theidea.
    88. 88. ... and evenmore ...PackagingInfrastructureUpdatesReleasesDeployment
    89. 89. WetwareRefactoring
    90. 90. Work together
    91. 91. Own together
    92. 92. Eat together http://www.flickr.com/photos/vilavelosa/ 3815032524/
    93. 93. Plantogether Kontinuierliche Entiwcklung - und dann? I Mayflower GmbH I 28. Oktober 2010 I 103
    94. 94. MayflowerDevOpsification
    95. 95. Wetware - Mayflower 1-2 Ops per team • Admin & Development tasks • full time working within the team
    96. 96. Wetware - MayflowerClose cooperation with central operationsFull root access for any development-infrastructre
    97. 97. Wetware - Mayflower1+n Puppet-Master • central company master • Team puppet master per team / project • company master is starting point for the project puppet configuration
    98. 98. Wetware - MayflowerExample Setup: • local Developer VM • CI-Deployment-Server in the DMZ • Staging in the private Cloud • Beta happens in the Amazon-Cloud • Production in AWS, too
    99. 99. Wetware - Mayflowerlocal GIT- / Gitorious-ServerEucalyptus-Cloud in the DMZ - as Self-Service!
    100. 100. Wetware - Mayflower Vagrant for Development Scrum => KanBan Puppet Nagios

    ×