DevOps for Developers
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

DevOps for Developers

on

  • 6,917 views

 

Statistics

Views

Total Views
6,917
Views on SlideShare
6,016
Embed Views
901

Actions

Likes
16
Downloads
166
Comments
0

7 Embeds 901

http://agile.dzone.com 863
http://paper.li 21
http://a0.twimg.com 9
http://php.dzone.com 3
http://www.dzone.com 2
http://www.scoop.it 2
http://ruby.dzone.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • \n
  • \n
  • \n
  • I am johann. a few people already know me. a warm welcome to you, too! \nI am founder and CTO of mayflower, sorry for not having a coffee lounge here this year. i am responsible for 65 php developers, so it‘s my job to improve the processes, and that‘s why i am talking about devops.\n\n
  • Because you are more interesting.\n
  • Who is a developer? \nWho is a administrator? Should you wear a beard?\nWho is neither a developer nor a system administrator? \nHey, nice, what are you? And why are you here? \n\n
  • Let‘s see what the development background is. How many of you are php developers? You can tell the truth, i am a PHP developer, too. sounds a bit like the alcoholics anonymous - „Hi, i am johann, and i am a PHP developer“.\nA java developer? Are there still jobs around for java developers? \n
  • \n
  • \n
  • Obviously You should do it because it‘s cool. DevOps is much of a hype right now, and you can be part of it! It‘s like the Google Wave developer hype without the disappointment later\n
  • The truth is a lot more boring - it‘s because we need it. Let me tell you a story about the dark age.\n
  • Do You remember the dark ages of development? How did development happen that days? \n(by the way: he does look a bit like benjamin eberlei, doesn‘t he?)\n
  • We used an basic vmware image. it was downloadable at some local fileserver, several gigabytes big and everything you needed for development was already installed. \n
  • This golden image always looked good in the beginning, but your application started to change. stuff was added, some kind of improvements were made. a default database was supplied, too. there was a default user used by everybody. \n
  • - but changes were needed - bash scripts to change the configuration\n- database update scripts - versioned database update scripts\n- a lot of bugs were solved by „you need to run the update script“\n- and the same amount of bugs were created by running the update script. \n- from time to time a new golden image was needed and some of the devs used it.\n
  • - after a while every developer had his own improved version of the image\n- incompatible, different versions, only the local version management sandbox was up to date\n\n
  • \n
  • \n
  • And in the good old days our application infrastructure was simple 3-tier\nweb server and database server were happening on one host.\n
  • Suddenly we had to add stuff. Like a hip NoSQL Server\n
  • And a memcache server, for Caching.\n
  • memcache became unhip, so it was replaced by redis\nan asynchronous messagequeue like gearman was introduced\n
  • Gearman wasn‘t so enterprisey in the end, so it was replaced by ActiveMQ. \nAn eJabberD was introduced for browser-side pubsub.\n
  • And actually it was 4 Servers now. \n
  • Ending up in 4 different bash scripted setup routines based on a set of 3 golden images.\n
  • On the other hand side, there wasn‘t just development, there was continious integration and production as well. sometimes with a different deployment mechanism.\n
  • And there were different Versions deployed, anyway. \n
  • With different tools, and software versions to work with different tools. Your application version happens in your version management system, your configuration in some adminstrators bash script. both are not in sync. \n
  • This wasn‘t any fun anymore. the number of wtf/minute was constantly increasing. We did not like it a lot.\n
  • 10th floor test: throw a random computer out of the windows and wait how long it takes everything is up & running again. We did not actually do it, since we are in a 5 store building. If your building is higher, try it out, it‘s a good benchmark.\n
  • That‘s our collection of fails. No simplicity, no failsafety - if a configuration is screwed it‘s screwed. \n
  • But how do we get there?\n
  • DevOps for the win!\n
  • (Danger: Code ahead)\n
  • (Danger: Code ahead). It works good on any linux, bsd etc. including Mac. Windows, especially with 64 bits is a bit hard to do, you have to use jruby. \n
  • With a cool logo!\n
  • First install vagrant and veewee. this is done using the default ruby gem install. \nlist baseboxes, choose yours and use it as your default box.\n\n
  • \n
  • \n
  • \n
  • First thing to know: configuration is code. it‘s not a setup anymore. \n
  • That are the two main players. like linux and freebsd, like gnome and kde everything opensource gets better when there are two of a kind. Does anyone still know cfengine?\n
  • On first sight chef and puppet look like twins. (Those are my sons, btw, sorry to show you, you know how proud parents are :-) )\n\n
  • \n
  • \n
  • \n
  • There are several tests and comparisons available online. half of the time puppet wins, half of the time chef does. there is no winner. have a look at it and take the tool you like. if you are an experenced ruby developer, chef is the better choice, if not, puppet can be. \n
  • That‘s how the puppet DSL looks like. You‘ll see some more examples later.\n
  • And that‘s how chef syntax looks like. The difference is:\nThis is ruby code. You have the full flexibility of the language available.\n
  • \n
  • That is the first Vagrantfile generated by vagrant init\n
  • Here we are talking vagrant. \n- puppet as a machine provisioner, with a link to the puppet directory and the default manifest for this machine - and more machines are possible\n- name of the server, network configuration, port forwarding and mount points.\n\n
  • This is the puppet configuration file for my nodes ( servers). i can include directores, and i can include other classes in my classes. \n
  • That‘s the included definition of the web class. see, there is inheritance.\nThe apache-include is a puppet module and provides for example the vhost configuration\nThe package is a resource wrapper for apt here, since this is an ubuntu natty setup.\n
  • This is an example for a custom package provider for pear packages.\n
  • \n
  • This is an example based on the zend framework default directory layout. Two parts are going to change - the configs will contain the server setup as well, and there is a new vms folder within the scripts directory, containing a Vagrantfile. let‘s cd into scripts/configuration and start to work\n
  • \n
  • That‘s our collection of fails. No simplicity, no failsafety - if a configuration is screwed it‘s screwed. \n
  • \n
  • \n
  • And all the developer has to do is a vagrant up to get his vms from the source\nif there have been configuration changes just do a vagrant provision\nno more. \nNO NEED TO SAVE YOUR VM ANYMORE!\n
  • \n
  • this is an additional module for vagrant to give you a chance to screw your vms.\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Right now libvirt-based, in future ec2-support is going to happen\n
  • \n
  • That‘s how you create a new machine. Or your developers do in self service. \nDo you remember how long this took before? \n\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Danke an Jimdo für das Beispiel.. \n
  • \n
  • Wer setzt Jenkins ein? (sonst erklären)\n
  • \n
  • Sebastian wird hierüber noch mehr erzählen. \n
  • \n
  • \n
  • - gemeinsame Standups\n- gegenseitige Teilnahme an den Sprint Plannings & Retros\n- gleiche Räume, wenn möglich\n
  • Der Code gehört auch den Admins, die Konfiguration und die Verlässlichkeit auch den Developern.\n
  • Wie bekommt man Respekt hin?\n- Soziale Interaktion, Feiern, Teambuilding\nWenn ich jemand persönliche kenne nehme ich auf seine Interessen Rücksicht\n
  • Die langfristige Planung wird gemeinsam gemacht. Es werden gemeinsame Ziele definiert, und die Lösungsstrategien gemeinsam erstellt.\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n

DevOps for Developers Presentation Transcript

  • 1. Johann-Peter Hartmann / Mayflower GmbHPractical DevOps forDevelopers
  • 2. Hi!
  • 3. Nice to meet You!
  • 4. I am Johann.
  • 5. Who are You?
  • 6. Developers? Ops?Anything else?
  • 7. Developers: PHP? Python? Ruby? Java?Anything else?
  • 8. What do you expect to hear today? (if you got a good explanationi‘ll leave out the motivation slides)
  • 9. Motivation orWhy it‘s more fun to do DevOps as a Dev right now.
  • 10. Hype!Googles CIO Ben Fried 11 days ago: „Cult Of DevOps“
  • 11. Actually:Because we need it.
  • 12. Development in the dark ages ...
  • 13. Golden ImageVMware /KVM
  • 14. It wasn‘t reliable anymore.
  • 15. And our applications changed
  • 16. BrowserApache with PHP MySQL
  • 17. Browser HipApache with PHP MySQL NoSQL Server
  • 18. BrowserApache with PHP MySQL Hip Memcache NoSQL Performance FTW! Server
  • 19. Browser HipApache with PHP MySQL NoSQL Server Memcache, oops, Gearman Redis now
  • 20. Browser HipApache with PHP MySQL NoSQL ServerMemcache, oops, Gearman, ahum, eJabberD Redis now ActiveMQ
  • 21. Browser HipApache with Apache with MySQL NoSQL PHP PHP ServerMemcache, Memcache, Gearman, oops, oops, ahum, eJabberDRedis now Redis now ActiveMQ
  • 22. Browser HipApache with Apache with MySQL NoSQL PHP PHP ServerMemcache, Memcache, Gearman, oops, oops, ahum, eJabberDRedis now Redis now ActiveMQ
  • 23. Apa Apa MyS Hip Apa Apa MyS HipMe Me Gea eJa Me Me Gea eJa Development Continuous Integration Apa Apa Apa Apa MyS MyS Hip Hip Me Me Me Me Gea Gea eJa eJa Production
  • 24. Version X+2 Version X+1 Development Continuous Integration Version X Production
  • 25. New! With Redis! Not so new with MemcachedVersion X+2 Version X+1 Development Continuous Integration Still without proper Caching :-( Version X Production
  • 26. 10th Floor Test http://www.flickr.com/photos/75905404@N00/
  • 27. Collection of Fails Failsafety Simplicity Fast Setup TimeRepeatability Self-ServiceConsistency Version Management http://www.flickr.com/photos/turtlemom_nancy/
  • 28. ?
  • 29. DevOps for Devs in3 (actually not so easy) steps
  • 30. 1. Manage your Dev-Setup with Vagrant and VeeWee (Install VirtualBox and Ruby first)
  • 31. Vagrant Virtualbox based automatic creation and management of VMs based on Puppet / Chef
  • 32. Better use rvm, see http://www.jedi.be/blog/2011/03/28/using-vagrant-as-a-team/~# gem install vagrant~# gem install veewee~# vagrant basebox templatesvagrant basebox define <boxname> archlinux-i386... (40 baseboxes)vagrant basebox define <boxname> windows-2008R2-amd64~# vagrant basebox define natty ubuntu-11.04-server-amd64~# vagrant basebox build natty~# vagrant basebox export natty~# vagrant box add natty natty.box
  • 33. http://vagrantbox.es
  • 34. ~# vagrant init natty~# vagrant up~# vagrant sshWelcome to Ubuntu 11.04 (GNU/Linux 2.6.38-8-server x86_64)* Documentation: http://www.ubuntu.com/server/docvagrant@natty:~$
  • 35. ~# vagrantTasks: vagrant basebox # Commands to manage baseboxes vagrant box # Commands to manage system boxes vagrant destroy # Destroy the environment, deleting the created virtualmachines vagrant halt # Halt the running VMs in the environment vagrant help [TASK] # Describe available tasks or one specific task vagrant init [box_name] [box_url] # Initializes the current folder forVagrant usage vagrant package # Package a Vagrant environment for distribution vagrant provision # Rerun the provisioning scripts on a running VM vagrant reload # Reload the environment, halting it then restartingit. vagrant resume # Resume a suspended Vagrant environment. vagrant ssh # SSH into the currently running Vagrant environment. vagrant ssh_config # outputs .ssh/config valid syntax for connecting tothis environment via ssh vagrant status # Shows the status of the current Vagrant environment. vagrant suspend # Suspend a running Vagrant environment. vagrant up # Creates the Vagrant environment vagrant version # Prints the Vagrant version information
  • 36. 2. Manage your configuration
  • 37. Similarities
  • 38. • Configuration as (Ruby-)Code• Client-only or Client-server Setup• there is a series-b funded company in the background• both are officially supported by amazon• there are a lot of BIG customers using the tool• good documentation• good, vibrant communities
  • 39. • both know the current configuration (ohai and facter)• You define your nodes (Servers)• using a lot of pre-existing resources• and a lot of default community-built cookbooks / modules• it‘s easy to extend using ruby• use configuration file templates• use providers as platform abstractions (for packaging, ...)
  • 40. • Chef is actually Ruby, Puppet provides a DSL• puppet has the bigger community• puppet has more documentation• but chef is growing fast in both regards• puppet = europe, chef = usa• chef is more flexible because of native ruby• chef is more flexible because of clever data structures
  • 41. There is no „better“ tool.
  • 42. There is no „better“ tool.But we prefer Puppet. Less Ruby :-)
  • 43. user { johann: ensure => present, uid => 507, gid => admin, shell => /bin/bash, home => /home/johann, managehome => true,}
  • 44. user "johann" do username "johann" password "$1$P$WXmqrQEVj88fVTHevErxq." shell "/bin/bash" system true supports :manage_home => trueend
  • 45. Back to your setup ...
  • 46. Vagrant::Config.run do |config| config.vm.box = "natty"end
  • 47. ~# cat VagrantfileVagrant::Config.run do |config| config.vm.provision :puppet, :module_path => "modules" do|puppet| puppet.manifests_path = "manifests" puppet.manifest_file = "development.pp" end config.vm.define :web do |web_config| web_config.vm.box = "natty" web_config.vm.host_name = "webserver01" # web_config.vm.boot_mode = :gui web_config.vm.network "33.33.33.10" web_config.vm.forward_port "http", 80, 8080 web_config.vm.forward_port "ssh", 22, 20022 web_config.vm.share_folder "v-data", "/srv/www", "../silex-demo" endend
  • 48. ~# cat manifests/development.ppimport "classes/*"node "webserver01" { include web}node "dbserver01" { include db}node "ciserver01" { include ci}
  • 49. ~# cat manifests/classes/web.ppclass web inherits basenode { include apache include apache::php apache::vhost { silex-demo.local: port => 80, docroot => /srv/www/docroot, } package { ["mysql-client", "php5-cli", "phpunit", "php5-curl", "php5-dev", "php5-gd", "php5-imagick", "php5-mcrypt","php5-mysql", "php5-xdebug","php5-suhosin", "php-pear","php-codesniffer" ]: ensure => present, }}
  • 50. ~# cat manifests/classes/ci.ppclass ci inherits basenode { include apache include apache::php exec { "pear_autodiscover": command => "/usr/bin/pear config-set auto_discover 1", } package { ["pear.phpunit.de/PHP_CodeBrowser","pear.phpunit.de/PHPUnit_MockObject", "pear.phpunit.de/PHPUnit_Selenium", "pear.phpunit.de/PHP_CodeCoverage","pear.phpunit.de/PHP_Timer", "pear.phpunit.de/phpcpd","pear.phpunit.de/phploc"]: ensure => latest, provider => "pear", require => Exec["pear_autodiscover"] }}
  • 51. 3. Make the configuration part of your sourcecode
  • 52. • application• data• docs• library• public• scripts • jobs • build • configuration • VagrantFile • manifests • modules• temp
  • 53. Ok, that was a lot of work. Why did i do that again?
  • 54. Collection of Wins Failsafety Simplicity Fast Setup TimeRepeatability Self-ServiceConsistency Version Management http://www.flickr.com/photos/turtlemom_nancy/
  • 55. There is no golden Image anymore
  • 56. There is just one directory in your source ...
  • 57. „vagrant up“„vagrant provision“
  • 58. Even more vagrant fun ...
  • 59. Fail Safety with Vagrant-Snapvagrant snap take -d “snap1“vagrant snap listvagrant snap go “snap1“
  • 60. What if i need tosimulate 20
  • 61. McCloudWrapper likeVagrant around FogTransparent local & cloud usageSupports EC2, OpenStack, KVM, etc
  • 62. What if i need tosimulate 2000machines?
  • 63. mCollectivedssh/ssh-for-loop on steroidsfast management for loads of serversuses puppet/facter or chef/ohai, MQ- based
  • 64. $ mc-package -W "architecture=x86" status apache * [ ============================================================> ] 10 / 10host01.example.com version = apache-2.2.9-7host02.example.com version = apache-2.2.9-7host03.example.com version = apache-2.2.9-7host04.example.com version = apache-2.2.9-7host05.example.com version = apache-2.2.9-7host06.example.com version = apache-2.2.9-7host07.example.com version = apache-2.2.9-7host08.example.com version = apache-2.2.9-7host09.example.com version = apache-2.2.9-7host10.example.com version = apache-2.2.9-7---- package agent summary ---- Nodes: 10 / 10 Versions: 10 * 0.25.5-1.el5 Elapsed Time: 1.03 s
  • 65. The Foreman - Machine Life CycleManagementActs as a web front end for PuppetShows You the system inventoryCreates new machines and takes care of provisioning
  • 66. Knowing everything about yourserver configuration is great!
  • 67. Create new servers
  • 68. Your (possible) Todolist
  • 69. 1. Install Vagrant, Veewee etc
  • 70. 2. Add anconfigurationfolder to your
  • 71. 3. Configure Your Vagrant images using chef or puppet
  • 72. 4. Setup a chef-or puppet-server using this configuration
  • 73. 5. Move your CI, Staging andProduction-Setup to chef/puppet, too
  • 74. 6. Create a self-service-plattform for thedevelopment team
  • 75. More cool stuff you can do:
  • 76. Test-Driven infrastructure using chef/puppet-cucumber
  • 77. Automatedmonitoring usingnagios or munin
  • 78. Automatedreporting using graphite
  • 79. Thanks!Johann-Peter Hartmannjohann__ @ freenode I hope you enjoyed it!Mail / Jabber:johann-peter.hartmann@mayflower.deFurther reading:http://www.planetdevops.nethttp://dzone.comhttp://twitter.com/#!/DEVOPS_BORAThttp://github.com/johannhartmann/
  • 80. Vegetable Test DrivenInfrastructure Kontinuierliche Entiwcklung - und dann? I Mayflower GmbH I 28. Oktober 2010 I 87
  • 81. actually a behavior driven design toolused for test driven infrastructurecucumber-puppetchef-cucumber
  • 82. Feature: Manualsearch In order to find an article As an developer I want to use the search function Scenario: Search for bdd and check resulting page Given I go to "http://it-republik.de/php/" When I fill in "search_itr" with "bdd" And I click "search2" Then I should see "Suche"
  • 83. Given /^I go to "([^"]*)"$/ do |url| visit urlendWhen /^I fill in "([^"]*)" with "([^"]*)"$/ do |field, value| fill_in field, :with => valueendWhen /^I click "([^"]*)"$/ do |button| click_button(button)endThen /^I should see "([^"]*)"$/ do |text| response_body.should include(text)end
  • 84. johann$ cucumberFeature: Manualsearch In order to find an article As an developer I want to use the search function Scenario: Search for bdd and check resulting page # features/search.feature:5 Given I go to "http://it-republik.de/php/" # features/step_definitions/search_steps.rb:1 When I fill in "search_itr" with "bdd" # features/step_definitions/search_steps.rb:5 And I click "search2" # features/step_definitions/search_steps.rb:9 Then I should see "Suche" # features/step_definitions/search_steps.rb:131 scenario (1 passed)4 steps (4 passed)0m1.615s
  • 85. Feature: Install inetd In order to serve the web the httpd service must be installed Scenario: Setup httpd Given a node of class “web“ When I compile the catalog Then package “httpd“ should be “installed“ Then there should be a resource “Service[httpd]“ And the service should have “enable“ set to “true“ And State should be “running“ And the service should require “Package[httpd]“ Kontinuierliche Entiwcklung - und dann? I Mayflower GmbH I 28. Oktober 2010 I 92
  • 86. Hey, i understandOperations - and i‘m working in marketing
  • 87. Unit TestsAcceptance TestsMetricsCoding Style... you get theidea.
  • 88. ... and evenmore ...PackagingInfrastructureUpdatesReleasesDeployment
  • 89. WetwareRefactoring
  • 90. Work together
  • 91. Own together
  • 92. Eat together http://www.flickr.com/photos/vilavelosa/ 3815032524/
  • 93. Plantogether Kontinuierliche Entiwcklung - und dann? I Mayflower GmbH I 28. Oktober 2010 I 103
  • 94. MayflowerDevOpsification
  • 95. Wetware - Mayflower 1-2 Ops per team • Admin & Development tasks • full time working within the team
  • 96. Wetware - MayflowerClose cooperation with central operationsFull root access for any development-infrastructre
  • 97. Wetware - Mayflower1+n Puppet-Master • central company master • Team puppet master per team / project • company master is starting point for the project puppet configuration
  • 98. Wetware - MayflowerExample Setup: • local Developer VM • CI-Deployment-Server in the DMZ • Staging in the private Cloud • Beta happens in the Amazon-Cloud • Production in AWS, too
  • 99. Wetware - Mayflowerlocal GIT- / Gitorious-ServerEucalyptus-Cloud in the DMZ - as Self-Service!
  • 100. Wetware - Mayflower Vagrant for Development Scrum => KanBan Puppet Nagios