Practical introduction to dev ops with chef

12,486 views

Published on

0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
12,486
On SlideShare
0
From Embeds
0
Number of Embeds
8,942
Actions
Shares
0
Downloads
53
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • Agile Software Studio & Coaching\nWeb, Mobile, Embedded Desktop\nCommunity Meetups, GiveCamp\n
  • What is DevOps?\n\n
  • traditionally separate teams\nDevelopment - creates software\nOperations - puts it intro production and keeps it running\nfigurative (sometimes literal) wall between \n
  • Development - creates software\nOperations - puts it intro production and keeps it running\n
  • portmanteau word capturing the idea of integrated development AND operations\nmates with Agile/Lean principles\nfrequent releases, shorter cycle times\nautomation\n“holistic” view of development and production\n
  • \n
  • \n
  • To make this discussion concrete and practical, we’ll be using a small web application for the examples. \n
  • using Apache to serve our static content and mod_wsgi to serve our Python Django app\nthere are lots of different deployment options here (nginx, gunicorn, etc.); this is pretty common\nusing postgresql for the Django persistence\n
  • \n
  • in our configuration runs as a daemon under Apache\n
  • we’re using postgresql\n
  • \n
  • But often necessary\n
  • \n
  • \n
  • \n
  • \n
  • package name == package version\ndjango - the web framework we’re using\npsycopg2 is a Python client interface for postgresql\n\n
  • \n
  • \n
  • editing lots of text files...\n
  • \n
  • Now we’re almost ready to actually do something\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • very simple Dept/Course/Student/Instructor Roster web application\n
  • \n
  • can always use your instructions to create a master image that can be “frozen” or bundled\n
  • \n
  • \n
  • actually, a great way to start\n
  • \n
  • \n
  • install the needed packages\nthe script is useful and goes a long way to documenting process\n
  • \n
  • \n
  • sure, you do some horrible shell scripts with sed and awk and such, but really?\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • we’d like to declare the state of the resource, like a database username, password, and database access, and let someone else figure out how to “make it so!”\n\nthat’s essentially what Chef is\n
  • very little Ruby experience is required\nothers to be aware of are: cfengine, puppet, and now salt\n
  • Chef Server (opscode or your own)\n“knife” is a Ruby command-line program\nchef-client runs on your “cluster”\n
  • really, it looks like this...\nCouchDB\nsolr\nRabbitMQ\n
  • \n
  • \n
  • \n
  • \n
  • an instance of a particular machine\nMostly Linux centric, but some Windows support\n
  • to name just a few\n
  • \n
  • resource type of “package”\n
  • resource name of “libpq-dev”\n
  • Chef calls yum, apt-get, etc. as appropriate for the platform\n
  • \n
  • \n
  • \n
  • “executed” top to bottom\n
  • \n
  • Yes, they have Django support, but we wouldn’t learn as much from using that\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Test Driven Infrastructure book by O’Reilly\n\n
  • \n
  • Today’s talk intentionally not a tutorial - lots of fine Chef tutorials out there\n
  • Practical introduction to dev ops with chef

    1. 1. A PRACTICALINTRODUCTION TO DEVOPS WITH CHEF Nick Barendt - LeanDog CWRU ACM Link-State 2012
    2. 2. WHO AM I• CWRU Alumnus - Wayward Electrical Engineer• LeanDog - Director of Labs• CWRU EECS Adjunct Faculty• Embedded Linux (Bootloaders, Kernel Drivers, Daemons)• Web/Cloud (AWS, Python Django)
    3. 3. LEANDOG http://leandog.comhttp://leandog.com/dogtreats/ Copyright 2012, Nick Barendt
    4. 4. DEVOPS
    5. 5. DEVELOPMENT + OPERATIONS
    6. 6. DEVELOPMENT + OPERATIONS
    7. 7. DEVOPS
    8. 8. • Agile/Lean Development• Frequent Releases (Minimize WIP)• Continuous Integration → Continuous Delivery• Reduced Stress: Release is Non-Event; TDD/BDD• Holistic - code/infrastructure are inextricably linked
    9. 9. TOPICS FOR TODAY• Configuration Management• Deployment Automation
    10. 10. SAMPLE APPLICATION• Ubuntu 12.04 LTS• Apache2• Python Django• PostgreSQL
    11. 11. Apache2static content mod_wsgi Postgresql
    12. 12. STATIC CONTENT• aka “files”• HTML, CSS, JavaScript, images, etc.
    13. 13. MOD_WSGI• WSGI - Web Server Gateway Interface • Python Universal HTTP interface• mod_wsgi Apache plugin• executes Python/Django code in response to HTTP requests
    14. 14. POSTGRESQL• Relational Database Management System• Django’s Object Relational Mapper (ORM) supports: • Postgresql • MySQL • SQLite • Oracle
    15. 15. HOW DO WE DEPLOY?
    16. 16. PART 1: THE HARD WAY
    17. 17. INSTALL DISTRO PACKAGES
    18. 18. CREATE THE VIRTUALENV
    19. 19. Virtualenv• Python “environment” isolated from rest of system• Analogous to Ruby’s rvm• Python Interpreter + 3rd party libraries• pip - Python Package Manager• requirements.txt - 3rd party library manifest, with versions
    20. 20. requirements.txt django==1.4.2 psycopg2==2.4.5
    21. 21. CONFIGURE POSTGRESQL
    22. 22. • Configure Postgresql to listen on TCP Port 5432• Configure Postgresql Authentication• Create User “roster”• Create Database “roster”
    23. 23. SOURCE!
    24. 24. THAT WASN’T PRETTY
    25. 25. OR EASILY REPEATABLE
    26. 26. PAUSE
    27. 27. WHAT DOES THE APP DO?
    28. 28. BACK TO THE STORY
    29. 29. CONFIGURATION• Repeatable and works on a minimal, standard install image• TIP: Use virtual machines for development/testing: • Snapshot Virtual Machine • Test Configuration Process • Verify and/or Fix Configuration Process • Rollback
    30. 30. SNAPSHOT TREE VMware Fusion Copyright 2012, Nick Barendt
    31. 31. HOW TO IMPROVE?
    32. 32. TAKE A FEW NOTES?
    33. 33. WE CAN DO SLIGHTLY BETTER
    34. 34. SCRIPTS!
    35. 35. “prepare_ubuntu”#!/bin/bash# must be run as root!apt-get install -y apache2 libapache2-mod-wsgiapt-get install -y python-dev python-virtualenvapt-get install -y gitapt-get install -y postgresql-9.1 libpq-devapt-get install -y vim
    36. 36. “create_virtualenv”#!/bin/bashENV_DIR=envrm -rf $ENV_DIRvirtualenv $ENV_DIR$ENV_DIR/bin/pip install -r requirements.txt
    37. 37. HOW ABOUT CONFIG FILES...
    38. 38. YEAH....ABOUT THAT
    39. 39. FINALLY, WE CAN TALK ABOUT CHEF
    40. 40. PART 2: CHEF!
    41. 41. LET’S WAIT JUST A LITTLE BIT LONGER
    42. 42. IN AN IDEAL WORLD, WHAT WOULD WE WANT?
    43. 43. Configuration IS Source• Configuration in Version Control• Text Files• Diff ’able• Tie Infrastructure Configuration and Production Releases
    44. 44. A DECLARATIVE DSLdatabase_user roster do    password spartans    database_name rosterend
    45. 45. CHEF• Opscode - Private company• 2009• Open Source - Apache License 2.0• Ruby DSL for configuration “recipes” or “cookbooks”
    46. 46. CHEF ARCHITECTURE Chef Server chef client knife
    47. 47. CHEF ARCHITECTURE
    48. 48. BUT I JUST WANT TOMANAGE A FEW MACHINES...
    49. 49. CHEF-SOLO
    50. 50. chef-solo• Essentially chef-client, but operates without chef server• Only uses local resources• Simpler to start learning• “Recipes” are inherently the same as “full blown” chef
    51. 51. CHEF CONCEPTS• NODES - “servers”, “computers”, “machines”• RESOURCES - abstraction of the “thing” you’re configuring• RECIPES - collection of Resource descriptions• COOKBOOKS - collections of Recipes and Attributes
    52. 52. NODES• Ubuntu • Scientific Linux• Debian • OS X• Red Hat • Windows• Fedora• CentOS• Oracle Copyright 2012, Nick Barendt
    53. 53. RESOURCES• File • User and Group• Template • Git• Execute • Subversion• Cron• Link• Directory Copyright 2012, Nick Barendt
    54. 54. RECIPESpackage "libpq-dev"python_virtualenv node[:roster]["VIRTUALENV_DIR"] do    interpreter "python2.7"    action :createendpip_cmd = File.join(node[:roster]["VIRTUALENV_DIR"], "bin", "pip")requirements_txt = File.join(node[:roster]["DEPLOY_DIR"], "requirements.txt")execute "#{pip_cmd} install -r #{requirements_txt}" doend
    55. 55. RECIPESpackage "libpq-dev"python_virtualenv node[:roster]["VIRTUALENV_DIR"] do    interpreter "python2.7"    action :createendpip_cmd = File.join(node[:roster]["VIRTUALENV_DIR"], "bin", "pip")requirements_txt = File.join(node[:roster]["DEPLOY_DIR"], "requirements.txt")execute "#{pip_cmd} install -r #{requirements_txt}" doend
    56. 56. RECIPESpackage "libpq-dev"python_virtualenv node[:roster]["VIRTUALENV_DIR"] do    interpreter "python2.7"    action :createendpip_cmd = File.join(node[:roster]["VIRTUALENV_DIR"], "bin", "pip")requirements_txt = File.join(node[:roster]["DEPLOY_DIR"], "requirements.txt")execute "#{pip_cmd} install -r #{requirements_txt}" doend
    57. 57. RECIPES Install Distro Packagepackage "libpq-dev"python_virtualenv node[:roster]["VIRTUALENV_DIR"] do    interpreter "python2.7"    action :createendpip_cmd = File.join(node[:roster]["VIRTUALENV_DIR"], "bin", "pip")requirements_txt = File.join(node[:roster]["DEPLOY_DIR"], "requirements.txt")execute "#{pip_cmd} install -r #{requirements_txt}" doend
    58. 58. RECIPES create virtualenv @package "libpq-dev" specified pathpython_virtualenv node[:roster]["VIRTUALENV_DIR"] do    interpreter "python2.7"    action :createendpip_cmd = File.join(node[:roster]["VIRTUALENV_DIR"], "bin", "pip")requirements_txt = File.join(node[:roster]["DEPLOY_DIR"], "requirements.txt")execute "#{pip_cmd} install -r #{requirements_txt}" doend
    59. 59. RECIPES some variablespackage "libpq-dev" and Ruby codepython_virtualenv node[:roster]["VIRTUALENV_DIR"] do    interpreter "python2.7"    action :createendpip_cmd = File.join(node[:roster]["VIRTUALENV_DIR"], "bin", "pip")requirements_txt = File.join(node[:roster]["DEPLOY_DIR"], "requirements.txt")execute "#{pip_cmd} install -r #{requirements_txt}" doend
    60. 60. RECIPESpackage "libpq-dev"python_virtualenv node[:roster]["VIRTUALENV_DIR"] do    interpreter "python2.7"    action :createendpip_cmd = File.join(node[:roster]["VIRTUALENV_DIR"], "bin", "pip")requirements_txt = File.join(node[:roster]["DEPLOY_DIR"], "requirements.txt")execute "#{pip_cmd} install -r #{requirements_txt}" doend execute command-line
    61. 61. RECIPESpackage "libpq-dev"python_virtualenv node[:roster]["VIRTUALENV_DIR"] do    interpreter "python2.7"    action :createendpip_cmd = File.join(node[:roster]["VIRTUALENV_DIR"], "bin", "pip")requirements_txt = File.join(node[:roster]["DEPLOY_DIR"], "requirements.txt")execute "#{pip_cmd} install -r #{requirements_txt}" doend
    62. 62. COOKBOOKS• Collections of • Attributes • Recipes • Templates Copyright 2012, Nick Barendt
    63. 63. TONS OF COOKBOOKS HTTPS://GITHUB.COM/OPSCODE-COOKBOOKS• Java • Erlang• Python • Django• Nagios • Rsyslog• openssh • Nginx• Apache2 • Perl• Postfix • Many, Many More Copyright 2012, Nick Barendt
    64. 64. CONVERGENCE• Chef attempts to converge a machine’s configuration to the desired state• Withtemporary errors, repeated runs will drive towards convergence• Resources are (generally) idempotent• Robust• Try, try again
    65. 65. HOW DOES THIS ACTUALLY WORK?
    66. 66. • Merge Defaults and Node Attributes• Recipe Compilation• Recipe Execution
    67. 67. sudo chef-solo -c solo.rb -j solo-configs/standalone.jsonsolo.rb# minimal configuration file for chef-solocookbook_path File.expand_path File.dirname(__FILE__) + "/cookbooks"standalone.json{ "roster": { "app_server": { "load_default_data": true } }, "postgresql": { "password":{ "postgres": "roster_123" } }, "run_list": [ "recipe[roster::database]", "recipe[roster::app_server]" ]}
    68. 68. app_server.rbinclude_recipe "apache2"include_recipe "apache2::mod_wsgi"include_recipe "python"node[:roster]["DEPLOY_DIR"] = File.expand_path( File.join( File.expand_path( File.dirname(__FILE__) ), "..","..","..",".."))node[:roster]["VIRTUALENV_DIR"] = File.expand_path( File.join(node[:roster]["DEPLOY_DIR"], "env"))# disable default, regardless of how it is symlinkedapache_site "default" do    enable falseendapache_site "000-default" do    enable falseend# we need a few packagespackage "libpq-dev"python_virtualenv node[:roster]["VIRTUALENV_DIR"] do    interpreter "python2.7"    action :createend
    69. 69. continuedpip_cmd = File.join(node[:roster]["VIRTUALENV_DIR"], "bin","pip")requirements_txt = File.join(node[:roster]["DEPLOY_DIR"],"requirements.txt")execute "#{pip_cmd} install -r #{requirements_txt}" doendweb_app "roster" do    template "roster_apache.conf.erb"enddjango_admin_cmd = File.join(node[:roster]["DEPLOY_DIR"],"cwru_acm", "manage.py")python_exec = File.join(node[:roster]["VIRTUALENV_DIR"], "bin","python")execute "#{python_exec} #{django_admin_cmd} syncdb --noinput"doendexecute "#{python_exec} #{django_admin_cmd} collectstatic --noinput" doendexecute "#{python_exec} #{django_admin_cmd} loaddata rostersome_initial_data" do    only_if {node[:roster][:app_server][:load_default_data]}end
    70. 70. database.rbinclude_recipe "postgresql::client"include_recipe "postgresql::server"r = package "ruby-pg" do  package_name "libpgsql-ruby"  action :nothingendr.run_action(:upgrade)# Snippet from opscode to reload gemsrequire rubygemsGem.clear_pathsrequire "pg"include_recipe "database"postgresql_connection_info = {:host => "127.0.0.1", :port => 5432, :username =>postgres, :password => node[postgresql][password][postgres]}postgresql_database "roster" do    connection postgresql_connection_info    action :createenddatabase_user roster do    connection postgresql_connection_info    password spartans    database_name roster    provider Chef::Provider::Database::PostgresqlUser    action :createenddatabase_user roster do    connection postgresql_connection_info    password spartans    provider Chef::Provider::Database::PostgresqlUser    action :grantend
    71. 71. But Wait, There’s More• Parametric Configs. (environments): Dev, Staging, Production• Roles - collect recipes based on how server is used (e.g., Apache2 + Memcache for “front-end”)• Test Driven Infrastructure - Configuration is just code!• Bootstrapping in AWS EC2 and other Clouds• Chef Server, Knife
    72. 72. Questions?nick@barendt.com Twitter: @nickbarendt Code From Today: https://github.com/nbarendt/cwru_acm_2012
    73. 73. FOR MORE INFO• http://wiki.opscode.com• Fast Start Linux Guide: http://wiki.opscode.com/display/chef/ Fast+Start+Guide• AWSEC2 Bootstrapping: http://wiki.opscode.com/display/chef/ EC2+Bootstrap+Fast+Start+Guide• O’Reilly: Test-Driven Infrastructure with Chef

    ×