Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The Kitchen Cloud How To: Automating Joyent SmartMachines with Chef


Published on

Learning a new OS can be intimidating, especially one with less support in terms of open source Chef cookbooks. At Wanelo we’ve found the rewards of using Chef with Joyent’s SmartOS to be well worth the effort.

SmartOS is an open source fork of Illumos (think Solaris) that runs in the Joyent Public Cloud. Over the last year we’ve grown to love SmartOS as a deployment environment, and with the help of Chef have grown Wanelo’s infrastructure more than ten times in six months to meet the demands our exponential user growth. In the next year, we expect to grow our infrastructure by another factor of ten. On another public cloud, our business growth would have required a significantly larger infrastructure at every step.

In this session I’ll explain why we appreciate SmartOS so much and how you can get started. What’s the terminology? What plugins do you need, and how do you use them? What providers should you learn and where can you find them? I’ll provide bootstrap scripts, basic roles and cookbooks on Github to get people provisioning and using SmartMachines immediately. For larger infrastructures, I’ll walk through some of the dependencies that have made our lives easier, and explain why.

By the end, you should have the code at your fingertips to deploy a Ruby or Rails application to the Joyent Public Cloud, with all of the dependent services up and running.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

The Kitchen Cloud How To: Automating Joyent SmartMachines with Chef

  1. 1. Proprietary andConfidentialAutomating JoyentSmartMachines with ChefChef on SmartOSEric Saxby@sax @ecdysone @sax
  2. 2. Who am I?Proprietary andConfidential■ Application developeroperational experience with manytechnologies, project by project■ BSD/AIX/UbuntuSolaris in 2002, but I was very muchout of my element■ Switched to DevOps-y team 18 months agoMultiple back end services for a large e-commerce site,transitioning to SmartOS■ Now I’m at Wanelo
  3. 3. From a certain point of view...Proprietary andConfidential
  4. 4. What is Wanelo?Proprietary andConfidential■ Wanelo (“Wah-nee-lo” from Want, NeedLove) is a global platform for shopping.
  5. 5. Proprietary andConfidentialMarketing-free shopping across 100s ofthousands of unique stores
  6. 6. Proprietary andConfidentialPersonal feed of products from anystore on the internet
  7. 7. Technology overviewProprietary andConfidential■ MRI Ruby 1.9.3 & Rails 3.2■ PostgreSQL 9.2.4, Solr 3.6■ Joyent Cloud, SmartOSZFS, ARC, raw IO performance, SmartOS, CPU bursting, dTrace■ Circonus, Chef + OpscodeMonitoring, graphing, alerting, automation■ Amazon S3 + Fastly CDN■ NewRelic, statsd, Graphite, nagios
  8. 8. What’s SmartOS?Proprietary andConfidential■ Illumos branch optimized for cloudcomputing■ Developed by Joyent for their publiccloud
  9. 9. What’s Illumos?Proprietary andConfidential■ It’s what OpenSolaris became after Oraclekilled the project■ Umbrella for various distributions, eachcommitted to pushing their improvementsupstream■
  10. 10. What does SmartOS lookProprietary andConfidential■ Compute Node — physical server■ Global Zone — host OS (SmartOS)■ Non-Global Zone — like a virtual machine, withnative system calls (no fake hardware layer)■ Very secure■ Can run KVM for guest OS (Ubuntu, Centos)
  11. 11. How is it deployed?Proprietary andConfidential■ Can manage from global zone (imgadm,zoneadm)■ Tools provide APIs■ Smart Data Center (Joyent’s tools, can be licensed)■ Project FIFO (SDC API in free package)■ Joyent Public Cloud■ Many compute nodes working in a cluster,PXE booted from a head node
  12. 12. ■ Service Management Facility (SMF)If init.d and monit and god were one thing, andactually awesomeWhy should I care?Proprietary andConfidential■ Visibility toolsdtrace, kstat, snoop, truss■ ZFSFile system built for speed and data integrity■ Application LatencyZones are OS virtualization, so fasterProcesses are scheduled in global zone kernel,not in a hardware virtualization layer
  13. 13. ■ # cores, RAM required =~# processesLower latency == less costProprietary andConfidential■ # processes required =~requests/second of site■ Requests/second of single process =~request latency$$$
  14. 14. On to Chef!Proprietary andConfidential
  15. 15. TerminologyProprietary andConfidential■ Image / Dataset — OS at a particular version,snapshotted at base state■ Flavor / Package— RAM, CPU shares■ API URL — Each data center has its own URL■ Server ID / Zonename — Each zone gets aUUID
  16. 16. knife-joyentProprietary andConfidential
  17. 17. Installation/ConfigurationProprietary andConfidential■ Update knife.rb■ Add to Gemfileknife[:joyent_username] = saxknife[:joyent_keyname] = EricSaxbyknife[:joyent_keyfile] = "#{ENV[HOME]}/.ssh/id_rsa"knife[:joyent_api_url] =■ Add first public key in cloud APIhttps://my.joyentcloud.comgem knife-joyent
  18. 18. Managing keysProprietary andConfidential■ No role based access, but at least you canmake each user upload their own keyknife joyent key add -f ~/.ssh/id_rsa -k KeyNameknife joyent key delete KeyName■ Passphrase protected keys are annoyingEach API request includes data signed with the privatekey. Ruby does not have a good way of signing privatekeys with ssh-agent.
  19. 19. Creating servers!Proprietary andConfidential■ See what images are availableknife joyent image listcf7e2f40-9276-11e2-af9a-0bad2233fb0b base64 1.9.1 smartosf4bc70ca-5e2c-11e1-8380-fb28785857cb smartosplus64 3.1.0 smartosda144ada-a558-11e2-8762-538b60994628 ubuntu-12.04 2.4.1 linux■ base / base64 — minimal install, you add whatyou need■ smartosplus — many more things pre-installed, but can get in the way13328c9a-9173-11e2-a9a5-2ff43d306c21 ws2008ent-r2-sp1 2.0.2 windows
  20. 20. Creating servers!Proprietary andConfidential■ See what flavors are availableknife joyent flavor listName RAM Disk SwapExtra Small 512 MB 0 GB 15 GB 1 GBSmall 1GB 1 GB 30 GB 2 GBMedium 2GB 2 GB 60 GB 4 GBMedium 4GB 4 GB 120 GB 8 GBLarge 8GB 8 GB 240 GB 16 GBLarge 16GB 16 GB 480 GB 32 GB■ Custom networking can be done in a customflavor (ie public or private VLAN, routes)
  21. 21. Creating servers already!Proprietary andConfidentialknife joyent server create--image cf7e2f40-9276-11e2-af9a-0bad2233fb0b--flavor Medium 2GB-N environment-d distro-r run_list■ No Omnibus, so you have to provide your owndistro bootstrap template
  22. 22. knife joyent server listSee whats there...Proprietary andConfidentiala597a3a7-3fdf-481f-af08-e7c1e0ae7dca running smartmachinesdc:sdc:base64:1.8.1 8 GB 240 GB5c066e6e-8af2-4d4f-a81e-c8e2691ae8a0 running smartmachinesdc:sdc:base64:1.8.1 8 GB 240 GBb3370d52-3bed-462e-857a-e17eba15ab06 running smartmachinesdc:sdc:base64:1.8.1 8 GB 240 GB■ ID / zonename■ Name■ Run state■ Type■ Image■ IP addresses■ RAM■ Disk
  23. 23. Other managementProprietary andConfidentialknife joyent server delete <server_id>knife joyent server start <server_id>knife joyent server stop <server_id>knife joyent server reboot <server_id>knife joyent server resize <server_id> -f <flavor>knife joyent snapshot create <server_id> <snapshot_name>■ Snapshots are full ZFS snapshotsCopy-on-write snapshot of local file system.Each snapshot is locally mounted in zone at/checkpoints
  24. 24. So now you have asmartmachine...Proprietary andConfidential
  25. 25. Whats different?Proprietary andConfidential■ Things you expect in /usr/local are in /opt/local■ For historical reasons■ If youre used to Linux, this can be annoying■ Joyent is working on a more Linux friendly image■ For now, add /opt/local/bin to PATH■ Many configs are in /opt/local/etc instead of /etc■ Some utilities are different■ This is not the grep youre looking for....■ Symlink your "correct" version into /opt/local/bin■ Add /opt/local/lib to CFLAGS and LDFLAGS
  26. 26. Caveats?Proprietary andConfidential■ Zones inside of zones inside of...■ Vagrant does not currently work with SmartOS■ VirtualBox only works in Bridged network mode■ Local integration tests do not work
  27. 27. Where are all the things?Proprietary andConfidential■ Services■ svcs -a■ svcadm < enable | disable | clear > service■ Packages■ pkgin search packagename■ pkgin -y install packagename
  28. 28. Public vs. Private IPProprietary andConfidential■ ipaddr_extensions gem■ Adds privateaddress attribute to ohai■ Useful to add this to bootstrap■ Smartmachines may have a public IP and aprivate IP■ Recipes can be configured to use ipaddress orprivateaddress
  29. 29. System preparationProprietary andConfidential■ smartos cookbook■■ fixes chef providers■ smartmachine_functions■ links nicer utils into /opt/local/bin■■ fixes chef providers■ provides access to Joyent metadata APIor
  30. 30. Useful LWRPsProprietary andConfidential
  31. 31. SMFProprietary andConfidential■■ Chef knows how to use SMF, not how to configure it■ Uses nokogiri, which requires libxsltsmf postgres douser postgresgroup postgresproject postgresstart_command startstop_command stopworking_directory /var/pgsql/dataenvironment PATH => /opt/postgres/binend
  32. 32. SMF (cntd)Proprietary andConfidentialsmf postgres douser postgresgroup postgresproject postgresstart_command startstop_command stopstop_timeout 120restart_command restartrefresh_command reloadworking_directory /var/pgsql/dataenvironment PATH => /opt/postgres/binendservice postgres dosupports :status => true,:restart => true, :reload => trueend
  33. 33. Resource Control /Proprietary andConfidential■■ configure max file descriptors, shared memory, etc■ Bunch up master/worker processes to view inprstat -Jresource_control_project "postgres" docomment "PostgreSQL 9.2"users "postgres"project_limits "max-shm-memory" => 12000000,"max-lwps" => 6process_limits "max-file-descriptor" => {"value" => 32768, "deny" => true}action :createend
  34. 34. Role Based Access ControlProprietary andConfidential■■ Allows delegation of authority without sudo■ Implementation currently too simple, only useful for SMFdelegationrbac solr douser waneloaction :add_management_permissionsend
  35. 35. Contributing to cookbooksProprietary andConfidential■ ~95% just require SMF, correct package names■ ~5% of those need a special init script■ The rest usually require custom compile`postgres -D /path/to/data` not granular enough`pg_ctl -D /path/to/data < start | stop | reload | refresh >`--with-libraries=/opt/local/lib--with-includes=/opt/local/includeLDFLAGS=-R/opt/local/lib -L/opt/local/lib
  36. 36. Comments? Questions? Findme. andConfidential@sax @ecdysone @sax