Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Chef on SmartOS

1,665 views

Published on

Published in: Technology
  • Be the first to comment

Chef on SmartOS

  1. 1. Proprietary andChef on SmartOSEric Saxby@sax @ecdysone @sax
  2. 2. Who am I?Proprietary and■ Application developeroperational experience with manytechnologies, project by project■ BSD/AIX/UbuntuSolaris in 2002, but I was very muchout of my element■ Switched to DevOps-y team 18 months agoMultiple back end services for a large e-commerce site,transitioning to SmartOS■ Now I’m at Wanelo
  3. 3. From a certain point of view...Proprietary and
  4. 4. From a certain point of view...Proprietary and
  5. 5. What is Wanelo?Proprietary and■ Wanelo (“Wah-nee-lo” from Want, NeedLove) is a global platform for shopping.
  6. 6. Proprietary andMarketing-free shopping across 100s ofthousands of unique stores
  7. 7. Proprietary andPersonal feed of products from any store onthe internet
  8. 8. Technology overviewProprietary and■ MRI Ruby 1.9.3 & Rails 3.2■ PostgreSQL 9.2.4, Solr 3.6■ Joyent Cloud, SmartOSZFS, ARC, raw IO performance, SmartOS, CPU bursting, dTrace■ Circonus, Chef + OpscodeMonitoring, graphing, alerting, automation■ Amazon S3 + Fastly CDN■ NewRelic, statsd, Graphite, nagios
  9. 9. What’s SmartOS?Proprietary and■ Illumos branch optimized for cloudcomputing■ Developed by Joyent for their publiccloud
  10. 10. What’s Illumos?Proprietary and■ It’s what OpenSolaris became after Oraclekilled the project■ Umbrella for various distributions, eachcommitted to pushing their improvementsupstream■ http://wiki.illumos.org/display/illumos/About+illumos
  11. 11. What does SmartOS look like?Proprietary and■ Compute Node — physical server■ Global Zone — host OS (SmartOS)■ Non-Global Zone — like a virtual machine, withnative system calls (no fake hardware layer)■ Very secure■ Can run KVM for guest OS (Ubuntu, Centos)
  12. 12. How is it deployed?Proprietary and■ Can manage from global zone (imgadm,zoneadm)■ Tools provide APIs■ Smart Data Center (Joyent’s tools, can be licensed)■ Project FIFO (SDC API in free package)■ Joyent Public Cloud■ Many compute nodes working in a cluster,PXE booted from a head node
  13. 13. Why should I care?Proprietary and
  14. 14. Why should I care?Proprietary and■ ZFSFile system built for speed and data integrity
  15. 15. Why should I care?Proprietary and■ Visibility toolsdtrace, kstat, snoop, truss■ ZFSFile system built for speed and data integrity
  16. 16. ■ Service Management Facility (SMF)If init.d and monit and god were one thing, andactually awesomeWhy should I care?Proprietary and■ Visibility toolsdtrace, kstat, snoop, truss■ ZFSFile system built for speed and data integrity
  17. 17. ■ Service Management Facility (SMF)If init.d and monit and god were one thing, andactually awesomeWhy should I care?Proprietary and■ Visibility toolsdtrace, kstat, snoop, truss■ ZFSFile system built for speed and data integrity■ Application LatencyZones are OS virtualization, so fasterProcesses are scheduled in global zone kernel,not in a hardware virtualization layer
  18. 18. ■ Service Management Facility (SMF)If init.d and monit and god were one thing, andactually awesomeWhy should I care?Proprietary and■ Visibility toolsdtrace, kstat, snoop, truss■ ZFSFile system built for speed and data integrity■ Application LatencyZones are OS virtualization, so fasterProcesses are scheduled in global zone kernel,not in a hardware virtualization layer
  19. 19. ■ Service Management Facility (SMF)If init.d and monit and god were one thing, andactually awesomeWhy should I care?Proprietary and■ Visibility toolsdtrace, kstat, snoop, truss■ ZFSFile system built for speed and data integrity■ Application LatencyZones are OS virtualization, so fasterProcesses are scheduled in global zone kernel,not in a hardware virtualization layer
  20. 20. Lower latency == less costProprietary and
  21. 21. Lower latency == less costProprietary and■ Requests/second of single process =~request latency
  22. 22. Lower latency == less costProprietary and■ # processes required =~requests/second of site■ Requests/second of single process =~request latency
  23. 23. ■ # cores, RAM required =~# processesLower latency == less costProprietary and■ # processes required =~requests/second of site■ Requests/second of single process =~request latency
  24. 24. ■ # cores, RAM required =~# processesLower latency == less costProprietary and■ # processes required =~requests/second of site■ Requests/second of single process =~request latency$$$
  25. 25. On to Chef!Proprietary and
  26. 26. TerminologyProprietary and■ Image / Dataset — OS at a particular version,snapshotted at base state■ Flavor / Package— RAM, CPU shares■ API URL — Each data center has its own URL■ Server ID / Zonename — Each zone gets aUUID
  27. 27. knife-joyentProprietary and
  28. 28. Installation/ConfigurationProprietary and■ Update knife.rb■ Add to Gemfileknife[:joyent_username] = saxknife[:joyent_keyname] = EricSaxbyknife[:joyent_keyfile] = "#{ENV[HOME]}/.ssh/id_rsa"knife[:joyent_api_url] = https://us-sw-1.api.joyentcloud.com/■ Add first public key in cloud APIhttps://my.joyentcloud.comgem knife-joyent
  29. 29. Managing keysProprietary and■ No role based access, but at least you canmake each user upload their own keyknife joyent key add -f ~/.ssh/id_rsa -k KeyNameknife joyent key delete KeyName■ Passphrase protected keys are annoyingEach API request includes data signed with the privatekey. Ruby does not have a good way of signing privatekeys with ssh-agent.
  30. 30. Creating servers!Proprietary and■ See what images are availableknife joyent image listcf7e2f40-9276-11e2-af9a-0bad2233fb0b base64 1.9.1 smartosf4bc70ca-5e2c-11e1-8380-fb28785857cb smartosplus64 3.1.0 smartosda144ada-a558-11e2-8762-538b60994628 ubuntu-12.04 2.4.1 linux■ base / base64 — minimal install, you add whatyou need■ smartosplus — many more things pre-installed, but can get in the way13328c9a-9173-11e2-a9a5-2ff43d306c21 ws2008ent-r2-sp1 2.0.2 windows
  31. 31. Creating servers!Proprietary and■ See what flavors are availableknife joyent flavor listName RAM Disk SwapExtra Small 512 MB 0 GB 15 GB 1 GBSmall 1GB 1 GB 30 GB 2 GBMedium 2GB 2 GB 60 GB 4 GBMedium 4GB 4 GB 120 GB 8 GBLarge 8GB 8 GB 240 GB 16 GBLarge 16GB 16 GB 480 GB 32 GB■ Custom networking can be done in a customflavor (ie public or private VLAN, routes)
  32. 32. Creating servers already!Proprietary andknife joyent server create--image cf7e2f40-9276-11e2-af9a-0bad2233fb0b--flavor Medium 2GB-N server.domain.com-E environment-d distro-r run_list■ No Omnibus, so you have to provide your owndistro bootstrap templatehttps://gist.github.com/sax/5457464
  33. 33. knife joyent server listSee whats there...Proprietary anda597a3a7-3fdf-481f-af08-e7c1e0ae7dca admin.prod running smartmachinesdc:sdc:base64:1.8.1 8.19.1.1 10.100.1.1 8 GB 240 GB5c066e6e-8af2-4d4f-a81e-c8e2691ae8a0 demo.dev running smartmachinesdc:sdc:base64:1.8.1 10.12.1.1 165.225.1.1 8 GB 240 GBb3370d52-3bed-462e-857a-e17eba15ab06 app010.c1.prod running smartmachinesdc:sdc:base64:1.8.1 10.100.1.2 165.225.1.2 8 GB 240 GB■ ID / zonename■ Name■ Run state■ Type■ Image■ IP addresses■ RAM■ Disk
  34. 34. Other managementProprietary andknife joyent server delete <server_id>knife joyent server start <server_id>knife joyent server stop <server_id>knife joyent server reboot <server_id>knife joyent server resize <server_id> -f <flavor>knife joyent snapshot create <server_id> <snapshot_name>■ Snapshots are full ZFS snapshotsCopy-on-write snapshot of local file system.Each snapshot is locally mounted in zone at/checkpoints
  35. 35. So now you have asmartmachine...Proprietary and
  36. 36. Whats different?Proprietary and■ Things you expect in /usr/local are in /opt/local■ For historical reasons■ If youre used to Linux, this can be annoying■ Joyent is working on a more Linux friendly image■ For now, add /opt/local/bin to PATH■ Many configs are in /opt/local/etc instead of /etc■ Some utilities are different■ This is not the grep youre looking for....■ Symlink your "correct" version into /opt/local/bin■ Add /opt/local/lib to CFLAGS and LDFLAGS
  37. 37. Caveats?Proprietary and■ Zones inside of zones inside of...■ Vagrant does not currently work with SmartOS■ VirtualBox only works in Bridged network mode■ Local integration tests do not work
  38. 38. Where are all the things?Proprietary and■ Services■ svcs -a■ svcadm < enable | disable | clear > service■ Packages■ pkgin search packagename■ pkgin -y install packagename
  39. 39. Public vs. Private IPProprietary and■ ipaddr_extensions gem■ Adds privateaddress attribute to ohai■ Useful to add this to bootstrap■ Smartmachines may have a public IP and aprivate IP■ Recipes can be configured to use ipaddress orprivateaddress
  40. 40. System preparationProprietary and■ smartos cookbook■ https://github.com/modcloth-cookbooks/smartos■ fixes chef providers■ smartmachine_functions■ links nicer utils into /opt/local/bin■ https://github.com/higanworks-cookbooks/smartmachine_functions■ fixes chef providers■ provides access to Joyent metadata APIor
  41. 41. Useful LWRPsProprietary and
  42. 42. SMFProprietary and■ https://github.com/modcloth-cookbooks/smf■ Chef knows how to use SMF, not how to configure it■ Uses nokogiri, which requires libxml2smf postgres douser postgresgroup postgresproject postgresstart_command postgres-service.sh startstop_command postgres-service.sh stopworking_directory /var/pgsql/dataenvironment PATH => /opt/postgres/binend
  43. 43. SMF (cntd)Proprietary andsmf postgres douser postgresgroup postgresproject postgresstart_command postgres-service.sh startstop_command postgres-service.sh stopstop_timeout 120restart_command postgres-service.sh restartrefresh_command postgres-service.sh reloadworking_directory /var/pgsql/dataenvironment PATH => /opt/postgres/binendservice postgres dosupports :status => true,:restart => true, :reload => trueend
  44. 44. SMF (cntd)Proprietary andsmf postgres douser postgresgroup postgresproject postgresstart_command postgres-service.sh startstop_command postgres-service.sh stopstop_timeout 120restart_command postgres-service.sh restartrefresh_command postgres-service.sh reloadworking_directory /var/pgsql/dataenvironment PATH => /opt/postgres/binendservice postgres dosupports :status => true,:restart => true, :reload => trueend
  45. 45. Resource Control / ProjectsProprietary and■ https://github.com/wanelo-chef/resource-control■ configure max file descriptors, shared memory, etc■ Bunch up master/worker processes to view inprstat -Jresource_control_project "postgres" docomment "PostgreSQL 9.2"users "postgres"project_limits "max-shm-memory" => 12000000,"max-lwps" => 6process_limits "max-file-descriptor" => {"value" => 32768, "deny" => true}action :createend
  46. 46. Role Based Access ControlProprietary and■ https://github.com/modcloth-cookbooks/rbac■ Allows delegation of authority without sudo■ Implementation currently too simple, only useful for SMFdelegationrbac solr douser waneloaction :add_management_permissionsend
  47. 47. Contributing to cookbooksProprietary and■ ~95% just require SMF, correct package names■ ~5% of those need a special init script■ The rest usually require custom compile`postgres -D /path/to/data` not granular enough`pg_ctl -D /path/to/data < start | stop | reload | refresh >`--with-libraries=/opt/local/lib--with-includes=/opt/local/includeLDFLAGS=-R/opt/local/lib -L/opt/local/lib
  48. 48. Comments? Questions? Find me.https://github.com/wanelohttps://github.com/wanelo-chefhttps://github.com/wanelo-chef/smartos-chef-repoProprietary and@sax @ecdysone @sax

×