Proprietary andChef on SmartOSEric Saxby@sax @ecdysone @sax
Who am I?Proprietary and■ Application developeroperational experience with manytechnologies, project by project■ BSD/AIX/U...
From a certain point of view...Proprietary and
From a certain point of view...Proprietary and
What is Wanelo?Proprietary and■ Wanelo (“Wah-nee-lo” from Want, NeedLove) is a global platform for shopping.
Proprietary andMarketing-free shopping across 100s ofthousands of unique stores
Proprietary andPersonal feed of products from any store onthe internet
Technology overviewProprietary and■ MRI Ruby 1.9.3 & Rails 3.2■ PostgreSQL 9.2.4, Solr 3.6■ Joyent Cloud, SmartOSZFS, ARC,...
What’s SmartOS?Proprietary and■ Illumos branch optimized for cloudcomputing■ Developed by Joyent for their publiccloud
What’s Illumos?Proprietary and■ It’s what OpenSolaris became after Oraclekilled the project■ Umbrella for various distribu...
What does SmartOS look like?Proprietary and■ Compute Node — physical server■ Global Zone — host OS (SmartOS)■ Non-Global Z...
How is it deployed?Proprietary and■ Can manage from global zone (imgadm,zoneadm)■ Tools provide APIs■ Smart Data Center (J...
Why should I care?Proprietary and
Why should I care?Proprietary and■ ZFSFile system built for speed and data integrity
Why should I care?Proprietary and■ Visibility toolsdtrace, kstat, snoop, truss■ ZFSFile system built for speed and data in...
■ Service Management Facility (SMF)If init.d and monit and god were one thing, andactually awesomeWhy should I care?Propri...
■ Service Management Facility (SMF)If init.d and monit and god were one thing, andactually awesomeWhy should I care?Propri...
■ Service Management Facility (SMF)If init.d and monit and god were one thing, andactually awesomeWhy should I care?Propri...
■ Service Management Facility (SMF)If init.d and monit and god were one thing, andactually awesomeWhy should I care?Propri...
Lower latency == less costProprietary and
Lower latency == less costProprietary and■ Requests/second of single process =~request latency
Lower latency == less costProprietary and■ # processes required =~requests/second of site■ Requests/second of single proce...
■ # cores, RAM required =~# processesLower latency == less costProprietary and■ # processes required =~requests/second of ...
■ # cores, RAM required =~# processesLower latency == less costProprietary and■ # processes required =~requests/second of ...
On to Chef!Proprietary and
TerminologyProprietary and■ Image / Dataset — OS at a particular version,snapshotted at base state■ Flavor / Package— RAM,...
knife-joyentProprietary and
Installation/ConfigurationProprietary and■ Update knife.rb■ Add to Gemfileknife[:joyent_username] = saxknife[:joyent_keynam...
Managing keysProprietary and■ No role based access, but at least you canmake each user upload their own keyknife joyent ke...
Creating servers!Proprietary and■ See what images are availableknife joyent image listcf7e2f40-9276-11e2-af9a-0bad2233fb0b...
Creating servers!Proprietary and■ See what flavors are availableknife joyent flavor listName RAM Disk SwapExtra Small 512 MB...
Creating servers already!Proprietary andknife joyent server create--image cf7e2f40-9276-11e2-af9a-0bad2233fb0b--flavor Med...
knife joyent server listSee whats there...Proprietary anda597a3a7-3fdf-481f-af08-e7c1e0ae7dca admin.prod running smartmach...
Other managementProprietary andknife joyent server delete <server_id>knife joyent server start <server_id>knife joyent ser...
So now you have asmartmachine...Proprietary and
Whats different?Proprietary and■ Things you expect in /usr/local are in /opt/local■ For historical reasons■ If youre used ...
Caveats?Proprietary and■ Zones inside of zones inside of...■ Vagrant does not currently work with SmartOS■ VirtualBox only...
Where are all the things?Proprietary and■ Services■ svcs -a■ svcadm < enable | disable | clear > service■ Packages■ pkgin ...
Public vs. Private IPProprietary and■ ipaddr_extensions gem■ Adds privateaddress attribute to ohai■ Useful to add this to ...
System preparationProprietary and■ smartos cookbook■ https://github.com/modcloth-cookbooks/smartos■ fixes chef providers■ s...
Useful LWRPsProprietary and
SMFProprietary and■ https://github.com/modcloth-cookbooks/smf■ Chef knows how to use SMF, not how to configure it■ Uses nok...
SMF (cntd)Proprietary andsmf postgres douser postgresgroup postgresproject postgresstart_command postgres-service.sh start...
SMF (cntd)Proprietary andsmf postgres douser postgresgroup postgresproject postgresstart_command postgres-service.sh start...
Resource Control / ProjectsProprietary and■ https://github.com/wanelo-chef/resource-control■ configure max file descriptors,...
Role Based Access ControlProprietary and■ https://github.com/modcloth-cookbooks/rbac■ Allows delegation of authority witho...
Contributing to cookbooksProprietary and■ ~95% just require SMF, correct package names■ ~5% of those need a special init s...
Comments? Questions? Find me.https://github.com/wanelohttps://github.com/wanelo-chefhttps://github.com/wanelo-chef/smartos...
Upcoming SlideShare
Loading in …5
×

Chef on SmartOS

1,424 views
1,322 views

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,424
On SlideShare
0
From Embeds
0
Number of Embeds
30
Actions
Shares
0
Downloads
20
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Chef on SmartOS

  1. 1. Proprietary andChef on SmartOSEric Saxby@sax @ecdysone @sax
  2. 2. Who am I?Proprietary and■ Application developeroperational experience with manytechnologies, project by project■ BSD/AIX/UbuntuSolaris in 2002, but I was very muchout of my element■ Switched to DevOps-y team 18 months agoMultiple back end services for a large e-commerce site,transitioning to SmartOS■ Now I’m at Wanelo
  3. 3. From a certain point of view...Proprietary and
  4. 4. From a certain point of view...Proprietary and
  5. 5. What is Wanelo?Proprietary and■ Wanelo (“Wah-nee-lo” from Want, NeedLove) is a global platform for shopping.
  6. 6. Proprietary andMarketing-free shopping across 100s ofthousands of unique stores
  7. 7. Proprietary andPersonal feed of products from any store onthe internet
  8. 8. Technology overviewProprietary and■ MRI Ruby 1.9.3 & Rails 3.2■ PostgreSQL 9.2.4, Solr 3.6■ Joyent Cloud, SmartOSZFS, ARC, raw IO performance, SmartOS, CPU bursting, dTrace■ Circonus, Chef + OpscodeMonitoring, graphing, alerting, automation■ Amazon S3 + Fastly CDN■ NewRelic, statsd, Graphite, nagios
  9. 9. What’s SmartOS?Proprietary and■ Illumos branch optimized for cloudcomputing■ Developed by Joyent for their publiccloud
  10. 10. What’s Illumos?Proprietary and■ It’s what OpenSolaris became after Oraclekilled the project■ Umbrella for various distributions, eachcommitted to pushing their improvementsupstream■ http://wiki.illumos.org/display/illumos/About+illumos
  11. 11. What does SmartOS look like?Proprietary and■ Compute Node — physical server■ Global Zone — host OS (SmartOS)■ Non-Global Zone — like a virtual machine, withnative system calls (no fake hardware layer)■ Very secure■ Can run KVM for guest OS (Ubuntu, Centos)
  12. 12. How is it deployed?Proprietary and■ Can manage from global zone (imgadm,zoneadm)■ Tools provide APIs■ Smart Data Center (Joyent’s tools, can be licensed)■ Project FIFO (SDC API in free package)■ Joyent Public Cloud■ Many compute nodes working in a cluster,PXE booted from a head node
  13. 13. Why should I care?Proprietary and
  14. 14. Why should I care?Proprietary and■ ZFSFile system built for speed and data integrity
  15. 15. Why should I care?Proprietary and■ Visibility toolsdtrace, kstat, snoop, truss■ ZFSFile system built for speed and data integrity
  16. 16. ■ Service Management Facility (SMF)If init.d and monit and god were one thing, andactually awesomeWhy should I care?Proprietary and■ Visibility toolsdtrace, kstat, snoop, truss■ ZFSFile system built for speed and data integrity
  17. 17. ■ Service Management Facility (SMF)If init.d and monit and god were one thing, andactually awesomeWhy should I care?Proprietary and■ Visibility toolsdtrace, kstat, snoop, truss■ ZFSFile system built for speed and data integrity■ Application LatencyZones are OS virtualization, so fasterProcesses are scheduled in global zone kernel,not in a hardware virtualization layer
  18. 18. ■ Service Management Facility (SMF)If init.d and monit and god were one thing, andactually awesomeWhy should I care?Proprietary and■ Visibility toolsdtrace, kstat, snoop, truss■ ZFSFile system built for speed and data integrity■ Application LatencyZones are OS virtualization, so fasterProcesses are scheduled in global zone kernel,not in a hardware virtualization layer
  19. 19. ■ Service Management Facility (SMF)If init.d and monit and god were one thing, andactually awesomeWhy should I care?Proprietary and■ Visibility toolsdtrace, kstat, snoop, truss■ ZFSFile system built for speed and data integrity■ Application LatencyZones are OS virtualization, so fasterProcesses are scheduled in global zone kernel,not in a hardware virtualization layer
  20. 20. Lower latency == less costProprietary and
  21. 21. Lower latency == less costProprietary and■ Requests/second of single process =~request latency
  22. 22. Lower latency == less costProprietary and■ # processes required =~requests/second of site■ Requests/second of single process =~request latency
  23. 23. ■ # cores, RAM required =~# processesLower latency == less costProprietary and■ # processes required =~requests/second of site■ Requests/second of single process =~request latency
  24. 24. ■ # cores, RAM required =~# processesLower latency == less costProprietary and■ # processes required =~requests/second of site■ Requests/second of single process =~request latency$$$
  25. 25. On to Chef!Proprietary and
  26. 26. TerminologyProprietary and■ Image / Dataset — OS at a particular version,snapshotted at base state■ Flavor / Package— RAM, CPU shares■ API URL — Each data center has its own URL■ Server ID / Zonename — Each zone gets aUUID
  27. 27. knife-joyentProprietary and
  28. 28. Installation/ConfigurationProprietary and■ Update knife.rb■ Add to Gemfileknife[:joyent_username] = saxknife[:joyent_keyname] = EricSaxbyknife[:joyent_keyfile] = "#{ENV[HOME]}/.ssh/id_rsa"knife[:joyent_api_url] = https://us-sw-1.api.joyentcloud.com/■ Add first public key in cloud APIhttps://my.joyentcloud.comgem knife-joyent
  29. 29. Managing keysProprietary and■ No role based access, but at least you canmake each user upload their own keyknife joyent key add -f ~/.ssh/id_rsa -k KeyNameknife joyent key delete KeyName■ Passphrase protected keys are annoyingEach API request includes data signed with the privatekey. Ruby does not have a good way of signing privatekeys with ssh-agent.
  30. 30. Creating servers!Proprietary and■ See what images are availableknife joyent image listcf7e2f40-9276-11e2-af9a-0bad2233fb0b base64 1.9.1 smartosf4bc70ca-5e2c-11e1-8380-fb28785857cb smartosplus64 3.1.0 smartosda144ada-a558-11e2-8762-538b60994628 ubuntu-12.04 2.4.1 linux■ base / base64 — minimal install, you add whatyou need■ smartosplus — many more things pre-installed, but can get in the way13328c9a-9173-11e2-a9a5-2ff43d306c21 ws2008ent-r2-sp1 2.0.2 windows
  31. 31. Creating servers!Proprietary and■ See what flavors are availableknife joyent flavor listName RAM Disk SwapExtra Small 512 MB 0 GB 15 GB 1 GBSmall 1GB 1 GB 30 GB 2 GBMedium 2GB 2 GB 60 GB 4 GBMedium 4GB 4 GB 120 GB 8 GBLarge 8GB 8 GB 240 GB 16 GBLarge 16GB 16 GB 480 GB 32 GB■ Custom networking can be done in a customflavor (ie public or private VLAN, routes)
  32. 32. Creating servers already!Proprietary andknife joyent server create--image cf7e2f40-9276-11e2-af9a-0bad2233fb0b--flavor Medium 2GB-N server.domain.com-E environment-d distro-r run_list■ No Omnibus, so you have to provide your owndistro bootstrap templatehttps://gist.github.com/sax/5457464
  33. 33. knife joyent server listSee whats there...Proprietary anda597a3a7-3fdf-481f-af08-e7c1e0ae7dca admin.prod running smartmachinesdc:sdc:base64:1.8.1 8.19.1.1 10.100.1.1 8 GB 240 GB5c066e6e-8af2-4d4f-a81e-c8e2691ae8a0 demo.dev running smartmachinesdc:sdc:base64:1.8.1 10.12.1.1 165.225.1.1 8 GB 240 GBb3370d52-3bed-462e-857a-e17eba15ab06 app010.c1.prod running smartmachinesdc:sdc:base64:1.8.1 10.100.1.2 165.225.1.2 8 GB 240 GB■ ID / zonename■ Name■ Run state■ Type■ Image■ IP addresses■ RAM■ Disk
  34. 34. Other managementProprietary andknife joyent server delete <server_id>knife joyent server start <server_id>knife joyent server stop <server_id>knife joyent server reboot <server_id>knife joyent server resize <server_id> -f <flavor>knife joyent snapshot create <server_id> <snapshot_name>■ Snapshots are full ZFS snapshotsCopy-on-write snapshot of local file system.Each snapshot is locally mounted in zone at/checkpoints
  35. 35. So now you have asmartmachine...Proprietary and
  36. 36. Whats different?Proprietary and■ Things you expect in /usr/local are in /opt/local■ For historical reasons■ If youre used to Linux, this can be annoying■ Joyent is working on a more Linux friendly image■ For now, add /opt/local/bin to PATH■ Many configs are in /opt/local/etc instead of /etc■ Some utilities are different■ This is not the grep youre looking for....■ Symlink your "correct" version into /opt/local/bin■ Add /opt/local/lib to CFLAGS and LDFLAGS
  37. 37. Caveats?Proprietary and■ Zones inside of zones inside of...■ Vagrant does not currently work with SmartOS■ VirtualBox only works in Bridged network mode■ Local integration tests do not work
  38. 38. Where are all the things?Proprietary and■ Services■ svcs -a■ svcadm < enable | disable | clear > service■ Packages■ pkgin search packagename■ pkgin -y install packagename
  39. 39. Public vs. Private IPProprietary and■ ipaddr_extensions gem■ Adds privateaddress attribute to ohai■ Useful to add this to bootstrap■ Smartmachines may have a public IP and aprivate IP■ Recipes can be configured to use ipaddress orprivateaddress
  40. 40. System preparationProprietary and■ smartos cookbook■ https://github.com/modcloth-cookbooks/smartos■ fixes chef providers■ smartmachine_functions■ links nicer utils into /opt/local/bin■ https://github.com/higanworks-cookbooks/smartmachine_functions■ fixes chef providers■ provides access to Joyent metadata APIor
  41. 41. Useful LWRPsProprietary and
  42. 42. SMFProprietary and■ https://github.com/modcloth-cookbooks/smf■ Chef knows how to use SMF, not how to configure it■ Uses nokogiri, which requires libxml2smf postgres douser postgresgroup postgresproject postgresstart_command postgres-service.sh startstop_command postgres-service.sh stopworking_directory /var/pgsql/dataenvironment PATH => /opt/postgres/binend
  43. 43. SMF (cntd)Proprietary andsmf postgres douser postgresgroup postgresproject postgresstart_command postgres-service.sh startstop_command postgres-service.sh stopstop_timeout 120restart_command postgres-service.sh restartrefresh_command postgres-service.sh reloadworking_directory /var/pgsql/dataenvironment PATH => /opt/postgres/binendservice postgres dosupports :status => true,:restart => true, :reload => trueend
  44. 44. SMF (cntd)Proprietary andsmf postgres douser postgresgroup postgresproject postgresstart_command postgres-service.sh startstop_command postgres-service.sh stopstop_timeout 120restart_command postgres-service.sh restartrefresh_command postgres-service.sh reloadworking_directory /var/pgsql/dataenvironment PATH => /opt/postgres/binendservice postgres dosupports :status => true,:restart => true, :reload => trueend
  45. 45. Resource Control / ProjectsProprietary and■ https://github.com/wanelo-chef/resource-control■ configure max file descriptors, shared memory, etc■ Bunch up master/worker processes to view inprstat -Jresource_control_project "postgres" docomment "PostgreSQL 9.2"users "postgres"project_limits "max-shm-memory" => 12000000,"max-lwps" => 6process_limits "max-file-descriptor" => {"value" => 32768, "deny" => true}action :createend
  46. 46. Role Based Access ControlProprietary and■ https://github.com/modcloth-cookbooks/rbac■ Allows delegation of authority without sudo■ Implementation currently too simple, only useful for SMFdelegationrbac solr douser waneloaction :add_management_permissionsend
  47. 47. Contributing to cookbooksProprietary and■ ~95% just require SMF, correct package names■ ~5% of those need a special init script■ The rest usually require custom compile`postgres -D /path/to/data` not granular enough`pg_ctl -D /path/to/data < start | stop | reload | refresh >`--with-libraries=/opt/local/lib--with-includes=/opt/local/includeLDFLAGS=-R/opt/local/lib -L/opt/local/lib
  48. 48. Comments? Questions? Find me.https://github.com/wanelohttps://github.com/wanelo-chefhttps://github.com/wanelo-chef/smartos-chef-repoProprietary and@sax @ecdysone @sax

×