Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Cooking Up Drama

1,587 views

Published on

Chef Conf 2105 talk - see http://bridgetkromhout.com/speaking/2015/chefconf/

Presented with Peter Shannon

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Cooking Up Drama

  1. 1. Cooking Up Drama Bridget Kromhout & Peter Shannon
  2. 2. @bridgetkromhout @pietroshannon Bridget Kromhout Peter Shannon @devopsdaysMSP @arresteddevops devopsdays.org Go Data Ops & Zip Ties
  3. 3. @pietroshannon@bridgetkromhout DramaFever: streaming international content
  4. 4. peak load: tens of thousands of requests per second traffic variance: swings 10-20x throughout the week @bridgetkromhout @pietroshannon
  5. 5. @bridgetkromhout @pietroshannon
  6. 6. @bridgetkromhout @pietroshannon
  7. 7. Architecture ● All services in AWS ● Python (Django) main website ● Go microservices (video analytics ingest, on- the-fly image processing, bookmarking…) ● Upstreams routed via nginx ● Celery + SQS for async tasks ● Streaming delivery through Akamai @bridgetkromhout @pietroshannon
  8. 8. What do you mean, NO CM!? Docker Jenkins Graphite & ELK AWS Config Management @bridgetkromhout @pietroshannon
  9. 9. CONFIG MANAGEMENT EVERYWHERE @bridgetkromhout @pietroshannon
  10. 10. Why did we need CM? ● Single source of truth to build AMIs and provision AWS instances. ● Consistent configuration across ephemeral instances. ● Hand-crafted, longer-lived instances are hard to reproduce. @bridgetkromhout @pietroshannon
  11. 11. @bridgetkromhout @pietroshannon CM selection showdown: fight!
  12. 12. @bridgetkromhout @pietroshannon cooperation, not competition
  13. 13. @bridgetkromhout @pietroshannon decisions: hosted chef
  14. 14. @bridgetkromhout @pietroshannon decisions: hosted chef (but not “Starter Kit”)
  15. 15. @bridgetkromhout @pietroshannon decisions: environments
  16. 16. @bridgetkromhout @pietroshannon decisions: environments roles
  17. 17. @bridgetkromhout @pietroshannon decisions: environments roles minimal notifies
  18. 18. @bridgetkromhout @pietroshannon workflow
  19. 19. @bridgetkromhout @pietroshannon workflow: chef-dk
  20. 20. @bridgetkromhout @pietroshannon workflow: chef-dk test kitchen
  21. 21. @bridgetkromhout @pietroshannon workflow: chef-dk test kitchen jenkins
  22. 22. @bridgetkromhout @pietroshannon workflow: chef-dk test kitchen jenkins via github
  23. 23. @bridgetkromhout @pietroshannon workflow: chef-dk test kitchen jenkins via github rubocop foodcritic
  24. 24. @bridgetkromhout @pietroshannon Docker images built from Dockerfiles in Jenkins jobs deployment via fabric & upstart
  25. 25. pain points? @bridgetkromhout @pietroshannon where to start...
  26. 26. pain points? visible wins? @bridgetkromhout @pietroshannon where to start...
  27. 27. @bridgetkromhout @pietroshannon where to start... production!
  28. 28. @bridgetkromhout @pietroshannon autoscaling: scorched earth
  29. 29. #!/bin/bash cat <<EOF > /etc/init/django.conf description "Run Django containers for www" start on started docker-reg stop on runlevel [!2345] or stopped docker respawn limit 5 30 [...] replacing 100s of lines of userdata... @bridgetkromhout @pietroshannon
  30. 30. #!/bin/bash # upstart configs are now created by chef rm /etc/chef/client.pem mkdir -p /var/log/chef chef-client -r 'role[rolename]' -E 'environment' -L /var/log/chef/chef-client.log ...with a chef-client run. @bridgetkromhout @pietroshannon
  31. 31. deregistering nodes (knife) /usr/bin/knife node delete -y -c /etc/chef/knife.rb <%= node['base']['chefname'] %> /usr/bin/knife client delete -y -c /etc/chef/knife.rb <%= node['base']['chefname'] %> @bridgetkromhout @pietroshannon
  32. 32. deregistering nodes (rc script) template '/etc/init.d/unregister_chef_instance' do source 'default/unregister_chef_instance.erb' end link '/etc/rc0.d/K99unregister_chef_instance' do to '/etc/init.d/unregister_chef_instance' end @bridgetkromhout @pietroshannon
  33. 33. ami factory @bridgetkromhout @pietroshannon Autoscaling Packer AMI EC2 Instances Jenkins GitHub Chef
  34. 34. @bridgetkromhout @pietroshannon apt-get install: only for AMI...
  35. 35. @bridgetkromhout @pietroshannon apt-get install: only for AMI… ...and test kitchen
  36. 36. Docker: what to chef? not Docker images... not application settings... LWRP: upstarts admin wrappers @bridgetkromhout @pietroshannon
  37. 37. Docker & LWRPs @bridgetkromhout @pietroshannon docker run -e DJANGO_ENVIRON=PROD -e HAPROXY=df/haproxy-prod.cfg -p 8000:8000 -v /var/log/containers:/var/log --name django localhost-alias.com:5000/www:prod /var/www/bin/start-django
  38. 38. docker run <% if @docker_rm == true -%> --rm <% end %> <% @docker_env.each do |k, v| -%> -e <%= k %>=<%= v %> <% end %> <% @docker_port.each do |p| -%> -p <%= p %> <% end %> @bridgetkromhout @pietroshannon upstart template
  39. 39. <% @docker_volume.each do |v| -%> -v <%= v %> <% end %> --name <%= @application_name %> localhost-alias.com:<%= @registry_port %>/<%= @docker_image %>:<%= @docker_tag %> <%= @docker_command %> @bridgetkromhout @pietroshannon upstart template (cont)
  40. 40. @bridgetkromhout @pietroshannon attribute :docker_command, :kind_of => String, :required => true attribute :docker_env, :kind_of => Hash, :default => {} attribute :docker_port, :kind_of => Array, :default => [] attribute :docker_volume, :kind_of => Array, :default => ['/var/log/containers:/var/log'] attribute :docker_rm, :kind_of => [TrueClass, FalseClass], : default => false attribute :docker_name_override, :kind_of => String, :default => nil attribute :docker_image_override, :kind_of => String, :default => nil attribute :docker_tag_override, :kind_of => String, :default => nil so many overrides
  41. 41. @bridgetkromhout @pietroshannon if new_resource.docker_name_override.nil? name = new_resource.name upstart = name else name = new_resource.docker_name_override upstart = "#{new_resource.name}-#{name}" end if this, then just don’t
  42. 42. @bridgetkromhout @pietroshannon attribute :command, :kind_of => String, :required => true attribute :env, :kind_of => Hash, :default => {} attribute :port, :kind_of => Array, :default => [] attribute :volume, :kind_of => Array, :default => ['/var/log/containers:/var/log'] attribute :rm, :kind_of => [TrueClass, FalseClass], :default => false attribute :image, :kind_of => String, :required => true attribute :tag, :kind_of => String, :required => true attribute :type, :kind_of => String, :required => true attribute :cron, :kind_of => [TrueClass, FalseClass], :default => false using attributes
  43. 43. recipe using LWRP base_docker node['www']['django']['name'] do command node['www']['django']['command'] env node['www'][service]['django'][env]['env'] image node['www']['django']['image'] port node['www'][service]['django'][env]['port'] tag node['www'][service]['django'][env]['tag'] type node['www']['django']['type'] end @bridgetkromhout @pietroshannon
  44. 44. Wrapping Community Cookbooks @bridgetkromhout @pietroshannon cookbook_path ["#{ENV['DFHOME']} /chef/cookbooks", "#{ENV['DFHOME']} /chef/community_cookbooks"] clear delineation no forking of community cookbooks
  45. 45. wrapper cookbook - leroy ● depends on community cookbooks (jenkins, etc) ● recipes include: builds, packer, plugins ● fun with FC001 ● variations on our base (fstab, docker registry) @bridgetkromhout @pietroshannon
  46. 46. private docker registry # this goes in /etc/default/docker to control docker's upstart config DOCKER_OPTS="--graph=/mnt/docker --insecure- registry=localhost-alias.com:5000" ● localhost-alias.com in DNS with A record to 127.0.0.1 ● OS X /etc/hosts: use the boot2docker host-only network IP @bridgetkromhout @pietroshannon
  47. 47. registry upstart docker pull public_registry_image docker run -p 5000:5000 --name registry -v /etc/docker-reg:/registry-conf -e DOCKER_REGISTRY_CONFIG=/registry-conf/config.yml public_registry_image @bridgetkromhout @pietroshannon
  48. 48. config.yml s3_region: us-east-1 s3_access_key: <aws-accesskey> s3_secret_key: <aws-secretkey> s3_bucket: <bucketname> standalone: true storage: s3 storage_path: /registry @bridgetkromhout @pietroshannon
  49. 49. packer$HOME/packer/packer build -var "account_id=$AWS_ACCOUNT_ID" -var "aws_access_key_id=$AWS_ACCESS_KEY_ID" -var "aws_secret_key=$AWS_SECRET_ACCESS_KEY" -var "x509_cert_path=$AWS_X509_CERT_PATH" -var "x509_key_path=$AWS_X509_KEY_PATH" -var "s3_bucket=bucketname" -var "ami_name=$AMI_NAME" -var "source_ami=$SOURCE_AMI" -var "chef_validation=$CHEF_VAL" -var "chef_client=$HOME/packer/client.rb" -only=amazon-instance $HOME/packer/prod.json @bridgetkromhout @pietroshannon
  50. 50. packer for ami building { "type": "chef-client", "server_url": "https://api.opscode. com/organizations/dramafever", "run_list": [ "base::ami" ], "validation_key_path": "{{user `chef_validation`}}", "validation_client_name": "dramafever-validator", "node_name": "packer-ami" } @bridgetkromhout @pietroshannon
  51. 51. limiting packer IAM permissions "Action":[ "ec2:TerminateInstances", "ec2:StopInstances", "ec2:DeleteSnapshot", "ec2:DetachVolume", "ec2:DeleteVolume", "ec2:ModifyImageAttribute" ], "Effect":"Allow", "Resource":"*", "Condition":{ "StringEquals":{ "ec2:ResourceTag/name":"Packer Builder" } } @bridgetkromhout @pietroshannon
  52. 52. so, about that Jenkins server... @bridgetkromhout @pietroshannon
  53. 53. @bridgetkromhout @pietroshannon (un-)containerize some of the things
  54. 54. sentrybase_docker_upstart 'sentry-udp' do docker_env( 'DJANGO_ENVIRON' => 'DRAMAFEVER' ) docker_image_override 'sentry' docker_tag_override 'dev' docker_port ['9001:9001/udp'] docker_command 'sentry --config=/config/sentry.conf.py start udp' end @bridgetkromhout @pietroshannon
  55. 55. chef for devs @bridgetkromhout @pietroshannon
  56. 56. self-service visibility @bridgetkromhout @pietroshannon
  57. 57. (re)cycle of life @bridgetkromhout @pietroshannon
  58. 58. @bridgetkromhout @pietroshannon
  59. 59. @bridgetkromhout @pietroshannon thanks!

×