ChefConf 2014 - AWS OpsWorks Under The Hood

3,718 views

Published on

AWS OpsWorks under the hood - presented at ChefConf 2014

ChefConf 2014 - AWS OpsWorks Under The Hood

  1. 1. © 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc. AWS OpsWorks Under the Hood Jonathan Weiss @jweiss Amazon Web Services
  2. 2. “Integrated application management service on EC2 – powered by Chef”
  3. 3. Chef Setup Chef Client/Zero & OpsWorks Backend Chef Client & Chef Server
  4. 4. OpsWorks Architecture OpsWorks Backend OpsWorks Agent EC2, EBS, EIP, VPC, ELB, … Auto-Scaling, Auto-Healing,… On-instance execution via Chef client/zero Command JSON Command Log+Status
  5. 5. Chef Setup in OpsWorks •  Supported Chef versions: 0.9, 11.4 or 11.10 •  Built-in convenience cookbooks / bring your own •  Chef run is triggered by life cycle event firing •  Event comes with stack state JSON
  6. 6. Stacks & Layers
  7. 7. Modeling in OpsWorks
  8. 8. Layers Group of instances with common behavior & settings –  Recipes / run_list –  Settings / attributes –  Similar to Chef role
  9. 9. Built-in Layers Open Source at http://github.com/aws/opsworks-cookbooks Rails MySQL PHP HAProxy Node.js Memcached Java Ganglia
  10. 10. Custom Layers Define your own layers and their run_list –  Erlang app server –  Cassandra DB cluster –  C daemon –  Custom PHP install –  …
  11. 11. Custom Layers
  12. 12. Custom Layers
  13. 13. Custom Layers
  14. 14. Event Life Cycle
  15. 15. Events Events are triggered when your stack changes: •  Give you fine-grained control •  Faster to execute •  Context: run_list per event per layer (aka role)
  16. 16. Life Cycle Events 1 6 setup configure deploy undeploy shutdown
  17. 17. Instance Life Cycle new
  18. 18. Instance Life Cycle new
  19. 19. Instance Life Cycle new online setup configure
  20. 20. Instance Life Cycle new online setup configure deploy
  21. 21. Instance Life Cycle new/stopped online setup configure terminating shutting down deploy configure
  22. 22. Setup Event •  Sent when instance boots •  Includes deploy event •  Use for initial installation of software & services
  23. 23. Setup Event – Recipe Execution Order AWS OpsWorks setup recipes Your setup recipes AWS OpsWorks deploy recipes Your deploy recipes
  24. 24. Configure Event •  Sent to all instances when any instance enters or leaves online state •  Use for making sure the configuration is up-to-date
  25. 25. Deploy Event •  Sent you deploy via UI/API also part of each setup •  Use for custom deployment
  26. 26. Undeploy Event •  Sent via UI/API when apps are deleted •  Use to remove apps from running instances
  27. 27. Shutdown Event •  Sent when an instance is shut down •  ~45s to execute •  Use for clean shutdown
  28. 28. Stack State JSON Each event gets JSON / attributes that define the current stack state: node[:opsworks] node[:opsworks][:layers] node[:opsworks][:instance] node[:opsworks][:stack]
  29. 29. Chef Integration
  30. 30. Search Stack state JSON available through search search(:node, “name:web1”) search(:node, “name:web*”) Attributes generated on nodes are not available
  31. 31. Search appserver = search(:node, "role:php-app").first Chef::Log.info(”Private IP: #{appserver[:private_ip]}") Exposes: hostname/fqdn, IP/DNS, private IP/DNS, instance type, AMI ID, AZ, …
  32. 32. Roles OpsWorks layers mapped as roles search(:node, “role:rails-app”) search(:node, “role:custom-foo”) Complete role functionality depends on supporting a “full” chef repo
  33. 33. Data Bags Define in custom JSON { "opsworks": { "data_bags": { "bag_name1": { "item_name1: { "key1" : “value1”, "key2" : “value2”, ... } }, "bag_name2": { "item_name1": { "key1" : “value1”, "key2" : “value2”, ... } }, ... } } }
  34. 34. { "opsworks": { "data_bags": { "myapp": { "mysql": { "username": "default-user", "password": "default-pass" } } } } } mything = data_bag_item("myapp", "mysql") Chef::Log.info("username: #{mything['username']}") Recipe Custom JSON
  35. 35. Encrypted Data Bags Not supported – Alternative handling: •  Upload encrypted JSON to S3 •  Have instances access via IAM roles in a recipe
  36. 36. Store Secrets on Amazon S3 Access from instance via IAM instance profiles bucket = node['acme']['bucket'] key = node['acme']['key'] s3 = AWS::S3.new obj = s3.buckets[bucket].objects[key] obj.read
  37. 37. Berkshelf Integration Enable Berkshelf in stack settings Supports any version, ships pre-compiled for some
  38. 38. Berkshelf Integration Cookbook directories /opt/aws/opsworks/current/cookbooks /opt/aws/opsworks/current/site-cookbooks /opt/aws/opsworks/current/berkshelf-cookbooks
  39. 39. Berksfile cookbook 'apt' cookbook 'bluepill', '>= 2.3.1' cookbook 'ark', git: 'git://github.com/opscode-cookbooks/ark.git' cookbook 'build-essential', '>= 1.4.2', git: 'git://github.com/opscode-cookbooks/build-essential.git', tag: 'v1.4.2'
  40. 40. Environments •  OpsWorks only supports the implicit _default env •  We are looking into adding proper env support
  41. 41. Recap
  42. 42. Main Differences To Chef Server •  One run vs. discrete events •  Push vs. pull •  Discovery: search & AWS OpsWorks attribute tree •  Encrypted data bags •  Environments & roles
  43. 43. AWS OpsWorks •  Life cycle framework •  Highly customizable – in the end everything is a Chef run •  Expect us to integrate more over time
  44. 44. More information about AWS OpsWorks •  Follow us on twitter @AWSOpsWorks •  Find us on YouTube •  Docs: http://aws.amazon.com/documentation/opsworks •  Blog: http://blogs.aws.amazon.com/application-management
  45. 45. Thank You@jweiss aws.amazon.com/opsworks

×