Your SlideShare is downloading. ×
AWS OpsWorks Under the Hood (DMG304) | AWS re:Invent 2013
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

AWS OpsWorks Under the Hood (DMG304) | AWS re:Invent 2013


Published on

AWS OpsWorks lets you model your application with layers that define the building blocks of your application: load balancers, application servers, databases, etc. But did you know that you can also …

AWS OpsWorks lets you model your application with layers that define the building blocks of your application: load balancers, application servers, databases, etc. But did you know that you can also extend OpsWorks layers or build your own custom layers? Whether you need to perform a specific task or install a new software package, OpsWorks gives you the tools to install and configure your instances consistently, and evolve them in an automated and predictable fashion through your application’s lifecycle. We'll dive into the development process including how to use attributes, recipes, and lifecycle events; show how to develop your environment locally; and provide troubleshooting steps that reduce your development time.

Published in: Technology, Self Improvement

  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. DMG304 - AWS OpsWorks Under the Hood Jonathan Weiss & Reza Spagnolo, Amazon Web Services November 14th, 2013 © 2013, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of, Inc.
  • 2. Agenda • AWS OpsWorks event life cycle • How to use custom cookbooks: MongoDB • Chef in AWS OpsWorks vs. Chef server
  • 3. Event Life Cycle
  • 4. The Heart of the Service Agent on each Agent on each Amazon EC2 instance EC2 instance Understands a set of commands that are triggered by AWS OpsWorks. The agent then runs a Chef solo run.
  • 5. Life Cycle Events setup configure deploy undeploy shutdown
  • 6. new Continuous Configuration
  • 7. new Continuous Configuration
  • 8. Continuous Configuration configure online new setup
  • 9. Continuous Configuration configure online new setup deploy
  • 10. Continuous Configuration configure shutting down online new / stopped setup terminating configure deploy
  • 11. Setup Event • Sent when instance boots • Includes deploy event • Use for initial installation of software & services
  • 12. Setup Event – Recipe Execution Order AWS OpsWorks setup recipes Your setup recipes AWS OpsWorks deploy recipes Your deploy recipes
  • 13. Configure Event • Sent to all instances when any instance enters or leaves online state • Use for making sure the configuration is up-to-date
  • 14. Deploy Event • Sent by deploy via UI/API and is also part of each setup • Use for custom deployment
  • 15. Undeploy Event • Sent via UI/API when apps are deleted • Use to remove apps from running instances
  • 16. Shutdown Event • Sent when an instance is shut down • ~45s to execute • Use for clean shutdown
  • 17. Customizing AWS OpsWorks
  • 18. control simple Customization Options Built-in layers Override Chef attributes via custom JSON Override Chef attributes via custom cookbook Overwrite Chef template file Deploy hooks Provide custom recipe to extend built-in layer Provide custom recipe to create custom layer
  • 19. Built-in Layers Rails MySQL PHP HAProxy Node.js Memcached Java Ganglia Open Source at
  • 20. Chef Templates Template for /etc/apache2/apache2.conf
  • 21. Custom JSON Override Chef attributes of built-in cookbooks # in apache2/attributes/default.rb default[:apache][:timeout] = 120 node.apache.timeout # => 60 { ‘apache2‘: {‘timeout‘: 60 } }
  • 22. Custom Cookbook Attributes Override Chef attributes of built-in cookbooks # in apache2/attributes/default.rb default[:apache][:timeout] = 120 node.apache.timeout # => 60 # in mycookbook/attributes/apache.rb set[:apache][:timeout] = 60
  • 23. Overwriting Chef Templates Provide custom file at same location $ cat mycookbook-repo/apache2/templates/apache2.conf.erb ... # # Example custom content # LogLevel error # change default type DefaultType text/json
  • 24. Deploy Hooks $ ls myapp/deploy/ before_migrate.rb before_symlink.rb before_restart.rb after_restart.rb Hooks use Capistrano syntax
  • 25. Example Deploy Hook Rails asset pipeline support $ cat deploy/before_symlink.rb run "cd #{release_path} && RAILS_ENV=production bundle exec rake assets:precompile sudo "/etc/init.d/myservice restart"
  • 26. Custom Cookbooks
  • 27. Custom Layers Run anything that is scriptable with Chef – – – – – Erlang app server Cassandra DB cluster C daemon Custom PHP install …
  • 28. Custom Layers
  • 29. Custom Layers
  • 30. Custom Layers $ ls mycookbook-repo/erlang/ recipes/install.rb recipes/uninstall.rb $ ls mycookbook-repo/e-app/ recipes/deploy.rb recipes/undeploy.rb
  • 31. Custom Layers
  • 32. Different Configuration Managers Call bash script from Chef bash "legacy bootstrap" do user "root" cwd "/tmp" code <<-EOH wget tar -zxf installer.tar.gz cd installer ./configure make make install EOH end
  • 33. AWS OpsWorks Attribute Tree Sources: Custom JSON Stack configuration JSON Deployment JSON Cookbook attributes Chef - Ohai
  • 34. AWS OpsWorks Attribute Tree - Categories Multiple categories and namespaces node[:opsworks] node[:opsworks][:layers] node[:opsworks][:instance] node[:opsworks][:stack] Built in layer specific -> i.e. node[:opsworks_java] Built in recipe specific -> i.e. node[:apache2]
  • 35. Walkthrough: MongoDB on AWS OpsWorks
  • 36. Architecture – MongoDB Replica Set
  • 37. Objectives • • • • MongoDB as custom layer Use community cookbook Extend Java application layer Manage system life cycle
  • 38. MongoDB Custom Layer • Chef community cookbook for MongoDB – • Cookbook functionality – – – – Install Register with peers Service startup Clean shutdown
  • 39. MongoDB Custom Layer - Recipes
  • 40. MongoDB Custom Layer - Changes Enable node discovery through AWS OpsWorks JSON def replicaset_members(node) replicaset_layer = node['opsworks']['instance']['layers'].first instances = node['opsworks']['layers'][replicaset_layer]['instances'] do |name, instance| member = member.default['hostname'] = name member.default['fqdn'] = instance['private_dns_name'] member.default['ipaddress'] = instance['private_ip'] member end end
  • 41. MongoDB Custom Layer - Configuration Custom JSON in MongoDB stack Made available to every Chef recipe Integrates out of the box with community cookbook
  • 42. MongoDB Custom Layer – EBS Setup AWS OpsWorks support for EBS RAID Setup the mount point passed through custom JSON
  • 43. Extend Application Layer - Configure Leverage configure event definition of built-in Java layer
  • 44. Extend Application Layer - Template Override application context template of built-in Java layer $ cat opsworks_java/templates/default/webapp_context.xml.erb <% replicaset_name = node['mongodb']['replicaset_name'] mongo_nodes = node['opsworks']['layers'][replicaset_name] ['instances']{|name| "#{name}:27017"}.join(",") %> <Context> <Environment name="mongoNodes" type="java.lang.String" value="<%= mongo_nodes %>" /> </Context>
  • 45. Package Your Custom Cookbook Package your cookbook and its dependencies in a single archive java-mongodb-compound-cookbook/ ├── apt/ ├── build-essential/ ├── python/ ├── yum/ ├── opsworks_java/ └── chef-mongodb/
  • 46. Done !
  • 47. Chef in AWS OpsWorks vs. Chef Server
  • 48. Main Differences • • • • • Chef setup One run vs. discrete events Push vs. pull Discovery: search & AWS OpsWorks attribute tree Data bags
  • 49. Chef Setup Chef Solo & AWS OpsWorks Backend Chef Client & Chef Server
  • 50. Life Cycle Events • Give you fine-grained control • Faster to execute • Context
  • 51. Push vs. Pull • On-demand and automatic • Respond immediately to changes in the stack
  • 52. Discovery: Chef Search AWS OpsWorks does not offer attribute search Alternative: node[:opsworks] with similar capabilities to partial_search
  • 53. AWS OpsWorks Attribute Tree Find all Rails application servers rails_servers = node['opsworks']['layers']['rails-app']['instances'] rails_ips ={|i| i['private_ip'] } template "/etc/rails-server.conf" do ... variables({ :ips => rails_server_ips }) end
  • 54. Encrypted Data Bags • Upload encrypted JSON to S3 • Have instances access via IAM roles in a recipe
  • 55. Custom JSON Arbitrary JSON on stack that is available in Chef { ‘foo‘: { ‘bar‘: true, ‘baz‘: false } } if ... elsif ... end
  • 56. Store Secrets on Amazon S3 Access from instance via instance profiles bucket = node['acme']['bucket'] key = node['acme']['key'] s3 = obj = s3.buckets[bucket].objects[key]
  • 57. Recap
  • 58. AWS OpsWorks • Life cycle framework • Highly customizable • Run anything
  • 59. More Information about AWS OpsWorks • • • • • • Do the AWS OpsWorks lab! Find us on the AWS Booth Follow us on twitter @AWSOpsWorks Find us on YouTube Blog: Survey:
  • 60. Recommend Sessions DMG305 - How Intuit Leveraged AWS OpsWorks as the Engine of Our PaaS Capen Brinkley & Rick Mendes of Intuit, inc. Thursday, Nov 14, 4:15 PM - 5:15 PM – Murano 3206
  • 61. Please give us your feedback on this presentation DMG304 As a thank you, we will select prize winners daily for completed surveys!