AWS OpsWorks Under the Hood (DMG304) | AWS re:Invent 2013
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

AWS OpsWorks Under the Hood (DMG304) | AWS re:Invent 2013

on

  • 3,573 views

AWS OpsWorks lets you model your application with layers that define the building blocks of your application: load balancers, application servers, databases, etc. But did you know that you can also ...

AWS OpsWorks lets you model your application with layers that define the building blocks of your application: load balancers, application servers, databases, etc. But did you know that you can also extend OpsWorks layers or build your own custom layers? Whether you need to perform a specific task or install a new software package, OpsWorks gives you the tools to install and configure your instances consistently, and evolve them in an automated and predictable fashion through your application’s lifecycle. We'll dive into the development process including how to use attributes, recipes, and lifecycle events; show how to develop your environment locally; and provide troubleshooting steps that reduce your development time.

Statistics

Views

Total Views
3,573
Views on SlideShare
3,554
Embed Views
19

Actions

Likes
8
Downloads
47
Comments
0

1 Embed 19

https://twitter.com 19

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

AWS OpsWorks Under the Hood (DMG304) | AWS re:Invent 2013 Presentation Transcript

  • 1. DMG304 - AWS OpsWorks Under the Hood Jonathan Weiss & Reza Spagnolo, Amazon Web Services November 14th, 2013 © 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • 2. Agenda • AWS OpsWorks event life cycle • How to use custom cookbooks: MongoDB • Chef in AWS OpsWorks vs. Chef server
  • 3. Event Life Cycle
  • 4. The Heart of the Service Agent on each Agent on each Amazon EC2 instance EC2 instance Understands a set of commands that are triggered by AWS OpsWorks. The agent then runs a Chef solo run.
  • 5. Life Cycle Events setup configure deploy undeploy shutdown
  • 6. new Continuous Configuration
  • 7. new Continuous Configuration
  • 8. Continuous Configuration configure online new setup
  • 9. Continuous Configuration configure online new setup deploy
  • 10. Continuous Configuration configure shutting down online new / stopped setup terminating configure deploy
  • 11. Setup Event • Sent when instance boots • Includes deploy event • Use for initial installation of software & services
  • 12. Setup Event – Recipe Execution Order AWS OpsWorks setup recipes Your setup recipes AWS OpsWorks deploy recipes Your deploy recipes
  • 13. Configure Event • Sent to all instances when any instance enters or leaves online state • Use for making sure the configuration is up-to-date
  • 14. Deploy Event • Sent by deploy via UI/API and is also part of each setup • Use for custom deployment
  • 15. Undeploy Event • Sent via UI/API when apps are deleted • Use to remove apps from running instances
  • 16. Shutdown Event • Sent when an instance is shut down • ~45s to execute • Use for clean shutdown
  • 17. Customizing AWS OpsWorks
  • 18. control simple Customization Options Built-in layers Override Chef attributes via custom JSON Override Chef attributes via custom cookbook Overwrite Chef template file Deploy hooks Provide custom recipe to extend built-in layer Provide custom recipe to create custom layer
  • 19. Built-in Layers Rails MySQL PHP HAProxy Node.js Memcached Java Ganglia Open Source at http://github.com/aws/opsworks-cookbooks
  • 20. Chef Templates Template for /etc/apache2/apache2.conf
  • 21. Custom JSON Override Chef attributes of built-in cookbooks # in apache2/attributes/default.rb default[:apache][:timeout] = 120 node.apache.timeout # => 60 { ‘apache2‘: {‘timeout‘: 60 } }
  • 22. Custom Cookbook Attributes Override Chef attributes of built-in cookbooks # in apache2/attributes/default.rb default[:apache][:timeout] = 120 node.apache.timeout # => 60 # in mycookbook/attributes/apache.rb set[:apache][:timeout] = 60
  • 23. Overwriting Chef Templates Provide custom file at same location $ cat mycookbook-repo/apache2/templates/apache2.conf.erb ... # # Example custom content # LogLevel error # change default type DefaultType text/json
  • 24. Deploy Hooks $ ls myapp/deploy/ before_migrate.rb before_symlink.rb before_restart.rb after_restart.rb Hooks use Capistrano syntax
  • 25. Example Deploy Hook Rails asset pipeline support $ cat deploy/before_symlink.rb run "cd #{release_path} && RAILS_ENV=production bundle exec rake assets:precompile sudo "/etc/init.d/myservice restart"
  • 26. Custom Cookbooks
  • 27. Custom Layers Run anything that is scriptable with Chef – – – – – Erlang app server Cassandra DB cluster C daemon Custom PHP install …
  • 28. Custom Layers
  • 29. Custom Layers
  • 30. Custom Layers $ ls mycookbook-repo/erlang/ recipes/install.rb recipes/uninstall.rb $ ls mycookbook-repo/e-app/ recipes/deploy.rb recipes/undeploy.rb
  • 31. Custom Layers
  • 32. Different Configuration Managers Call bash script from Chef bash "legacy bootstrap" do user "root" cwd "/tmp" code <<-EOH wget http://www.example.com/installer.tar.gz tar -zxf installer.tar.gz cd installer ./configure make make install EOH end
  • 33. AWS OpsWorks Attribute Tree Sources: Custom JSON Stack configuration JSON Deployment JSON Cookbook attributes Chef - Ohai
  • 34. AWS OpsWorks Attribute Tree - Categories Multiple categories and namespaces node[:opsworks] node[:opsworks][:layers] node[:opsworks][:instance] node[:opsworks][:stack] Built in layer specific -> i.e. node[:opsworks_java] Built in recipe specific -> i.e. node[:apache2]
  • 35. Walkthrough: MongoDB on AWS OpsWorks
  • 36. Architecture – MongoDB Replica Set
  • 37. Objectives • • • • MongoDB as custom layer Use community cookbook Extend Java application layer Manage system life cycle
  • 38. MongoDB Custom Layer • Chef community cookbook for MongoDB – https://github.com/edelight/chef-mongodb • Cookbook functionality – – – – Install Register with peers Service startup Clean shutdown
  • 39. MongoDB Custom Layer - Recipes
  • 40. MongoDB Custom Layer - Changes Enable node discovery through AWS OpsWorks JSON def replicaset_members(node) replicaset_layer = node['opsworks']['instance']['layers'].first instances = node['opsworks']['layers'][replicaset_layer]['instances'] instances.map do |name, instance| member = Chef::Node.new member.name(name) member.default['hostname'] = name member.default['fqdn'] = instance['private_dns_name'] member.default['ipaddress'] = instance['private_ip'] member end end
  • 41. MongoDB Custom Layer - Configuration Custom JSON in MongoDB stack Made available to every Chef recipe Integrates out of the box with community cookbook
  • 42. MongoDB Custom Layer – EBS Setup AWS OpsWorks support for EBS RAID Setup the mount point passed through custom JSON
  • 43. Extend Application Layer - Configure Leverage configure event definition of built-in Java layer
  • 44. Extend Application Layer - Template Override application context template of built-in Java layer $ cat opsworks_java/templates/default/webapp_context.xml.erb <% replicaset_name = node['mongodb']['replicaset_name'] mongo_nodes = node['opsworks']['layers'][replicaset_name] ['instances'].keys.map{|name| "#{name}:27017"}.join(",") %> <Context> <Environment name="mongoNodes" type="java.lang.String" value="<%= mongo_nodes %>" /> </Context>
  • 45. Package Your Custom Cookbook Package your cookbook and its dependencies in a single archive java-mongodb-compound-cookbook/ ├── apt/ ├── build-essential/ ├── python/ ├── yum/ ├── opsworks_java/ └── chef-mongodb/
  • 46. Done !
  • 47. Chef in AWS OpsWorks vs. Chef Server
  • 48. Main Differences • • • • • Chef setup One run vs. discrete events Push vs. pull Discovery: search & AWS OpsWorks attribute tree Data bags
  • 49. Chef Setup Chef Solo & AWS OpsWorks Backend Chef Client & Chef Server
  • 50. Life Cycle Events • Give you fine-grained control • Faster to execute • Context
  • 51. Push vs. Pull • On-demand and automatic • Respond immediately to changes in the stack
  • 52. Discovery: Chef Search AWS OpsWorks does not offer attribute search Alternative: node[:opsworks] with similar capabilities to partial_search
  • 53. AWS OpsWorks Attribute Tree Find all Rails application servers rails_servers = node['opsworks']['layers']['rails-app']['instances'] rails_ips = rails_servers.map{|i| i['private_ip'] } template "/etc/rails-server.conf" do ... variables({ :ips => rails_server_ips }) end
  • 54. Encrypted Data Bags • Upload encrypted JSON to S3 • Have instances access via IAM roles in a recipe
  • 55. Custom JSON Arbitrary JSON on stack that is available in Chef { ‘foo‘: { ‘bar‘: true, ‘baz‘: false } } if node.foo.bar ... elsif node.foo.baz ... end
  • 56. Store Secrets on Amazon S3 Access from instance via instance profiles bucket = node['acme']['bucket'] key = node['acme']['key'] s3 = AWS::S3.new obj = s3.buckets[bucket].objects[key] obj.read
  • 57. Recap
  • 58. AWS OpsWorks • Life cycle framework • Highly customizable • Run anything
  • 59. More Information about AWS OpsWorks • • • • • • Do the AWS OpsWorks lab! Find us on the AWS Booth Follow us on twitter @AWSOpsWorks Find us on YouTube Blog: http://blogs.aws.amazon.com/application-management Survey: http://tinyurl.com/OpsWorksSurvey2013
  • 60. Recommend Sessions DMG305 - How Intuit Leveraged AWS OpsWorks as the Engine of Our PaaS Capen Brinkley & Rick Mendes of Intuit, inc. Thursday, Nov 14, 4:15 PM - 5:15 PM – Murano 3206
  • 61. Please give us your feedback on this presentation DMG304 As a thank you, we will select prize winners daily for completed surveys!