Real life aws_user_data_auto_scale

1,183 views

Published on

Real Life AWS usages of user-data and AutoScale.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,183
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
18
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Real life aws_user_data_auto_scale

  1. 1. Real Life AWS: user-data & AutoScale Adam Neumann Solutions Architect at IndexMedia [email_address] &
  2. 2. Part 1: user-data <ul><ul><li>What is user-data? </li></ul></ul><ul><ul><li>A problem </li></ul></ul><ul><ul><li>user-data in the solution </li></ul></ul><ul><ul><li>Further work </li></ul></ul>
  3. 3. What is user-data? <ul><ul><li>user specified instance attribute </li></ul></ul><ul><ul><li>accessible via HTTP and curl/wget </li></ul></ul><ul><ul><li>usage wrapped by cloud-init </li></ul></ul><ul><ul><li>'runs' the user data once on boot </li></ul></ul><ul><ul><li>great for configuring instances </li></ul></ul>
  4. 4. The concept <ul><ul><li>any script, or init-cloud format </li></ul></ul><ul><ul><li>#! /bin/bash etc etc </li></ul></ul><ul><ul><li>cmd line arg or via UI: </li></ul></ul>
  5. 5. A Problem <ul><ul><li>too hard to spin up new instances </li></ul></ul><ul><ul><li>complex, undocumented, manual </li></ul></ul><ul><ul><li>fab, bash, and misc scripts </li></ul></ul><ul><ul><li>slow: 30 min per instance </li></ul></ul><ul><ul><li>first: consolidate, then: user-data </li></ul></ul>
  6. 6. IndexMedia example 1 <ul><ul><li>small user-data script, bootstrap </li></ul></ul><ul><ul><li>load big script from S3 </li></ul></ul><ul><ul><li>setup apache, hadoop, virtualenv,... </li></ul></ul><ul><ul><li>16 mins, vanilla ubuntu to serving </li></ul></ul><ul><ul><li>automated: yes, optimal: no </li></ul></ul><ul><ul><li>could be improved... </li></ul></ul>
  7. 7. IndexMedia example 2 <ul><ul><li>split script into 2 scripts :) </li></ul></ul><ul><ul><li>1: heavy, general, static = 15 min </li></ul></ul><ul><ul><li>runs on vanilla ubuntu, creates AMI </li></ul></ul><ul><ul><li>2: light, instance specific = 90 secs </li></ul></ul><ul><ul><li>runs on AMI </li></ul></ul><ul><ul><li>from 30 min manual to 90 sec auto </li></ul></ul>
  8. 8. Further work <ul><ul><li>investigate cloud-init format </li></ul></ul><ul><ul><li>include Puppet/Chef (first, learn Puppet/Chef...) </li></ul></ul><ul><ul><li>integrate into CI process (environment becomes versioned, deployable, etc) </li></ul></ul>
  9. 9. user-data summary <ul><ul><li>great way to configure instances </li></ul></ul><ul><ul><li>UI and cmd line accessible </li></ul></ul><ul><ul><li>enables automated+repeatable deployment </li></ul></ul><ul><ul><li>reduced deployment time, 30 mins to 90 secs </li></ul></ul>
  10. 10. Part 2: AutoScale <ul><ul><li>What is AutoScale </li></ul></ul><ul><ul><li>Our (common) problem </li></ul></ul><ul><ul><li>Show me the examples! </li></ul></ul><ul><ul><li>Learnings </li></ul></ul>
  11. 11. Part 2: AutoScale <ul><ul><li>cmd line only = no UI = hidden = unknown = scary </li></ul></ul><ul><ul><li>basically: </li></ul></ul><ul><ul><li>launching & terminating instances </li></ul></ul><ul><ul><li>within some bounds </li></ul></ul><ul><ul><li>based upon metrics </li></ul></ul><ul><ul><li>a second order system :) </li></ul></ul>
  12. 12. AutoScale concepts <ul><li>1. Launch Configuration </li></ul><ul><li>individual: instance type, AMI, user-data, etc </li></ul>2. AutoScale Group collective: ideal/max/min, AZs 3. Policy how: a reaction, increase, decrease 4. Alerts when: an event, high/low load, CW metrics
  13. 13. Our (common) problem <ul><ul><li>maintain consistent experience for users, ie. Pip load time </li></ul></ul><ul><ul><li>turn on servers when load is high </li></ul></ul><ul><ul><li>bootstrapped, must turn off servers when load is low! </li></ul></ul><ul><ul><li>enter AutoScale... </li></ul></ul>
  14. 14. Solution = AutoScale <ul><ul><li>servers start/stop based upon current system state </li></ul></ul><ul><ul><li>spending on IT matches user demand exactly! </li></ul></ul><ul><ul><li>4 commands to set it up... </li></ul></ul>
  15. 15. 1/4: Launch Configuration <ul><li>as-create-launch-config gorilla_config_7d2 </li></ul><ul><li>--image-id ami-XXXXXXX </li></ul><ul><li>         --instance-type t1.micro </li></ul><ul><li>--group &quot;gorilla-live&quot;  </li></ul><ul><li>         --key indexmedia-live </li></ul><ul><li>--user-data-file /path/to/bootstrap.sh </li></ul>
  16. 16. 2/4: Group <ul><li>as-create-auto-scaling-group as_group </li></ul><ul><li>--launch-configuration gorilla_config_7d2 </li></ul><ul><li>--availability-zones us-west-1a us-west-1b </li></ul><ul><li>--min-size 10 --max-size 20 </li></ul><ul><li>--load-balancers my_balancer </li></ul><ul><li>--health-check-type EC2 </li></ul><ul><li>         --grace-period 120 </li></ul>
  17. 17. 3/4: Policy <ul><li>as-put-scaling-policy gorilla_scaleup_policy </li></ul><ul><li>--auto-scaling-group as_group  </li></ul><ul><li>         --type ChangeInCapacity  </li></ul><ul><li>--adjustment=1 </li></ul>
  18. 18. 4/4: Alarm <ul><li>mon-put-metric-alarm &quot;gorilla high load&quot; </li></ul><ul><li>     --dimensions &quot;AutoScalingGroupName=as_group&quot; </li></ul><ul><li>     --namespace &quot;AWS/EC2&quot; </li></ul><ul><li>     --statistic Average --metric-name CPUUtilization </li></ul><ul><li>     --comparison-operator GreaterThanThreshold </li></ul><ul><li>     --threshold 80 </li></ul><ul><li>     --evaluation-periods 5 --period 60 </li></ul><ul><li>     --alarm-actions ${POLICY-ARN} </li></ul>
  19. 19. ... and done! Learnings <ul><ul><li>4 steps (well 6...) = elastic system </li></ul></ul><ul><ul><li>feedback = second order system </li></ul></ul><ul><ul><li>avoid bad harmonics! </li></ul></ul><ul><ul><li>shorter instance boot time == quicker system  response time </li></ul></ul><ul><ul><li>validate alarms & behaviour w/ load tests on AWS </li></ul></ul>
  20. 20. AutoScale summary <ul><ul><li>cmd line only; just jump in! </li></ul></ul><ul><ul><li>4 distinct concepts, one per cmd </li></ul></ul><ul><ul><li>quick boot = shorter grace period = quicker scaling up (enter user-data...) </li></ul></ul><ul><ul><li>key to our multi-AZ, HA elastic cluster, problem solved! </li></ul></ul>
  21. 21. Thanks + Questions? Adam Neumann Solutions Architect at IndexMedia [email_address]

×