Five Years of EC2 Distilled
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Five Years of EC2 Distilled

on

  • 4,981 views

My experiences running large-scale system infrastructures in EC2 and in traditional data centers, with lessons learned the hard way.

My experiences running large-scale system infrastructures in EC2 and in traditional data centers, with lessons learned the hard way.

Statistics

Views

Total Views
4,981
Views on SlideShare
4,831
Embed Views
150

Actions

Likes
8
Downloads
38
Comments
1

3 Embeds 150

https://twitter.com 137
https://www.rebelmouse.com 12
http://www.twylah.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • excellent summary & tips with real case studies & actual experience. Thanks for sharing this & tips. www.attribo.com
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Five Years of EC2 Distilled Presentation Transcript

  • 1. Five years of EC2 distilled Grig GheorghiuSilicon Valley Cloud Computing Meetup, Feb. 19th 2013 @griggheo agiletesting.blogspot.com
  • 2. whoami• Dir of Technology at Reliam (managed hosting)• Sr Sys Architect at OpenX• VP Technical Ops at Evite• VP Technical Ops at Nasty Gal
  • 3. EC2 creds• Started with personal m1.small instance in 2008• Still around!• UPTIME:• 5:13:52 up 438 days, 23:33, 1 user, load average: 0.03, 0.09, 0.08
  • 4. EC2 at OpenX• end of 2008• 100s then 1000s of instances• one of largest AWS customers at the time• NAMING is very important • terminated DB server by mistake • in ideal world naming doesn’t matter
  • 5. EC2 at OpenX (cont.)• Failures are very frequent at scale• Forced to architect for failure and horizontal scaling• Hard to scale at all layers at the same time (scaling app server layer can overwhelm DB layer; play wack-a-mole)• Elasticity: easier to scale out than scale back
  • 6. EC2 at OpenX (cont.)• Automation and configuration management become critical • Used little-known tool - ‘slack’ • Rolled own EC2 management tool in Python, wrapped around EC2 Java API • Testing deployments is critical (one mistake can get propagated everywhere)
  • 7. EC2 at OpenX (cont.)• Hard to scale at the DB layer (MySQL) • mysql-proxy for r/w split • slaves behind HAProxy for reads• HAProxy for LB, then ELB • ELB melted initially, had to be gradually warmed up
  • 8. EC2 at Evite• Sharded MySQL at DB layer; application very write-intensive• Didn’t do proper capacity planning/dark launching; had to move quickly from data center to EC2 to scale horizontally• Engaged Percona at the same time
  • 9. EC2 at Evite (cont.)• Started with EBS volumes (separate for data, transaction logs, temp files)• EBS horror stories• CPU Wait up to 100%, instances AWOL• I/O very inconsistent, unpredictable• Striped EBS volumes in RAID0 helps with performance but not with reliability
  • 10. EC2 at Evite (cont.)• EBS apocalypse in April 2011• Hit us even with masters and slaves in diff. availability zones (but all in single region - mistake!)• IMPORTANT: rebuilding redundancy into your system is HARD• For DB servers, reloading data on new server is a lengthy process
  • 11. EC2 at Evite (cont.)• General operation: very frequent failures (once a week); nightmare for pager duty• Got very good at disaster recovery! • Failover of master to slave • Rebuilding of slave from master (xtrabackup)• Local disks striped in RAID0 better than EBS
  • 12. EC2 at Evite (cont.)• Ended up moving DB servers back to data center• Bare metal (Dell C2100, 144 GB RAM, RAID10); 2 MySQL instances per server• Lots of tuning help from Percona• BUT: EC2 was great for capacity planning! (Zynga does the same)
  • 13. EC2 at Evite (cont.)• Relational databases are not ready for the cloud (reliability, I/O performance)• Still keep MySQL slaves in EC2 for DR• Ryan Macktechnologies so“Wecould better understood (Facebook): we chose well- predict capacity needs and rely on our existing monitoring and operational tool kits."
  • 14. EC2 at Evite (cont.)• Didn’t use provisioned IOPS for EBS• Didn’t use VPC• Great experience with Elastic Map Reduce, S3, Route 53 DNS• Not so great experience with DynamoDB• ELB OK but still need HAProxy behind it
  • 15. EC2 at NastyGal• VPC - really good idea! • Extension of data center infrastructure • Currently using it for dev/staging + some internal backend production • Challenging to set up VPN tunnels to various firewall vendors (Cisco, Fortinet) - not much debugging on VPC side
  • 16. Interacting with AWS• AWS API (mostly Java based, but also Ruby and Python)• Multi-cloud libraries: jclouds (Java), libcloud (Python), deltacloud (Ruby)• Chef knife• Vagrant EC2 provider• Roll your own
  • 17. Proper infrastructure care and feeding• Monitoring - alerting, logging, graphing• It’s not in production if it’s not monitored and graphed• Monitoring is for ops what testing is for dev • Great way to learn a new infrastructure • Dev and ops on pager
  • 18. Proper infrastructure care and feeding• Going from #monitoringsucks to #monitoringlove and @monitorama• Modern monitoring/graphing/logging tools • Sensu, Graphite, Boundary, Server Density, New Relic, Papertrail, Pingdom, Dead Man’s Snitch
  • 19. Proper infrastructure care and feeding• Dashboards!• Mission Control page with graphs based on Graphite and Google Visualization API• Correlate spikes and dips in graphs with errors (external and internal monitoring) • Akamai HTTP 500 alerts correlated with Web server 500 errors and DB server I/O wait increase
  • 20. Proper infrastructure care and feeding• HTTP 500 errors as a percentage of all HTTP requests across all app servers in the last 60 minutes
  • 21. Proper infrastructure care and feeding• Expect failures and recover quickly• Capacity planning • Dark launching • Measure baselines • Correlate external symptoms (HTTP 500) with metrics (CPU I/O Wait) then keep metrics under certain thresholds by adding resources
  • 22. Proper infrastructure care and feeding• Automate, automate, automate! - Chef, Puppet, CFEngine, Jenkins, Capistrano, Fabric• Chef - can be single source of truth for infrastructure • Running chef-client continuously on nodes requires discipline • Logging into remote node is anti-pattern (hard!)
  • 23. Proper infrastructure care and feeding• Chef best practices • Use knife - no snowflakes! • Deploy new nodes, don’t do massive updates in place• BUT! beware of OS monoculture • kernel bug after 200+ days • leapocalypse
  • 24. Is the cloud worth the hype?• It’s a game changer, but it’s not magical; try before you buy! (benchmarks could surprise you)• Cloud expert? Carry pager or STFU• Forces you to think about failure recovery, horizontal scalability, automation• Something to be said about abstracting away the physical network - the most obscure bugs are network-related (ARP caching, routing tables)
  • 25. So...when should I use the cloud?• Great for dev/staging/testing• Great for layers of infrastructure that contain many identical nodes and that are forgiving of node failures (web farms, Hadoop nodes, distributed databases)• Not great for ‘snowflake’-type systems• Not great for RDBMS (esp. write-intensive)
  • 26. If you still want to use the cloud• Watch that monthly bill!• Use multiple cloud vendors• Design your infrastructure to scale horizontally and to be portable across cloud vendors • Shared nothing • No SAN, NAS
  • 27. If you still want to use the cloud• Don’t get locked into vendor-proprietary services • EC2, S3, Route 53, EMR are OK • Data stores are not OK (DynamoDB) • OpsWorks - debatable (based on Chef, but still locks you in) • Wrap services in your own RESTful endpoints
  • 28. Does EC2 have rivals?• No (or at least not yet)• Anybody use GCE?• Other public clouds are either toys or smaller, with less features (no names named)• Perception matters - not a contender unless featured on High Scalability blog• APIs matter less (can use multi-cloud libs)
  • 29. Does EC2 have rivals?• OpenStack, CloudStack, Eucalyptus all seem promising• Good approach: private infrastructure (bare metal, private cloud) for performance/ reliability + extension into public cloud for elasticity/agility (EC2 VPC, Rack Connect)• How about PaaS? • Personally: too hard to relinquish control