Effective EC2
                          PyCon Italia Qu4ttro - May 7/8/9 2010
                                   Valentino...
Buzz
                               Why I’ll never own another server...
                                                 ...
Buzzwords
                                    Cost                          Reliability

                                 ...
Competition...



         Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
Google AppEngine

                   • Transparently scalable
                   • No maintenance headaches
              ...
Rackspace Cloud

                   • CloudServers/CloudFiles/CloudSites
                   • Some nice additional feature...
Others




         Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
Amazon AWS

                   • 4 independent regions - 10 data centers
                   • 3rd party vendor support
   ...
Amazon AWS

                               Amazon Web Services can be as big
                                    as our re...
AdRoll

                   • Scaling High ROI Display Advertising
                   • 60M paid campaigns impressions
    ...
AdRoll 2008

                   • ServePath Housing
                   • NaviSite CDN with mod_python
                   •...
AdRoll 2009
                   • ServePath Housing
                   • Amazon EC2 auto-managed AdServers
                ...
Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
AdRoll 2010

                   • Fully Amazon EC2
                   • Akamai with Amazon S3 as origin
                  ...
Campaigns/Reports/Plan




                                                                                               ...
AdRoll 2010

                                                    Network Uptime: 100%




               Avg Resp Time: 96...
Migration
                                              Phase I - May 2009




         Effective EC2 - Valentino Volonghi...
New Ad Servers

                   • Very Quick & faultless boot procedure
                   • Easy software upgrade
    ...
New Ad Servers

                   • Custom bundled AMI stored in S3
                   • Python deploys the AdServer on b...
Boto monitoring
                                         def regional_max_allocation(start_adserver=ec2.Image.start_adserv...
Migration
                                              Phase 2 - March 2010




         Effective EC2 - Valentino Volong...
Preparation

                   • SQLAlchemy is awesome
                   • PostgreSQL 5-10x faster than MySQL
          ...
Automated Deploy

                   • Stock Ubuntu AMI
                   • Setup scripts on S3
                   • Fabr...
Automated Deploy
                          Deploy DB
     #!/bin/bash -ex

     wget -O- run.alestic.com/install/runurl | ...
DB Instance

                   • m2.2xlarge - 34GB RAM - 4 VCPU
                   • 8 EBS RAID 1+0
                   • ...
Generic Instance

                   • m1.large - 7.5 GB RAM - 2 VCPU
                   • 1 EBS with 500GB space
        ...
Web Instance

                   • c1.medium - 1.7 GB RAM - 2 VCPU
                   • No EBS
                   • Amazon...
AWS Tips/Tricks



         Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
runurl
                  by Eric Hammond. Download and run any script from
                                     any URL.

...
ec2-consistent-snapshot
                    by Eric Hammond, atomic snapshot of EBS volumes
                              ...
Eric Hammond
                          If you want to know more about Amazon AWS,
                                  he’s t...
Use Public addresses
                 Public addresses are automatically resolved to internal
                            ...
Use ELB and
                                              Cloudwatch
                   Very useful for latency and load m...
EBS RAID
                 EBS is cheap. Instead of one big EBS create one big
                soft-RAID/LVM volume from ma...
SMTPAuth / SendGrid
                  Mail sending from Amazon EC2 was/is crippled. Use a
                       third par...
Use /etc/hosts
                Setting up a DNS is a lot of work at the beginning. Use
               SSH key names to set...
new_lines = ["127.0.0.1 localhost"]
                          for reservation in conn.get_all_instances():
               ...
AWS Wishlist



         Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
Elastic DNS Service
                          Did I say that setting up your own DNS is hard?




         Effective EC2 -...
Dedicated Rack
                                        Housing
                                Try to get a Netezza rack i...
Improve ELB
                                                  It’s a CNAME…
                                              ...
Thank you!



         Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
Answers?



         Effective EC2 - Valentino Volonghi
Wednesday, May 19, 2010
Upcoming SlideShare
Loading in...5
×

Effective EC2

985

Published on

Valentino Volonghi

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
985
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
10
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Effective EC2

  1. 1. Effective EC2 PyCon Italia Qu4ttro - May 7/8/9 2010 Valentino Volonghi - valentino@adroll.com Wednesday, May 19, 2010
  2. 2. Buzz Why I’ll never own another server... Joe Stump - Digg Lead Architect, CTO SimpleGeo http://stu.mp/2010/04/why-ill-never-own-another-server.html Cloud computing economies of scale James Hamilton - VP & Distinguished Engineer, Amazon Web Services http://live.visitmix.com/MIX10/Sessions/EX01 Moving to the cloud Reddit Team http://blog.reddit.com/2009/11/moving-to-cloud.html Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  3. 3. Buzzwords Cost Reliability Multi-tenancy Agility Scalability Location Independence Security Maintenance Metering Effective EC2 - Valentino Volonghi Source: Wikipedia Wednesday, May 19, 2010
  4. 4. Competition... Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  5. 5. Google AppEngine • Transparently scalable • No maintenance headaches • Software must be written ad hoc • Little control over infrastructure Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  6. 6. Rackspace Cloud • CloudServers/CloudFiles/CloudSites • Some nice additional features • Fewer datacenters and can’t pick them • 12/11/07, 09/07/08, 03/11/09 Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  7. 7. Others Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  8. 8. Amazon AWS • 4 independent regions - 10 data centers • 3rd party vendor support • Well integrated services accessible via API • “Poor” single-node performance • Flexible upper limits Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  9. 9. Amazon AWS Amazon Web Services can be as big as our retail business Andy Jassy - Amazon Senior VP Cloud Computing Business Revenue: Market Share: $650.000.000 77% Effective EC2 - Valentino Volonghi Source: www.businessweek.com/technology/content/apr2010/tc20100428_085106.htm Wednesday, May 19, 2010
  10. 10. AdRoll • Scaling High ROI Display Advertising • 60M paid campaigns impressions • 200M advertiser pixels impressions • 99% of requests below 3ms think time • Realtime Dynamic Ads Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  11. 11. AdRoll 2008 • ServePath Housing • NaviSite CDN with mod_python • Self-Hosted DNS • MySQL DB • NFS Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  12. 12. AdRoll 2009 • ServePath Housing • Amazon EC2 auto-managed AdServers • Amazon S3/CloudFront CDN • Dynect DNS Global Server Load Blancing • MySQL DB • NFS Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  13. 13. Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  14. 14. AdRoll 2010 • Fully Amazon EC2 • Akamai with Amazon S3 as origin • Dynect DNS GSLB • PostgreSQL DB • Amazon S3 instead of NFS Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  15. 15. Campaigns/Reports/Plan platforms 3rd party Syncing AdRoll.com Scheduling Plan Ads Akamai DB S3 Plan/ Log Processing Config AdServers Logs/Performance Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  16. 16. AdRoll 2010 Network Uptime: 100% Avg Resp Time: 96ms Effective EC2 - Valentino Volonghi Measured by Pingdom Wednesday, May 19, 2010
  17. 17. Migration Phase I - May 2009 Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  18. 18. New Ad Servers • Very Quick & faultless boot procedure • Easy software upgrade • Low latency network • Real time monitoring Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  19. 19. New Ad Servers • Custom bundled AMI stored in S3 • Python deploys the AdServer on boot • Hidden real load test • Boto monitoring Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  20. 20. Boto monitoring def regional_max_allocation(start_adserver=ec2.Image.start_adserver): zones = [zone.name for zone in ec2.bconn.get_all_zones()] (availables_with_ip, availables_without_ip, software_failures, startings) = get_instances_by_status() def setup_candidate(failure=None, free_ip=None): if availables_without_ip: new_instance = get_instance_in_unused_zone(availables_with_ip, availables_without_ip, lenient=True) if failure is not None: free_ip = failure.disassociate_elastic() availables_without_ip.remove(new_instance) availables_with_ip.add(new_instance) new_instance.associate_elastic(free_ip) return if startings: startings.pop() return start_adserver("adserver-" + utils.uuid(), zone=get_unused_zone(availables_with_ip, zones)) for software_failure in software_failures: setup_candidate(failure=software_failure) free_ips = ec2.get_all_free_elastic_ip(default=[]) for free_ip in free_ips: Effective EC2 - Valentino Volonghi setup_candidate(free_ip=free_ip) Wednesday, May 19, 2010
  21. 21. Migration Phase 2 - March 2010 Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  22. 22. Preparation • SQLAlchemy is awesome • PostgreSQL 5-10x faster than MySQL • 1 EBS is slower than 8 in RAID!! :) • m1.small instances are useless Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  23. 23. Automated Deploy • Stock Ubuntu AMI • Setup scripts on S3 • Fabric ➡ 5 minutes to boot a new instance ➡ No maintenance overhead Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  24. 24. Automated Deploy Deploy DB #!/bin/bash -ex wget -O- run.alestic.com/install/runurl | bash runurl "https://$base_s3/setup/add-keys-and-cert?SIGNATURE" runurl $base_s3/machine/db runurl $base_s3/deploy/db Deploy Web #!/bin/bash -ex wget -O- run.alestic.com/install/runurl | bash runurl "https://${base_s3}/setup/add-keys-and-cert?SIGNATURE" runurl $base_s3/machine/web Deploy App runurl $base_s3/deploy/web #!/bin/bash -ex runurl $base_s3/start/service production.ini paster wget -O- run.alestic.com/install/runurl | bash runurl "https://${base_s3}/setup/add-keys-and-cert?SIGNATURE" runurl $base_s3/machine/app runurl $base_s3/deploy/app runurl $base_s3/start/service production.ini watchwoman_inotify watchwoman_pixels watchwoman_amqp spaceman crawlman billman generalman turtl quexecutor sphinx Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  25. 25. DB Instance • m2.2xlarge - 34GB RAM - 4 VCPU • 8 EBS RAID 1+0 • PostgreSQL 8.4 log shipping to S3 • RAID snapshot every 6 hours • Nightly SQL Backup Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  26. 26. Generic Instance • m1.large - 7.5 GB RAM - 2 VCPU • 1 EBS with 500GB space • Store frequently changed data • Exact copies of each other Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  27. 27. Web Instance • c1.medium - 1.7 GB RAM - 2 VCPU • No EBS • Amazon ELB frontend • Easily replaceable • Logs aggregated separately in real time Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  28. 28. AWS Tips/Tricks Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  29. 29. runurl by Eric Hammond. Download and run any script from any URL. Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  30. 30. ec2-consistent-snapshot by Eric Hammond, atomic snapshot of EBS volumes with xfs filesystem. Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  31. 31. Eric Hammond If you want to know more about Amazon AWS, he’s the one you want to talk to. Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  32. 32. Use Public addresses Public addresses are automatically resolved to internal addresses when possible. Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  33. 33. Use ELB and Cloudwatch Very useful for latency and load monitoring… It also makes scaling the web frontend extremely easy! Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  34. 34. EBS RAID EBS is cheap. Instead of one big EBS create one big soft-RAID/LVM volume from many of them, it’s faster and safer. Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  35. 35. SMTPAuth / SendGrid Mail sending from Amazon EC2 was/is crippled. Use a third party service to improve deliverability. Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  36. 36. Use /etc/hosts Setting up a DNS is a lot of work at the beginning. Use SSH key names to setup a custom /etc/hosts file in each instance. Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  37. 37. new_lines = ["127.0.0.1 localhost"] for reservation in conn.get_all_instances(): for instance in reservation.instances: if instance.state == "running": KEYS.discard(instance.key_name) new_lines.append( "%s %s.internal %s" % ( socket.gethostbyname(instance.public_dns_name), instance.key_name, instance.key_name) ) for missing_key in KEYS: new_lines.append( "127.0.0.1 %s.internal %s" % (missing_key, missing_key) ) f = open('/etc/hosts.new', 'wb') f.write("n".join(new_lines)) f.write("n") f.flush() f.close() shutil.move('/etc/hosts.new', '/etc/hosts') Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  38. 38. AWS Wishlist Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  39. 39. Elastic DNS Service Did I say that setting up your own DNS is hard? Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  40. 40. Dedicated Rack Housing Try to get a Netezza rack in AWS today... Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  41. 41. Improve ELB It’s a CNAME… No HTTPS termination Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  42. 42. Thank you! Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  43. 43. Answers? Effective EC2 - Valentino Volonghi Wednesday, May 19, 2010
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×