Amazon Aws Presentation Drupal


Published on

Amazon Aws Presentation Drupal

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Amazon Aws Presentation Drupal

  1. 1. Amazon Web Services <ul><ul><li>An overview of Amazon's Web Services, and how one can put them to use </li></ul></ul><ul><ul><li>June 28, 2007 </li></ul></ul><ul><ul><li>Alex Harford </li></ul></ul><ul><ul><li>[email_address] </li></ul></ul>
  2. 2. Overview of Amazon Web Services <ul><li>MTurk: Mechanical Turk </li></ul><ul><li>SQS: Simple Queue Service </li></ul><ul><li>S3: Simple Storage Service </li></ul><ul><li>EC2: Elastic Compute Cloud </li></ul>
  3. 3. Overview of Amazon Web Services <ul><li>Why do I like it? </li></ul><ul><ul><li>Cross platform support: Libraries in Python, Java, etc </li></ul></ul><ul><ul><ul><li>Or roll your own through REST and SOAP </li></ul></ul></ul><ul><ul><li>Enthusiastic developer community: Message boards are very active </li></ul></ul><ul><ul><li>Pay as you go, for as much or as little as you need </li></ul></ul>
  4. 4. Mechanical Turk <ul><li>There is a story of the Mechanical Turk, a chess playing automaton, that toured Europe in the 18 th century. It beat many famous people, including Napoleon Bonaparte and Benjamin Franklin. </li></ul>
  5. 5. Mechanical Turk <ul><li>It was a hoax, there was a person hidden inside that controlled the Automaton! </li></ul>
  6. 6. Mechanical Turk <ul><li>Amazon's offering is similar, it uses 'Artificial Artificial Intelligence'. </li></ul><ul><li>Submitters enter “Human Intelligence Tasks” into the queue and Workers solve them. </li></ul><ul><li>Workers can be tested for capabilities before they are trusted. </li></ul><ul><li>Submitters can refuse results if they find them unsatisfactory. </li></ul><ul><li>There is a feedback system in place so Workers know if the Submitters will pay. </li></ul>
  7. 7. Mechanical Turk <ul><li>Some example uses are: </li></ul><ul><ul><li>podcast transcription services </li></ul></ul><ul><ul><li>image recognition for mapping service </li></ul></ul><ul><ul><li>Amazon has used it themselves for internal projects </li></ul></ul>
  8. 8. SQS: Simple Queue Service <ul><li>From their website: Amazon Simple Queue Service (Amazon SQS) offers a reliable, highly scalable hosted queue for storing messages as they travel between computers. By using Amazon SQS, developers can simply move data between distributed application components performing different tasks, without losing messages or requiring each component to be always available. </li></ul>
  9. 9. SQS: Simple Queue Service <ul><li>What can you do? </li></ul><ul><ul><li>CreateQueue </li></ul></ul><ul><ul><li>ListQueue </li></ul></ul><ul><ul><li>DeleteQueue </li></ul></ul><ul><ul><li>SendMessage </li></ul></ul><ul><ul><li>ReceiveMessage </li></ul></ul><ul><ul><li>DeleteMessage </li></ul></ul><ul><ul><li>PeekMessage </li></ul></ul><ul><ul><li>SetVisibilityTimeout </li></ul></ul><ul><ul><li>AddGrant </li></ul></ul>
  10. 10. SQS: Simple Queue Service <ul><li>Made for communicating to your various web services. </li></ul><ul><li>Accessible through SOAP and REST APIs </li></ul><ul><li>How much does it cost? </li></ul><ul><ul><li>Pay only for what you use. There is no minimum fee, and no start-up cost. </li></ul></ul><ul><ul><li>$0.10 per 1,000 messages sent ($0.0001 per message sent) </li></ul></ul><ul><ul><li>$0.20 per GB of data transferred </li></ul></ul>
  11. 11. SQS: Simple Queue Service <ul><li>CreateQueue: Create queues for your own use, or to share with others. </li></ul><ul><li>ListQueues: List your existing queues. </li></ul><ul><li>DeleteQueue: Delete one of your queues. </li></ul><ul><li>SendMessage: Add any data entry to a specified queue. </li></ul><ul><li>ReceiveMessage: Return one or more messages from a specified queue, which are returned in roughly the same order it was added to the queue. </li></ul>
  12. 12. SQS: Simple Queue Service <ul><li>DeleteMessage: Remove a message from a specified queue. </li></ul><ul><li>PeekMessage: Return a specific entry from the queue without locking it. </li></ul><ul><li>SetVisibilityTimeout: Control the amount of time after a message has been read that is locked from being read again. </li></ul><ul><li>AddGrant: Allow other users to send messages to or receive messages from your queue. </li></ul>
  13. 13. S3: Simple Storage Service <ul><li>Simple Storage Service lets you store data in 'buckets' </li></ul><ul><li>Cheap! </li></ul><ul><ul><li>$0.15 per GB-Month of storage used. </li></ul></ul><ul><ul><li>$0.10 per GB of data uploaded. </li></ul></ul><ul><ul><li>$0.18 per GB of data downloaded (price breaks for volume over 10TB) </li></ul></ul><ul><ul><li>$0.01 per 1000 PUT or LIST requests </li></ul></ul><ul><ul><li>$0.01 per 10,000 GET requests </li></ul></ul>
  14. 14. S3: Simple Storage Service <ul><li>S3 is NOT a filesystem </li></ul><ul><ul><li>You store your data in buckets </li></ul></ul><ul><ul><ul><li>Bucket names are globally unique! </li></ul></ul></ul><ul><ul><li>Individual chunks of data have a 'key' associated with them </li></ul></ul><ul><ul><li>Keys can be 4KB long </li></ul></ul><ul><ul><li>Chunks of data are a max of 5GB </li></ul></ul>
  15. 15. S3: Simple Storage Service <ul><li>Permissions can be set on buckets and objects to be public, private, or for individual users. </li></ul><ul><li>Files can be accessed over HTTP or BitTorrent! </li></ul><ul><li>Data is replicated across multiple datacenters. </li></ul><ul><ul><li>The more it is accessed, the more it is replicated across different datacenters. </li></ul></ul>
  16. 16. S3: Simple Storage Service <ul><li>Accessing S3: </li></ul><ul><ul><li>HTTP </li></ul></ul><ul><ul><li>BitTorrent </li></ul></ul><ul><ul><li>Python API </li></ul></ul><ul><ul><li>Java API </li></ul></ul><ul><ul><li>Linux FUSE (ok, so it can look like a filesystem) </li></ul></ul><ul><ul><li>s3sync, an rsync like program </li></ul></ul><ul><ul><li>JungleDisk </li></ul></ul><ul><ul><li>S3 Organizer for Firefox </li></ul></ul><ul><ul><li>MySQL storage backend (very preliminary!) </li></ul></ul>
  17. 17. S3: Simple Storage Service <ul><li>How do you run a website from S3? </li></ul><ul><ul><li>#1 caveat: No index.html support </li></ul></ul><ul><ul><li>So it is only useful to offload traffic and storage from your main site. </li></ul></ul>
  18. 18. S3: Simple Storage Service <ul><li>Set up a bucket 'vanlug' </li></ul><ul><li>Store a key in it 'tux.png' </li></ul><ul><ul><li>Make it publicly accessible. </li></ul></ul><ul><li>Set up a CNAME '' and point it to '' </li></ul><ul><li>Load up </li></ul><ul><li>That's it! </li></ul>
  19. 19. S3: Simple Storage Service <ul><li>BitTorrent: </li></ul><ul><ul><li>remember our Tux image? </li></ul></ul><ul><ul><li> </li></ul></ul><ul><ul><li>Turn it into a torrent: </li></ul></ul><ul><ul><ul><li> ?torrent </li></ul></ul></ul><ul><li>Done. Amazon creates a tracker for you! </li></ul><ul><li>To save bandwidth, after you have served the file a few times, hide the original file. Amazon will keep hosting it for you, and the people sharing the file will seed it for you. </li></ul>
  20. 20. S3: Simple Storage Service <ul><li>JungleDisk </li></ul><ul><ul><li>This application lets you mount S3 like it is a local drive. </li></ul></ul><ul><ul><li>Cross platform, supports Linux, Mac, Windows </li></ul></ul><ul><ul><li>Encryption can be pre or post upload </li></ul></ul>
  21. 21. S3: Simple Storage Service <ul><li>Who is using it? </li></ul><ul><ul><li>Amazon (duh) </li></ul></ul><ul><ul><li>SmugMug </li></ul></ul><ul><ul><li>Linden Labs (Second Life) </li></ul></ul>
  22. 22. S3: Simple Storage Service <ul><li>Drawbacks: </li></ul><ul><ul><li>S3 cannot have data POSTed from clients, so if users are to upload data, they need to send it to another webserver first. This means that the data is transmitted twice to get into S3. </li></ul></ul><ul><ul><li>Difficult to track per key bandwidth. </li></ul></ul><ul><ul><li>Bandwidth cannot be throttled. </li></ul></ul>
  23. 23. EC2: Elastic Compute Cloud <ul><li>This is what I'm most excited about! </li></ul><ul><li>What is it? </li></ul><ul><ul><li>An on-demand Linux cluster </li></ul></ul><ul><ul><li>Uses Xen technology </li></ul></ul><ul><ul><li>'Instances' are stored on S3, you request an instance, and an IP address / hostname is returned </li></ul></ul><ul><ul><li>Basically 1.7Ghz x86 processor, 1.75GB of RAM, 160GB of local disk, and 250Mb/s of network bandwidth. </li></ul></ul><ul><ul><li>Traffic between different EC2 instances and S3 is free! Even if they are not in the same datacenter! </li></ul></ul>
  24. 24. EC2: Elastic Compute Cloud <ul><li>How much does it cost? </li></ul><ul><ul><li>$0.10 per instance-hour consumed (or part of an hour consumed). </li></ul></ul><ul><ul><li>Same prices for traffic and storage as S3 </li></ul></ul>
  25. 25. EC2: Elastic Compute Cloud <ul><li>What can you do with it? </li></ul><ul><ul><li>You are root, do anything! </li></ul></ul><ul><ul><li>Install software: </li></ul></ul><ul><ul><ul><li>Apache </li></ul></ul></ul><ul><ul><ul><li>VPN </li></ul></ul></ul><ul><ul><ul><li>UT2004 / CounterStrike / Call of Duty </li></ul></ul></ul><ul><ul><ul><li>Jabber </li></ul></ul></ul><ul><ul><ul><li>Asterisk </li></ul></ul></ul><ul><ul><li>Load kernel modules </li></ul></ul>
  26. 26. EC2: Elastic Compute Cloud <ul><li>Creating a custom Instance </li></ul><ul><ul><li>Boot a current instance </li></ul></ul><ul><ul><li>Customize it </li></ul></ul><ul><ul><li>Bundle it </li></ul></ul><ul><ul><li>Save it to S3 </li></ul></ul><ul><ul><li>Register it </li></ul></ul><ul><ul><li>Boot it </li></ul></ul>
  27. 27. EC2: Elastic Compute Cloud <ul><li>One Step Further: Creating custom instances without repackaging </li></ul><ul><ul><li>Pass an environment variable in that indicates: </li></ul></ul><ul><ul><ul><li>filename of a tarball stored on S3 </li></ul></ul></ul><ul><ul><ul><li>what SQS queue to grab startup data from </li></ul></ul></ul><ul><ul><ul><li>what Subversion repository to grab data from </li></ul></ul></ul><ul><ul><ul><li>what startup file to use </li></ul></ul></ul>
  28. 28. EC2: Elastic Compute Cloud <ul><li>The Ephemeral Nature </li></ul><ul><ul><li>Your data is not stored anywhere permanent </li></ul></ul><ul><ul><li>Instances can be rebooted, preserving data </li></ul></ul><ul><ul><li>When they are terminated, they are gone forever </li></ul></ul><ul><ul><li>Hardware failures are rare but they can occur (just like on any server) </li></ul></ul><ul><li>What to do? </li></ul><ul><ul><li>S3 FUSE </li></ul></ul><ul><ul><li>MySQL replication </li></ul></ul><ul><ul><li>other failover systems </li></ul></ul>
  29. 29. EC2: Elastic Compute Cloud <ul><li>How To Access EC2 </li></ul><ul><ul><li>command line tools </li></ul></ul><ul><ul><li>EC2UI for Firefox </li></ul></ul><ul><ul><li> </li></ul></ul>
  30. 30. Questions?
  31. 31. More Info? <ul><li>Presentation and links are available on my blog: </li></ul><ul><li>Contact me: </li></ul>