The document provides an overview of Amazon Web Services, describing various services including Simple Storage Service (S3) for storage, Simple Queue Service (SQS) for message queuing, SimpleDB for database services, Elastic Compute Cloud (EC2) for compute capacity, Mechanical Turk for human intelligence tasks, and other payment and marketplace services. It discusses features, usage examples, and pricing for each service.

1. Amazon Web Services – Introduction & Discussion

2. Amazon Web Services – Overview of Amazon's Services Explore each type of service Not implementation specific [1] Open Discussion [1] http://highscalability.com/amazon-architecture

3. Amazon Web Services – The Picture Associates Alexa Marketplace DevPay WebStore Prime Grocery Historical Pricing [1] http://highscalability.com/paper-dynamo-amazon-s-highly-available-key-value-store

4. Small Pieces Loosely Joined – “ ...There are intriguing assertions: that the Web is "broken on purpose" and that its many pockets of erroneous information and its available forums [...] let people feel more comfortable with their own inherent imperfections . [...] Weinberger's analysis, though occasionally facile and too relentlessly optimistic and overstated , is surely destined to be the subject of furious debate in chat rooms the cyber-world over. --H. O'Billovich” [1] [1] http://www.amazon.com/Small-Pieces-Loosely-Joined-Unified/dp/B00008NRGI/ref=pd_bbs_sr_1?ie=UTF8&s=books&qid=1204731761&sr=8-1 Amazon Web Services – “Defined”

5. Small Pieces Loosely Joined Think Unix Model – “ Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.” -- Doug McIlroy [1] [1] [Salus] Peter H. Salus. A Quarter-Century of Unix. Addison-Wesley. 1994. ISBN 0-201-54777-5 Amazon Web Services – “Defined”

6. Small Pieces Loosely Joined Think Unix Model “ Single” Deployment Embrace the Power of Simplicity Open Collaboration Leverages “Momentum Makers” “ Force Multiplication” Estimates of 40% of sales from affiliates [1] Amazon Web Services – “Defined” [1] http://en.wikipedia.org/wiki/Amazon.com

7. Amazon Web Services – From the Ground Up Provides a Storage Service – Since April 2006 [1] Geographically distributed content network U.S. and European Union REST and SOAP interfaces over HTTP A BitTorrent(TM) protocol interface is provided Does have a Service Level Agreement [2] 25% for < 99% 10% for > 99% and < 99.9% “ 3 nines” implies <= 8 hours 46 minutes down time [1] http://en.wikipedia.org/wiki/Amazon_S3 [2] http://www.amazon.com/gp/browse.html%3fnode%3d379654011&token=3A0F170E7CEFE27BDC730D3D7344512BC1296B83

8. Uses – Extensive use internally SmugMug [1] Many independent blogs & “startups” [2] Even Firefox extensions as interfaces ! Cost – Less then you'd think [3] U.S. – $0.15 per GB/month In: $0.10 /GB, Out: graduated $0.13 - 0.18 GB/month $.01 per 1,000 PUT or 10,000 GET requests 20GB & 2GB is ~$40 per year [4] Webmail.us reportedly saved 75% with S3 No cost to transfer between Amazon Services ! [1] http://blogs.smugmug.com/don/2006/11/10/amazon-s3-show-me-the-money/ [2] http://calculator.s3.amazonaws.com/calc5.html [3] http://www.codinghorror.com/blog/archives/000808.html [4] http://www.jungledisk.com/ Amazon Web Services – From the Ground Up

9. Amazon Web Services – From the Ground Up Files (objects) are placed in “buckets” Unlimited objects in a bucket Buckets host objects between 1byte to 5GB Objects can be public, private or specific rights Retrieved via a unique key – global address space Typically linked with an MD5sum Subdirectories are done via strategic naming “ jay” and “jay/photos” are logically independent but might be related at an application level Another interesting solution involves loopback mounts [1] [1] http://developer.amazonwebservices.com/connect/thread.jspa?threadID=10271&tstart=30

10. Examples: CLI [1] ./s3-get -k MyAWSAccessKeyId -s AWSSecretAccessKeyIdFile /bucket/pathtoobject Python [2] >>> from boto.s3.connection import S3Connection, Key >>> conn = S3Connection('<aws access key>', '<aws secret key>') >>> bucket = conn.create_bucket('garnaat_fileit') >>> k = Key(bucket) >>> k.key = 'foobar' >>> k.set_contents_from_string('This is a test of S3') >>> k.ghet_contents_as_string() >>> k.key = 'myfile' >>> k.set_contents_from_filename('foo.jpg') >>> k.get_contents_to_filename('bar.jpg') [1] http://code.google.com/p/s3-bash/ [2] http://boto.googlecode.com/svn/trunk/doc/s3_tut.txt Amazon Web Services – From the Ground Up

11. Amazon Web Services – Redefining the Dictionary ... if databases were straightforward and easy Web Services interface – Queries and Submissions No schema – Automatic indexes Loose enforcement of structure Consistency is “eventual” (MVCC) [1] Automatic scaling & caching No configuration & management required For “real-time data” applications; e.g. shopping carts Heavily reliant on S3 [1] http://en.wikipedia.org/wiki/Multiversion_concurrency_control http://www.sriramkrishnan.com/blog/2007/12/amazon-simpledb-technical-overview.html

12. CREATE a domain to store data Beta constraint – 100 domains max 10 GB each GET, PUT or DELETE items Items have attribute-value pairs “ size = large” “ color = blue, red” Each item can have up to 256 attribute values. Each attribute value can range from 1 to 1,024 bytes. QUERY – max 5 seconds Operators: =, !=, <, > <=, >=, STARTS-WITH, AND, OR, NOT, INTERSECTION AND UNION. http://www.amazon.com/b/ref=sc_fe_l_2?ie=UTF8&node=342335011&no=3435361&me=A36L942TSJ2AJA Amazon Web Services – Redefining the Dictionary

13. Provides a Storage Queue – Simple! For $1 a user can transmit over 500,000 messages [1] $0.01 per 10,000 Amazon SQS Requests ($0.000001 per Request) Other bandwidth fees still apply (similar to S3 #'s) Message Characteristics Max messages size - 8KB Latency should be anticipated [2] Transmit and Receive order not guaranteed Messages held for 4 days max Data Flow – Find an Amazon SQS queue SendMessage to the queue ReceiveMessage is called ReceiveMessage returns a message Locks that message until the visibility timeout has passed. DeleteMessage removes the message from the queue [1] http://www.amazon.com/Simple-Queue-Service-home-page/b?ie=UTF8&node=13584001 [2] http://en.wikipedia.org/wiki/Amazon_SQS Amazon Web Services – “Ground Control to Major Tom...”

14. Example: Python [1] >>> from boto.sqs.connection import SQSConnection, Message >>> conn = SQSConnection('<aws access key>', '<aws secret key>') >>> q = conn.create_queue('myqueue', 60) >>> m = Message() >>> m.set_body('This is my first message.') >>> rs = q.write(m) >>> rs = q.get_messages() >>> rs[0].get_body() >>> q.delete_message(rs[0]) [1] http://boto.googlecode.com/svn/trunk/doc/sqs_tut.txt [2] http://aws.typepad.com/aws/2007/05/sqs_super_queue.html Amazon Web Services – “Ground Control to Major Tom...”

15. Amazon Web Services – “Show me the Money” Supports multiple forms of payment Permits true micro-payment [1] Costs vary based on Amazon's costs Amazon manages security and fraud responsibility “ GateKeeper” enforces policies for senders and receivers Explicit “Payment Instructions” language “ only five transactions or $50 per month&quot; or &quot;refuse payment after August 1&quot; Three token exchange Integrates with Amazon's existing customer experience “ Tens of Millions of existing customers” “ Pay Now” Widget Promotes “market places” for intermediaries [1] http://paulstamatiou.com/2007/08/04/why-you-shouldnt-ignore-amazons-new-fps [2] http://www.amazon.com/gp/browse.html?node=342430011 [3] http://docs.amazonwebservices.com/AmazonFPS/2007-01-08/FPSGettingStartedGuide/

16. Amazon Web Services – The Sky's the Limit [1] http://www.amazon.com/b/?node=201590011 Provides programmatic compute capacity Including pre-configured appliance images “ ... allows you to obtain and configure capacity with minimal friction” [1] The Basic Process Build or Identify an appropriate AMI Configure Security and Access Start, Monitor and Stop Instances Pay

17. Amazon Web Services – The Sky's the Limit What's an AMI ? - Amazon Machine Instance “ Golden Master” of the image you want to run Linux only, Windows is a hack [1] AMI's can be private [2] – Vendor tools like rPath [3] Leverage an Open “Operating Systems” bazaar Some AMI's can have “associate” costs per instance Multiple Instances can run from a single AMI By default, limited to 20, simple request for more Getting started with EC2 [4] Mike Culver - Introductory video [5] [1] http://www.enomaly.net/wiki.1137+M5c1693b2f05.0.html [2]http://docs.amazonwebservices.com/AmazonEC2/dg/2006-06-26/creating-an-ami.html [3] http://highscalability.com/product-rpath-creating-and-managing-virtual-appliances [4] http://docs.amazonwebservices.com/AWSEC2/2007-08-29/DeveloperGuide/ [5] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=583&categoryID=100

18. Amazon Web Services – The Sky's the Limit Example: > ec2-describe-images -x all IMAGE ami-23b6534a ec2-public-images/fedora-7-apache-manafest.xml > ec2-add-keypair j-keypair (generates ssh keypairs) > ec2-run-instance ami-23b6534a -k j-keypair (could take up to a few minutes) > ec2-authorize default -p 22 (Authorize port 22 for the default security group) > ec2-describe-instances (Shows instance info including DNS name) > ec2-terminate-instances <image name> > ec2run ami-e3a5408a -n 20 -g appserver

19. Amazon Web Services – The Sky's the Limit [1] http://www.amazon.com/b/?node=201590011 [2] http://www.amazon.com/b/?node=201590011#measure [3] http://www.amazon.com/b/ref=sc_fe_c_0_201590011_2?ie=UTF8&node=370375011&no=201590011&me=A36L942TSJ2AJA Small Instance - $.10 / hour 1.7 GB of memory, 1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit) 160 GB of instance storage, 32-bit platform Large Instance - $0.40 / hour 7.5 GB of memory 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each) 850 GB of instance storage, 64-bit platform Extra Large Instance - $0.80 / hour 15 GB of memory 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each) 1690 GB of instance storage, 64-bit platform 1 EC2 Compute Unit is ~ 1.0-1.2 GHz 2007 Opteron or 2007 Xeon Bandwidth costs of still apply at 256Mb / sec

20. Amazon Web Services – The Sky's the Limit Uses x509 certificates for Developer Auth. SSH keypairs for instance authenticaion Instances can belong to “security groups” Network is dynamic via NAT IP's and hostnames change Momentum RedHat is deploying a “cloud” on EC2 [1] Facebook [2] Hadoop [3] EC2 for Python Programmers [4] [1] http://www.redhat.com/solutions/cloud/ [2] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1044&ref=featured http://www.amazon.com/gp/browse.html?node=391557011 [3] http://blog.awswebshop.com/2008/02/27/taking-massive-distributed-computing-to-the-common-man-hadoop-on-amazon-ec2s3/ [4] http://jimmyg.org/2007/09/01/amazon-ec2-for-people-who-prefer-debian-and-python-over-fedora-and-java/

21. Amazon Web Services – From the Ground Up Amazon Mechanical Turk Launched in late 2005 [1] [1] http://www.egge.net/~savory/blog_mar_04.htm

22. Amazon Web Services – From the Ground Up Amazon Mechanical Turk Launched in late 2005 [1] “ Artificial Artificial Intelligence” - Jeff Bezos [1] http://en.wikipedia.org/wiki/Amazon.com

23. Amazon Web Services – From the Ground Up [1] http://www.amazon.com/gp/browse.html?node=15879911 [2] http://www.techcrunch.com/2007/09/08/search-for-steve-fossett-expands-to-amazons-mechanical-turk/ Humans augment computational requirements Let someone else solve complex AI problems Get along with the business of making money Built around Human Intelligence Tasks (HITs) Examples; “ Transcribe audio data” “ Are these paragraphs synonymous?” “ Find the business in this photo” “ Write a review...” “ Search for survivors” “ So far, searchers have found half a dozen previously unknown crash sites, but no sign of Fossett’s plane.” [2]

24. Amazon Web Services – The Example [1] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=691&categoryID=100 Mitch Garnaat Author of Bono “ DVS” transcoding Tutorial [1]

25. Read a message from the input queue - SQS Based on message, retrieve an input file - S3 Perform our video conversion processing - EC2 Store the generated files - S3 Write a message the the output queue - SQS Delete the input message from the input queue - SQS Amazon Web Services – The Example [1] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=691&categoryID=100

26. Amazon Web Services – The Example [1] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=691&categoryID=100 Input Message Bucket: garnaat_fileit InputKey: f84e4a20b571abc69baf2277d193e596 Date: Tue, 20 Feb 2007 17:21:21 GMT OriginalFileName: MVI_3113.AVI Size: 1126472 Output Message Bucket: garnaat_fileit InputKey: f84e4a20b571abc69baf2277d193e596 Date: Tue, 20 Feb 2007 17:21:21 GMT OriginalFileName: MVI_3113.AVI Size: 1126472 OutputKey: e69e376be5af6f88f81d3e31adf27988;type=video/quicktime Server: ConvertVideo Host: domU-12-31-34-00-02-82 Service-Read: Wed, 21 Feb 2007 01:28:14 GMT Service-Write: Wed, 21 Feb 2007 01:28:27 GMT

27. Amazon Web Services – The Example [1] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=691&categoryID=100 Define the Service Instance from boto.services.service import Service import os class ConvertVideo(Service): ProcessingTime = 30 Command = &quot;&quot;&quot;ffmpeg -y -i %s -f mov -r 29.97 -b 1200kb -mbd 2 -flags \ +4mv+trell -aic 2 -cmp 2 -subcmp 2 -ar 48000 -ab 192 -s 320x240 \ -vcodec mpeg4 -acodec aac %s&quot;&quot;&quot; def process_file(self, in_file_name, msg): out_file_name = os.path.join(self.working_dir, 'out.mov') command = self.Command % (in_file_name, out_file_name) os.system(command) return [(out_file_name, 'video/quicktime')]

28. Amazon Web Services – The Example [1] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=691&categoryID=100 $ boto/services/submit_files.py -b myvideos -q vc-input ~/movies ... 50 files successfully submitted. $ boto/services/start_service.py -r -m boto.services.convertvideo \ -c ConvertVideo -a ami-2eba5f47 -i vc-input -o vc-output \ -e mitch@garnaat.com -n 1 python boto/services/get_results.py -q vc-status ~/movies retrieving file: MVI_3110.mov ... 50 results successfully retrieved. Minimum Processing Time: 2 Maximum Processing Time: 58 Average Processing Time: 17.820000 Elapsed Time: 896 Throughput: 3.348214 transactions / minute $ python boto/services/get_results.py -q test-status ~/movies retrieving file: MVI_3110.mov 500 results successfully retrieved. ... Minimum Processing Time: 2 Maximum Processing Time: 60 Average Processing Time: 17.794000 Elapsed Time: 928 Throughput: 32.327586 transactions / minute Submit Files and Setup Queue Startup EC2 worker(s) 1 Instance 10 Instances, 500 files

29. Amazon Web Services – The Example [1] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=691&categoryID=100 Okay... so what ? $.004 per movie, $8k server => 2,000,000 movies 24 hrs * 60 min = 1440 movies / day * 365 days 525600 movies / year * 4 cameras = 2.1M

30. This matters to Amazon ! For the period of 4Q07, bandwidth utilized for S3 & EC2 was greater than the combined bandwidth of Amazon.com’s global websites. [1] In Dec '07 Amazon was the #7 website in the US [2] Same period 330,000+ new developers registered Up 30,000+ from the previous last quarter. Success Stories [3] Startup Challenge [4] Amazon Web Services – “So Who, What, Why” [1] http://biz.yahoo.com/bw/080130/20080130006013.html [2] http://www.comscore.com/press/release.asp?press=2000 [3] http://www.amazon.com/gp/browse.html?node=182241011 [4] http://www.amazon.com/gp/browse.html?node=377634011

31. It's not just Amazon Google [1] Microsoft! [2] Yahoo IBM [3][4][5][6][7] Additional Examples BigTable, GFS, MapReduce CouchDB Hadoop Amazon Web Services – “So Who, What, Why” [1] http://feedblog.org/2007/12/14/google-vs-amazon-in-open-infrastructure/ [2] http://www.microsoft.com/sql/dataservices/default.mspx [3] http://redmonk.com/jgovernor/2007/04/19/why-ibm-should-acquire-amazon/ [4] http://www.ogf.org/OGF22/materials/1137/Irving+Wladawsky-Berger+Keynote.pdf [5] http://www.news.com/8301-10784_3-9817905-7.html [6] http://www-03.ibm.com/press/us/en/pressrelease/22613.wss [7] http://blogs.zdnet.com/SAAS/?p=415

32. Amazon Web Services – Take Away Customers can leverage the “cloud” ! Energy getting invested “up the stack” “ The Industry” is embracing “simplicity” “ Functional programming” renaissance Analogs to the resurgence of the mainframe Application Trend - trust (but verify) Intuition over specificity Amazon doesn't re-instantiate components Insanity - “doing the same thing over and over and expecting different results.” Define Commodity - ...

33. Amazon Web Services – Thinkovers [1] http://www.kitchensoap.com/2008/02/27/when-do-you-get-too-big-to-use-cloudy-stuff/ [2] http://www.readwriteweb.com/archives/amazon_haas_hardware_as_a_service.php [1] [2]