Amazon WS Overview

Slide 1: A mazon Web S ervices – Introduction & Discussion

Slide 2: A mazon Web S ervices – Overview of A mazon's S ervices Explore each type of service Not implementation specific [1] Open Discussion [1] http:/ highscalability.com/ / amazon-architecture

Slide 3: A mazon Web S ervices – The Picture Grocery A lexa M arketplace Prime DevPay A ssociates WebS tore Historical Pricing M echanical Turk Flexible Payment S ystems - FPS Elastic Compute Cloud - EC2 S imple Queue S ervice - S imple S torage S ervice - S 3 S imple DB S QS Dynamo [1] [1] http:/ highscalability.com/ / paper-dynamo-amazon-s-highly-available-key-value-store

Slide 4: A mazon Web S ervices – FPS EC2 M Turk “ Defined” S QS S3 S impleDB S mall Pieces L oosely Joined – “...There are intriguing assertions: that the Web is \"broken on purpose\" and that its many pockets of erroneous information and its available forums [...] let people feel more comfortable with their own inherent imperfections. [...] Weinberger's analysis, though occasionally facile and too relentlessly optimistic and overstated, is surely destined to be the subject of furious debate in chat rooms the cyber-world over. --H. O'Billovich” [1] [1] http:/ www.amazon.com/ mall-Pieces-Loosely-Joined-Unified/ B 00008NRGI/ / S dp/ ref=pd_bbs_sr_1? ie=UTF8& s=books& qid=1204731761& sr=8-1

Slide 5: A mazon Web S ervices – FPS EC2 M Turk “ Defined” S QS S3 S impleDB S mall Pieces L oosely Joined Think Unix M odel – “Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.” -- Doug McIlroy [1] [1] [S alus] Peter H. S alus. A Quarter-Century of Unix. A ddison-Wesley. 1994. IS B N 0-201-54777-5

Slide 6: A mazon Web S ervices – FPS EC2 M Turk “ Defined” S QS S3 S impleDB S mall Pieces Loosely Joined Think Unix M odel “Single” Deployment Embrace the Power of Simplicity Open Collaboration Leverages “Momentum Makers” “Force Multiplication” Estimates of 40% of sales from affiliates [1] [1] http:/ en.wikipedia.org/ / wiki/ mazon.com A

Slide 7: A mazon Web S ervices – S3 From the Ground Up Provides a S torage S ervice – S ince A pril 2006 [1] Geographically distributed content network U.S . and European Union RES T and S OA P interfaces over HTTP A B itTorrent(TM ) protocol interface is provided Does have a S ervice L evel A greement [2] 25% for < 99% 10% for > 99% and < 99.9% “ 3 nines” implies <= 8 hours 46 minutes down time [1] http:/ en.wikipedia.org/ / wiki/ mazon_S 3 A [2] http:/ www.amazon.com/ browse.html%3fnode%3d379654011& token=3A 0F170E7CEFE27B DC730D3D7344512B C1296B 83 / gp/

Slide 8: A mazon Web S ervices – S3 From the Ground Up Uses – Extensive use internally S mugM ug [1] M any independent blogs & “ startups” [2] Even Firefox extensions as interfaces ! Cost – L ess then you'd think [3] U.S . – $0.15 per GB /month In: $0.10 / , Out: graduated $0.13 - 0.18 GB / GB month $.01 per 1,000 PUT or 10,000 GE T requests 20GB & 2GB is ~$40 per year [4] Webmail.us reportedly saved 75% with S 3 No cost to transfer between Amazon Services ! [1] http:/ blogs.smugmug.com/ / don/2006/ 10/ 11/ amazon-s3-show-me-the-money/ [2] http:/ calculator.s3.amazonaws.com/ / calc5.html [3] http:/ www.codinghorror.com/ / blog/archives/000808.html [4] http:/ www.jungledisk.com/ /

Slide 9: A mazon Web S ervices – S3 From the Ground Up F iles (objects) are placed in “ buckets” Unlimited objects in a bucket B uckets host objects between 1byte to 5GB Objects can be public, private or specific rights Retrieved via a unique key – global address space Typically linked with an M D5sum S ubdirectories are done via strategic naming “ jay” and “ jay/photos” are logically independent but might be related at an application level A nother interesting solution involves loopback mounts [1] [1] http:/ developer.amazonwebservices.com/ / connect/thread.jspa? threadID=10271& tstart=30

Slide 10: A mazon Web S ervices – S3 From the Ground Up E xamples: CLI [1] ./s3-get -k M yA WS A ccessKeyId -s A WS S ecretA ccessKeyIdFile /bucket/pathtoobject Python [2] >>> from boto.s3.connection import S 3Connection, Key >>> conn = S 3Connection('<aws access key>', '<aws secret key>') >>> bucket = conn.create_bucket('garnaat_fileit') >>> k = Key(bucket) >>> k.key = 'foobar' >>> k.set_contents_from_string('This is a test of S 3') >>> k.ghet_contents_as_string() >>> k.key = 'myfile' >>> k.set_contents_from_filename('foo.jpg') >>> k.get_contents_to_filename('bar.jpg') [1] http:/ code.google.com/ s3-bash/ / p/ [2] http:/ boto.googlecode.com/ trunk/ s3_tut.txt / svn/ doc/

Slide 11: A mazon Web S ervices – S impleDB Redefining the Dictionary ... if databases were straightforward and easy Web S ervices interface – Queries and S ubmissions No schema – A utomatic indexes Loose enforcement of structure Consistency is “ eventual” (M V CC) [1] A utomatic scaling & caching No configuration & management required For “ real-time data” applications; e.g. shopping carts Heavily reliant on S 3 [1] http:/ en.wikipedia.org/ / wiki/ ultiversion_concurrency_control M http:/ www.sriramkrishnan.com/ / blog/2007/ amazon-simpledb-technical-overview.html 12/

Slide 12: A mazon Web S ervices – S impleDB Redefining the Dictionary 1) CRE A TE a domain to store data B eta constraint – 100 domains max 10 GB each 2) GE T, PUT or DE L E TE items Items have attribute-value pairs “ size = large” “ color = blue, red” Each item can have up to 256 attribute values. Each attribute value can range from 1 to 1,024 bytes. 3) QUE RY – max 5 seconds Operators: =, !=, <, > <=, >=, S TA RTS -WITH, A ND, OR, NOT, INTERS ECTION A ND UNION. http:/ www.amazon.com/ ref=sc_fe_l_2? ie=UTF8& node=342335011& no=3435361& me=A 36L 942TS J2A JA / b/

Slide 13: A mazon Web S ervices – S QS “ Ground Control to Major Tom...” Provides a S torage Queue – S imple! For $1 a user can transmit over 500,000 messages [1] $0.01 per 10,000 A mazon S QS Requests ($0.000001 per Request) Other bandwidth fees still apply (similar to S 3 #'s) M essage Characteristics M ax messages size - 8KB Latency should be anticipated [2] Transmit and Receive order not guaranteed M essages held for 4 days max Data Flow – 1) Find an A mazon S QS queue 2) S endM essage to the queue 3) ReceiveM essage is called 4) ReceiveM essage returns a message L ocks that message until the visibility timeout has passed. 5) DeleteM essage removes the message from the queue [1] http:/ www.amazon.com/ imple-Queue-S ervice-home-page/ ie=UTF8& node=13584001 / S b? [2] http:/ en.wikipedia.org/ / wiki/ mazon_S QS A

Slide 14: A mazon Web S ervices – S QS “ Ground Control to Major Tom...” E xample: Python [1] >>> from boto.sqs.connection import S QS Connection, M essage >>> conn = S QS Connection('<aws access key>', '<aws secret key>') >>> q = conn.create_queue('myqueue', 60) >>> m = M essage() >>> m.set_body('This is my first message.') >>> rs = q.write(m) >>> rs = q.get_messages() >>> rs[0].get_body() >>> q.delete_message(rs[0]) [1] http:/ boto.googlecode.com/ trunk/ sqs_tut.txt / svn/ doc/ [2] http:/ aws.typepad.com/ / aws/2007/ sqs_super_queue.html 05/

Slide 15: A mazon Web S ervices – FPS “ S how me the Money” S upports multiple forms of payment Permits true micro-payment [1] Costs vary based on A mazon's costs A mazon manages security and fraud responsibility “ GateKeeper” enforces policies for senders and receivers Explicit “ Payment Instructions” language “ only five transactions or $50 per month\" or \"refuse payment after A ugust 1\" Three token exchange Integrates with A mazon's existing customer experience “ Tens of M illions of existing customers” “ Pay Now” Widget Promotes “ market places” for intermediaries [1] http:/ paulstamatiou.com/ / 2007/ 04/ 08/ why-you-shouldnt-ignore-amazons-new-fps [2] http:/ www.amazon.com/ browse.html? node=342430011 / gp/ [3] http:/ docs.amazonwebservices.com/ mazonFPS / / A 2007-01-08/ GettingS tartedGuide/ FPS

Slide 16: A mazon Web S ervices – EC2 The S ky's the Limit Provides programmatic compute capacity Including pre-configured appliance images “ ... allows you to obtain and configure capacity with minimal friction” [1] The B asic Process 1) B uild or Identify an appropriate A M I 2) Configure S ecurity and A ccess 3) S tart, M onitor and S top Instances 4) Pay [1] http:/ www.amazon.com/ ? node=201590011 / b/

Slide 17: A mazon Web S ervices – EC2 The S ky's the Limit What's an A M I ? - A mazon Machine Instance “ Golden Master” of the image you want to run Linux only, Windows is a hack [1] A MI's can be private [2] – V endor tools like rPath [3] Leverage an Open “ Operating S ystems” bazaar S ome A M I's can have “ associate” costs per instance M ultiple Instances can run from a single A M I B y default, limited to 20, simple request for more Getting started with E C2 [4] M ike Culver - Introductory video [5] [1] http:/ www.enomaly.net/ / wiki.1137+M 5c1693b2f05.0.html [2]http:/ docs.amazonwebservices.com/ mazonEC2/ 2006-06-26/ / A dg/ creating-an-ami.html [3] http:/ highscalability.com/ / product-rpath-creating-and-managing-virtual-appliances [4] http:/ docs.amazonwebservices.com/ WS EC2/ / A 2007-08-29/ DeveloperGuide/ [5] http:/ developer.amazonwebservices.com/ / connect/ entry.jspa? externalID=583& categoryID=100

Slide 18: A mazon Web S ervices – EC2 The S ky's the Limit E xample: > ec2-describe-images -x all IM A GE ami-23b6534a ec2-public-images/ fedora-7-apache-manafest.xml > ec2-add-keypair j-keypair (generates ssh keypairs) > ec2-run-instance ami-23b6534a -k j-keypair (could take up to a few minutes) > ec2-authorize default -p 22 (A uthorize port 22 for the default security group) > ec2-describe-instances (S hows instance info including DNS name) > ec2-terminate-instances <image name> > ec2run ami-e3a5408a -n 20 -g appserver

Slide 19: A mazon Web S ervices – EC2 The S ky's the Limit S mall Instance - $.10 / hour 1.7 GB of memory, 1 E C2 Compute Unit (1 virtual core with 1 E C2 Compute Unit) 160 GB of instance storage, 32-bit platform L arge Instance - $0.40 / hour 7.5 GB of memory 4 E C2 Compute Units (2 virtual cores with 2 E C2 Compute Units each) 850 GB of instance storage, 64-bit platform E xtra L arge Instance - $0.80 /hour 15 GB of memory 8 E C2 Compute Units (4 virtual cores with 2 E C2 Compute Units each) 1690 GB of instance storage, 64-bit platform 1 E C2 Compute Unit is ~ 1.0-1.2 GHz 2007 Opteron or 2007 X eon B andwidth costs of still apply at 256M b / sec [1] http:/ www.amazon.com/ ? node=201590011 / b/ [2] http:/ www.amazon.com/ ? node=201590011#measure / b/ [3] http:/ www.amazon.com/ ref=sc_fe_c_0_201590011_2? ie=UTF8& node=370375011& no=201590011& me=A 36L 942TS J2A JA / b/

Slide 20: A mazon Web S ervices – EC2 The S ky's the Limit Uses x509 certificates for Developer A uth. S S H keypairs for instance authenticaion Instances can belong to “ security groups” Network is dynamic via NA T IP's and hostnames change M omentum RedHat is deploying a “ cloud” on EC2 [1] Facebook [2] Hadoop [3] EC2 for Python Programmers [4] [1] http:/ www.redhat.com/ / solutions/cloud/ [2] http:/ developer.amazonwebservices.com/ / connect/entry.jspa? externalID=1044& ref=featured http:/ www.amazon.com/ browse.html? node=391557011 / gp/ [3] http:/ blog.awswebshop.com/ / 2008/ 27/ 02/ taking-massive-distributed-computing-to-the-common-man-hadoop-on-amazon-ec2s3/ [4] http:/ jimmyg.org/ / 2007/ 01/ 09/ amazon-ec2-for-people-who-prefer-debian-and-python-over-fedora-and-java/

Slide 21: A mazon Web S ervices – M Turk From the Ground Up A mazon M echanical Turk L aunched in late 2005 [1] [1] http:/ www.egge.net/ / ~savory/blog_mar_04.htm

Slide 22: A mazon Web S ervices – M Turk From the Ground Up A mazon M echanical Turk L aunched in late 2005 [1] “ A rtificial A rtificial Intelligence” - Jeff B ezos [1] http:/ en.wikipedia.org/ / wiki/ mazon.com A

Slide 23: A mazon Web S ervices – M Turk From the Ground Up Humans augment computational requirements Let someone else solve complex A I problems Get along with the business of making money B uilt around Human Intelligence Tasks (HITs) Examples; “ Transcribe audio data” “ A re these paragraphs synonymous? ” “ Find the business in this photo” “ Write a review...” “ S earch for survivors” “ S o far, searchers have found half a dozen previously unknown crash sites, but no sign of Fossett’s plane.” [2] [1] http:/ www.amazon.com/ browse.html? node=15879911 / gp/ [2] http:/ www.techcrunch.com/ / 2007/ 08/ 09/ search-for-steve-fossett-expands-to-amazons-mechanical-turk/

Slide 24: A mazon Web S ervices – The Example EC2 S QS S3 M itch Garnaat A uthor of B ono “ DV S ” transcoding Tutorial [1] [1] http:/ developer.amazonwebservices.com/ / connect/entry.jspa? externalID=691& categoryID=100

Slide 25: A mazon Web S ervices – The Example EC2 S QS S3 1) Read a message from the input queue - S QS 2) B ased on message, retrieve an input file - S 3 3) Perform our video conversion processing - EC2 4) S tore the generated files - S 3 5) Write a message the the output queue - S QS 6) Delete the input message from the input queue - S QS [1] http:/ developer.amazonwebservices.com/ / connect/entry.jspa? externalID=691& categoryID=100

Slide 26: A mazon Web S ervices – The Example EC2 S QS S3 Input Message B ucket: garnaat_fileit InputK ey: f84e4a20b571abc69baf2277d193e596 Date: Tue, 20 Feb 2007 17:21:21 GM T OriginalFileName: M V I_3113.A V I S ize: 1126472 Output M essage B ucket: garnaat_fileit InputK ey: f84e4a20b571abc69baf2277d193e596 Date: Tue, 20 Feb 2007 17:21:21 GM T OriginalFileName: M V I_3113.A V I S ize: 1126472 OutputK ey: e69e376be5af6f88f81d3e31adf27988;type=video/quicktime S erver: ConvertV ideo Host: domU-12-31-34-00-02-82 S ervice-Read: Wed, 21 Feb 2007 01:28:14 GM T S ervice-Write: Wed, 21 F eb 2007 01:28:27 GM T [1] http:/ developer.amazonwebservices.com/ / connect/entry.jspa? externalID=691& categoryID=100

Slide 27: A mazon Web S ervices – The Example EC2 Define the S ervice Instance S QS S3 from boto.services.service import S ervice import os class ConvertV ideo(S ervice): ProcessingTime = 30 Command = \"\"\"ffmpeg -y -i %s -f mov -r 29.97 -b 1200kb -mbd 2 -flags \\ +4mv+trell -aic 2 -cmp 2 -subcmp 2 -ar 48000 -ab 192 -s 320x240 \\ -vcodec mpeg4 -acodec aac %s\"\"\" def process_file(self, in_file_name, msg): out_file_name = os.path.join(self.working_dir, 'out.mov') command = self.Command % (in_file_name, out_file_name) os.system(command) return [(out_file_name, 'video/ quicktime')] [1] http:/ developer.amazonwebservices.com/ / connect/entry.jspa? externalID=691& categoryID=100

Slide 28: A mazon Web S ervices – The Example EC2 S ubmit Files and S etup Queue S QS S3 $ boto/services/submit_files.py -b myvideos -q vc-input ~/movies ... 50 files successfully submitted. S tartup EC2 worker(s) $ boto/services/start_service.py -r -m boto.services.convertvideo \\ -c ConvertV ideo -a ami-2eba5f47 -i vc-input -o vc-output \\ -e mitch@garnaat.com -n 1 1 Instance python boto/services/get_results.py -q vc-status ~/movies retrieving file: M V I_3110.mov ... 10 Instances, 500 files 50 results successfully retrieved. $ python boto/services/get_results.py -q test-status ~/movies M inimum Processing Time: 2 retrieving file: M V I_3110.mov M aximum Processing Time: 58 500 results successfully retrieved. A verage Processing Time: 17.820000 ... E lapsed Time: 896 M inimum Processing Time: 2 Throughput: 3.348214 transactions / minute M aximum Processing Time: 60 A verage Processing Time: 17.794000 Elapsed Time: 928 Throughput: 32.327586 transactions /minute [1] http:/ developer.amazonwebservices.com/ / connect/entry.jspa? externalID=691& categoryID=100

Slide 29: A mazon Web S ervices – The Example EC2 S QS S3 Okay... so what ? S torage 2.5 GB ytes $0.38/ onth M Transfer 2.5 GB ytes $0.50 M essages 1000 $0.10 Compute Time 160 min $0.80   Total: $1.78 $.004 per movie, $8k server => 2,000,000 movies 24 hrs * 60 min = 1440 movies /day * 365 days 525600 movies /year * 4 cameras = 2.1M [1] http:/ developer.amazonwebservices.com/ / connect/entry.jspa? externalID=691& categoryID=100

Slide 30: FPS EC2 M Turk A mazon Web S ervices – S QS S3 S impleDB “ S o Who, What, Why” This matters to A mazon ! For the period of 4Q07, bandwidth utilized for S 3 & EC2 was greater than the combined bandwidth of A mazon.com’s global websites. [1] In Dec '07 A mazon was the #7 website in the US [2] S ame period 330,000+ new developers registered Up 30,000+ from the previous last quarter. S uccess S tories [3] S tartup Challenge [4] [1] http:/ biz.yahoo.com/ 080130/ / bw/ 20080130006013.html [2] http:/ www.comscore.com/ / press/release.asp? press=2000 [3] http:/ www.amazon.com/ browse.html? node=182241011 / gp/ [4] http:/ www.amazon.com/ browse.html? node=377634011 / gp/

Slide 31: FPS EC2 M Turk A mazon Web S ervices – S QS S3 S impleDB “ S o Who, What, Why” It's not just A mazon Google [1] M icrosoft! [2] Y ahoo IB M [3][4][5][6][7] A dditional E xamples B igTable, GFS , M apReduce CouchDB Hadoop [1] http:/ feedblog.org/ / 2007/ 14/ 12/ google-vs-amazon-in-open-infrastructure/ [2] http:/ www.microsoft.com/ dataservices/ / sql/ default.mspx [3] http:/ redmonk.com/ / jgovernor/ 2007/ 19/ 04/ why-ibm-should-acquire-amazon/ [4] http:/ www.ogf.org/ / OGF22/ materials/1137/Irving+Wladawsky-B erger+K eynote.pdf [5] http:/ www.news.com/ / 8301-10784_3-9817905-7.html [6] http:/ www-03.ibm.com/ / press/ en/ us/ pressrelease/ 22613.wss [7] http:/ blogs.zdnet.com/ A A S / p=415 / S ?

Slide 32: A mazon Web S ervices – Take A way Customers can leverage the “ cloud” ! Energy getting invested “ up the stack” “ The Industry” is embracing “ simplicity” “ Functional programming” renaissance A nalogs to the resurgence of the mainframe A pplication Trend - trust (but verify) Intuition over specificity A mazon doesn't re-instantiate components Insanity - “ doing the same thing over and over and expecting different results.” Define Commodity - ...

Slide 33: A mazon Web S ervices – Thinkovers [1] [2] [1] http:/ www.kitchensoap.com/ / 2008/ 27/ 02/ when-do-you-get-too-big-to-use-cloudy-stuff/ [2] http:/ www.readwriteweb.com/ / archives/amazon_haas_hardware_as_a_service.php