Amazon Web Services – Introduction & Discussion

Amazon Web Services – Overview of Amazon's Services Explore each type of service Not implementation specific [1] Open Discussion [1] http://highscalability.com/amazon-architecture

Overview of Amazon's Services

Explore each type of service

Not implementation specific [1]

Open Discussion

Amazon Web Services – The Picture Associates Alexa Marketplace DevPay WebStore Prime Grocery Historical Pricing [1] http://highscalability.com/paper-dynamo-amazon-s-highly-available-key-value-store

Small Pieces Loosely Joined – “ ...There are intriguing assertions: that the Web is "broken on purpose" and that its many pockets of erroneous information and its available forums [...] let people feel more comfortable with their own inherent imperfections . [...] Weinberger's analysis, though occasionally facile and too relentlessly optimistic and overstated , is surely destined to be the subject of furious debate in chat rooms the cyber-world over. --H. O'Billovich” [1] [1] http://www.amazon.com/Small-Pieces-Loosely-Joined-Unified/dp/B00008NRGI/ref=pd_bbs_sr_1?ie=UTF8&s=books&qid=1204731761&sr=8-1 Amazon Web Services – “Defined”

Small Pieces Loosely Joined –

“ ...There are intriguing assertions: that the Web is "broken on purpose" and that its many pockets of erroneous information and its available forums [...] let people feel more comfortable with their own inherent imperfections . [...] Weinberger's analysis, though occasionally facile and too relentlessly optimistic and overstated , is surely destined to be the subject of furious debate in chat rooms the cyber-world over. --H. O'Billovich” [1]

Small Pieces Loosely Joined Think Unix Model – “ Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.” -- Doug McIlroy [1] [1] [Salus] Peter H. Salus. A Quarter-Century of Unix. Addison-Wesley. 1994. ISBN 0-201-54777-5 Amazon Web Services – “Defined”

Small Pieces Loosely Joined

Think Unix Model –

“ Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.” -- Doug McIlroy [1]

Small Pieces Loosely Joined Think Unix Model “ Single” Deployment Embrace the Power of Simplicity Open Collaboration Leverages “Momentum Makers” “ Force Multiplication” Estimates of 40% of sales from affiliates [1] Amazon Web Services – “Defined” [1] http://en.wikipedia.org/wiki/Amazon.com

Small Pieces Loosely Joined

Think Unix Model

“ Single” Deployment

Embrace the Power of Simplicity

Open Collaboration

Leverages “Momentum Makers”

“ Force Multiplication”

Estimates of 40% of sales from affiliates [1]

Amazon Web Services – From the Ground Up Provides a Storage Service – Since April 2006 [1] Geographically distributed content network U.S. and European Union REST and SOAP interfaces over HTTP A BitTorrent(TM) protocol interface is provided Does have a Service Level Agreement [2] 25% for < 99% 10% for > 99% and < 99.9% “ 3 nines” implies <= 8 hours 46 minutes down time [1] http://en.wikipedia.org/wiki/Amazon_S3 [2] http://www.amazon.com/gp/browse.html%3fnode%3d379654011&token=3A0F170E7CEFE27BDC730D3D7344512BC1296B83

Provides a Storage Service – Since April 2006 [1]

Geographically distributed content network

U.S. and European Union

REST and SOAP interfaces over HTTP

A BitTorrent(TM) protocol interface is provided

Does have a Service Level Agreement [2]

25% for < 99%

10% for > 99% and < 99.9%

“ 3 nines” implies <= 8 hours 46 minutes down time

Uses – Extensive use internally SmugMug [1] Many independent blogs & “startups” [2] Even Firefox extensions as interfaces ! Cost – Less then you'd think [3] U.S. – $0.15 per GB/month In: $0.10 /GB, Out: graduated $0.13 - 0.18 GB/month $.01 per 1,000 PUT or 10,000 GET requests 20GB & 2GB is ~$40 per year [4] Webmail.us reportedly saved 75% with S3 No cost to transfer between Amazon Services ! [1] http://blogs.smugmug.com/don/2006/11/10/amazon-s3-show-me-the-money/ [2] http://calculator.s3.amazonaws.com/calc5.html [3] http://www.codinghorror.com/blog/archives/000808.html [4] http://www.jungledisk.com/ Amazon Web Services – From the Ground Up

Uses – Extensive use internally

SmugMug [1]

Many independent blogs & “startups” [2]

Even Firefox extensions as interfaces !

Cost – Less then you'd think [3]

U.S. – $0.15 per GB/month

In: $0.10 /GB, Out: graduated $0.13 - 0.18 GB/month

$.01 per 1,000 PUT or 10,000 GET requests

20GB & 2GB is ~$40 per year [4]

Webmail.us reportedly saved 75% with S3

No cost to transfer between Amazon Services !

Amazon Web Services – From the Ground Up Files (objects) are placed in “buckets” Unlimited objects in a bucket Buckets host objects between 1byte to 5GB Objects can be public, private or specific rights Retrieved via a unique key – global address space Typically linked with an MD5sum Subdirectories are done via strategic naming “ jay” and “jay/photos” are logically independent but might be related at an application level Another interesting solution involves loopback mounts [1] [1] http://developer.amazonwebservices.com/connect/thread.jspa?threadID=10271&tstart=30

Files (objects) are placed in “buckets”

Unlimited objects in a bucket

Buckets host objects between 1byte to 5GB

Objects can be public, private or specific rights

Retrieved via a unique key – global address space

Typically linked with an MD5sum

Subdirectories are done via strategic naming

“ jay” and “jay/photos” are logically independent but might be related at an application level

Another interesting solution involves loopback mounts [1]

Examples: CLI [1] ./s3-get -k MyAWSAccessKeyId -s AWSSecretAccessKeyIdFile /bucket/pathtoobject Python [2] >>> from boto.s3.connection import S3Connection, Key >>> conn = S3Connection('<aws access key>', '<aws secret key>') >>> bucket = conn.create_bucket('garnaat_fileit') >>> k = Key(bucket) >>> k.key = 'foobar' >>> k.set_contents_from_string('This is a test of S3') >>> k.ghet_contents_as_string() >>> k.key = 'myfile' >>> k.set_contents_from_filename('foo.jpg') >>> k.get_contents_to_filename('bar.jpg') [1] http://code.google.com/p/s3-bash/ [2] http://boto.googlecode.com/svn/trunk/doc/s3_tut.txt Amazon Web Services – From the Ground Up

Examples:

CLI [1]

./s3-get -k MyAWSAccessKeyId -s AWSSecretAccessKeyIdFile /bucket/pathtoobject

Python [2]

>>> from boto.s3.connection import S3Connection, Key

>>> conn = S3Connection('<aws access key>', '<aws secret key>')

>>> bucket = conn.create_bucket('garnaat_fileit')

>>> k = Key(bucket)

>>> k.key = 'foobar'

>>> k.set_contents_from_string('This is a test of S3')

>>> k.ghet_contents_as_string()

>>> k.key = 'myfile'

>>> k.set_contents_from_filename('foo.jpg')

>>> k.get_contents_to_filename('bar.jpg')

Amazon Web Services – Redefining the Dictionary ... if databases were straightforward and easy Web Services interface – Queries and Submissions No schema – Automatic indexes Loose enforcement of structure Consistency is “eventual” (MVCC) [1] Automatic scaling & caching No configuration & management required For “real-time data” applications; e.g. shopping carts Heavily reliant on S3 [1] http://en.wikipedia.org/wiki/Multiversion_concurrency_control http://www.sriramkrishnan.com/blog/2007/12/amazon-simpledb-technical-overview.html

... if databases were straightforward and easy

Web Services interface – Queries and Submissions

No schema –

Automatic indexes

Loose enforcement of structure

Consistency is “eventual” (MVCC) [1]

Automatic scaling & caching

No configuration & management required

For “real-time data” applications; e.g. shopping carts

Heavily reliant on S3

CREATE a domain to store data Beta constraint – 100 domains max 10 GB each GET, PUT or DELETE items Items have attribute-value pairs “ size = large” “ color = blue, red” Each item can have up to 256 attribute values. Each attribute value can range from 1 to 1,024 bytes. QUERY – max 5 seconds Operators: =, !=, <, > <=, >=, STARTS-WITH, AND, OR, NOT, INTERSECTION AND UNION. http://www.amazon.com/b/ref=sc_fe_l_2?ie=UTF8&node=342335011&no=3435361&me=A36L942TSJ2AJA Amazon Web Services – Redefining the Dictionary

CREATE a domain to store data

Beta constraint – 100 domains max 10 GB each

GET, PUT or DELETE items

Items have attribute-value pairs

“ size = large”

“ color = blue, red”

Each item can have up to 256 attribute values.

Each attribute value can range from 1 to 1,024 bytes.

QUERY – max 5 seconds

Operators: =, !=, <, > <=, >=, STARTS-WITH, AND, OR, NOT, INTERSECTION AND UNION.

Provides a Storage Queue – Simple! For $1 a user can transmit over 500,000 messages [1] $0.01 per 10,000 Amazon SQS Requests ($0.000001 per Request) Other bandwidth fees still apply (similar to S3 #'s) Message Characteristics Max messages size - 8KB Latency should be anticipated [2] Transmit and Receive order not guaranteed Messages held for 4 days max Data Flow – Find an Amazon SQS queue SendMessage to the queue ReceiveMessage is called ReceiveMessage returns a message Locks that message until the visibility timeout has passed. DeleteMessage removes the message from the queue [1] http://www.amazon.com/Simple-Queue-Service-home-page/b?ie=UTF8&node=13584001 [2] http://en.wikipedia.org/wiki/Amazon_SQS Amazon Web Services – “Ground Control to Major Tom...”

Provides a Storage Queue – Simple!

For $1 a user can transmit over 500,000 messages [1]

$0.01 per 10,000 Amazon SQS Requests ($0.000001 per Request)

Other bandwidth fees still apply (similar to S3 #'s)

Message Characteristics

Max messages size - 8KB

Latency should be anticipated [2]

Transmit and Receive order not guaranteed

Messages held for 4 days max

Data Flow –

Find an Amazon SQS queue

SendMessage to the queue

ReceiveMessage is called

ReceiveMessage returns a message

Locks that message until the visibility timeout has passed.

DeleteMessage removes the message from the queue

Example: Python [1] >>> from boto.sqs.connection import SQSConnection, Message >>> conn = SQSConnection('<aws access key>', '<aws secret key>') >>> q = conn.create_queue('myqueue', 60) >>> m = Message() >>> m.set_body('This is my first message.') >>> rs = q.write(m) >>> rs = q.get_messages() >>> rs[0].get_body() >>> q.delete_message(rs[0]) [1] http://boto.googlecode.com/svn/trunk/doc/sqs_tut.txt [2] http://aws.typepad.com/aws/2007/05/sqs_super_queue.html Amazon Web Services – “Ground Control to Major Tom...”

Example:

Python [1]

>>> from boto.sqs.connection import SQSConnection, Message

>>> conn = SQSConnection('<aws access key>', '<aws secret key>')

>>> q = conn.create_queue('myqueue', 60)

>>> m = Message()

>>> m.set_body('This is my first message.')

>>> rs = q.write(m)

>>> rs = q.get_messages()

>>> rs[0].get_body()

>>> q.delete_message(rs[0])

Amazon Web Services – “Show me the Money” Supports multiple forms of payment Permits true micro-payment [1] Costs vary based on Amazon's costs Amazon manages security and fraud responsibility “ GateKeeper” enforces policies for senders and receivers Explicit “Payment Instructions” language “ only five transactions or $50 per month&quot; or &quot;refuse payment after August 1&quot; Three token exchange Integrates with Amazon's existing customer experience “ Tens of Millions of existing customers” “ Pay Now” Widget Promotes “market places” for intermediaries [1] http://paulstamatiou.com/2007/08/04/why-you-shouldnt-ignore-amazons-new-fps [2] http://www.amazon.com/gp/browse.html?node=342430011 [3] http://docs.amazonwebservices.com/AmazonFPS/2007-01-08/FPSGettingStartedGuide/

Supports multiple forms of payment

Permits true micro-payment [1]

Costs vary based on Amazon's costs

Amazon manages security and fraud responsibility

“ GateKeeper” enforces policies for senders and receivers

Explicit “Payment Instructions” language

“ only five transactions or $50 per month&quot; or &quot;refuse payment after August 1&quot;

Three token exchange

Integrates with Amazon's existing customer experience

“ Tens of Millions of existing customers”

“ Pay Now” Widget

Promotes “market places” for intermediaries

Amazon Web Services – The Sky's the Limit [1] http://www.amazon.com/b/?node=201590011 Provides programmatic compute capacity Including pre-configured appliance images “ ... allows you to obtain and configure capacity with minimal friction” [1] The Basic Process Build or Identify an appropriate AMI Configure Security and Access Start, Monitor and Stop Instances Pay

Provides programmatic compute capacity

Including pre-configured appliance images

“ ... allows you to obtain and configure capacity with minimal friction” [1]

The Basic Process

Build or Identify an appropriate AMI

Configure Security and Access

Start, Monitor and Stop Instances

Pay

Amazon Web Services – The Sky's the Limit What's an AMI ? - Amazon Machine Instance “ Golden Master” of the image you want to run Linux only, Windows is a hack [1] AMI's can be private [2] – Vendor tools like rPath [3] Leverage an Open “Operating Systems” bazaar Some AMI's can have “associate” costs per instance Multiple Instances can run from a single AMI By default, limited to 20, simple request for more Getting started with EC2 [4] Mike Culver - Introductory video [5] [1] http://www.enomaly.net/wiki.1137+M5c1693b2f05.0.html [2]http://docs.amazonwebservices.com/AmazonEC2/dg/2006-06-26/creating-an-ami.html [3] http://highscalability.com/product-rpath-creating-and-managing-virtual-appliances [4] http://docs.amazonwebservices.com/AWSEC2/2007-08-29/DeveloperGuide/ [5] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=583&categoryID=100

What's an AMI ? - Amazon Machine Instance

“ Golden Master” of the image you want to run

Linux only, Windows is a hack [1]

AMI's can be private [2] – Vendor tools like rPath [3]

Leverage an Open “Operating Systems” bazaar

Some AMI's can have “associate” costs per instance

Multiple Instances can run from a single AMI

By default, limited to 20, simple request for more

Getting started with EC2 [4]

Mike Culver - Introductory video [5]

Amazon Web Services – The Sky's the Limit Example: > ec2-describe-images -x all IMAGE ami-23b6534a ec2-public-images/fedora-7-apache-manafest.xml > ec2-add-keypair j-keypair (generates ssh keypairs) > ec2-run-instance ami-23b6534a -k j-keypair (could take up to a few minutes) > ec2-authorize default -p 22 (Authorize port 22 for the default security group) > ec2-describe-instances (Shows instance info including DNS name) > ec2-terminate-instances <image name> > ec2run ami-e3a5408a -n 20 -g appserver

Example:

> ec2-describe-images -x all

IMAGE ami-23b6534a ec2-public-images/fedora-7-apache-manafest.xml

> ec2-add-keypair j-keypair

(generates ssh keypairs)

> ec2-run-instance ami-23b6534a -k j-keypair

(could take up to a few minutes)

> ec2-authorize default -p 22

(Authorize port 22 for the default security group)

> ec2-describe-instances

(Shows instance info including DNS name)

> ec2-terminate-instances <image name>

> ec2run ami-e3a5408a -n 20 -g appserver

Amazon Web Services – The Sky's the Limit [1] http://www.amazon.com/b/?node=201590011 [2] http://www.amazon.com/b/?node=201590011#measure [3] http://www.amazon.com/b/ref=sc_fe_c_0_201590011_2?ie=UTF8&node=370375011&no=201590011&me=A36L942TSJ2AJA Small Instance - $.10 / hour 1.7 GB of memory, 1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit) 160 GB of instance storage, 32-bit platform Large Instance - $0.40 / hour 7.5 GB of memory 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each) 850 GB of instance storage, 64-bit platform Extra Large Instance - $0.80 / hour 15 GB of memory 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each) 1690 GB of instance storage, 64-bit platform 1 EC2 Compute Unit is ~ 1.0-1.2 GHz 2007 Opteron or 2007 Xeon Bandwidth costs of still apply at 256Mb / sec

Small Instance - $.10 / hour

1.7 GB of memory,

1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit)

160 GB of instance storage, 32-bit platform

Large Instance - $0.40 / hour

7.5 GB of memory

4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each)

850 GB of instance storage, 64-bit platform

Extra Large Instance - $0.80 / hour

15 GB of memory

8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each)

1690 GB of instance storage, 64-bit platform

1 EC2 Compute Unit is ~ 1.0-1.2 GHz 2007 Opteron or 2007 Xeon

Bandwidth costs of still apply at 256Mb / sec

Amazon Web Services – The Sky's the Limit Uses x509 certificates for Developer Auth. SSH keypairs for instance authenticaion Instances can belong to “security groups” Network is dynamic via NAT IP's and hostnames change Momentum RedHat is deploying a “cloud” on EC2 [1] Facebook [2] Hadoop [3] EC2 for Python Programmers [4] [1] http://www.redhat.com/solutions/cloud/ [2] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1044&ref=featured http://www.amazon.com/gp/browse.html?node=391557011 [3] http://blog.awswebshop.com/2008/02/27/taking-massive-distributed-computing-to-the-common-man-hadoop-on-amazon-ec2s3/ [4] http://jimmyg.org/2007/09/01/amazon-ec2-for-people-who-prefer-debian-and-python-over-fedora-and-java/

Uses x509 certificates for Developer Auth.

SSH keypairs for instance authenticaion

Instances can belong to “security groups”

Network is dynamic via NAT

IP's and hostnames change

Momentum

RedHat is deploying a “cloud” on EC2 [1]

Facebook [2]

Hadoop [3]

EC2 for Python Programmers [4]

Amazon Web Services – From the Ground Up Amazon Mechanical Turk Launched in late 2005 [1] [1] http://www.egge.net/~savory/blog_mar_04.htm

Amazon Mechanical Turk

Launched in late 2005 [1]

Amazon Web Services – From the Ground Up Amazon Mechanical Turk Launched in late 2005 [1] “ Artificial Artificial Intelligence” - Jeff Bezos [1] http://en.wikipedia.org/wiki/Amazon.com

Amazon Mechanical Turk

Launched in late 2005 [1]

“ Artificial Artificial Intelligence” - Jeff Bezos

Amazon Web Services – From the Ground Up [1] http://www.amazon.com/gp/browse.html?node=15879911 [2] http://www.techcrunch.com/2007/09/08/search-for-steve-fossett-expands-to-amazons-mechanical-turk/ Humans augment computational requirements Let someone else solve complex AI problems Get along with the business of making money Built around Human Intelligence Tasks (HITs) Examples; “ Transcribe audio data” “ Are these paragraphs synonymous?” “ Find the business in this photo” “ Write a review...” “ Search for survivors” “ So far, searchers have found half a dozen previously unknown crash sites, but no sign of Fossett’s plane.” [2]

Humans augment computational requirements

Let someone else solve complex AI problems

Get along with the business of making money

Built around Human Intelligence Tasks (HITs)

Examples;

“ Transcribe audio data”

“ Are these paragraphs synonymous?”

“ Find the business in this photo”

“ Write a review...”

“ Search for survivors”

“ So far, searchers have found half a dozen previously unknown crash sites, but no sign of Fossett’s plane.” [2]

Amazon Web Services – The Example [1] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=691&categoryID=100 Mitch Garnaat Author of Bono “ DVS” transcoding Tutorial [1]

Mitch Garnaat

Author of Bono

“ DVS” transcoding

Tutorial [1]

Read a message from the input queue - SQS Based on message, retrieve an input file - S3 Perform our video conversion processing - EC2 Store the generated files - S3 Write a message the the output queue - SQS Delete the input message from the input queue - SQS Amazon Web Services – The Example [1] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=691&categoryID=100

Read a message from the input queue - SQS

Based on message, retrieve an input file - S3

Perform our video conversion processing - EC2

Store the generated files - S3

Write a message the the output queue - SQS

Delete the input message from the input queue - SQS

Amazon Web Services – The Example [1] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=691&categoryID=100 Input Message Bucket: garnaat_fileit InputKey: f84e4a20b571abc69baf2277d193e596 Date: Tue, 20 Feb 2007 17:21:21 GMT OriginalFileName: MVI_3113.AVI Size: 1126472 Output Message Bucket: garnaat_fileit InputKey: f84e4a20b571abc69baf2277d193e596 Date: Tue, 20 Feb 2007 17:21:21 GMT OriginalFileName: MVI_3113.AVI Size: 1126472 OutputKey: e69e376be5af6f88f81d3e31adf27988;type=video/quicktime Server: ConvertVideo Host: domU-12-31-34-00-02-82 Service-Read: Wed, 21 Feb 2007 01:28:14 GMT Service-Write: Wed, 21 Feb 2007 01:28:27 GMT

Input Message

Bucket: garnaat_fileit

InputKey: f84e4a20b571abc69baf2277d193e596

Date: Tue, 20 Feb 2007 17:21:21 GMT

OriginalFileName: MVI_3113.AVI

Size: 1126472

Output Message

Bucket: garnaat_fileit

InputKey: f84e4a20b571abc69baf2277d193e596

Date: Tue, 20 Feb 2007 17:21:21 GMT

OriginalFileName: MVI_3113.AVI

Size: 1126472

OutputKey: e69e376be5af6f88f81d3e31adf27988;type=video/quicktime

Server: ConvertVideo

Host: domU-12-31-34-00-02-82

Service-Read: Wed, 21 Feb 2007 01:28:14 GMT

Service-Write: Wed, 21 Feb 2007 01:28:27 GMT

Amazon Web Services – The Example [1] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=691&categoryID=100 Define the Service Instance from boto.services.service import Service import os class ConvertVideo(Service): ProcessingTime = 30 Command = &quot;&quot;&quot;ffmpeg -y -i %s -f mov -r 29.97 -b 1200kb -mbd 2 -flags +4mv+trell -aic 2 -cmp 2 -subcmp 2 -ar 48000 -ab 192 -s 320x240 -vcodec mpeg4 -acodec aac %s&quot;&quot;&quot; def process_file(self, in_file_name, msg): out_file_name = os.path.join(self.working_dir, 'out.mov') command = self.Command % (in_file_name, out_file_name) os.system(command) return [(out_file_name, 'video/quicktime')]

Define the Service Instance

from boto.services.service import Service

import os

class ConvertVideo(Service):

ProcessingTime = 30

Command = &quot;&quot;&quot;ffmpeg -y -i %s -f mov -r 29.97 -b 1200kb -mbd 2 -flags

+4mv+trell -aic 2 -cmp 2 -subcmp 2 -ar 48000 -ab 192 -s 320x240

-vcodec mpeg4 -acodec aac %s&quot;&quot;&quot;

def process_file(self, in_file_name, msg):

out_file_name = os.path.join(self.working_dir, 'out.mov')

command = self.Command % (in_file_name, out_file_name)

os.system(command)

return [(out_file_name, 'video/quicktime')]

Amazon Web Services – The Example [1] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=691&categoryID=100 $ boto/services/submit_files.py -b myvideos -q vc-input ~/movies ... 50 files successfully submitted. $ boto/services/start_service.py -r -m boto.services.convertvideo -c ConvertVideo -a ami-2eba5f47 -i vc-input -o vc-output -e mitch@garnaat.com -n 1 python boto/services/get_results.py -q vc-status ~/movies retrieving file: MVI_3110.mov ... 50 results successfully retrieved. Minimum Processing Time: 2 Maximum Processing Time: 58 Average Processing Time: 17.820000 Elapsed Time: 896 Throughput: 3.348214 transactions / minute $ python boto/services/get_results.py -q test-status ~/movies retrieving file: MVI_3110.mov 500 results successfully retrieved. ... Minimum Processing Time: 2 Maximum Processing Time: 60 Average Processing Time: 17.794000 Elapsed Time: 928 Throughput: 32.327586 transactions / minute Submit Files and Setup Queue Startup EC2 worker(s) 1 Instance 10 Instances, 500 files

$ boto/services/submit_files.py -b myvideos -q vc-input ~/movies

...

50 files successfully submitted.

$ boto/services/start_service.py -r -m boto.services.convertvideo

-c ConvertVideo -a ami-2eba5f47 -i vc-input -o vc-output

-e mitch@garnaat.com -n 1

python boto/services/get_results.py -q vc-status ~/movies

retrieving file: MVI_3110.mov

...

50 results successfully retrieved.

Minimum Processing Time: 2

Maximum Processing Time: 58

Average Processing Time: 17.820000

Elapsed Time: 896

Throughput: 3.348214 transactions / minute

$ python boto/services/get_results.py -q test-status ~/movies

retrieving file: MVI_3110.mov

500 results successfully retrieved.

...

Minimum Processing Time: 2

Maximum Processing Time: 60

Average Processing Time: 17.794000

Elapsed Time: 928

Throughput: 32.327586 transactions / minute

Amazon Web Services – The Example [1] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=691&categoryID=100 Okay... so what ? $.004 per movie, $8k server => 2,000,000 movies 24 hrs * 60 min = 1440 movies / day * 365 days 525600 movies / year * 4 cameras = 2.1M

Okay... so what ?

$.004 per movie, $8k server => 2,000,000 movies

24 hrs * 60 min = 1440 movies / day * 365 days

525600 movies / year * 4 cameras = 2.1M

This matters to Amazon ! For the period of 4Q07, bandwidth utilized for S3 & EC2 was greater than the combined bandwidth of Amazon.com’s global websites. [1] In Dec '07 Amazon was the #7 website in the US [2] Same period 330,000+ new developers registered Up 30,000+ from the previous last quarter. Success Stories [3] Startup Challenge [4] Amazon Web Services – “So Who, What, Why” [1] http://biz.yahoo.com/bw/080130/20080130006013.html [2] http://www.comscore.com/press/release.asp?press=2000 [3] http://www.amazon.com/gp/browse.html?node=182241011 [4] http://www.amazon.com/gp/browse.html?node=377634011

This matters to Amazon !

For the period of 4Q07, bandwidth utilized for S3 & EC2 was greater than the combined bandwidth of Amazon.com’s global websites. [1]

In Dec '07 Amazon was the #7 website in the US [2]

Same period 330,000+ new developers registered

Up 30,000+ from the previous last quarter.

Success Stories [3]

Startup Challenge [4]

It's not just Amazon Google [1] Microsoft! [2] Yahoo IBM [3][4][5][6][7] Additional Examples BigTable, GFS, MapReduce CouchDB Hadoop Amazon Web Services – “So Who, What, Why” [1] http://feedblog.org/2007/12/14/google-vs-amazon-in-open-infrastructure/ [2] http://www.microsoft.com/sql/dataservices/default.mspx [3] http://redmonk.com/jgovernor/2007/04/19/why-ibm-should-acquire-amazon/ [4] http://www.ogf.org/OGF22/materials/1137/Irving+Wladawsky-Berger+Keynote.pdf [5] http://www.news.com/8301-10784_3-9817905-7.html [6] http://www-03.ibm.com/press/us/en/pressrelease/22613.wss [7] http://blogs.zdnet.com/SAAS/?p=415

It's not just Amazon

Google [1]

Microsoft! [2]

Yahoo

IBM [3][4][5][6][7]

Additional Examples

BigTable, GFS, MapReduce

CouchDB

Hadoop

Amazon Web Services – Take Away Customers can leverage the “cloud” ! Energy getting invested “up the stack” “ The Industry” is embracing “simplicity” “ Functional programming” renaissance Analogs to the resurgence of the mainframe Application Trend - trust (but verify) Intuition over specificity Amazon doesn't re-instantiate components Insanity - “doing the same thing over and over and expecting different results.” Define Commodity - ...

Customers can leverage the “cloud” !

Energy getting invested “up the stack”

“ The Industry” is embracing “simplicity”

“ Functional programming” renaissance

Analogs to the resurgence of the mainframe

Application Trend - trust (but verify)

Intuition over specificity

Amazon doesn't re-instantiate components

Insanity - “doing the same thing over and over and expecting different results.”

Define Commodity - ...

Amazon Web Services – Thinkovers [1] http://www.kitchensoap.com/2008/02/27/when-do-you-get-too-big-to-use-cloudy-stuff/ [2] http://www.readwriteweb.com/archives/amazon_haas_hardware_as_a_service.php [1] [2]

[1]

[2]