Amazon WS Overview

Amazon Web Services – Introduction & Discussion

Amazon Web Services –
Overview of Amazon's Services
Explore each type of service
Not implementation specific [1]
Open Discussion
[1] http://highscalability.com/amazon-architecture

Amazon Web Services – The Picture Associates Alexa Marketplace DevPay WebStore Prime Grocery Historical Pricing [1] http://highscalability.com/paper-dynamo-amazon-s-highly-available-key-value-store

Small Pieces Loosely Joined –
“ ...There are intriguing assertions: that the Web is "broken on purpose" and that its many pockets of erroneous information and its available forums [...] let people feel more comfortable with their own inherent imperfections . [...] Weinberger's analysis, though occasionally facile and too relentlessly optimistic and overstated , is surely destined to be the subject of furious debate in chat rooms the cyber-world over. --H. O'Billovich” [1]
[1] http://www.amazon.com/Small-Pieces-Loosely-Joined-Unified/dp/B00008NRGI/ref=pd_bbs_sr_1?ie=UTF8&s=books&qid=1204731761&sr=8-1 Amazon Web Services – “Defined”

Small Pieces Loosely Joined
Think Unix Model –
“ Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.” -- Doug McIlroy [1]
[1] [Salus] Peter H. Salus. A Quarter-Century of Unix. Addison-Wesley. 1994. ISBN 0-201-54777-5 Amazon Web Services – “Defined”

Small Pieces Loosely Joined
Think Unix Model
“ Single” Deployment
Embrace the Power of Simplicity
Open Collaboration
Leverages “Momentum Makers”
“ Force Multiplication”
Estimates of 40% of sales from affiliates [1]
Amazon Web Services – “Defined” [1] http://en.wikipedia.org/wiki/Amazon.com

Amazon Web Services – From the Ground Up
Provides a Storage Service – Since April 2006 [1]
Geographically distributed content network
U.S. and European Union
REST and SOAP interfaces over HTTP
A BitTorrent(TM) protocol interface is provided
Does have a Service Level Agreement [2]
25% for < 99%
10% for > 99% and < 99.9%
“ 3 nines” implies <= 8 hours 46 minutes down time
[1] http://en.wikipedia.org/wiki/Amazon_S3 [2] http://www.amazon.com/gp/browse.html%3fnode%3d379654011&token=3A0F170E7CEFE27BDC730D3D7344512BC1296B83

Uses – Extensive use internally
SmugMug [1]
Many independent blogs & “startups” [2]
Even Firefox extensions as interfaces !
Cost – Less then you'd think [3]
U.S. – $0.15 per GB/month
In: $0.10 /GB, Out: graduated $0.13 - 0.18 GB/month
$.01 per 1,000 PUT or 10,000 GET requests
20GB & 2GB is ~$40 per year [4]
Webmail.us reportedly saved 75% with S3
No cost to transfer between Amazon Services !
[1] http://blogs.smugmug.com/don/2006/11/10/amazon-s3-show-me-the-money/ [2] http://calculator.s3.amazonaws.com/calc5.html [3] http://www.codinghorror.com/blog/archives/000808.html [4] http://www.jungledisk.com/ Amazon Web Services – From the Ground Up

Files (objects) are placed in “buckets”
Unlimited objects in a bucket
Buckets host objects between 1byte to 5GB
Objects can be public, private or specific rights
Retrieved via a unique key – global address space
Typically linked with an MD5sum
Subdirectories are done via strategic naming
“ jay” and “jay/photos” are logically independent but might be related at an application level
Another interesting solution involves loopback mounts [1]
[1] http://developer.amazonwebservices.com/connect/thread.jspa?threadID=10271&tstart=30

Examples:
CLI [1]
./s3-get -k MyAWSAccessKeyId -s AWSSecretAccessKeyIdFile /bucket/pathtoobject
Python [2]
>>> from boto.s3.connection import S3Connection, Key
>>> conn = S3Connection('<aws access key>', '<aws secret key>')
>>> bucket = conn.create_bucket('garnaat_fileit')
>>> k = Key(bucket)
>>> k.key = 'foobar'
>>> k.set_contents_from_string('This is a test of S3')
>>> k.ghet_contents_as_string()
>>> k.key = 'myfile'
>>> k.set_contents_from_filename('foo.jpg')
>>> k.get_contents_to_filename('bar.jpg')
[1] http://code.google.com/p/s3-bash/ [2] http://boto.googlecode.com/svn/trunk/doc/s3_tut.txt Amazon Web Services – From the Ground Up

Amazon Web Services – Redefining the Dictionary
... if databases were straightforward and easy
Web Services interface – Queries and Submissions
No schema –
Automatic indexes
Loose enforcement of structure
Consistency is “eventual” (MVCC) [1]
Automatic scaling & caching
No configuration & management required
For “real-time data” applications; e.g. shopping carts
Heavily reliant on S3
[1] http://en.wikipedia.org/wiki/Multiversion_concurrency_control http://www.sriramkrishnan.com/blog/2007/12/amazon-simpledb-technical-overview.html

CREATE a domain to store data
Beta constraint – 100 domains max 10 GB each
GET, PUT or DELETE items
Items have attribute-value pairs
“ size = large”
“ color = blue, red”
Each item can have up to 256 attribute values.
Each attribute value can range from 1 to 1,024 bytes.
QUERY – max 5 seconds
Operators: =, !=, <, > <=, >=, STARTS-WITH, AND, OR, NOT, INTERSECTION AND UNION.
http://www.amazon.com/b/ref=sc_fe_l_2?ie=UTF8&node=342335011&no=3435361&me=A36L942TSJ2AJA Amazon Web Services – Redefining the Dictionary

Provides a Storage Queue – Simple!
For $1 a user can transmit over 500,000 messages [1]
$0.01 per 10,000 Amazon SQS Requests ($0.000001 per Request)
Other bandwidth fees still apply (similar to S3 #'s)
Message Characteristics
Max messages size - 8KB
Latency should be anticipated [2]
Transmit and Receive order not guaranteed
Messages held for 4 days max
Data Flow –
Find an Amazon SQS queue
SendMessage to the queue
ReceiveMessage is called
ReceiveMessage returns a message
Locks that message until the visibility timeout has passed.
DeleteMessage removes the message from the queue
[1] http://www.amazon.com/Simple-Queue-Service-home-page/b?ie=UTF8&node=13584001 [2] http://en.wikipedia.org/wiki/Amazon_SQS Amazon Web Services – “Ground Control to Major Tom...”

Example:
Python [1]
>>> from boto.sqs.connection import SQSConnection, Message
>>> conn = SQSConnection('<aws access key>', '<aws secret key>')
>>> q = conn.create_queue('myqueue', 60)
>>> m = Message()
>>> m.set_body('This is my first message.')
>>> rs = q.write(m)
>>> rs = q.get_messages()
>>> rs[0].get_body()
>>> q.delete_message(rs[0])
[1] http://boto.googlecode.com/svn/trunk/doc/sqs_tut.txt [2] http://aws.typepad.com/aws/2007/05/sqs_super_queue.html Amazon Web Services – “Ground Control to Major Tom...”

Amazon Web Services – “Show me the Money”
Supports multiple forms of payment
Permits true micro-payment [1]
Costs vary based on Amazon's costs
Amazon manages security and fraud responsibility
“ GateKeeper” enforces policies for senders and receivers
Explicit “Payment Instructions” language
“ only five transactions or $50 per month" or "refuse payment after August 1"
Three token exchange
Integrates with Amazon's existing customer experience
“ Tens of Millions of existing customers”
“ Pay Now” Widget
Promotes “market places” for intermediaries
[1] http://paulstamatiou.com/2007/08/04/why-you-shouldnt-ignore-amazons-new-fps [2] http://www.amazon.com/gp/browse.html?node=342430011 [3] http://docs.amazonwebservices.com/AmazonFPS/2007-01-08/FPSGettingStartedGuide/

Amazon Web Services – The Sky's the Limit [1] http://www.amazon.com/b/?node=201590011
Provides programmatic compute capacity
Including pre-configured appliance images
“ ... allows you to obtain and configure capacity with minimal friction” [1]
The Basic Process
Build or Identify an appropriate AMI
Configure Security and Access
Start, Monitor and Stop Instances
Pay

Amazon Web Services – The Sky's the Limit
What's an AMI ? - Amazon Machine Instance
“ Golden Master” of the image you want to run
Linux only, Windows is a hack [1]
AMI's can be private [2] – Vendor tools like rPath [3]
Leverage an Open “Operating Systems” bazaar
Some AMI's can have “associate” costs per instance
Multiple Instances can run from a single AMI
By default, limited to 20, simple request for more
Getting started with EC2 [4]
Mike Culver - Introductory video [5]
[1] http://www.enomaly.net/wiki.1137+M5c1693b2f05.0.html [2]http://docs.amazonwebservices.com/AmazonEC2/dg/2006-06-26/creating-an-ami.html [3] http://highscalability.com/product-rpath-creating-and-managing-virtual-appliances [4] http://docs.amazonwebservices.com/AWSEC2/2007-08-29/DeveloperGuide/ [5] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=583&categoryID=100

Example:
> ec2-describe-images -x all
IMAGE ami-23b6534a ec2-public-images/fedora-7-apache-manafest.xml
> ec2-add-keypair j-keypair
(generates ssh keypairs)
> ec2-run-instance ami-23b6534a -k j-keypair
(could take up to a few minutes)
> ec2-authorize default -p 22
(Authorize port 22 for the default security group)
> ec2-describe-instances
(Shows instance info including DNS name)
> ec2-terminate-instances <image name>
> ec2run ami-e3a5408a -n 20 -g appserver

Amazon Web Services – The Sky's the Limit [1] http://www.amazon.com/b/?node=201590011 [2] http://www.amazon.com/b/?node=201590011#measure [3] http://www.amazon.com/b/ref=sc_fe_c_0_201590011_2?ie=UTF8&node=370375011&no=201590011&me=A36L942TSJ2AJA
Small Instance - $.10 / hour
1.7 GB of memory,
1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit)
160 GB of instance storage, 32-bit platform
Large Instance - $0.40 / hour
7.5 GB of memory
4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each)
Extra Large Instance - $0.80 / hour
15 GB of memory
8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each)
1 EC2 Compute Unit is ~ 1.0-1.2 GHz 2007 Opteron or 2007 Xeon
Bandwidth costs of still apply at 256Mb / sec

Uses x509 certificates for Developer Auth.
SSH keypairs for instance authenticaion
Instances can belong to “security groups”
Network is dynamic via NAT
IP's and hostnames change
Momentum
RedHat is deploying a “cloud” on EC2 [1]
Facebook [2]
Hadoop [3]
EC2 for Python Programmers [4]
[1] http://www.redhat.com/solutions/cloud/ [2] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1044&ref=featured http://www.amazon.com/gp/browse.html?node=391557011 [3] http://blog.awswebshop.com/2008/02/27/taking-massive-distributed-computing-to-the-common-man-hadoop-on-amazon-ec2s3/ [4] http://jimmyg.org/2007/09/01/amazon-ec2-for-people-who-prefer-debian-and-python-over-fedora-and-java/

Amazon Mechanical Turk
Launched in late 2005 [1]
[1] http://www.egge.net/~savory/blog_mar_04.htm

Amazon Mechanical Turk
Launched in late 2005 [1]
“ Artificial Artificial Intelligence” - Jeff Bezos
[1] http://en.wikipedia.org/wiki/Amazon.com

Amazon Web Services – From the Ground Up [1] http://www.amazon.com/gp/browse.html?node=15879911 [2] http://www.techcrunch.com/2007/09/08/search-for-steve-fossett-expands-to-amazons-mechanical-turk/
Humans augment computational requirements
Let someone else solve complex AI problems
Get along with the business of making money
Built around Human Intelligence Tasks (HITs)
Examples;
“ Transcribe audio data”
“ Are these paragraphs synonymous?”
“ Find the business in this photo”
“ Write a review...”
“ Search for survivors”
“ So far, searchers have found half a dozen previously unknown crash sites, but no sign of Fossett’s plane.” [2]

Amazon Web Services – The Example [1] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=691&categoryID=100
Mitch Garnaat
Author of Bono
“ DVS” transcoding
Tutorial [1]

Read a message from the input queue - SQS
Based on message, retrieve an input file - S3
Perform our video conversion processing - EC2
Store the generated files - S3
Write a message the the output queue - SQS
Delete the input message from the input queue - SQS

Input Message
Bucket: garnaat_fileit
InputKey: f84e4a20b571abc69baf2277d193e596
Date: Tue, 20 Feb 2007 17:21:21 GMT
OriginalFileName: MVI_3113.AVI
Size: 1126472
Output Message
Bucket: garnaat_fileit
InputKey: f84e4a20b571abc69baf2277d193e596
Date: Tue, 20 Feb 2007 17:21:21 GMT
OriginalFileName: MVI_3113.AVI
Size: 1126472
OutputKey: e69e376be5af6f88f81d3e31adf27988;type=video/quicktime
Server: ConvertVideo
Host: domU-12-31-34-00-02-82
Service-Read: Wed, 21 Feb 2007 01:28:14 GMT
Service-Write: Wed, 21 Feb 2007 01:28:27 GMT

Define the Service Instance
from boto.services.service import Service
import os
class ConvertVideo(Service):
ProcessingTime = 30
Command = """ffmpeg -y -i %s -f mov -r 29.97 -b 1200kb -mbd 2 -flags
+4mv+trell -aic 2 -cmp 2 -subcmp 2 -ar 48000 -ab 192 -s 320x240
-vcodec mpeg4 -acodec aac %s"""
def process_file(self, in_file_name, msg):
out_file_name = os.path.join(self.working_dir, 'out.mov')
command = self.Command % (in_file_name, out_file_name)
os.system(command)
return [(out_file_name, 'video/quicktime')]

$ boto/services/submit_files.py -b myvideos -q vc-input ~/movies
...
50 files successfully submitted.
$ boto/services/start_service.py -r -m boto.services.convertvideo
-c ConvertVideo -a ami-2eba5f47 -i vc-input -o vc-output
-e mitch@garnaat.com -n 1
python boto/services/get_results.py -q vc-status ~/movies
retrieving file: MVI_3110.mov
...
50 results successfully retrieved.
Minimum Processing Time: 2
Maximum Processing Time: 58
Average Processing Time: 17.820000
Elapsed Time: 896
Throughput: 3.348214 transactions / minute
$ python boto/services/get_results.py -q test-status ~/movies
retrieving file: MVI_3110.mov
500 results successfully retrieved.
...
Minimum Processing Time: 2
Maximum Processing Time: 60
Average Processing Time: 17.794000
Elapsed Time: 928
Throughput: 32.327586 transactions / minute
Submit Files and Setup Queue Startup EC2 worker(s) 1 Instance 10 Instances, 500 files

Okay... so what ?
$.004 per movie, $8k server => 2,000,000 movies
24 hrs * 60 min = 1440 movies / day * 365 days
525600 movies / year * 4 cameras = 2.1M

This matters to Amazon !
For the period of 4Q07, bandwidth utilized for S3 & EC2 was greater than the combined bandwidth of Amazon.com’s global websites. [1]
In Dec '07 Amazon was the #7 website in the US [2]
Same period 330,000+ new developers registered
Up 30,000+ from the previous last quarter.
Success Stories [3]
Startup Challenge [4]
Amazon Web Services – “So Who, What, Why” [1] http://biz.yahoo.com/bw/080130/20080130006013.html [2] http://www.comscore.com/press/release.asp?press=2000 [3] http://www.amazon.com/gp/browse.html?node=182241011 [4] http://www.amazon.com/gp/browse.html?node=377634011

It's not just Amazon
Google [1]
Microsoft! [2]
Yahoo
IBM [3][4][5][6][7]
Additional Examples
BigTable, GFS, MapReduce
CouchDB
Hadoop
Amazon Web Services – “So Who, What, Why” [1] http://feedblog.org/2007/12/14/google-vs-amazon-in-open-infrastructure/ [2] http://www.microsoft.com/sql/dataservices/default.mspx [3] http://redmonk.com/jgovernor/2007/04/19/why-ibm-should-acquire-amazon/ [4] http://www.ogf.org/OGF22/materials/1137/Irving+Wladawsky-Berger+Keynote.pdf [5] http://www.news.com/8301-10784_3-9817905-7.html [6] http://www-03.ibm.com/press/us/en/pressrelease/22613.wss [7] http://blogs.zdnet.com/SAAS/?p=415

Amazon Web Services – Take Away
Customers can leverage the “cloud” !
Energy getting invested “up the stack”
“ The Industry” is embracing “simplicity”
“ Functional programming” renaissance
Analogs to the resurgence of the mainframe
Application Trend - trust (but verify)
Intuition over specificity
Amazon doesn't re-instantiate components
Insanity - “doing the same thing over and over and expecting different results.”
Define Commodity - ...

Amazon Web Services – Thinkovers [1] http://www.kitchensoap.com/2008/02/27/when-do-you-get-too-big-to-use-cloudy-stuff/ [2] http://www.readwriteweb.com/archives/amazon_haas_hardware_as_a_service.php
[1]
[2]

http://www.slideshare.net	14
http://tracyrobin.blogspot.com	3
http://webcache.googleusercontent.com	1
http://www.techgig.com	1

Amazon WS Overview

by Jay on Mar 18, 2008

Accessibility

Categories

Upload Details

Usage Rights

4 Embeds 19

Statistics

Amazon WS Overview Presentation Transcript