Amazon WS Overview
Upcoming SlideShare
Loading in...5
×
 

Amazon WS Overview

on

  • 10,932 views

A compact overview of Amazon's WebServices offerings.

A compact overview of Amazon's WebServices offerings.

Statistics

Views

Total Views
10,932
Views on SlideShare
10,912
Embed Views
20

Actions

Likes
6
Downloads
178
Comments
0

5 Embeds 20

http://www.slideshare.net 14
http://tracyrobin.blogspot.com 3
http://webcache.googleusercontent.com 1
http://www.techgig.com 1
http://www.slideee.com 1

Accessibility

Categories

Upload Details

Uploaded via as OpenOffice

Usage Rights

CC Attribution-ShareAlike LicenseCC Attribution-ShareAlike License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Amazon WS Overview Amazon WS Overview Presentation Transcript

  • Amazon Web Services – Introduction & Discussion
  • Amazon Web Services –
    • Overview of Amazon's Services
      • Explore each type of service
    • Not implementation specific [1]
    • Open Discussion
    [1] http://highscalability.com/amazon-architecture
  • Amazon Web Services – The Picture Associates Alexa Marketplace DevPay WebStore Prime Grocery Historical Pricing [1] http://highscalability.com/paper-dynamo-amazon-s-highly-available-key-value-store
    • Small Pieces Loosely Joined –
    • “ ...There are intriguing assertions: that the Web is "broken on purpose" and that its many pockets of erroneous information and its available forums [...] let people feel more comfortable with their own inherent imperfections . [...] Weinberger's analysis, though occasionally facile and too relentlessly optimistic and overstated , is surely destined to be the subject of furious debate in chat rooms the cyber-world over. --H. O'Billovich” [1]
    [1] http://www.amazon.com/Small-Pieces-Loosely-Joined-Unified/dp/B00008NRGI/ref=pd_bbs_sr_1?ie=UTF8&s=books&qid=1204731761&sr=8-1 Amazon Web Services – “Defined”
    • Small Pieces Loosely Joined
    • Think Unix Model –
      • “ Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.” -- Doug McIlroy [1]
    [1] [Salus] Peter H. Salus. A Quarter-Century of Unix. Addison-Wesley. 1994. ISBN 0-201-54777-5 Amazon Web Services – “Defined”
    • Small Pieces Loosely Joined
    • Think Unix Model
    • “ Single” Deployment
    • Embrace the Power of Simplicity
    • Open Collaboration
    • Leverages “Momentum Makers”
    • “ Force Multiplication”
      • Estimates of 40% of sales from affiliates [1]
    Amazon Web Services – “Defined” [1] http://en.wikipedia.org/wiki/Amazon.com
  • Amazon Web Services – From the Ground Up
    • Provides a Storage Service – Since April 2006 [1]
    • Geographically distributed content network
      • U.S. and European Union
      • REST and SOAP interfaces over HTTP
      • A BitTorrent(TM) protocol interface is provided
    • Does have a Service Level Agreement [2]
      • 25% for < 99%
      • 10% for > 99% and < 99.9%
      • “ 3 nines” implies <= 8 hours 46 minutes down time
    [1] http://en.wikipedia.org/wiki/Amazon_S3 [2] http://www.amazon.com/gp/browse.html%3fnode%3d379654011&token=3A0F170E7CEFE27BDC730D3D7344512BC1296B83
    • Uses – Extensive use internally
      • SmugMug [1]
      • Many independent blogs & “startups” [2]
      • Even Firefox extensions as interfaces !
    • Cost – Less then you'd think [3]
      • U.S. – $0.15 per GB/month
        • In: $0.10 /GB, Out: graduated $0.13 - 0.18 GB/month
          • $.01 per 1,000 PUT or 10,000 GET requests
        • 20GB & 2GB is ~$40 per year [4]
      • Webmail.us reportedly saved 75% with S3
      • No cost to transfer between Amazon Services !
    [1] http://blogs.smugmug.com/don/2006/11/10/amazon-s3-show-me-the-money/ [2] http://calculator.s3.amazonaws.com/calc5.html [3] http://www.codinghorror.com/blog/archives/000808.html [4] http://www.jungledisk.com/ Amazon Web Services – From the Ground Up
  • Amazon Web Services – From the Ground Up
    • Files (objects) are placed in “buckets”
      • Unlimited objects in a bucket
        • Buckets host objects between 1byte to 5GB
        • Objects can be public, private or specific rights
      • Retrieved via a unique key – global address space
        • Typically linked with an MD5sum
      • Subdirectories are done via strategic naming
        • “ jay” and “jay/photos” are logically independent but might be related at an application level
        • Another interesting solution involves loopback mounts [1]
    [1] http://developer.amazonwebservices.com/connect/thread.jspa?threadID=10271&tstart=30
    • Examples:
      • CLI [1]
        • ./s3-get -k MyAWSAccessKeyId -s AWSSecretAccessKeyIdFile /bucket/pathtoobject
      • Python [2]
        • >>> from boto.s3.connection import S3Connection, Key
        • >>> conn = S3Connection('<aws access key>', '<aws secret key>')
        • >>> bucket = conn.create_bucket('garnaat_fileit')
        • >>> k = Key(bucket)
        • >>> k.key = 'foobar'
        • >>> k.set_contents_from_string('This is a test of S3')
        • >>> k.ghet_contents_as_string()
        • >>> k.key = 'myfile'
        • >>> k.set_contents_from_filename('foo.jpg')
        • >>> k.get_contents_to_filename('bar.jpg')
    [1] http://code.google.com/p/s3-bash/ [2] http://boto.googlecode.com/svn/trunk/doc/s3_tut.txt Amazon Web Services – From the Ground Up
  • Amazon Web Services – Redefining the Dictionary
    • ... if databases were straightforward and easy
    • Web Services interface – Queries and Submissions
    • No schema –
      • Automatic indexes
      • Loose enforcement of structure
      • Consistency is “eventual” (MVCC) [1]
    • Automatic scaling & caching
      • No configuration & management required
      • For “real-time data” applications; e.g. shopping carts
      • Heavily reliant on S3
    [1] http://en.wikipedia.org/wiki/Multiversion_concurrency_control http://www.sriramkrishnan.com/blog/2007/12/amazon-simpledb-technical-overview.html
    • CREATE a domain to store data
      • Beta constraint – 100 domains max 10 GB each
    • GET, PUT or DELETE items
      • Items have attribute-value pairs
        • “ size = large”
        • “ color = blue, red”
      • Each item can have up to 256 attribute values.
        • Each attribute value can range from 1 to 1,024 bytes.
    • QUERY – max 5 seconds
      • Operators: =, !=, <, > <=, >=, STARTS-WITH, AND, OR, NOT, INTERSECTION AND UNION.
    http://www.amazon.com/b/ref=sc_fe_l_2?ie=UTF8&node=342335011&no=3435361&me=A36L942TSJ2AJA Amazon Web Services – Redefining the Dictionary
    • Provides a Storage Queue – Simple!
      • For $1 a user can transmit over 500,000 messages [1]
      • $0.01 per 10,000 Amazon SQS Requests ($0.000001 per Request)
      • Other bandwidth fees still apply (similar to S3 #'s)
    • Message Characteristics
      • Max messages size - 8KB
      • Latency should be anticipated [2]
      • Transmit and Receive order not guaranteed
      • Messages held for 4 days max
    • Data Flow –
      • Find an Amazon SQS queue
      • SendMessage to the queue
      • ReceiveMessage is called
      • ReceiveMessage returns a message
        • Locks that message until the visibility timeout has passed.
      • DeleteMessage removes the message from the queue
    [1] http://www.amazon.com/Simple-Queue-Service-home-page/b?ie=UTF8&node=13584001 [2] http://en.wikipedia.org/wiki/Amazon_SQS Amazon Web Services – “Ground Control to Major Tom...”
    • Example:
      • Python [1]
        • >>> from boto.sqs.connection import SQSConnection, Message
        • >>> conn = SQSConnection('<aws access key>', '<aws secret key>')
        • >>> q = conn.create_queue('myqueue', 60)
        • >>> m = Message()
        • >>> m.set_body('This is my first message.')
        • >>> rs = q.write(m)
        • >>> rs = q.get_messages()
        • >>> rs[0].get_body()
        • >>> q.delete_message(rs[0])
    [1] http://boto.googlecode.com/svn/trunk/doc/sqs_tut.txt [2] http://aws.typepad.com/aws/2007/05/sqs_super_queue.html Amazon Web Services – “Ground Control to Major Tom...”
  • Amazon Web Services – “Show me the Money”
    • Supports multiple forms of payment
      • Permits true micro-payment [1]
      • Costs vary based on Amazon's costs
    • Amazon manages security and fraud responsibility
    • “ GateKeeper” enforces policies for senders and receivers
    • Explicit “Payment Instructions” language
      • “ only five transactions or $50 per month&quot; or &quot;refuse payment after August 1&quot;
      • Three token exchange
    • Integrates with Amazon's existing customer experience
      • “ Tens of Millions of existing customers”
      • “ Pay Now” Widget
      • Promotes “market places” for intermediaries
    [1] http://paulstamatiou.com/2007/08/04/why-you-shouldnt-ignore-amazons-new-fps [2] http://www.amazon.com/gp/browse.html?node=342430011 [3] http://docs.amazonwebservices.com/AmazonFPS/2007-01-08/FPSGettingStartedGuide/
  • Amazon Web Services – The Sky's the Limit [1] http://www.amazon.com/b/?node=201590011
    • Provides programmatic compute capacity
      • Including pre-configured appliance images
    • “ ... allows you to obtain and configure capacity with minimal friction” [1]
    • The Basic Process
      • Build or Identify an appropriate AMI
      • Configure Security and Access
      • Start, Monitor and Stop Instances
      • Pay
  • Amazon Web Services – The Sky's the Limit
    • What's an AMI ? - Amazon Machine Instance
      • “ Golden Master” of the image you want to run
      • Linux only, Windows is a hack [1]
      • AMI's can be private [2] – Vendor tools like rPath [3]
      • Leverage an Open “Operating Systems” bazaar
        • Some AMI's can have “associate” costs per instance
    • Multiple Instances can run from a single AMI
      • By default, limited to 20, simple request for more
    • Getting started with EC2 [4]
    • Mike Culver - Introductory video [5]
    [1] http://www.enomaly.net/wiki.1137+M5c1693b2f05.0.html [2]http://docs.amazonwebservices.com/AmazonEC2/dg/2006-06-26/creating-an-ami.html [3] http://highscalability.com/product-rpath-creating-and-managing-virtual-appliances [4] http://docs.amazonwebservices.com/AWSEC2/2007-08-29/DeveloperGuide/ [5] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=583&categoryID=100
  • Amazon Web Services – The Sky's the Limit
    • Example:
      • > ec2-describe-images -x all
        • IMAGE ami-23b6534a ec2-public-images/fedora-7-apache-manafest.xml
      • > ec2-add-keypair j-keypair
        • (generates ssh keypairs)
      • > ec2-run-instance ami-23b6534a -k j-keypair
        • (could take up to a few minutes)
      • > ec2-authorize default -p 22
        • (Authorize port 22 for the default security group)
      • > ec2-describe-instances
        • (Shows instance info including DNS name)
      • > ec2-terminate-instances <image name>
      • > ec2run ami-e3a5408a -n 20 -g appserver
  • Amazon Web Services – The Sky's the Limit [1] http://www.amazon.com/b/?node=201590011 [2] http://www.amazon.com/b/?node=201590011#measure [3] http://www.amazon.com/b/ref=sc_fe_c_0_201590011_2?ie=UTF8&node=370375011&no=201590011&me=A36L942TSJ2AJA
    • Small Instance - $.10 / hour
      • 1.7 GB of memory,
      • 1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit)
      • 160 GB of instance storage, 32-bit platform
    • Large Instance - $0.40 / hour
      • 7.5 GB of memory
      • 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each)
      • 850 GB of instance storage, 64-bit platform
    • Extra Large Instance - $0.80 / hour
      • 15 GB of memory
      • 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each)
      • 1690 GB of instance storage, 64-bit platform
    • 1 EC2 Compute Unit is ~ 1.0-1.2 GHz 2007 Opteron or 2007 Xeon
    • Bandwidth costs of still apply at 256Mb / sec
  • Amazon Web Services – The Sky's the Limit
    • Uses x509 certificates for Developer Auth.
      • SSH keypairs for instance authenticaion
    • Instances can belong to “security groups”
    • Network is dynamic via NAT
      • IP's and hostnames change
    • Momentum
      • RedHat is deploying a “cloud” on EC2 [1]
      • Facebook [2]
      • Hadoop [3]
      • EC2 for Python Programmers [4]
    [1] http://www.redhat.com/solutions/cloud/ [2] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1044&ref=featured http://www.amazon.com/gp/browse.html?node=391557011 [3] http://blog.awswebshop.com/2008/02/27/taking-massive-distributed-computing-to-the-common-man-hadoop-on-amazon-ec2s3/ [4] http://jimmyg.org/2007/09/01/amazon-ec2-for-people-who-prefer-debian-and-python-over-fedora-and-java/
  • Amazon Web Services – From the Ground Up
    • Amazon Mechanical Turk
      • Launched in late 2005 [1]
    [1] http://www.egge.net/~savory/blog_mar_04.htm
  • Amazon Web Services – From the Ground Up
    • Amazon Mechanical Turk
      • Launched in late 2005 [1]
    • “ Artificial Artificial Intelligence” - Jeff Bezos
    [1] http://en.wikipedia.org/wiki/Amazon.com
  • Amazon Web Services – From the Ground Up [1] http://www.amazon.com/gp/browse.html?node=15879911 [2] http://www.techcrunch.com/2007/09/08/search-for-steve-fossett-expands-to-amazons-mechanical-turk/
    • Humans augment computational requirements
    • Let someone else solve complex AI problems
      • Get along with the business of making money
    • Built around Human Intelligence Tasks (HITs)
    • Examples;
      • “ Transcribe audio data”
      • “ Are these paragraphs synonymous?”
      • “ Find the business in this photo”
      • “ Write a review...”
      • “ Search for survivors”
        • “ So far, searchers have found half a dozen previously unknown crash sites, but no sign of Fossett’s plane.” [2]
  • Amazon Web Services – The Example [1] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=691&categoryID=100
    • Mitch Garnaat
      • Author of Bono
    • “ DVS” transcoding
    • Tutorial [1]
    • Read a message from the input queue - SQS
    • Based on message, retrieve an input file - S3
    • Perform our video conversion processing - EC2
    • Store the generated files - S3
    • Write a message the the output queue - SQS
    • Delete the input message from the input queue - SQS
    Amazon Web Services – The Example [1] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=691&categoryID=100
  • Amazon Web Services – The Example [1] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=691&categoryID=100
    • Input Message
      • Bucket: garnaat_fileit
      • InputKey: f84e4a20b571abc69baf2277d193e596
      • Date: Tue, 20 Feb 2007 17:21:21 GMT
      • OriginalFileName: MVI_3113.AVI
      • Size: 1126472
    • Output Message
      • Bucket: garnaat_fileit
      • InputKey: f84e4a20b571abc69baf2277d193e596
      • Date: Tue, 20 Feb 2007 17:21:21 GMT
      • OriginalFileName: MVI_3113.AVI
      • Size: 1126472
      • OutputKey: e69e376be5af6f88f81d3e31adf27988;type=video/quicktime
      • Server: ConvertVideo
      • Host: domU-12-31-34-00-02-82
      • Service-Read: Wed, 21 Feb 2007 01:28:14 GMT
      • Service-Write: Wed, 21 Feb 2007 01:28:27 GMT
  • Amazon Web Services – The Example [1] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=691&categoryID=100
    • Define the Service Instance
        • from boto.services.service import Service
        • import os
        • class ConvertVideo(Service):
        • ProcessingTime = 30
        • Command = &quot;&quot;&quot;ffmpeg -y -i %s -f mov -r 29.97 -b 1200kb -mbd 2 -flags
        • +4mv+trell -aic 2 -cmp 2 -subcmp 2 -ar 48000 -ab 192 -s 320x240
        • -vcodec mpeg4 -acodec aac %s&quot;&quot;&quot;
        • def process_file(self, in_file_name, msg):
        • out_file_name = os.path.join(self.working_dir, 'out.mov')
        • command = self.Command % (in_file_name, out_file_name)
        • os.system(command)
        • return [(out_file_name, 'video/quicktime')]
  • Amazon Web Services – The Example [1] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=691&categoryID=100
    • $ boto/services/submit_files.py -b myvideos -q vc-input ~/movies
    • ...
    • 50 files successfully submitted.
    • $ boto/services/start_service.py -r -m boto.services.convertvideo
    • -c ConvertVideo -a ami-2eba5f47 -i vc-input -o vc-output
    • -e mitch@garnaat.com -n 1
    • python boto/services/get_results.py -q vc-status ~/movies
    • retrieving file: MVI_3110.mov
    • ...
    • 50 results successfully retrieved.
    • Minimum Processing Time: 2
    • Maximum Processing Time: 58
    • Average Processing Time: 17.820000
    • Elapsed Time: 896
    • Throughput: 3.348214 transactions / minute
    • $ python boto/services/get_results.py -q test-status ~/movies
    • retrieving file: MVI_3110.mov
    • 500 results successfully retrieved.
    • ...
    • Minimum Processing Time: 2
    • Maximum Processing Time: 60
    • Average Processing Time: 17.794000
    • Elapsed Time: 928
    • Throughput: 32.327586 transactions / minute
    Submit Files and Setup Queue Startup EC2 worker(s) 1 Instance 10 Instances, 500 files
  • Amazon Web Services – The Example [1] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=691&categoryID=100
    • Okay... so what ?
    • $.004 per movie, $8k server => 2,000,000 movies
    • 24 hrs * 60 min = 1440 movies / day * 365 days
    • 525600 movies / year * 4 cameras = 2.1M
    • This matters to Amazon !
      • For the period of 4Q07, bandwidth utilized for S3 & EC2 was greater than the combined bandwidth of Amazon.com’s global websites. [1]
        • In Dec '07 Amazon was the #7 website in the US [2]
      • Same period 330,000+ new developers registered
        • Up 30,000+ from the previous last quarter.
    • Success Stories [3]
    • Startup Challenge [4]
    Amazon Web Services – “So Who, What, Why” [1] http://biz.yahoo.com/bw/080130/20080130006013.html [2] http://www.comscore.com/press/release.asp?press=2000 [3] http://www.amazon.com/gp/browse.html?node=182241011 [4] http://www.amazon.com/gp/browse.html?node=377634011
    • It's not just Amazon
      • Google [1]
      • Microsoft! [2]
      • Yahoo
    • IBM [3][4][5][6][7]
    • Additional Examples
      • BigTable, GFS, MapReduce
      • CouchDB
      • Hadoop
    Amazon Web Services – “So Who, What, Why” [1] http://feedblog.org/2007/12/14/google-vs-amazon-in-open-infrastructure/ [2] http://www.microsoft.com/sql/dataservices/default.mspx [3] http://redmonk.com/jgovernor/2007/04/19/why-ibm-should-acquire-amazon/ [4] http://www.ogf.org/OGF22/materials/1137/Irving+Wladawsky-Berger+Keynote.pdf [5] http://www.news.com/8301-10784_3-9817905-7.html [6] http://www-03.ibm.com/press/us/en/pressrelease/22613.wss [7] http://blogs.zdnet.com/SAAS/?p=415
  • Amazon Web Services – Take Away
    • Customers can leverage the “cloud” !
      • Energy getting invested “up the stack”
    • “ The Industry” is embracing “simplicity”
      • “ Functional programming” renaissance
        • Analogs to the resurgence of the mainframe
      • Application Trend - trust (but verify)
        • Intuition over specificity
    • Amazon doesn't re-instantiate components
      • Insanity - “doing the same thing over and over and expecting different results.”
    • Define Commodity - ...
  • Amazon Web Services – Thinkovers [1] http://www.kitchensoap.com/2008/02/27/when-do-you-get-too-big-to-use-cloudy-stuff/ [2] http://www.readwriteweb.com/archives/amazon_haas_hardware_as_a_service.php
    • [1]
    • [2]