Building Highly Scalable Web Applications

Amazon Web Services: Building Highly Scalable Web Applications Institutional Web Management Workshop July 2007 Jeff Barr Senior Web Services Evangelist [email_address]

Today
Introduction
Amazon Web Services overview
Amazon EC2
Amazon S3
Amazon SQS
Application Architecture
Questions

Who am I?
Software development background
5 Years with Amazon:
Senior Developer
Web Services Evangelist
Senior Web Services Evangelist
Veteran of several startups
Microsoft .Net Team
Visix Software co-founder

Amazon?
Consumers (Buy)
Associates, Sellers & Merchants (Sell)
Developers (Innovate)

Issues Facing Developers
70% of Web Development Effort is “Muck”:
Data Centers
Bandwidth / Power / Cooling
Operations
Staffing
Scaling is Difficult and Expensive:
Large Up-Front Investment
Invest Ahead of Demand
Load is Unpredictable

Dream or Nightmare?
Slashdot/Digg/TechCrunch Effect
Rapid, unexpected customer demand/growth

Seasonal Spikes

More Seasonal Spikes
“ Every year, we take the busiest minute of the busiest hour of the busiest day and build capacity on that, we built our systems to (handle that load) and we went above and beyond that.” *
-- Scott Gulbransen
Intuit Spokesman
* http://news.com.com/2100-1038_3-6177341.html

Solution: Web-Scale Computing
Scale capacity on demand
Turn fixed costs into variable costs
Always available
Rock-solid reliability
Simple APIs and conceptual models
Cost-effective
Reduced time to market
Focus on product & core competencies

Amazon Web Services Are...
A set of APIs and business models which give developer-level access to Amazon’s infrastructure and content:
Data As A Service
Amazon E-Commerce Service
Amazon Historical Pricing
Search As A Service
Alexa Web Information Service
Alexa Top Sites
Alexa Site Thumbnail
Alexa Web Search Platform
Infrastructure As A Service
Amazon Simple Queue Service
Amazon Simple Storage Service
Amazon Elastic Compute Cloud
People As A Service
Amazon Mechanical Turk

The Commercial Side
Standard licensing terms
Commercially usable
Aggressive pricing
Monthly credit card billing
Self-serve model:
Sign up as developer
Choose services
Agree to service licenses
Enter payment info
Start coding

Web Services Billing

Infrastructure Services Compute Store Message Elastic Compute Cloud Simple Storage Service Simple Queue Service

Amazon Simple Storage Service S3

Amazon Simple Storage Service $.15 per GB per month storage
Object-Based Storage
1 B – 5 GB / object
Fast, Reliable, Scalable
Redundant, Dispersed
99.99% Availability Goal
Private or Public
Per-object URLs & ACLs
BitTorrent Support
$.10 - $.18 per GB data transfer $.01 for 1000 to 10000 requests

Amazon S3 Concepts
Objects:
Opaque data to be stored (1 byte … 5 Gigabytes)
Authentication and access controls
Buckets:
Object container – any number of objects
100 buckets per account / buckets are “owned”
Keys:
Unique object identifier within bucket
Up to 1024 bytes long
Flat object storage model
Standards-Based Interfaces:
REST and SOAP
URL-Addressability – every object has a URL

S3 SOAP/Query API
Service:
ListAllMyBuckets
Buckets:
CreateBucket
DeleteBucket
ListBucket
GetBucketAccessControlPolicy
SetBucketAccessControlPolicy
GetBucketLoggingStatus
SetBucketLoggingStatus
Objects:
PutObject
PutObjectInline
GetObject
GetObjectExtended
DeleteObject
GetObjectAccessControlPolicy
SetObjectAccessControlPolicy

Amazon Simple Queue Service SQS

Amazon Simple Queue Service $.10 per 1000 messages
Scalable Queuing
Elastic Capacity
Reliable, Simple, Secure
Inter-process messaging, data buffering, architecture component
$.10 - $.18 per GB data transfer

Amazon SQS Concepts
Queues:
Named message container
Persistent
Messages:
Up to 256KB of data per message
Peek / Lock access model
Scalable:
Unlimited number of queues per account
Unlimited number of messages per queue

Amazon SQS At Work

SQS SOAP/Query API
Queues:
ListQueues
DeleteQueue
SetVisibilityTimeout
GetVisibilityTimeout
Messages:
SendMessage
ReceiveMessage
DeleteMessage
PeekMessage
Security:
AddGrant
ListGrants
RemoveGrant

Amazon Elastic Compute Cloud EC2

Amazon Elastic Compute Cloud $.10 per server hour
Virtual Compute Cloud
Elastic Capacity
1.7 GHz x86
1.7 GB RAM
160 GB Disk
250 MB/Second Network
Network Security Model
Time or Traffic-based Scaling, Load testing, Simulation and Analysis, Rendering, Software as a Service Platform, Hosting
$.10 - $.18 per GB data transfer

Amazon EC2 Concepts
Amazon Machine Image (AMI):
Bootable root disk
Pre-defined or user-built
Catalog of user-built AMIs
OS: Fedora, Centos, Gentoo, Debian, Ubuntu, Windows Server
App Stack: LAMP, mpiBLAST, Hadoop
Instance:
Running copy of an AMI
Launch in less than 2 minutes
Start/stop programmatically
Network Security Model:
Explicit access control
Security groups
Inter-service bandwidth is free

Root-level access

Amazon EC2 At Work
Startups
Cruxy – Media transcoding
GigaVox Media – Podcast Management
Fortune 500 clients:
High-Impact, S hort-Term Projects
Development Host
Science / Research:
Hadoop / MapReduce
mpiBLAST
Load-Management and Load Balancing Tools:
Pound
Weogeo
Rightscale

EC2 SOAP/Query API
Images:
RegisterImage
DescribeImages
DeregisterImage
Instances:
RunInstances
DescribeInstances
TerminateInstances
GetConsoleOutput
RebootInstances
Keypairs:
CreateKeyPair
DescribeKeyPairs
DeleteKeyPair
Image Attributes:
ModifyImageAttribute
DescribeImageAttribute
ResetImageAttribute
Security Groups:
CreateSecurityGroup
DescribeSecurityGroups
DeleteSecurityGroup
AuthorizeSecurityGroupIngress
RevokeSecurityGroupIngress

Sample Web-Scale Architecture GigaVox Media

Web-Scale Architecture

GigaVox Economics
Implemented Amazon S3, Amazon EC2 and Amazon SQS in November 2006
Created an infinitely scalable infrastructure for less than $100 - building the same infrastructure themselves would have cost thousands of dollars
Reduced staffing requirements - far less responsibility for 24x7 operations

Thank You!
Pointers
Portal
http://aws.amazon.com
Blog
http:// aws.typepad.com
EC2
http://aws.amazon.com/ec2
S3
http://aws.amazon.com/s3
Resource Center
http://aws.amazon.com/resources
Forums
http://aws.amazon.com/forums
Jeff Barr Senior Web Services Evangelist [email_address]

http://www.slideshare.net	122
http://www.ukoln.ac.uk	13
http://lanyrd.com	3
http://jisi.dreamblog.jp	1
http://wildfire.gigya.com	1
http://webcache.googleusercontent.com	1

Building Highly Scalable Web Applications

by IWMW on Jul 11, 2007

Accessibility

Categories

Tags

Upload Details

Usage Rights

6 Embeds 141

Statistics

Building Highly Scalable Web Applications — Presentation Transcript