Amazon S3 and EC2 Rob Tweed M/Gateway Developments Ltd

Background March 2006: Amazon launched Simple Storage Service (S3) August 2006: Amazon launched Elastic Compute Cloud (EC2) Essentially making virtual processing and disk storage available from their existing infrastructure Low-cost, pay-as-you-go basis No up-front costs

March 2006: Amazon launched Simple Storage Service (S3)

August 2006: Amazon launched Elastic Compute Cloud (EC2)

Essentially making virtual processing and disk storage available from their existing infrastructure

Low-cost, pay-as-you-go basis

No up-front costs

S3 Remote data storage Low cost, pay-as-you go No up-front costs High-availability High bandwidth Uses Amazon’s own networking infrastructure

Remote data storage

Low cost, pay-as-you go

No up-front costs

High-availability

High bandwidth

Uses Amazon’s own networking infrastructure

S3 Costs Storage $0.15 per GB-Month of storage used Data Transfer $0.10 per GB - all data transfer in $0.18 per GB - first 10 TB / month data transfer out $0.16 per GB - next 40 TB / month data transfer out $0.13 per GB - data transfer out / month over 50 TB Requests $0.01 per 1,000 PUT or LIST requests $0.01 per 10,000 GET and all other requests*

Storage $0.15 per GB-Month of storage used Data Transfer $0.10 per GB - all data transfer in

$0.18 per GB - first 10 TB / month data transfer out $0.16 per GB - next 40 TB / month data transfer out $0.13 per GB - data transfer out / month over 50 TB

Requests $0.01 per 1,000 PUT or LIST requests $0.01 per 10,000 GET and all other requests*

S3 Access REST interface Secure: uses HMAC-SHA1 authentication keys Data can be private or made publicly accessible (read or read/write) If publicly accessible: simple URL

REST interface

Secure: uses HMAC-SHA1 authentication keys

Data can be private or made publicly accessible (read or read/write)

If publicly accessible:

simple URL

S3 Database Structure You must have an S3 Account Id Buckets Equivalent of directories Single, common namespace across S3 But bucketNames can include “/”, eg mgateway/backups/presentations Objects Equivalent of files Up to 5Gb in size Identified by key (== filename)

You must have an S3 Account Id

Buckets

Equivalent of directories

Single, common namespace across S3

But bucketNames can include “/”, eg

mgateway/backups/presentations

Objects

Equivalent of files

Up to 5Gb in size

Identified by key (== filename)

S3 Data Access Accessible using simple HTTP URLs http://s3.amazonaws.com/ bucket / key http:// bucket .s3.amazonaws.com/ key http:// bucket / key where bucket is a DNS CNAME record pointing to s3.amazonaws.com)

Accessible using simple HTTP URLs

http://s3.amazonaws.com/ bucket / key

http:// bucket .s3.amazonaws.com/ key

http:// bucket / key

where bucket is a DNS CNAME record pointing to s3.amazonaws.com)

S3: Uses Remote Backup Archiving Primary online storage Examples: SmugMug (similar to Flickr) Photos saved in S3 storage Estimated to have saved $1m in storage costs Pluggable Storage Engine for MySQL: made available as tables and rows in MySQL

Remote Backup

Archiving

Primary online storage

Examples:

SmugMug (similar to Flickr)

Photos saved in S3 storage

Estimated to have saved $1m in storage costs

Pluggable Storage Engine for MySQL:

made available as tables and rows in MySQL

S3 Value For Money Very cheap for small to medium volume Becomes expensive at very high data volumes and high rates of access

Very cheap for small to medium volume

Becomes expensive at very high data volumes and high rates of access

S3 Client for Cach é Part of Enterprise Web Developer (EWD) Free to download and use Web interface and command-line APIs Create Read Update Delete Both buckets and objects

Part of Enterprise Web Developer (EWD)

Free to download and use

Web interface and command-line APIs

Create

Read

Update

Delete

Both buckets and objects

EC2 Low-cost online virtual machines Pay-as-you-go No up-front costs High Availability High bandwidth (published as 250 Mbs!) Customisable AMIs (equivalent to VMs) You choose: How powerful each AMI is Number of instances you run

Low-cost online virtual machines

Pay-as-you-go

No up-front costs

High Availability

High bandwidth (published as 250 Mbs!)

Customisable AMIs (equivalent to VMs)

You choose:

How powerful each AMI is

Number of instances you run

EC2 Costs $0.10 - Small Instance (Default) 1.7 GB of memory, 1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit), 160 GB of instance storage, 32-bit platform $0.40 - Large Instance 7.5 GB of memory, 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each), 850 GB of instance storage, 64-bit platform $0.80 - Extra Large Instance 15 GB of memory, 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each), 1690 GB of instance storage, 64-bit platform Pricing is per instance-hour consumed for each instance type.

$0.10 - Small Instance (Default)

1.7 GB of memory, 1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit), 160 GB of instance storage, 32-bit platform

$0.40 - Large Instance

7.5 GB of memory, 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each), 850 GB of instance storage, 64-bit platform

$0.80 - Extra Large Instance

15 GB of memory, 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each), 1690 GB of instance storage, 64-bit platform

Pricing is per instance-hour consumed for each instance type.

EC2 Costs (cont’d) Internet Data Transfer $0.10 per GB - all data transfer in $0.18 per GB - first 10 TB / month data transfer out $0.16 per GB - next 40 TB / month data transfer out $0.13 per GB - data transfer out / month over 50 TB Data transfer "in" and "out" refers to transfer into and out of Amazon EC2.

Internet Data Transfer $0.10 per GB - all data transfer in $0.18 per GB - first 10 TB / month data transfer out $0.16 per GB - next 40 TB / month data transfer out $0.13 per GB - data transfer out / month over 50 TB

Data transfer "in" and "out" refers to transfer into and out of Amazon EC2.

EC2 Access Startup, configuration: Command prompt APIs Google Plug-in (recommended!) SSH, eg puTTY, WinSCP Root access RSA key-pair authentication Assigned an IP address by Amazon Register with domain name if required

Startup, configuration:

Command prompt APIs

Google Plug-in (recommended!)

SSH, eg puTTY, WinSCP

Root access

RSA key-pair authentication

Assigned an IP address by Amazon

Register with domain name if required

EC2 AMIs Public Generic machine configurations, mostly Linux Private Customised versions, pre-loaded with extra software or applications and/or custom-configured Stored in your S3 storage “ Paid-for” Negotiable with Amazon Determine a marked-up cost structure for usage Amazon collect the fees Amazon take their standard costs + small markup Amazon forward you the difference

Public

Generic machine configurations, mostly Linux

Private

Customised versions, pre-loaded with extra software or applications and/or custom-configured

Stored in your S3 storage

“ Paid-for”

Negotiable with Amazon

Determine a marked-up cost structure for usage

Amazon collect the fees

Amazon take their standard costs + small markup

Amazon forward you the difference

EC2 uses Web site Instantly scalable if peaks in demand by starting new instances Demos Test platforms Eg simulating large numbers of users

Web site

Instantly scalable if peaks in demand by starting new instances

Demos

Test platforms

Eg simulating large numbers of users

EC2 Issues Can be more costly than a leased or rented physical or virtual server Key issue is that when shut down, the instance vanishes and nothing is persisted Next time you start an instance it starts from scratch again If you want to persist data, it’s your responsibility

Can be more costly than a leased or rented physical or virtual server

Key issue is that when shut down, the instance vanishes and nothing is persisted

Next time you start an instance it starts from scratch again

If you want to persist data, it’s your responsibility

EC2 for Cach é We have a private AMI: Fedora Linux + Apache + PHP Cach é 2007.1 MGWSI Gateway EWD Considering making available as Paid AMI Issue is Cach é licensing InterSystems’ position? Cost structure?

We have a private AMI:

Fedora Linux + Apache + PHP

Cach é 2007.1

MGWSI Gateway

EWD

Considering making available as Paid AMI

Issue is Cach é licensing

InterSystems’ position?

Cost structure?

EC2 for Cach é How to implement persistence of Cache data on EC2 machines? Using S3? By applying journalling? Some other lower-level techniques? May be something ISC would have to implement?

How to implement persistence of Cache data on EC2 machines?

Using S3?

By applying journalling?

Some other lower-level techniques?

May be something ISC would have to implement?

Conclusions S3 and EC2 are important and useful technologies They are worth examining and evaluating S3 integration with Cach é works very well EC2 provides a quick and easy way of making Cach é available on the web But bear in mind the persistence issue! Watch for equivalent services emerging

S3 and EC2 are important and useful technologies

They are worth examining and evaluating

S3 integration with Cach é works very well

EC2 provides a quick and easy way of making Cach é available on the web

But bear in mind the persistence issue!

Watch for equivalent services emerging