Scalability with AWS Set Amazon’s Servers on Fire, Not Yours Parks Hall Fire, July 3, 2002 - http://www.acadweb.wwu.edu/dbrunner/

Set Amazon’s Servers on Fire, Not Yours

Bootstrapped. $10M+. Profitable. No debt. 500M+ photos. Petabytes at S3. Growing fast. Super Heroes. Who the heck are we?

Bootstrapped.

$10M+.

Profitable.

No debt.

500M+ photos.

Petabytes at S3.

Growing fast.

Super Heroes.

Our Love Affair with S3

Always on, global, infinite storage. Inexpensive. Less than $0.15/GB/month w/replicas. Easy. REST API. (SOAP too, but...) Fast. Not 15K-SCSI fast, but Internet fast. Company focused on product, not muck. Game changer. Our Love Affair with S3

Always on, global, infinite storage.

Inexpensive. Less than $0.15/GB/month w/replicas.

Easy. REST API. (SOAP too, but...)

Fast. Not 15K-SCSI fast, but Internet fast.

Company focused on product, not muck.

Game changer.

Reliability Not 100%. Close, though. More reliable than SmugFS which was quite reliable. Lots of failure points: SmugMug’s datacenter Internet backbones Amazon’s datacenter No other software, hardware, or service we use is 100%, either. Outages happen. Rare, but happen. Deal.

Not 100%. Close, though.

More reliable than SmugFS which was quite reliable.

Lots of failure points:

SmugMug’s datacenter

Internet backbones

Amazon’s datacenter

No other software, hardware, or service we use is 100%, either.

Outages happen. Rare, but happen. Deal.

SkyNet Lives! (EC2 @ SmugMug)

SkyNet Lives! SmugMug + EC2 originally called ‘RubberBand’ Completely autonomous scaling up/down Tried to take over the world: Launched 240 XLarge instances (1920 EC2 standard instances) in one call. Re-named SkyNet :) SkyNet Lives! (EC2 @ SmugMug)

SmugMug + EC2 originally called ‘RubberBand’

Completely autonomous scaling up/down

Tried to take over the world:

Launched 240 XLarge instances (1920 EC2 standard instances) in one call.

Re-named SkyNet :)

SkyNet Lives! SkyNet Lives! (EC2 @ SmugMug) Image processing. Millions of photos/day. BIG photos. Up to 24MB, 48Mpix. 50+ Terapixels/day processed Peaky traffic on weekends, holidays Ridiculously parallel

Image processing.

Millions of photos/day.

BIG photos. Up to 24MB, 48Mpix.

50+ Terapixels/day processed

Peaky traffic on weekends, holidays

Ridiculously parallel

SkyNet Lives! SkyNet Lives! (EC2 @ SmugMug) Video processing experiment True HD - 1920x1080p. Tons of CPU horsepower. Very expensive up-front investment Unknown uptake Couldn’t have done it without EC2 Big success - customers love it.

Video processing experiment

True HD - 1920x1080p.

Tons of CPU horsepower.

Very expensive up-front investment

Unknown uptake

Couldn’t have done it without EC2

Big success - customers love it.

SkyNet Lives! SkyNet Lives! (EC2 @ SmugMug) Totally autonomous is difficult (but fun!) Lots of inputs: Historical data Type of job(s) Queue status Recent EC2 activity (Run/Terminate/etc) EC2 has start-up lag - patience required Thresholds - no Terminators! Small buffer to handle small spikes

Totally autonomous is difficult (but fun!)

Lots of inputs:

Historical data

Type of job(s)

Queue status

Recent EC2 activity (Run/Terminate/etc)

EC2 has start-up lag - patience required

Thresholds - no Terminators!

Small buffer to handle small spikes

SkyNet Lives! SkyNet Lives! (EC2 @ SmugMug)

Amazon Payments: SmugVault Photographers shooting in RAW, not JPEG. RAW isn’t web-readable. Archival. RAW files are 4X or more in size. Videographers wanting to archive footage. Tons of formats, often not web-readable. Video can be 10X or more in size. Unlimited just doesn’t work. DevPay lets us re-sell S3 to our customers

Photographers shooting in RAW, not JPEG.

RAW isn’t web-readable. Archival.

RAW files are 4X or more in size.

Videographers wanting to archive footage.

Tons of formats, often not web-readable.

Video can be 10X or more in size.

Unlimited just doesn’t work.

DevPay lets us re-sell S3 to our customers

Amazon Payments: Subscriptions Amazon Simple Pay Subscriptions New service, using it to bill SmugMug subscriptions. Lots of people don’t want to enter CC details. Lots of people trust Amazon. :) 15% adoption rate first month. Amazon handles renewals, discounts, etc.

Amazon Simple Pay Subscriptions

New service, using it to bill SmugMug subscriptions.

Lots of people don’t want to enter CC details.

Lots of people trust Amazon. :)

15% adoption rate first month.

Amazon handles renewals, discounts, etc.

Thanks! Blog : http://blogs.smugmug.com/ Slides : I’ll give to someone at Amazon. Email : andrew AT smugmug Photo sharing : http://www.smugmug.com / Amazon 50% off coupon : Amazon50

Blog : http://blogs.smugmug.com/

Slides : I’ll give to someone at Amazon.

Email : andrew AT smugmug

Photo sharing : http://www.smugmug.com /

Amazon 50% off coupon : Amazon50