Use Somebody Else's Infrastructure:
Utilizing Amazon S3 and EC2
Jonathan Weiss
Peritor Wissensmanagement GmbH, www.peritor.com
Web 2.0 Expo Berlin, 2007

Peritor
• Consultancy from Berlin, Germany
• Specialized on Architecture, Scaling and
Ruby on Rails
• A lot of our clients use Amazon Web
Services
2

www.peritormail.com
We run a Webmail portal on top of Amazon Web Services
3

Starting Point
One machine:
• Apache
• Ruby on Rails / PHP / Perl / Java / …
• MySQL
4

Worst Case Populariy
5

A Difficult Path
6

Problem: Backup
• High availability
• Redundancy
• Very big data sets
7

Problem: File System
• Important files have to be accessed by many servers
• NFS / Samba not practical
8

Problem: Messaging
• Many servers on different locations have to communicate
• Creating a reliable and scalable message queue is really hard
9

Problem: Spontaneous Traffic
10

Problem: Load Fluctuation
11

Don‘t reinvent the wheel!
12

Amazon Web Services
Simple Storage Service - S3 Elastic Compute Cloud - EC2 Simple Queue Service - SQS E-Commerce..
Redundant Data Store Virtual server per hour Reliable Message System …….
$0,15 per GB data per $0,10 per CPU hour $0,10 per 1,000 messages
month
$0,10 - $0,20 per GB $0,10 - $0,18 per GB
$0,10 - $0,20 per GB traffic traffic
traffic
13

S3 - Simple Storage Service
• Redundant storage - as much as you like
• max. 5 GB per object
• Organized in „Buckets“
• Web Service API for uploads
• Downloads via
• Web Service
• HTTP / HTTPS
• BitTorrent
14

S3 - Buckets
S3
• Unique over all S3
• Contains many
MyBucket_name www.peritor.com
key-value-metadata tupel
foto_5.jpg site/screen.css
• Cannot contain other buckets!
backups/januar.zip site/script.js
• Key can contain „/“ fotos/2007/001.png FreeBSD-6.2.iso
15

S3 with AWS::S3
Upload
Download
16

EC2 - Elastic Compute Cloud
• Based on XEN virtualization
• On demand virtual servers - controlled with Web Service API
• Use your favorite Linux distro (Linux 2.6.16),
Amazon Machine Images (AMI) are stored on S3
• ACLs for hosts/ports access control
17

EC2 Tools
List availabe images
Start a new instance
Login with SSH
Shutdown instance
18

amazon-ec2 gem
Setup
Usage
19

SQS - Simple Queue Service
• Distributed, highly scalable and reliable message system
• Unlimited number of queues and messages
• Locking / releasing built-in
• 256 KB message payload
• Basic permission model
20

SQS gem
Setup
Usage
21

How does this solve
my problems?
22

S3 - Backup
• s3sync.rb • S3Browser
• Brackup • Firefox S3 Organizer extension
• Jungle Disk •…
• S3 FUSE
• s3DAV
• Duplicity
23

s3sync.rb
Backup
Restore
24

S3 Asset Host
25

S3 Asset Host
Setup DNS
Rails configuration
26

S3 Asset Host
welcome.rhtml template
Output
27

S3 - Authenticated User Data
28

attachment_fu Rails plugin
Setup
29

attachment_fu Rails plugin
Upload
Download
30

On-Demand Computing with EC2
Time based, e.g. with cron
31

On-Demand Computing with EC2
Load based, e.g. with Monit
32

On-Demand Computing with EC2
33

EC2 for extra capacity
34

Peritor Wissensmanagement GmbH
Lenbachstrasse 2
12157 Berlin
Internet: www.peritor.com Phone: +49 (0)30 69 40 11 94
Email: info@peritor.com Fax: +49 (0)30 69 40 11 95
© Peritor Wissensmanagement GmbH - All Rights Reserved
35