Use Somebody Else's Infrastructure - Utilizing Amazon S3 and EC2

Use Somebody Else's Infrastructure: Utilizing Amazon S3 and EC2 Jonathan Weiss Peritor Wissensmanagement GmbH, www.peritor.com Web 2.0 Expo Berlin, 2007

Peritor • Consultancy from Berlin, Germany • Specialized on Architecture, Scaling and Ruby on Rails • A lot of our clients use Amazon Web Services 2

www.peritormail.com We run a Webmail portal on top of Amazon Web Services 3

Starting Point One machine: • Apache • Ruby on Rails / PHP / Perl / Java / … • MySQL 4

Worst Case Populariy 5

A Difﬁcult Path 6

Problem: Backup • High availability • Redundancy • Very big data sets 7

Problem: File System • Important ﬁles have to be accessed by many servers • NFS / Samba not practical 8

Problem: Messaging • Many servers on different locations have to communicate • Creating a reliable and scalable message queue is really hard 9

Problem: Spontaneous Trafﬁc 10

Problem: Load Fluctuation 11

Don‘t reinvent the wheel! 12

Amazon Web Services Simple Storage Service - S3 Elastic Compute Cloud - EC2 Simple Queue Service - SQS E-Commerce.. Redundant Data Store Virtual server per hour Reliable Message System ……. $0,15 per GB data per $0,10 per CPU hour $0,10 per 1,000 messages month $0,10 - $0,20 per GB $0,10 - $0,18 per GB $0,10 - $0,20 per GB traffic traffic traffic 13

S3 - Simple Storage Service • Redundant storage - as much as you like • max. 5 GB per object • Organized in „Buckets“ • Web Service API for uploads • Downloads via • Web Service • HTTP / HTTPS • BitTorrent 14

S3 - Buckets S3 • Unique over all S3 • Contains many MyBucket_name www.peritor.com key-value-metadata tupel foto_5.jpg site/screen.css • Cannot contain other buckets! backups/januar.zip site/script.js • Key can contain „/“ fotos/2007/001.png FreeBSD-6.2.iso 15

S3 with AWS::S3 Upload Download 16

EC2 - Elastic Compute Cloud • Based on XEN virtualization • On demand virtual servers - controlled with Web Service API • Use your favorite Linux distro (Linux 2.6.16), Amazon Machine Images (AMI) are stored on S3 • ACLs for hosts/ports access control 17

EC2 Tools List availabe images Start a new instance Login with SSH Shutdown instance 18

amazon-ec2 gem Setup Usage 19

SQS - Simple Queue Service • Distributed, highly scalable and reliable message system • Unlimited number of queues and messages • Locking / releasing built-in • 256 KB message payload • Basic permission model 20

SQS gem Setup Usage 21

How does this solve my problems? 22

S3 - Backup • s3sync.rb • S3Browser • Brackup • Firefox S3 Organizer extension • Jungle Disk •… • S3 FUSE • s3DAV • Duplicity 23

s3sync.rb Backup Restore 24

S3 Asset Host 25

S3 Asset Host Setup DNS Rails conﬁguration 26

S3 Asset Host welcome.rhtml template Output 27

S3 - Authenticated User Data 28

attachment_fu Rails plugin Setup 29

attachment_fu Rails plugin Upload Download 30

On-Demand Computing with EC2 Time based, e.g. with cron 31

On-Demand Computing with EC2 Load based, e.g. with Monit 32

On-Demand Computing with EC2 33

EC2 for extra capacity 34

Peritor Wissensmanagement GmbH Lenbachstrasse 2 12157 Berlin Internet: www.peritor.com Phone: +49 (0)30 69 40 11 94 Email: info@peritor.com Fax: +49 (0)30 69 40 11 95 © Peritor Wissensmanagement GmbH - All Rights Reserved 35

http://blog.innerewut.de	303
http://lars-schenk.com	94
http://www.planetrubyonrails.org	29
http://planetrubyonrails.org	7
http://www.slideshare.net	5
http://www.lars-schenk.com	4
http://130.149.43.21	1
http://vevis.lars-schenk.com	1
http://72.14.235.132	1
http://static.slideshare.net	1
http://web.archive.org	1

Use Somebody Else's Infrastructure - Utilizing Amazon S3 and EC2

by Jonathan Weiss

Accessibility

Categories

Upload Details

Usage Rights

11 Embeds 447

Statistics

Use Somebody Else’s Infrastructure - Utilizing Amazon S3 and EC2 Presentation Transcript