Use Somebody Else's Infrastructure - Utilizing Amazon S3 and EC2

Slide 1: Use Somebody Else's Infrastructure: Utilizing Amazon S3 and EC2 Jonathan Weiss Peritor Wissensmanagement GmbH, www.peritor.com Web 2.0 Expo Berlin, 2007

Slide 2: Peritor • Consultancy from Berlin, Germany • Specialized on Architecture, Scaling and Ruby on Rails • A lot of our clients use Amazon Web Services 2

Slide 3: www.peritormail.com We run a Webmail portal on top of Amazon Web Services 3

Slide 4: Starting Point One machine: • Apache • Ruby on Rails / PHP / Perl / Java / … • MySQL 4

Slide 5: Worst Case Populariy 5

Slide 6: A Difficult Path 6

Slide 7: Problem: Backup • High availability • Redundancy • Very big data sets 7

Slide 8: Problem: File System • Important files have to be accessed by many servers • NFS / Samba not practical 8

Slide 9: Problem: Messaging • Many servers on different locations have to communicate • Creating a reliable and scalable message queue is really hard 9

Slide 10: Problem: Spontaneous Traffic 10

Slide 11: Problem: Load Fluctuation 11

Slide 12: Don‘t reinvent the wheel! 12

Slide 13: Amazon Web Services Simple Storage Service - S3 Elastic Compute Cloud - EC2 Simple Queue Service - SQS E-Commerce.. Redundant Data Store Virtual server per hour Reliable Message System ……. $0,15 per GB data per $0,10 per CPU hour $0,10 per 1,000 messages month $0,10 - $0,20 per GB $0,10 - $0,18 per GB $0,10 - $0,20 per GB traffic traffic traffic 13

Slide 14: S3 - Simple Storage Service • Redundant storage - as much as you like • max. 5 GB per object • Organized in „Buckets“ • Web Service API for uploads • Downloads via • Web Service • HTTP / HTTPS • BitTorrent 14

Slide 15: S3 - Buckets S3 • Unique over all S3 • Contains many MyBucket_name www.peritor.com key-value-metadata tupel foto_5.jpg site/screen.css • Cannot contain other buckets! backups/januar.zip site/script.js • Key can contain „/“ fotos/2007/001.png FreeBSD-6.2.iso 15

Slide 16: S3 with AWS::S3 Upload Download 16

Slide 17: EC2 - Elastic Compute Cloud • Based on XEN virtualization • On demand virtual servers - controlled with Web Service API • Use your favorite Linux distro (Linux 2.6.16), Amazon Machine Images (AMI) are stored on S3 • ACLs for hosts/ports access control 17

Slide 18: EC2 Tools List availabe images Start a new instance Login with SSH Shutdown instance 18

Slide 19: amazon-ec2 gem Setup Usage 19

Slide 20: SQS - Simple Queue Service • Distributed, highly scalable and reliable message system • Unlimited number of queues and messages • Locking / releasing built-in • 256 KB message payload • Basic permission model 20

Slide 21: SQS gem Setup Usage 21

Slide 22: How does this solve my problems? 22

Slide 23: S3 - Backup • s3sync.rb • S3Browser • Brackup • Firefox S3 Organizer extension • Jungle Disk •… • S3 FUSE • s3DAV • Duplicity 23

Slide 24: s3sync.rb Backup Restore 24

Slide 25: S3 Asset Host 25

Slide 26: S3 Asset Host Setup DNS Rails configuration 26

Slide 27: S3 Asset Host welcome.rhtml template Output 27

Slide 28: S3 - Authenticated User Data 28

Slide 29: attachment_fu Rails plugin Setup 29

Slide 30: attachment_fu Rails plugin Upload Download 30

Slide 31: On-Demand Computing with EC2 Time based, e.g. with cron 31

Slide 32: On-Demand Computing with EC2 Load based, e.g. with Monit 32

Slide 33: On-Demand Computing with EC2 33

Slide 34: EC2 for extra capacity 34

Slide 35: Peritor Wissensmanagement GmbH Lenbachstrasse 2 12157 Berlin Internet: www.peritor.com Phone: +49 (0)30 69 40 11 94 Email: info@peritor.com Fax: +49 (0)30 69 40 11 95 © Peritor Wissensmanagement GmbH - All Rights Reserved 35