Use Somebody Else's Infrastructure:

Utilizing Amazon S3 and EC2


Jonathan Weiss
Peritor Wissensmanagement GmbH, www.peri...
Peritor

• Consultancy from Berlin, Germany

• Specialized on Architecture, Scaling and
  Ruby on Rails

• A lot of our cl...
www.peritormail.com




We run a Webmail portal on top of Amazon Web Services

                                           ...
Starting Point


                 One machine:

                 • Apache

                 • Ruby on Rails / PHP / Perl /...
Worst Case Populariy




                       5
A Difficult Path




                  6
Problem: Backup



• High availability

• Redundancy

• Very big data sets




                       7
Problem: File System


• Important files have to be accessed by many servers

• NFS / Samba not practical




             ...
Problem: Messaging


• Many servers on different locations have to communicate

• Creating a reliable and scalable message...
Problem: Spontaneous Traffic




                              10
Problem: Load Fluctuation




                            11
Don‘t reinvent the wheel!




                            12
Amazon Web Services




Simple Storage Service - S3   Elastic Compute Cloud - EC2   Simple Queue Service - SQS   E-Commerc...
S3 - Simple Storage Service


• Redundant storage - as much as you like

• max. 5 GB per object

• Organized in „Buckets“
...
S3 - Buckets

                                                             S3

• Unique over all S3

• Contains many
     ...
S3 with AWS::S3
Upload




Download




                  16
EC2 - Elastic Compute Cloud

• Based on XEN virtualization

• On demand virtual servers - controlled with Web Service API
...
EC2 Tools
 List availabe images



Start a new instance




Login with SSH




Shutdown instance



                      ...
amazon-ec2 gem
Setup




Usage




                 19
SQS - Simple Queue Service


• Distributed, highly scalable and reliable message system

• Unlimited number of queues and ...
SQS gem
Setup




Usage




          21
How does this solve
my problems?



                      22
S3 - Backup


• s3sync.rb     • S3Browser

• Brackup       • Firefox S3 Organizer extension

• Jungle Disk   •…

• S3 FUSE...
s3sync.rb
Backup




Restore




            24
S3 Asset Host




                25
S3 Asset Host

Setup DNS




Rails configuration




                     26
S3 Asset Host

welcome.rhtml template




Output




                         27
S3 - Authenticated User Data




                               28
attachment_fu Rails plugin

Setup




                             29
attachment_fu Rails plugin

Upload




Download




                             30
On-Demand Computing with EC2




Time based, e.g. with cron




                               31
On-Demand Computing with EC2




Load based, e.g. with Monit




                               32
On-Demand Computing with EC2




                               33
EC2 for extra capacity




                         34
Peritor Wissensmanagement GmbH
Lenbachstrasse 2
12157 Berlin
Internet: www.peritor.com       Phone: +49 (0)30 69 40 11 94
...
Upcoming SlideShare
Loading in …5
×

Use Somebody Else's Infrastructure - Utilizing Amazon S3 and EC2

12,611 views
12,290 views

Published on

Scaling a Web Application is a very hard problem, especially for small project and teams who do not have sufficient manpower, money, and time to solve this problem. Luckily Amazon already had to solve this problem in their datacenters and offers their services to other developers. This talk will introduce the two most important Amazon Web Services, the Elastic Compute Cloud (EC2) and the Simple Storage Service (S3), and will present different ways to leverage them in your own web application. Ruby on Rails will be used for the examples but they will apply to any web framework. Several use-cases will be covered that show how S3 and EC2 can be used to move load from your servers to Amazon's or even to completely host your application at Amazon.

Published in: Technology
1 Comment
22 Likes
Statistics
Notes
No Downloads
Views
Total views
12,611
On SlideShare
0
From Embeds
0
Number of Embeds
162
Actions
Shares
0
Downloads
623
Comments
1
Likes
22
Embeds 0
No embeds

No notes for slide

Use Somebody Else's Infrastructure - Utilizing Amazon S3 and EC2

  1. Use Somebody Else's Infrastructure: Utilizing Amazon S3 and EC2 Jonathan Weiss Peritor Wissensmanagement GmbH, www.peritor.com Web 2.0 Expo Berlin, 2007
  2. Peritor • Consultancy from Berlin, Germany • Specialized on Architecture, Scaling and Ruby on Rails • A lot of our clients use Amazon Web Services 2
  3. www.peritormail.com We run a Webmail portal on top of Amazon Web Services 3
  4. Starting Point One machine: • Apache • Ruby on Rails / PHP / Perl / Java / … • MySQL 4
  5. Worst Case Populariy 5
  6. A Difficult Path 6
  7. Problem: Backup • High availability • Redundancy • Very big data sets 7
  8. Problem: File System • Important files have to be accessed by many servers • NFS / Samba not practical 8
  9. Problem: Messaging • Many servers on different locations have to communicate • Creating a reliable and scalable message queue is really hard 9
  10. Problem: Spontaneous Traffic 10
  11. Problem: Load Fluctuation 11
  12. Don‘t reinvent the wheel! 12
  13. Amazon Web Services Simple Storage Service - S3 Elastic Compute Cloud - EC2 Simple Queue Service - SQS E-Commerce.. Redundant Data Store Virtual server per hour Reliable Message System ……. $0,15 per GB data per $0,10 per CPU hour $0,10 per 1,000 messages month $0,10 - $0,20 per GB $0,10 - $0,18 per GB $0,10 - $0,20 per GB traffic traffic traffic 13
  14. S3 - Simple Storage Service • Redundant storage - as much as you like • max. 5 GB per object • Organized in „Buckets“ • Web Service API for uploads • Downloads via • Web Service • HTTP / HTTPS • BitTorrent 14
  15. S3 - Buckets S3 • Unique over all S3 • Contains many MyBucket_name www.peritor.com key-value-metadata tupel foto_5.jpg site/screen.css • Cannot contain other buckets! backups/januar.zip site/script.js • Key can contain „/“ fotos/2007/001.png FreeBSD-6.2.iso 15
  16. S3 with AWS::S3 Upload Download 16
  17. EC2 - Elastic Compute Cloud • Based on XEN virtualization • On demand virtual servers - controlled with Web Service API • Use your favorite Linux distro (Linux 2.6.16), Amazon Machine Images (AMI) are stored on S3 • ACLs for hosts/ports access control 17
  18. EC2 Tools List availabe images Start a new instance Login with SSH Shutdown instance 18
  19. amazon-ec2 gem Setup Usage 19
  20. SQS - Simple Queue Service • Distributed, highly scalable and reliable message system • Unlimited number of queues and messages • Locking / releasing built-in • 256 KB message payload • Basic permission model 20
  21. SQS gem Setup Usage 21
  22. How does this solve my problems? 22
  23. S3 - Backup • s3sync.rb • S3Browser • Brackup • Firefox S3 Organizer extension • Jungle Disk •… • S3 FUSE • s3DAV • Duplicity 23
  24. s3sync.rb Backup Restore 24
  25. S3 Asset Host 25
  26. S3 Asset Host Setup DNS Rails configuration 26
  27. S3 Asset Host welcome.rhtml template Output 27
  28. S3 - Authenticated User Data 28
  29. attachment_fu Rails plugin Setup 29
  30. attachment_fu Rails plugin Upload Download 30
  31. On-Demand Computing with EC2 Time based, e.g. with cron 31
  32. On-Demand Computing with EC2 Load based, e.g. with Monit 32
  33. On-Demand Computing with EC2 33
  34. EC2 for extra capacity 34
  35. Peritor Wissensmanagement GmbH Lenbachstrasse 2 12157 Berlin Internet: www.peritor.com Phone: +49 (0)30 69 40 11 94 Email: info@peritor.com Fax: +49 (0)30 69 40 11 95 © Peritor Wissensmanagement GmbH - All Rights Reserved 35

×