Utilizing Amazon S3 and EC2
in Rails




Jonathan Weiss, Peritor Wissensmanagement GmbH
RailsConf Europe Berlin, 2007
Starting Point


                 One machine:

                 • Apache

                 • Ruby / Rails

              ...
Worst Case Populariy




                       3
A Difficult Path




                  4
Problem: Backup



• High availability

• Redundancy

• Very big data sets




                       5
Problem: File System


• Important files have to be accessed by many servers

• NFS / Samba not practical




             ...
Problem: Spontaneous Traffic




                              7
Problem: Load Fluctuation




                            8
Don‘t reinvent the wheel!




                            9
Amazon Web Services




    Simple Storage Service - S3    Elastic Compute Cloud - EC2   E-Commerce S..

  Redundant Data ...
S3 - Simple Storage Service


• Redundant storage - as much as you like

• max. 5 GB per object

• Organized in „Buckets“
...
S3 - Buckets

                                                             S3

• Unique over all S3

• Contains many
     ...
S3 with AWS::S3
Upload




Download




                  13
EC2 - Elastic Compute Cloud

• Based on XEN virtualization

• On demand virtual servers - controlled with Web Service API
...
EC2 Tools
 List availabe images



Start a new instance




Login with SSH




Shutdown instance



                      ...
amazon-ec2 gem
Setup




Usage




                 16
And now?

How does this solve
my problems?


                      17
S3 - Backup


• s3sync.rb     • S3Browser

• Brackup       • Firefox S3 Organizer extension

• Jungle Disk   •…

• S3 FUSE...
s3sync.rb
Backup




Restore




            19
S3 Asset Host




                20
S3 Asset Host

Setup DNS




Rails configuration




                     21
S3 Asset Host

welcome.rhtml template




Output




                         22
S3 - Authenticated User Data




                               23
attachment_fu Rails plugin

Setup




                             24
attachment_fu Rails plugin

Upload




Download




                             25
On-Demand Computing with EC2




Time based, e.g. with cron




                               26
On-Demand Computing with EC2




Load based, e.g. with Monit




                               27
On-Demand Computing with EC2




                               28
EC2 for extra capacity




                         29
Load Balancer / Proxy

                        Example mod_proxy_blancer:

                        • Talks to multiple bac...
Swiftiply
            Swiftiply Proxy:

            • Multiple backend clients have a
              persistent connection ...
Swiftiply Proxy

Installation      Configuration



Start




                                 32
Swiftiplied Mongrel

• Gem plugin that patches Mongrel

• Transforms Mongrel in Swiftiply client

• Experimental



Start
...
EC2 on Demand before Swiftiply




                                 34
EC2 on Demand with Swiftiply




                               35
EC2 on Demand with Swiftiply




                               36
Ressources

• Amazon Web Services
  http://aws.amazon.com

• Swiftiply
  http://swiftiply.swiftcore.org

• Attachment_fu
 ...
Peritor Wissensmanagement GmbH
Lenbachstraße 2
12157 Berlin
Internet: www.peritor.com       Telefon: +49 (0)30 69 40 11 94...
Upcoming SlideShare
Loading in …5
×

Rails Conf Europe 2007 - Utilizing Amazon S3 and EC2 in Rails

5,101
-1

Published on

Scaling a web application is a very hard problem, especially for small project and teams who do not have sufficient manpower, money, and time to solve this problem. Luckily Amazon already had to solve this problem in their datacenters and offers their services to other developers. This talk will introduce the two most important Amazon Web Services, the Elastic Compute Cloud (EC2) and the Simple Storage Service (S3), and will present different ways to leverage them in your own Rails application.

Presented at RailsConfEurope Berlin 2007

Published in: Technology
0 Comments
9 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
5,101
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
138
Comments
0
Likes
9
Embeds 0
No embeds

No notes for slide

Rails Conf Europe 2007 - Utilizing Amazon S3 and EC2 in Rails

  1. 1. Utilizing Amazon S3 and EC2 in Rails Jonathan Weiss, Peritor Wissensmanagement GmbH RailsConf Europe Berlin, 2007
  2. 2. Starting Point One machine: • Apache • Ruby / Rails • MySQL 2
  3. 3. Worst Case Populariy 3
  4. 4. A Difficult Path 4
  5. 5. Problem: Backup • High availability • Redundancy • Very big data sets 5
  6. 6. Problem: File System • Important files have to be accessed by many servers • NFS / Samba not practical 6
  7. 7. Problem: Spontaneous Traffic 7
  8. 8. Problem: Load Fluctuation 8
  9. 9. Don‘t reinvent the wheel! 9
  10. 10. Amazon Web Services Simple Storage Service - S3 Elastic Compute Cloud - EC2 E-Commerce S.. Redundant Data Store Virtual server per hour ……. $0,15 per GB data per month $0,10 per CPU hour $0,10 - $0,20 per GB traffic $0,10 - $0,20 per GB traffic 10
  11. 11. S3 - Simple Storage Service • Redundant storage - as much as you like • max. 5 GB per object • Organized in „Buckets“ • Web Service API for uploads • Downloads via • Web Service • HTTP / HTTPS • BitTorrent 11
  12. 12. S3 - Buckets S3 • Unique over all S3 • Contains many MyBucket_name www.peritor.com key-value-metadata tupel foto_5.jpg site/screen.css • Cannot contain other buckets! backups/januar.zip site/script.js • Key can contain „/“ fotos/2007/001.png FreeBSD-6.2.iso 12
  13. 13. S3 with AWS::S3 Upload Download 13
  14. 14. EC2 - Elastic Compute Cloud • Based on XEN virtualization • On demand virtual servers - controlled with Web Service API • 1,7 GHz • 1,75 GB RAM • 160 GB local file system (not persistent!) • Use your favorite Linux distro (Linux 2.6.16), Amazon Machine Images (AMI) are stored on S3 • ACLs for hosts/ports access control 14
  15. 15. EC2 Tools List availabe images Start a new instance Login with SSH Shutdown instance 15
  16. 16. amazon-ec2 gem Setup Usage 16
  17. 17. And now? How does this solve my problems? 17
  18. 18. S3 - Backup • s3sync.rb • S3Browser • Brackup • Firefox S3 Organizer extension • Jungle Disk •… • S3 FUSE • s3DAV • Duplicity 18
  19. 19. s3sync.rb Backup Restore 19
  20. 20. S3 Asset Host 20
  21. 21. S3 Asset Host Setup DNS Rails configuration 21
  22. 22. S3 Asset Host welcome.rhtml template Output 22
  23. 23. S3 - Authenticated User Data 23
  24. 24. attachment_fu Rails plugin Setup 24
  25. 25. attachment_fu Rails plugin Upload Download 25
  26. 26. On-Demand Computing with EC2 Time based, e.g. with cron 26
  27. 27. On-Demand Computing with EC2 Load based, e.g. with Monit 27
  28. 28. On-Demand Computing with EC2 28
  29. 29. EC2 for extra capacity 29
  30. 30. Load Balancer / Proxy Example mod_proxy_blancer: • Talks to multiple backend servers (Mongrel) • Central Proxy/Load-Balancer configuration that has knowledge about nodes • Typically proxy restart on config change 30
  31. 31. Swiftiply Swiftiply Proxy: • Multiple backend clients have a persistent connection to the backend port • Proxy listens on cluster port for requests and forwards them No re-configuration 31
  32. 32. Swiftiply Proxy Installation Configuration Start 32
  33. 33. Swiftiplied Mongrel • Gem plugin that patches Mongrel • Transforms Mongrel in Swiftiply client • Experimental Start or 33
  34. 34. EC2 on Demand before Swiftiply 34
  35. 35. EC2 on Demand with Swiftiply 35
  36. 36. EC2 on Demand with Swiftiply 36
  37. 37. Ressources • Amazon Web Services http://aws.amazon.com • Swiftiply http://swiftiply.swiftcore.org • Attachment_fu http://svn.techno-weenie.net/projects/plugins/attachment_fu/ • Elastic Rails http://www.elasticrails.com • Capazon http://capazon.rubyforge.org • RightScale http://www.rightscale.com 37
  38. 38. Peritor Wissensmanagement GmbH Lenbachstraße 2 12157 Berlin Internet: www.peritor.com Telefon: +49 (0)30 69 40 11 94 Email: info@peritor.com Telefax: +49 (0)30 69 40 11 95 © Peritor Wissensmanagement GmbH - All Rights Reserved 38
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×