S3 and EC2 Rails Scenarios


Published on

Thoughts on how to utilize Amazons EC2 (Elastic Compute Cloud) web service for hosting applications, especially with Ruby on Rails in mind. A major issue is the non persistence of local storage in the EC2 environment. We outlined several approaches on how to possibly solve this issue, minimizing potential data loss.

by Adam Groves and Martin Rehfeld

Published in: Economy & Finance, Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

S3 and EC2 Rails Scenarios

  1. Amazon S3 + EC2 + Rails = Dream Team? Thoughts by Adam Groves and Martin Rehfeld @ BRUG 05-Apr-2007
  2. Amazon S3 Simple Storage Service • Pricing • $0.15 per GB-Month NO CO ST (10 GB = $1.50 per month) DRIVE R • $0.20 per GB of data transferred BEWA RE HIGH TRAFF IC (100 GB = additional $20 per month) SITES
  3. Amazon S3 Common Terms • buckets: S3 global name space some bucket A • objects: accessed by key [~ path+name] my bucket #1 can have metadata object 1, key = „images/icons/smile.png“, encoding=... object 2, key = „images/icons/cry.png“, encoding=... • prefixes: } identical prefix searching by prefix emulates directory structure my bucket #2 • ACL: some bucket B ... everyone, authenticated users, owner / named users
  4. S3Fox Extension
  5. Amazon EC2 Elastic Compute Cloud • Pricing • $0.10 per instance-hour consumed IDEAL STAGIN FOR GA SCALIN ND (1 month 24x7 ~ $72) G • $0.20 per GB of data transferred (just internet traffic; TRAFF WARN IC no charge for EC2-S3 traffic) IN APPLIE G S
  6. Amazon EC2 Instance Specs • ~ 1.7 GHz x86 CPU ACTUA LLY A XEN VIRTUA INSTAN L CE • 1.75 GB RAM • 160 GB local disk space • 250 Mb/s network bandwidth
  7. Amazon EC2 Common Terms • images: named OS images stored in S3: „AMI“ • instances: virtual maschines running an AMI • bundling: saving customized images back to S3 • non-persistence: BUG O FEATU R local disk storage will not survive instance RE? shutdown or failure
  8. Amazon EC2 Network Security access group quot;defaultquot; access group quot;webquot; Internet access group quot;backendquot;
  9. Amazon EC2 Rails Scenario } EC2 S3 Web Static Server Content  Application Code Server memcached, backgrounDRb, ...  SQL HOW TO GET A PERSIS Database TENT DB?
  10. Amazon EC2 Database Persistence Flavor A: • on instance failure Frequent backup to S3 you will lose all changes since last backup S3 • backup using a lot backup job of resources DB • no automatic failover
  11. Amazon EC2 Database Persistence Flavor B: • on instance failure Backup write ahead logs to S3 you will lose all changes since last log switch • S3 DB backup is fast, but backup job recovery will take WAL longer • still no automatic failover
  12. Amazon EC2 Database Persistence Flavor C: • you might still lose all Shadow database changes since last log switch • Master Shadow backup is fast, recovery is usually not neccessary (only if master and slave continous DB recovery should fail at the same time) WAL transfer DB WAL • no load balancing between master and slave via • automatic failover can be achieved S3
  13. Amazon EC2 Database Persistence • Flavor D: data loss only on failure of all instances (Multi-Master-)Replication • reduced write performance Master #1 Master #2 • sophisticated configuration  can be tricky two-way • should be combined with replication DB flavor A or B for backup DB • load balancing can be backup job achieved • automatic failover can be achieved S3
  14. Amazon EC2 Database Persistence Flavor E (highly Experimental): • unless caching is used there should be no data DB files on S3DFS loss • (very?) reduced performance • reliability of S3DFS yet to S3 be proven • can be combined with DB S3DFS fuse filesystem flavors A or B for backup • can be combined with flavors C or D for failover and load balancing (flavor D only)
  15. Resources • Amazon EC2 / S3 aws.amazon.com/ec2 / aws.amazon.com/s3 • Rails & EC2 & Capistrano tasks • Amazon Web Services Ruby Gem (aws/s3) http://niblets.wordpress.com/2007/02/12/capistrano-ec2-sitting-in-a-tree-k-i-s-s-i-n-g/ http://amazon.rubyforge.org/ • S3DFS filesystem driver http://www.openfount.com/blog/s3dfs-for-ec2 • Discussion on persistence with EC2 http://blog.awswebshop.com/2007/01/07/thoughts-on-current-functionality-requests-for-future-functionality/ • Postgres WAL / continuous archiving http://www.postgresql.org/docs/8.2/static/wal-intro.html / http://www.postgresql.org/docs/8.2/static/continuous-archiving.html • MySQL multi-master replication http://capttofu.livejournal.com/1752.html
  16. Q &A ! Adam Groves adam.groves@gmail.com Martin Rehfeld martin.rehfeld@glnetworks.de