Rails in the Cloud

Rails in the Cloud Ian Warshak Developer @ RightScale http://twitter.com/iwarshak http://github.com/iwarshak/lsrc-cloud

The Clouds Cloud Infrastructure Cloud Services Amazon S3 Amazon Ec2 Amazon SQS Rackspace Cloudservers Amazon SimpleDB Flexiscale Google AppEngine GoGrid Heroku

Pictr Photo website. You upload photos, they are resized, watermarked, and put on the homepage Photos can be uploaded by the public Photos are converted asynchronously by separate processing servers Very simple app, but hopefully shows some ideas of how cloud computing apps work

Why Pictr? Lots of Disk Storage Lots of Bandwidth Ofﬂine processing I want to manage and host as little as possible It will be an instant hit. Needs to scale from day 1

Don’t get your hopes up about Pictr!

Why the Cloud? I don’t want to build or configure hardware. I don’t want to do much capacity planning. I want to pay as I go for servers and service. Offload as much as possible. Lots of Bandwidth Offline processing I want to manage and host as little as possible It will be an instant hit. Needs to scale from day 1

How is it built? Rails application running on Amazon EC2 servers Asynchronous image processing (resizing, effects) Database is Amazon SimpleDB Storage is Amazon S3 CDN is Amazon Cloudfront Messaging queue is Amazon SQS ImageMagick is used for image manipulation

gem install right_aws

Why Amazon? Simple. Right now, they are the market leader... But the competition is good. Tight integration between services RESTful APIs for their services Lots of documentation and libraries

Amazon Ec2 Servers “on demand”. Pay by the hour: $0.10 - $0.80/hour Non-persistent. Forces you to automate your conﬁguration. RightScale helps a lot with this. Elastic IP can give you a static IP Elastic Block Store gives you persistent data store

Amazon S3 Oldest of the publicly available web services Scalable, reliable data storage mechanism Bucket/object concept Pay for the data you are storing, as well as bandwidth in and out $0.15/GB stored. $0.10/GB transferred. $0.01/ request. Very ﬁne grained permissions

CloudFront Similar to LimeLight, Akamai Low latency, high data rate transfers 8 edge locations in the US Origin server is an S3 bucket 24 hour object expiration Very simple API acf = RightAws::AcfInterface.new acf.create_distribution('pictr.s3.amazonaws.com', 'Woo-Hoo!', true)

Amazon Simple Queue Service Reliable, scalable queue PUSH and PULL messages Messages can be up to 8K of text Messages are “locked” as they are processed, preventing them from being read multiple times.

Amazon SimpleDB Simple (but powerful) database Non-relational. No tables, only “domains”. No joins. No schema. Deﬁne it as you go. All data is stored as strings. Attributes can have up to 256 values. Automatically indexes all your data. Not exactly a speed daemon.

Amazon SimpleDB

ActiveSDB RightAws::ActiveSdb.establish_connection class Item < RightAws::ActiveSdb::Base; end Item.create_domain item = Item.create :Category => "Electronics", :Size => 47, :Manufacturer => "Sony" # Add a new "column" item["Illumination"] = "LCD" item.save # New category. Now, category is an "array" item["Category"] << "Televisions" item.save items = Item.select(:all, :conditions => ["Size = ?", 47]) # => [#<Item:0x1a8a5ec @new_record=false, @attributes={"Size"=>["47"], "Category"=>["Electronics", "Televisions"], "id"=>"788b5192-8a85-11de-a035-00264a1272e2", "Manufacturer"=>["Sony"], "Illumination"=>["LCD"]}>]

SimpleDB caveats All data is stored as a string. Response cannot be > 1MB Maximum query execution time: 5 sec Eventually consistent (a tradeoff for availability) Speed (network)

What does all this mean? I worry less about scaling I worry more about conﬁguration management and automation Database performance is very consistent S3, Cloudfront handles static ﬁle serving Minimal upfront investment Only pay for what I need

SQS SimpleDB Web Servers Converter Servers S3 Users

Step 1: Upload S3 EC2 INSTANCE REDIRECT ON SUCCESS POST PIC.JPG INITIATES PROCESSING

Step 2: Queue the jobs # Jobs are created jobs = [ConvertJob.new(key, "paint", "400", "-paint 5"), ConvertJob.new(key, "monochrome", "400", "-monochrome")] # Push the jobs into the queue sqs = RightAws::SqsGen2.new convert_queue = sqs.queue("convert") jobs.each do |job| convert_queue.send_message job.to_json end

Step 3: Create SimpleDB record # class Picture < RightAws::ActiveSdb::Base # ... # end picture = Picture.create(:key => key) picture["total_conversions"] = ["paint", "monochrome"] picture.save

SimpleDB record # Before conversion {"key"=>["uploads/f1"], "total_conversions"=>["paint", "monochrome"]} # After conversion {"key"=>["uploads/f1"], "total_conversions"=>["paint", "monochrome"], "paint"=>["processed/f1_paint.jpg"], "monochrome"=>["processed/f1_monochrome.jpg"] }

Step 4: Processor Daemon loop do sqs = RightAws::SqsGen2.new convert_queue = sqs.queue("convert") message = convert_queue.pop if message job = JSON.parse message.to_s puts "Found a new message #{job.key}" job.run! end sleep(5) end

Thanks! right_aws gem http://github.com/iwarshak/lsrc-cloud twitter.com/iwarshak

http://ianwarshak.com	695
http://ianwarshak.posterous.com	29
http://warshaksystems.com	12
http://www.slideshare.net	7
http://posterous.com	2
http://webcache.googleusercontent.com	1

Rails in the Cloud

by iwarshak

Accessibility

Categories

Upload Details

Usage Rights

6 Embeds 746

Statistics

Rails in the Cloud Presentation Transcript