Presentation at Lonestar Ruby Conf 2009

1. Rails in the Cloud
Ian Warshak
Developer @ RightScale
http://twitter.com/iwarshak
http://github.com/iwarshak/lsrc-cloud

2. The Clouds
Cloud Infrastructure Cloud Services
Amazon S3
Amazon Ec2
Amazon SQS
Rackspace Cloudservers
Amazon SimpleDB
Flexiscale
Google AppEngine
GoGrid
Heroku

3. Pictr
Photo website. You upload photos, they are
resized, watermarked, and put on the homepage
Photos can be uploaded by the public
Photos are converted asynchronously by
separate processing servers
Very simple app, but hopefully shows some
ideas of how cloud computing apps work

4. Why Pictr?
Lots of Disk Storage
Lots of Bandwidth
Offline processing
I want to manage and host as little as possible
It will be an instant hit. Needs to scale from day 1

5. Don’t get your hopes up
about Pictr!

6. Why the Cloud?
I don’t want to build or configure hardware.
I don’t want to do much capacity planning.
I want to pay as I go for servers and service.
Offload as much as possible.
Lots of Bandwidth
Offline processing
I want to manage and host as little as possible
It will be an instant hit. Needs to scale from day 1

7. How is it built?
Rails application running on Amazon EC2 servers
Asynchronous image processing (resizing, effects)
Database is Amazon SimpleDB
Storage is Amazon S3
CDN is Amazon Cloudfront
Messaging queue is Amazon SQS
ImageMagick is used for image manipulation

8. gem install right_aws

9. Why Amazon?
Simple. Right now, they are the market leader...
But the competition is good.
Tight integration between services
RESTful APIs for their services
Lots of documentation and libraries

10. Amazon Ec2
Servers “on demand”.
Pay by the hour: $0.10 - $0.80/hour
Non-persistent.
Forces you to automate your configuration.
RightScale helps a lot with this.
Elastic IP can give you a static IP
Elastic Block Store gives you persistent data
store

11. Amazon S3
Oldest of the publicly available web services
Scalable, reliable data storage mechanism
Bucket/object concept
Pay for the data you are storing, as well as
bandwidth in and out
$0.15/GB stored. $0.10/GB transferred. $0.01/
request.
Very fine grained permissions

12. CloudFront
Similar to LimeLight, Akamai
Low latency, high data rate transfers
8 edge locations in the US
Origin server is an S3 bucket
24 hour object expiration
Very simple API
acf = RightAws::AcfInterface.new
acf.create_distribution('pictr.s3.amazonaws.com', 'Woo-Hoo!', true)

13. Amazon Simple Queue Service
Reliable, scalable queue
PUSH and PULL messages
Messages can be up to 8K of text
Messages are “locked” as they are processed,
preventing them from being read multiple times.

14. Amazon SimpleDB
Simple (but powerful) database
Non-relational. No tables, only “domains”. No joins.
No schema. Define it as you go.
All data is stored as strings.
Attributes can have up to 256 values.
Automatically indexes all your data.
Not exactly a speed daemon.

15. Amazon SimpleDB

16. ActiveSDB
RightAws::ActiveSdb.establish_connection
class Item < RightAws::ActiveSdb::Base; end
Item.create_domain
item = Item.create :Category => "Electronics", :Size => 47, :Manufacturer => "Sony"
# Add a new "column"
item["Illumination"] = "LCD"
item.save
# New category. Now, category is an "array"
item["Category"] << "Televisions"
item.save
items = Item.select(:all, :conditions => ["Size = ?", 47])
# => [#<Item:0x1a8a5ec @new_record=false, @attributes={"Size"=>["47"], "Category"=>["Electronics",
"Televisions"], "id"=>"788b5192-8a85-11de-a035-00264a1272e2", "Manufacturer"=>["Sony"],
"Illumination"=>["LCD"]}>]

17. SimpleDB caveats
All data is stored as a string.
Response cannot be > 1MB
Maximum query execution time: 5 sec
Eventually consistent (a tradeoff for availability)
Speed (network)

18. What does all this mean?
I worry less about scaling
I worry more about configuration management
and automation
Database performance is very consistent
S3, Cloudfront handles static file serving
Minimal upfront investment
Only pay for what I need

19. SQS
SimpleDB
Web Servers Converter Servers
S3
Users

20. Step 1: Upload
S3
EC2 INSTANCE
REDIRECT ON
SUCCESS
POST
PIC.JPG
INITIATES PROCESSING

21. Step 2: Queue the jobs
# Jobs are created
jobs = [ConvertJob.new(key, "paint", "400", "-paint 5"),
ConvertJob.new(key, "monochrome", "400", "-monochrome")]
# Push the jobs into the queue
sqs = RightAws::SqsGen2.new
convert_queue = sqs.queue("convert")
jobs.each do |job|
convert_queue.send_message job.to_json
end

22. Step 3: Create SimpleDB record
# class Picture < RightAws::ActiveSdb::Base
# ...
# end
picture = Picture.create(:key => key)
picture["total_conversions"] = ["paint", "monochrome"]
picture.save

23. SimpleDB record
# Before conversion
{"key"=>["uploads/f1"],
"total_conversions"=>["paint", "monochrome"]}
# After conversion
{"key"=>["uploads/f1"],
"total_conversions"=>["paint", "monochrome"],
"paint"=>["processed/f1_paint.jpg"],
"monochrome"=>["processed/f1_monochrome.jpg"]
}

24. Step 4: Processor Daemon
loop do
sqs = RightAws::SqsGen2.new
convert_queue = sqs.queue("convert")
message = convert_queue.pop
if message
job = JSON.parse message.to_s
puts "Found a new message #{job.key}"
job.run!
end
sleep(5)
end

25. Demo

26. Thanks!
right_aws gem
http://github.com/iwarshak/lsrc-cloud
twitter.com/iwarshak