Getting Distributed (With Ruby On Rails)

Getting Distributed
With Ruby (On Rails)

by Martin Sadler

Implementing distributed processing at Working With Rails

DSC
• Hosting, Web application development, and
consultancy
• Host the crew email system and carried out
the intranet integration for Virgin Atlantic.
• Runs several large forums.
• e.g. pprune.org - 150k members, 2600 at one
time
• Also AskDirect, Mumsnet

http://www.workingwithrails.com

Working With Rails
• Largest index of Ruby on Rails in the world
• Over 7000 people listed
• From 104 countries
• Find out who’s who?
• Connect with others
• Find a developer for a project / employment
• Also lists groups, companies, and sites

Current

• Uses FeedTools (nice lib!)
• Great at parsing feed formats
• Good for small sites

Issues
• No longer supported by author

Issues
• Feeds fetched at the users expense (and
therefore Mongrels)

Issues
therefore Mongrels)
• Feeds are cached locally but are parsed on
request

Issues
therefore Mongrels)
• Feeds are cached locally but are parsed on
request
• Known probs when scaling (search on
Google)

The Result

• Occasional slow loading pages that include
third party feeds

The Result

third party feeds
• Stale feed items

The Result

third party feeds
• Inconstant feed items

The Result

third party feeds
• Inconstant feed items
• No good!

The Challenge
• To keep content fresh, push trafﬁc to WWR
and out to the blog owners
• Different feeds and sources to consider:
Flickr, Twitter, Blog, Delicious
• Each need to display in multiple places in
many ways
• But also want to do some funkier stuff (as
you’ll see a bit later)

Ruby on Rails distributed processing choices

RingyDingy
AP4R

Rinda DRB Starﬁsh

BackgroundRB Reliable-Message

DRB
Basic building block of all other Ruby
distributed libs.
“DRb literally stands for quot;Distributed Rubyquot;. It is a library that allows you
to send and receive messages from remote Ruby objects via TCP/IP. Sound
kind of like RPC, CORBA or Java's RMI? Probably so. This is Ruby's simple
as dirt answer to all of the above.”

http://chadfowler.com/ruby/drb.html

Quick DRB Example
Server
Client
require 'drb'
require 'drb'
class TestServer DRb.start_service()
obj = DRbObject.new(nil, 'druby://localhost:9000')
def doit # Now use obj
quot;Hello, Distributed Worldquot; p obj.doit
end
end

aServerObject = TestServer.new
DRb.start_service('druby://localhost:9000', aServerObject)
DRb.thread.join # Don't exit just yet!

Quick DRB Example
Server
Client
require 'drb'
require 'drb'
end
end


> ruby server.rb

Quick DRB Example
Server
Client
require 'drb'
require 'drb'
end
end


> ruby server.rb > ruby client.rb
“Hello Distributed World”

Basics

• Server
• Clients / Workers
• Communicate via messages
http://en.wikipedia.org/wiki/Distributed_computing

BackgroundRB
• Ruby job server and scheduler.
• Integrates with Rails
• Quite complex
• Some issues between versions but many
favor it above the other libs
• Most well known
http://backgroundrb.rubyforge.org/

Starﬁsh
• Inspired by Google’s MapReduce
• Easy to understand code
• Stability?
• No longer supported by author?

http://rufy.com/starﬁsh/doc/

reliable-message
• Solid library
• Easy to understand API
• Bit more involved to setup
• Can be integrated with Rails
• On going development
http://trac.labnotes.org/cgi-bin/trac.cgi/wiki/Ruby/ReliableMessaging

AP4R

• Asynchronous Processing for Ruby
• Lesser known lib from Japan (new kid on the
block)
• Integrates with Rails
• Built on top of reliable-message

AP4R
• AP4R, Asynchronous Processing for Ruby, is
the implementation of reliable asynchronous
message processing. It provides message
queuing, and message dispatching.
• Using asynchronous processing, we can cut
down turn-around-time of web applications
by queuing, or can utilize more machine
power by load-balancing.

AP4R Features

• Business logic can be implemented as simple Web applications, or ruby code, whether it's called
asynchronously or synchronously.

• Asynchronous messaging is reliable by RDBMS persistence (now MySQL only) or ﬁle
persistence, under the favor of reliable-msg.

• Load balancing over multiple AP4R processes on single/multiple servers is supported.

• Asynchronous logics are called via various protocols, such as XML-RPC, SOAP, HTTP PUT, and
more.

• Using store and forward function, at-least-omce QoS level is provided.

AP4R Process Flow

• A client(e.g. a web browser) makes a request to a web server (Apache, Lighttpd, etc...).

• A rails application (a synchronous logic) is executed on mongrel via mod_proxy or something.

• At the last of the synchronous logic, message(s) are put to AP4R (AP4R provides a helper).

• Once the synchronous logic is done, the clients receives a response immediately.

• AP4R queues the message, and requests it to the web server asynchronously.

• An asynchronous logic, implemented as usual rails action, is executed.

AP4R example
Hello World app comes with AP4R to get
you started.

Nice guide also here
http://rubyforge.org/frs/download.php/13312/AP4R_Users_Guide_EN.pdf

Rinda
• Rinda::Ring allows DRb services and clients
to automatically ﬁnd each other without
knowing where they live.
• DRb servers register themselves with a
RingServer which allows clients to ﬁnd the
servers they need. Many servers may
register themselves with the RingServer. The
DRb servers don't need to run on the same
machine.
http://segment7.net/projects/ruby/drb/rinda/ringserver.html

RingyDingy
• RingyDingy automatically registers a service
with a RingServer. If communication between
the RingServer and the RingyDingy is lost,
RingyDingy will re-register its service with
the RingServer when it reappears.

http://seattlerb.rubyforge.org/RingyDingy/

Feeds in WWR
AP4R
Server
Feed
@queue
Queue

Feed Fetcher Feed Fetcher Feed Fetcher
1 2 N

Running the code

ruby script/ap4r_start -c conﬁg/queues_mysql.cfg
rake background:feed_queue
rake background:feed_retrieve

Key points

• The Feed Queue fetches the urls of stale
feeds
• Each worker (client) has the Rails
environment loaded

With this solution

• Can scale as demand grows
• Flexible for any type of feed data
• Still - room for improvement

Possible Improvements

• Automatic spawning and killing of workers
as queue size grows or decreases
• Better handling of feed errors
• Dynamic polling intervals based on user
deﬁned prefs or some intelligent logic.

When to go distributed?

• Long running process or task
• Fetching external data
• Complex computations
• .... that can be broken into chunks or work
• You care about the user experience

Pitfalls
• Dependencies
• `connection closed' errors on Mac (IPV6) -
change all refs of localhost to 127.0.0.1 to
avoid. (had to patch reliable-message)
• Terminology to understand
• Memory requirements

Do you need
distributed?

• Maybe you would be better scheduling
instead?
• http://www.igvita.com/blog/2007/03/29/
scheduling-tasks-in-ruby-rails/

So where is all this
leading us to?

Group blog posts
Twitters
and so on.......

Thanks!
http://www.dsc.net
http://www.workingwithrails.com
Blog: http://beyondthetype.com

Enjoyed the talk? Recommend me on WWR
http://workingwithrails.com/person/5152-martin-sadler

Getting Distributed (With Ruby On Rails)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Getting Distributed (With Ruby On Rails)

Similar to Getting Distributed (With Ruby On Rails) (20)

Recently uploaded

Recently uploaded (20)

Getting Distributed (With Ruby On Rails)