• Save
Getting Distributed (With Ruby On Rails)
Upcoming SlideShare
Loading in...5
×
 

Getting Distributed (With Ruby On Rails)

on

  • 28,087 views

Implementing distributed processing at Working With Rails

Implementing distributed processing at Working With Rails

Statistics

Views

Total Views
28,087
Views on SlideShare
28,038
Embed Views
49

Actions

Likes
49
Downloads
0
Comments
4

5 Embeds 49

http://www.slideshare.net 38
http://phuha.net 4
http://www.techgig.com 3
http://blog.phuha.net 2
http://www.m.techgig.com 2

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

14 of 4 Post a comment

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • Very good presentation. Nicely done. I’m Ana Mui Stanley, working on my latest site on lyrics, www.lyrics-search.org/ . I enjoy reading the slide.
    Are you sure you want to
    Your message goes here
    Processing…
  • Excellent. You've shown your credibility on presentation with this slideshow. This one deserves thumbs up. I'm John, owner of www.freeringtones.ws/ . Perhaps I'll get to see more quality slides from you.

    Best wishes.
    Are you sure you want to
    Your message goes here
    Processing…
  • It's a bit complicated, but having reading this gives me more sight on ruby.

    Dave (CEO : www.freeringtonesforverizon.net/ )
    Are you sure you want to
    Your message goes here
    Processing…
  • Great display. I've taken a few of the structure graphics as well as adapted to my startup

    Janie
    http://financejedi.com
    http://healthjedi.com
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Getting Distributed (With Ruby On Rails) Getting Distributed (With Ruby On Rails) Presentation Transcript

    • Getting Distributed With Ruby (On Rails) by Martin Sadler Implementing distributed processing at Working With Rails
    • dsc.net
    • DSC • Hosting, Web application development, and consultancy • Host the crew email system and carried out the intranet integration for Virgin Atlantic. • Runs several large forums. • e.g. pprune.org - 150k members, 2600 at one time • Also AskDirect, Mumsnet
    • http://www.workingwithrails.com
    • Working With Rails • Largest index of Ruby on Rails in the world • Over 7000 people listed • From 104 countries • Find out who’s who? • Connect with others • Find a developer for a project / employment • Also lists groups, companies, and sites
    • Why Distributed?
    • Working With Rails
    • Distributed Ruby?
    • Some background
    • Some background
    • Current • Uses FeedTools (nice lib!) • Great at parsing feed formats • Good for small sites
    • Issues
    • Issues • No longer supported by author
    • Issues • No longer supported by author • Feeds fetched at the users expense (and therefore Mongrels)
    • Issues • No longer supported by author • Feeds fetched at the users expense (and therefore Mongrels) • Feeds are cached locally but are parsed on request
    • Issues • No longer supported by author • Feeds fetched at the users expense (and therefore Mongrels) • Feeds are cached locally but are parsed on request • Known probs when scaling (search on Google)
    • The Result
    • The Result • Occasional slow loading pages that include third party feeds
    • The Result • Occasional slow loading pages that include third party feeds • Stale feed items
    • The Result • Occasional slow loading pages that include third party feeds • Stale feed items • Inconstant feed items
    • The Result • Occasional slow loading pages that include third party feeds • Stale feed items • Inconstant feed items • No good!
    • The Challenge • To keep content fresh, push traffic to WWR and out to the blog owners • Different feeds and sources to consider: Flickr, Twitter, Blog, Delicious • Each need to display in multiple places in many ways • But also want to do some funkier stuff (as you’ll see a bit later)
    • Ruby on Rails distributed processing choices RingyDingy AP4R Rinda DRB Starfish BackgroundRB Reliable-Message
    • DRB Basic building block of all other Ruby distributed libs. “DRb literally stands for quot;Distributed Rubyquot;. It is a library that allows you to send and receive messages from remote Ruby objects via TCP/IP. Sound kind of like RPC, CORBA or Java's RMI? Probably so. This is Ruby's simple as dirt answer to all of the above.” http://chadfowler.com/ruby/drb.html
    • Quick DRB Example Server Client require 'drb' require 'drb' class TestServer DRb.start_service() obj = DRbObject.new(nil, 'druby://localhost:9000') def doit # Now use obj quot;Hello, Distributed Worldquot; p obj.doit end end aServerObject = TestServer.new DRb.start_service('druby://localhost:9000', aServerObject) DRb.thread.join # Don't exit just yet!
    • Quick DRB Example Server Client require 'drb' require 'drb' class TestServer DRb.start_service() obj = DRbObject.new(nil, 'druby://localhost:9000') def doit # Now use obj quot;Hello, Distributed Worldquot; p obj.doit end end aServerObject = TestServer.new DRb.start_service('druby://localhost:9000', aServerObject) DRb.thread.join # Don't exit just yet! > ruby server.rb
    • Quick DRB Example Server Client require 'drb' require 'drb' class TestServer DRb.start_service() obj = DRbObject.new(nil, 'druby://localhost:9000') def doit # Now use obj quot;Hello, Distributed Worldquot; p obj.doit end end aServerObject = TestServer.new DRb.start_service('druby://localhost:9000', aServerObject) DRb.thread.join # Don't exit just yet! > ruby server.rb > ruby client.rb “Hello Distributed World”
    • Basics • Server • Clients / Workers • Communicate via messages http://en.wikipedia.org/wiki/Distributed_computing
    • BackgroundRB • Ruby job server and scheduler. • Integrates with Rails • Quite complex • Some issues between versions but many favor it above the other libs • Most well known http://backgroundrb.rubyforge.org/
    • Starfish • Inspired by Google’s MapReduce • Easy to understand code • Stability? • No longer supported by author? http://rufy.com/starfish/doc/
    • reliable-message • Solid library • Easy to understand API • Bit more involved to setup • Can be integrated with Rails • On going development http://trac.labnotes.org/cgi-bin/trac.cgi/wiki/Ruby/ReliableMessaging
    • AP4R • Asynchronous Processing for Ruby • Lesser known lib from Japan (new kid on the block) • Integrates with Rails • Built on top of reliable-message
    • AP4R • AP4R, Asynchronous Processing for Ruby, is the implementation of reliable asynchronous message processing. It provides message queuing, and message dispatching. • Using asynchronous processing, we can cut down turn-around-time of web applications by queuing, or can utilize more machine power by load-balancing.
    • AP4R Features • Business logic can be implemented as simple Web applications, or ruby code, whether it's called asynchronously or synchronously. • Asynchronous messaging is reliable by RDBMS persistence (now MySQL only) or file persistence, under the favor of reliable-msg. • Load balancing over multiple AP4R processes on single/multiple servers is supported. • Asynchronous logics are called via various protocols, such as XML-RPC, SOAP, HTTP PUT, and more. • Using store and forward function, at-least-omce QoS level is provided.
    • AP4R Process Flow • A client(e.g. a web browser) makes a request to a web server (Apache, Lighttpd, etc...). • A rails application (a synchronous logic) is executed on mongrel via mod_proxy or something. • At the last of the synchronous logic, message(s) are put to AP4R (AP4R provides a helper). • Once the synchronous logic is done, the clients receives a response immediately. • AP4R queues the message, and requests it to the web server asynchronously. • An asynchronous logic, implemented as usual rails action, is executed.
    • AP4R example Hello World app comes with AP4R to get you started. Nice guide also here http://rubyforge.org/frs/download.php/13312/AP4R_Users_Guide_EN.pdf
    • Complimentary Services
    • Rinda • Rinda::Ring allows DRb services and clients to automatically find each other without knowing where they live. • DRb servers register themselves with a RingServer which allows clients to find the servers they need. Many servers may register themselves with the RingServer. The DRb servers don't need to run on the same machine. http://segment7.net/projects/ruby/drb/rinda/ringserver.html
    • RingyDingy • RingyDingy automatically registers a service with a RingServer. If communication between the RingServer and the RingyDingy is lost, RingyDingy will re-register its service with the RingServer when it reappears. http://seattlerb.rubyforge.org/RingyDingy/
    • Feeds in WWR AP4R Server Feed @queue Queue Feed Fetcher Feed Fetcher Feed Fetcher 1 2 N
    • Running the code ruby script/ap4r_start -c config/queues_mysql.cfg rake background:feed_queue rake background:feed_retrieve
    • Key points • The Feed Queue fetches the urls of stale feeds • Each worker (client) has the Rails environment loaded
    • With this solution • Can scale as demand grows • Flexible for any type of feed data • Still - room for improvement
    • Possible Improvements • Automatic spawning and killing of workers as queue size grows or decreases • Better handling of feed errors • Dynamic polling intervals based on user defined prefs or some intelligent logic.
    • When to go distributed? • Long running process or task • Fetching external data • Complex computations • .... that can be broken into chunks or work • You care about the user experience
    • Pitfalls • Dependencies • `connection closed' errors on Mac (IPV6) - change all refs of localhost to 127.0.0.1 to avoid. (had to patch reliable-message) • Terminology to understand • Memory requirements
    • Do you need distributed? • Maybe you would be better scheduling instead? • http://www.igvita.com/blog/2007/03/29/ scheduling-tasks-in-ruby-rails/
    • So where is all this leading us to?
    • Contextual Feed Aggregation
    • Group feed aggregation
    • Group blog posts
    • Group blog posts Twitters
    • Group blog posts Twitters and so on.......
    • Thanks! http://www.dsc.net http://www.workingwithrails.com Blog: http://beyondthetype.com Enjoyed the talk? Recommend me on WWR http://workingwithrails.com/person/5152-martin-sadler