The Code Of The Forking Paths (Asynchronous Processing with Resque and AMQP)

The Code of Forking PathsKarel Minařík

Karel Minařík→ Independent web designer and developer→ Ruby, Rails, Git and CouchDB propagandista in .cz→ Previously: Flash Developer; Art Director; Information Architect;… (see LinkedIn)→ @karmiq at Twitter→ karmi.cz The Code of Forking Paths

“He believed in an inﬁnite series of times, in a growing,dizzying net of divergent, convergent and paralleltimes. (...) We do not exist in the majority of thesetimes; in some you exist, and not I; in others I, and notyou; in others, both of us. In the present one, whicha favorable fate has granted me, you have arrived at myhouse; in another, while crossing the garden, you foundme dead; in still another, I utter these same words, butI am a mistake, a ghost.”

http://en.wikipedia.org/wiki/The_Garden_of_Forking_Paths The Code of Forking Paths

THE CODE OF FORKING PATHSParallel presence HTTP/1.1 503 Service Unavailable HTTP/1.1 201 Created The Code of Forking Paths

THE CODE OF FORKING PATHSAtaxo Social Insider The Code of Forking Paths

1 Asynchronous Task Processing The Code of Forking Paths

THE CODE OF FORKING PATHSAsynchronous Task ProcessingLong running requestsCanonical Example: video upload and conversion The Code of Forking Paths

Request WORKLOAD ResponseRequest NOTIFICATION Response WORKLOAD The Code of Forking Paths

THE CODE OF FORKING PATHS„Engineering“ solutionclass UploadController < ApplicationController def create # .... Thread.new do # *** WORK REALLY HARD <HERE> *** What could possibly go wrong? end render :text => "KTHXBAI!" endend The Code of Forking Paths

THE CODE OF FORKING PATHSThe Task Queueclass UploadController < ApplicationController def create # Store uploaded file, store record put_the_video_on_the_processing_queue(@video.id) render :text => "Thanks! Your video is being processed." endend The Code of Forking Paths

REDISHow Does It Work? RPUSH } LPOP O(1) Millions of items https://github.com/defunkt/resque/blob/v1.13.0/lib/resque.rb#L133-138http://redis.io/commands#list The Code of Forking Paths

TASK QUEUESPoor-man’s Queues 1 „Publisher”/usr/local/bin/redis-‐cli RPUSH "queue" "task-‐01" 2 „Worker”while true; do /usr/local/bin/redis-‐cli BLPOP "queue" 0; done The Code of Forking Paths

TASK QUEUESPoor-man’s Queues 1 „Publisher”/usr/local/bin/redis-‐cli RPUSH "queue" "task-‐01" 2 „Worker”while true; do /usr/local/bin/redis-‐cli BLPOP "queue" 0 /usr/local/bin/redis-‐cli PUBLISH "queue:messages" "Processed task."done 3 „Monitor”/usr/local/bin/redis-‐cli SUBSCRIBE "queue:messages" Demo The Code of Forking Paths

require "redis" require "nest" module Ost VERSION = "0.0.1" TIMEOUT = ENV["OST_TIMEOUT"] || 2 class Queue attr :ns def initialize(name) @ns = Nest.new(:ost)[name] end def push(value) redis.lpush(ns, value) end def each(&block) loop do _, item = redis.brpop(ns, TIMEOUT) next if item.nil? or item.empty? begin block.call(item) rescue Exception => e error = "#{Time.now} #{ns[item]} => #{e.inspect}" redis.rpush ns[:errors], error redis.publish ns[:errors], error end end end def errors redis.lrange ns[:errors], 0, -1 end alias << push alias pop each private def redis Ost.redis end end @queues = Hash.new do |hash, key| hash[key] = Queue.new(key) end def self.[](queue) @queues[queue] end def self.connect(options = {}) @redis = Redis.connect(options) end def self.redis @redis ||= Redis.connect end def self.redis=(redis) @redis = redis end endhttps://github.com/soveran/ost

Demohttps://github.com/karmi/resque-demo

Demohttp://git.karmi.cz/resque_job_polling_demo.git

2 „Messaging” The Code of Forking Paths

MESSAGING„Object Oriented Programming” Im sorry that I long ago coined the term "objects" for thistopic because it gets many people to focus on the lesser idea.e big idea is "messaging" (...).e key in making great and growable systems is much moreto design how its modules communicate rather than whattheir internal properties and behaviors should be.— Alan Kay, prototypes vs classes was: Re: Suns HotSpothttp://lists.squeakfoundation.org/pipermail/squeak-dev/1998-October/017019.html The Code of Forking Paths

MESSAGINGThree Overlooked Features of A Software SystemMaintainability (changing features)Extensibility (adding or removing features)Testability (validating features) The Code of Forking Paths

MESSAGINGAsynchronous Task Processing in A Real WorldWhen you place your order the cashier marks a coﬀee cupwith your order and places it into the queue. e queue isquite literally a queue of coﬀee cups lined up on top of theespresso machine. is queue decouples cashier and baristaand allows the cashier to keep taking orders even if thebarista is backed up for a moment.— Gregor Hohpe, Starbucks Does Not Use Two-Phase Commithttp://www.eaipatterns.com/ramblings/18_starbucks.html The Code of Forking Paths

Message Queue

Workers

MESSAGINGAMQP Alvaro Videla and Jason J.W. Williams, RabbitMQ in Action p. 16 The Code of Forking Paths

AMQPExchanges, Queues and Bindings Alvaro Videla and Jason J.W. Williams, RabbitMQ in Action p. 30 The Code of Forking Paths

The Code of Forking Paths

AMQPPublishers and Consumers 1 Publisherexchange = MQ.topic(log)exchange.publish( {:level => level, :time => Time.now, Message :body => message}.to_json, :key => "log.#{level}" ) Routing Key 2 Consumer (all logs)MQ.queue("all logs").bind( MQ.topic(log), :key => "log.*" ).subscribe do |header, message| ...end 3 Consumer (error logs)MQ.queue("error logs").bind( MQ.topic(log), :key => "log.error" ).subscribe do |message| ...end The Code of Forking Paths

Thumbs up!!!

ASYNCHRONOUS WORLDEvented Programming$.get(/data, function(data) { $(.result).html(data); alert(Loaded data from the server.);}); var mydata = $.get(/data); The Code of Forking Paths

Ry Dahl, Node.js (2009)http://nodejs.org

http://s3.amazonaws.com/four.livejournal/20091117/jsconf.pdf

module BalancingProxy # ... module Callbacks def on_select lambda do |backend| puts "Selected backend: #{backend.url}" backend.increment_counter if Backend.strategy == :balanced end end def on_connect lambda do |backend| module Server puts "Connected" # ... end end Backend.select do |backend| conn.server backend, :host => backend.host, :port => backend.port def on_data lambda do |data| conn.on_connect &Callbacks.on_connect puts "Receiving data" conn.on_data &Callbacks.on_data data end conn.on_response &Callbacks.on_response end conn.on_finish &Callbacks.on_finish end def on_response end lambda do |backend, resp| puts "Handling response" resp end end def on_finish lambda do |backend| puts "Finished" backend.decrement_counter if Backend.strategy == :balanced end end endendhttps://github.com/igrigorik/em-proxy/blob/master/examples/balancing.rb The Code of Forking Paths

THE CODE OF FORKING PATHSResume➡ Long-running Tasks➡ Task Queues (Redis, Resque)➡ Maintainability, Extensibility, Testability➡ “Monadic” Architecture (of loosely coupled parts)➡ Messaging: AMQP, Publisher/Consumer, Routing The Code of Forking Paths

THE CODE OF FORKING PATHSWatch OnlineWatch this lecture online (in Czech):http://multimedia.vse.cz/media/Viewer/?peid=51c06c512f4645289c4e9c749dc85acc1d The Code of Forking Paths

Thanks! d

http://www.techgig.com	3
http://paper.li	1
http://us-w1.rockmelt.com	1
http://twitter.com	1
http://news.taaza.com	1
http://www.linkedin.com	1

The Code Of The Forking Paths (Asynchronous Processing with Resque and AMQP)

by Karel Minarik

Accessibility

Categories

Upload Details

Usage Rights

6 Embeds 8

Statistics

The Code Of The Forking Paths (Asynchronous Processing with Resque and AMQP) Presentation Transcript