The Code Of The Forking Paths (Asynchronous Processing with Resque and AMQP)
Upcoming SlideShare
Loading in...5
×
 

The Code Of The Forking Paths (Asynchronous Processing with Resque and AMQP)

on

  • 4,362 views

Slides for a lecture on task queues, asynchronous processing, messaging and architecture at University of Economics in Prague. ...

Slides for a lecture on task queues, asynchronous processing, messaging and architecture at University of Economics in Prague.

Video available online: http://multimedia.vse.cz/media/Viewer/?peid=51c06c512f4645289c4e9c749dc85acc1d (Silverlight, so Windows only)

Statistics

Views

Total Views
4,362
Views on SlideShare
4,346
Embed Views
16

Actions

Likes
10
Downloads
59
Comments
2

7 Embeds 16

http://www.techgig.com 6
https://twitter.com 5
http://paper.li 1
http://us-w1.rockmelt.com 1
http://twitter.com 1
http://news.taaza.com 1
http://www.linkedin.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

The Code Of The Forking Paths (Asynchronous Processing with Resque and AMQP) The Code Of The Forking Paths (Asynchronous Processing with Resque and AMQP) Presentation Transcript

  • The Code of Forking PathsKarel Minařík
  • Karel Minařík→ Independent web designer and developer→ Ruby, Rails, Git and CouchDB propagandista in .cz→ Previously: Flash Developer; Art Director; Information Architect;… (see LinkedIn)→ @karmiq at Twitter→ karmi.cz The Code of Forking Paths
  • “He believed in an infinite series of times, in a growing,dizzying net of divergent, convergent and paralleltimes. (...) We do not exist in the majority of thesetimes; in some you exist, and not I; in others I, and notyou; in others, both of us. In the present one, whicha favorable fate has granted me, you have arrived at myhouse; in another, while crossing the garden, you foundme dead; in still another, I utter these same words, butI am a mistake, a ghost.”
  • http://en.wikipedia.org/wiki/The_Garden_of_Forking_Paths The Code of Forking Paths
  • THE CODE OF FORKING PATHSParallel presence HTTP/1.1  503  Service  Unavailable HTTP/1.1  201  Created The Code of Forking Paths
  • THE CODE OF FORKING PATHSAtaxo Social Insider The Code of Forking Paths
  • 1 Asynchronous Task Processing The Code of Forking Paths
  • THE CODE OF FORKING PATHSAsynchronous Task ProcessingLong running requestsCanonical Example: video upload and conversion The Code of Forking Paths
  • Request WORKLOAD ResponseRequest NOTIFICATION Response WORKLOAD The Code of Forking Paths
  • THE CODE OF FORKING PATHS„Engineering“ solutionclass  UploadController  <  ApplicationController    def  create        #  ....        Thread.new  do            #  ***  WORK  REALLY  HARD  <HERE>  *** What could possibly go wrong?        end        render  :text  =>  "KTHXBAI!"    endend The Code of Forking Paths
  • THE CODE OF FORKING PATHSThe Task Queueclass  UploadController  <  ApplicationController    def  create        #  Store  uploaded  file,  store  record        put_the_video_on_the_processing_queue(@video.id)        render  :text  =>  "Thanks!  Your  video  is  being  processed."    endend The Code of Forking Paths
  • REDISHow Does It Work? RPUSH } LPOP O(1) Millions of items https://github.com/defunkt/resque/blob/v1.13.0/lib/resque.rb#L133-138http://redis.io/commands#list The Code of Forking Paths
  • TASK QUEUESPoor-man’s Queues 1 „Publisher”/usr/local/bin/redis-­‐cli  RPUSH  "queue"  "task-­‐01" 2 „Worker”while  true;  do  /usr/local/bin/redis-­‐cli  BLPOP  "queue"  0;  done The Code of Forking Paths
  • TASK QUEUESPoor-man’s Queues 1 „Publisher”/usr/local/bin/redis-­‐cli  RPUSH  "queue"  "task-­‐01" 2 „Worker”while  true;  do    /usr/local/bin/redis-­‐cli  BLPOP  "queue"  0    /usr/local/bin/redis-­‐cli  PUBLISH  "queue:messages"  "Processed  task."done 3 „Monitor”/usr/local/bin/redis-­‐cli  SUBSCRIBE  "queue:messages" Demo The Code of Forking Paths
  • require "redis" require "nest" module Ost VERSION = "0.0.1" TIMEOUT = ENV["OST_TIMEOUT"] || 2 class Queue attr :ns def initialize(name) @ns = Nest.new(:ost)[name] end def push(value) redis.lpush(ns, value) end def each(&block) loop do _, item = redis.brpop(ns, TIMEOUT) next if item.nil? or item.empty? begin block.call(item) rescue Exception => e error = "#{Time.now} #{ns[item]} => #{e.inspect}" redis.rpush ns[:errors], error redis.publish ns[:errors], error end end end def errors redis.lrange ns[:errors], 0, -1 end alias << push alias pop each private def redis Ost.redis end end @queues = Hash.new do |hash, key| hash[key] = Queue.new(key) end def self.[](queue) @queues[queue] end def self.connect(options = {}) @redis = Redis.connect(options) end def self.redis @redis ||= Redis.connect end def self.redis=(redis) @redis = redis end endhttps://github.com/soveran/ost
  • Demohttps://github.com/karmi/resque-demo
  • Demohttp://git.karmi.cz/resque_job_polling_demo.git
  • 2 „Messaging” The Code of Forking Paths
  • MESSAGING„Object Oriented Programming” Im sorry that I long ago coined the term "objects" for thistopic because it gets many people to focus on the lesser idea.e big idea is "messaging" (...).e key in making great and growable systems is much moreto design how its modules communicate rather than whattheir internal properties and behaviors should be.— Alan Kay, prototypes vs classes was: Re: Suns HotSpothttp://lists.squeakfoundation.org/pipermail/squeak-dev/1998-October/017019.html The Code of Forking Paths
  • MESSAGINGThree Overlooked Features of A Software SystemMaintainability (changing features)Extensibility (adding or removing features)Testability (validating features) The Code of Forking Paths
  • MESSAGINGAsynchronous Task Processing in A Real WorldWhen you place your order the cashier marks a coffee cupwith your order and places it into the queue. e queue isquite literally a queue of coffee cups lined up on top of theespresso machine. is queue decouples cashier and baristaand allows the cashier to keep taking orders even if thebarista is backed up for a moment.— Gregor Hohpe, Starbucks Does Not Use Two-Phase Commithttp://www.eaipatterns.com/ramblings/18_starbucks.html The Code of Forking Paths
  • Message Queue
  • Workers
  • MESSAGINGAMQP Alvaro Videla and Jason J.W. Williams, RabbitMQ in Action p. 16 The Code of Forking Paths
  • AMQPExchanges, Queues and Bindings Alvaro Videla and Jason J.W. Williams, RabbitMQ in Action p. 30 The Code of Forking Paths
  • The Code of Forking Paths
  • AMQPPublishers and Consumers 1 Publisherexchange  =  MQ.topic(log)exchange.publish(  {:level  =>  level,                                      :time    =>  Time.now, Message                                      :body    =>  message}.to_json,                                      :key      =>  "log.#{level}"  ) Routing Key 2 Consumer (all logs)MQ.queue("all  logs").bind(  MQ.topic(log),  :key  =>  "log.*"  ).subscribe  do  |header,  message|    ...end 3 Consumer (error logs)MQ.queue("error  logs").bind(  MQ.topic(log),  :key  =>  "log.error"  ).subscribe  do  |message|    ...end The Code of Forking Paths
  • Thumbs up!!!
  • ASYNCHRONOUS WORLDEvented Programming$.get(/data,  function(data)  {    $(.result).html(data);    alert(Loaded  data  from  the  server.);}); var  mydata  =  $.get(/data); The Code of Forking Paths
  • Ry Dahl, Node.js (2009)http://nodejs.org
  • http://s3.amazonaws.com/four.livejournal/20091117/jsconf.pdf
  • module BalancingProxy # ... module Callbacks def on_select lambda do |backend| puts "Selected backend: #{backend.url}" backend.increment_counter if Backend.strategy == :balanced end end def on_connect lambda do |backend| module  Server puts "Connected"        #  ... end end        Backend.select  do  |backend|            conn.server  backend,  :host  =>  backend.host,  :port  =>  backend.port def on_data lambda do |data|            conn.on_connect    &Callbacks.on_connect puts "Receiving data"            conn.on_data          &Callbacks.on_data data end            conn.on_response  &Callbacks.on_response end            conn.on_finish      &Callbacks.on_finish        end def on_response    end lambda do |backend, resp| puts "Handling response" resp end end def on_finish lambda do |backend| puts "Finished" backend.decrement_counter if Backend.strategy == :balanced end end endendhttps://github.com/igrigorik/em-proxy/blob/master/examples/balancing.rb The Code of Forking Paths
  • THE CODE OF FORKING PATHSResume➡ Long-running Tasks➡ Task Queues (Redis, Resque)➡ Maintainability, Extensibility, Testability➡ “Monadic” Architecture (of loosely coupled parts)➡ Messaging: AMQP, Publisher/Consumer, Routing The Code of Forking Paths
  • THE CODE OF FORKING PATHSWatch OnlineWatch this lecture online (in Czech):http://multimedia.vse.cz/media/Viewer/?peid=51c06c512f4645289c4e9c749dc85acc1d The Code of Forking Paths
  • Thanks! d