The Code Of The Forking Paths (Asynchronous Processing with Resque and AMQP)

  • 3,880 views
Uploaded on

Slides for a lecture on task queues, asynchronous processing, messaging and architecture at University of Economics in Prague. …

Slides for a lecture on task queues, asynchronous processing, messaging and architecture at University of Economics in Prague.

Video available online: http://multimedia.vse.cz/media/Viewer/?peid=51c06c512f4645289c4e9c749dc85acc1d (Silverlight, so Windows only)

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
3,880
On Slideshare
0
From Embeds
0
Number of Embeds
3

Actions

Shares
Downloads
60
Comments
2
Likes
10

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. The Code of Forking PathsKarel Minařík
  • 2. Karel Minařík→ Independent web designer and developer→ Ruby, Rails, Git and CouchDB propagandista in .cz→ Previously: Flash Developer; Art Director; Information Architect;… (see LinkedIn)→ @karmiq at Twitter→ karmi.cz The Code of Forking Paths
  • 3. “He believed in an infinite series of times, in a growing,dizzying net of divergent, convergent and paralleltimes. (...) We do not exist in the majority of thesetimes; in some you exist, and not I; in others I, and notyou; in others, both of us. In the present one, whicha favorable fate has granted me, you have arrived at myhouse; in another, while crossing the garden, you foundme dead; in still another, I utter these same words, butI am a mistake, a ghost.”
  • 4. http://en.wikipedia.org/wiki/The_Garden_of_Forking_Paths The Code of Forking Paths
  • 5. THE CODE OF FORKING PATHSParallel presence HTTP/1.1  503  Service  Unavailable HTTP/1.1  201  Created The Code of Forking Paths
  • 6. THE CODE OF FORKING PATHSAtaxo Social Insider The Code of Forking Paths
  • 7. 1 Asynchronous Task Processing The Code of Forking Paths
  • 8. THE CODE OF FORKING PATHSAsynchronous Task ProcessingLong running requestsCanonical Example: video upload and conversion The Code of Forking Paths
  • 9. Request WORKLOAD ResponseRequest NOTIFICATION Response WORKLOAD The Code of Forking Paths
  • 10. THE CODE OF FORKING PATHS„Engineering“ solutionclass  UploadController  <  ApplicationController    def  create        #  ....        Thread.new  do            #  ***  WORK  REALLY  HARD  <HERE>  *** What could possibly go wrong?        end        render  :text  =>  "KTHXBAI!"    endend The Code of Forking Paths
  • 11. THE CODE OF FORKING PATHSThe Task Queueclass  UploadController  <  ApplicationController    def  create        #  Store  uploaded  file,  store  record        put_the_video_on_the_processing_queue(@video.id)        render  :text  =>  "Thanks!  Your  video  is  being  processed."    endend The Code of Forking Paths
  • 12. REDISHow Does It Work? RPUSH } LPOP O(1) Millions of items https://github.com/defunkt/resque/blob/v1.13.0/lib/resque.rb#L133-138http://redis.io/commands#list The Code of Forking Paths
  • 13. TASK QUEUESPoor-man’s Queues 1 „Publisher”/usr/local/bin/redis-­‐cli  RPUSH  "queue"  "task-­‐01" 2 „Worker”while  true;  do  /usr/local/bin/redis-­‐cli  BLPOP  "queue"  0;  done The Code of Forking Paths
  • 14. TASK QUEUESPoor-man’s Queues 1 „Publisher”/usr/local/bin/redis-­‐cli  RPUSH  "queue"  "task-­‐01" 2 „Worker”while  true;  do    /usr/local/bin/redis-­‐cli  BLPOP  "queue"  0    /usr/local/bin/redis-­‐cli  PUBLISH  "queue:messages"  "Processed  task."done 3 „Monitor”/usr/local/bin/redis-­‐cli  SUBSCRIBE  "queue:messages" Demo The Code of Forking Paths
  • 15. require "redis" require "nest" module Ost VERSION = "0.0.1" TIMEOUT = ENV["OST_TIMEOUT"] || 2 class Queue attr :ns def initialize(name) @ns = Nest.new(:ost)[name] end def push(value) redis.lpush(ns, value) end def each(&block) loop do _, item = redis.brpop(ns, TIMEOUT) next if item.nil? or item.empty? begin block.call(item) rescue Exception => e error = "#{Time.now} #{ns[item]} => #{e.inspect}" redis.rpush ns[:errors], error redis.publish ns[:errors], error end end end def errors redis.lrange ns[:errors], 0, -1 end alias << push alias pop each private def redis Ost.redis end end @queues = Hash.new do |hash, key| hash[key] = Queue.new(key) end def self.[](queue) @queues[queue] end def self.connect(options = {}) @redis = Redis.connect(options) end def self.redis @redis ||= Redis.connect end def self.redis=(redis) @redis = redis end endhttps://github.com/soveran/ost
  • 16. Demohttps://github.com/karmi/resque-demo
  • 17. Demohttp://git.karmi.cz/resque_job_polling_demo.git
  • 18. 2 „Messaging” The Code of Forking Paths
  • 19. MESSAGING„Object Oriented Programming” Im sorry that I long ago coined the term "objects" for thistopic because it gets many people to focus on the lesser idea.e big idea is "messaging" (...).e key in making great and growable systems is much moreto design how its modules communicate rather than whattheir internal properties and behaviors should be.— Alan Kay, prototypes vs classes was: Re: Suns HotSpothttp://lists.squeakfoundation.org/pipermail/squeak-dev/1998-October/017019.html The Code of Forking Paths
  • 20. MESSAGINGThree Overlooked Features of A Software SystemMaintainability (changing features)Extensibility (adding or removing features)Testability (validating features) The Code of Forking Paths
  • 21. MESSAGINGAsynchronous Task Processing in A Real WorldWhen you place your order the cashier marks a coffee cupwith your order and places it into the queue. e queue isquite literally a queue of coffee cups lined up on top of theespresso machine. is queue decouples cashier and baristaand allows the cashier to keep taking orders even if thebarista is backed up for a moment.— Gregor Hohpe, Starbucks Does Not Use Two-Phase Commithttp://www.eaipatterns.com/ramblings/18_starbucks.html The Code of Forking Paths
  • 22. Message Queue
  • 23. Workers
  • 24. MESSAGINGAMQP Alvaro Videla and Jason J.W. Williams, RabbitMQ in Action p. 16 The Code of Forking Paths
  • 25. AMQPExchanges, Queues and Bindings Alvaro Videla and Jason J.W. Williams, RabbitMQ in Action p. 30 The Code of Forking Paths
  • 26. The Code of Forking Paths
  • 27. AMQPPublishers and Consumers 1 Publisherexchange  =  MQ.topic(log)exchange.publish(  {:level  =>  level,                                      :time    =>  Time.now, Message                                      :body    =>  message}.to_json,                                      :key      =>  "log.#{level}"  ) Routing Key 2 Consumer (all logs)MQ.queue("all  logs").bind(  MQ.topic(log),  :key  =>  "log.*"  ).subscribe  do  |header,  message|    ...end 3 Consumer (error logs)MQ.queue("error  logs").bind(  MQ.topic(log),  :key  =>  "log.error"  ).subscribe  do  |message|    ...end The Code of Forking Paths
  • 28. Thumbs up!!!
  • 29. ASYNCHRONOUS WORLDEvented Programming$.get(/data,  function(data)  {    $(.result).html(data);    alert(Loaded  data  from  the  server.);}); var  mydata  =  $.get(/data); The Code of Forking Paths
  • 30. Ry Dahl, Node.js (2009)http://nodejs.org
  • 31. http://s3.amazonaws.com/four.livejournal/20091117/jsconf.pdf
  • 32. module BalancingProxy # ... module Callbacks def on_select lambda do |backend| puts "Selected backend: #{backend.url}" backend.increment_counter if Backend.strategy == :balanced end end def on_connect lambda do |backend| module  Server puts "Connected"        #  ... end end        Backend.select  do  |backend|            conn.server  backend,  :host  =>  backend.host,  :port  =>  backend.port def on_data lambda do |data|            conn.on_connect    &Callbacks.on_connect puts "Receiving data"            conn.on_data          &Callbacks.on_data data end            conn.on_response  &Callbacks.on_response end            conn.on_finish      &Callbacks.on_finish        end def on_response    end lambda do |backend, resp| puts "Handling response" resp end end def on_finish lambda do |backend| puts "Finished" backend.decrement_counter if Backend.strategy == :balanced end end endendhttps://github.com/igrigorik/em-proxy/blob/master/examples/balancing.rb The Code of Forking Paths
  • 33. THE CODE OF FORKING PATHSResume➡ Long-running Tasks➡ Task Queues (Redis, Resque)➡ Maintainability, Extensibility, Testability➡ “Monadic” Architecture (of loosely coupled parts)➡ Messaging: AMQP, Publisher/Consumer, Routing The Code of Forking Paths
  • 34. THE CODE OF FORKING PATHSWatch OnlineWatch this lecture online (in Czech):http://multimedia.vse.cz/media/Viewer/?peid=51c06c512f4645289c4e9c749dc85acc1d The Code of Forking Paths
  • 35. Thanks! d