Dataflow: Declarative concurrency in Ruby

2,796
-1

Published on

While Ruby is known for its flexibility due to high mutability and meta-programming capability, these features make writing thread-safe programs using manual locking very error-prone. For this reason some people are switching to languages with easier to manage concurrency paradigms, such as Erlang/Scala’s message passing, or Clojure/Haskell’s Software Transactional Memory (STM).

This talk is about Dataflow, a pure Ruby gem that adds dataflow variables to the Ruby language. Dataflow variables are write-once (or write multiple times with the same value), and suspend execution in the current thread/context if called before being assigned/bound. We will explore how this technique makes writing concurrent but thread-safe code easy, even making it possible to write tests that spawn threads without needing to worry.

Declarative concurrency is a relatively unknown programming model that is an alternative to message passing and STM. Ruby’s malleability makes it an ideal host for this model. Besides performance implications, dataflow variables also have an important impact on declarative program modeling. The talk will also go over the differences in performance and memory of the library in various Ruby implementations.

Published in: Technology, Education
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,796
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
31
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

Dataflow: Declarative concurrency in Ruby

  1. 1. Dataflow The declarative concurrent programming model
  2. 2. Larry Diehl {:larrytheliquid => %w[.com github twitter]}
  3. 3. Outline Purpose of presentation Gradual explanation of concepts Helpful tips
  4. 4. Purpose
  5. 5. Lexical Scope foo = :foo define_method :foo do foo end
  6. 6. Dynamic Scope def foo @foo end
  7. 7. Mutability def initialize @foo = :foo end def foo @foo end
  8. 8. Mutability def foo @foo = :foo @foo end
  9. 9. Mutability+Concurrency def initialize Thread.new { loop { @foo = :shazbot } } end def foo @foo = :foo @foo end
  10. 10. The Declarative Model
  11. 11. Declarative Synchronous my_var = :bound my_var = :rebind # NOT ALLOWED!
  12. 12. Declarative Synchronous local do |my_var| my_var.object_id # thread sleeps end
  13. 13. Declarative Synchronous local do |my_var| unify my_var, :bound unify my_var, :rebind # => # Dataflow::UnificationError, # ":bound != :rebind" end
  14. 14. Declarative Synchronous class MyClass declare :my_var def initialize unify my_var, :bound end end
  15. 15. Declarative Concurrent (MAGIC)
  16. 16. Declarative Concurrent local do |my_var| Thread.new { unify my_var, :bound } my_var.should == :bound end
  17. 17. Dependency Resolution local do |sentence, middle, tail| Thread.new { unify middle, "base are belong #{tail}" } Thread.new { unify tail, "to us" } Thread.new { unify sentence, "all your #{middle}" } sentence.should == "all your base are belong to us" end
  18. 18. Asynchronous Output def Worker.async(output=nil) Thread.new do result = # do hard work unify output, result if output end end local do |output| Worker.async(output) output.should == # hard work result end
  19. 19. Asynchronous Output local do |output| flow(output) do # do hard work end output.should == # hard work result end
  20. 20. Anonymous variables {'google.com' => Dataflow::Variable.new, 'bing.com' => Dataflow::Variable.new }.map do |domain,var| Thread.new do unify var, open("http://#{domain}").read end var end
  21. 21. need_later %w[google.com bing.com].map do |domain| need_later { open("http://#{domain}").read } end
  22. 22. Chunked Sequential Processing (1..100).each_slice(10).map do |chunk| sleep(1) chunk.inject(&:+) end.inject(&:+) # => ~10s
  23. 23. Chunked Parallel Processing (1..100).each_slice(10).map do |chunk| need_later do sleep(1) chunk.inject(&:+) end end.inject(&:+) # => ~1s
  24. 24. Leaving Declarative via Async
  25. 25. Ports & Streams local do |port, stream| unify port, Dataflow::Port.new(stream) port.send 1 port.send 2 stream.take(2).should == [1, 2] end
  26. 26. Ports & Streams (async) local do |port, stream| unify port, Dataflow::Port.new(stream) Thread.new do stream.each do |message| puts "received: #{message}" end end %w[x y z].each do |letter| Thread.new{ port.send letter } end stream.take(3).sort.should == %w[x y z] end
  27. 27. FutureQueue local do |queue, first, second, third| unify queue, FutureQueue.new queue.pop first queue.pop second queue.push 1 queue.push 2 queue.push 3 queue.pop third [first, second, third].should == [1, 2, 3] end
  28. 28. Actors Ping = Actor.new { Pong = Actor.new { 3.times { 3.times { case receive case receive when :ping when :pong puts "Ping" puts "Pong" Pong.send :pong Ping.send :ping end end } } } } Ping.send :ping
  29. 29. by_need def baz(num) might_get_used = by_need { Factory.gen } might_get_used.value if num%2 == 0 end
  30. 30. Tips
  31. 31. Modular local do |my_var| Thread.new { unify my_var, :bound } # my_var.wait my_var.should == :bound end
  32. 32. Debugging local do |my_var| my_var.inspect # => #<Dataflow::Variable:2637860 unbound> end
  33. 33. Class/Module methods Dataflow.local do |my_var| Dataflow.async do Dataflow.unify my_var, :bound end my_var.should == :bound end
  34. 34. Use Cases general purpose concurrency for elegant program structure with respect to coordination concurrency to make use of extra processors/cores (depending on Ruby implementation) web development worker daemons concurrently munging together data from various rest api's
  35. 35. Ruby Implementations Pure Ruby library, should work on any implementation JRuby in particular has a great GC, no GIL, native threads, and a tunable threadpool option. Rubinius has more code written in Ruby, so it proxies more method calls (e.g. Array#flatten).
  36. 36. class FutureQueue include Dataflow declare :push_port, :pop_port def initialize local do |pushed, popped| unify push_port, Dataflow::Port.new(pushed) unify pop_port, Dataflow::Port.new(popped) Thread.new { loop do barrier pushed.head, popped.head unify popped.head, pushed.head pushed, popped = pushed.tail, popped.tail end } end end def push(x) push_port.send x end def pop(x) pop_port.send x end end
  37. 37. The End sudo port install dataflow http://github.com /larrytheliquid /dataflow freenode: #dataflow-gem
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×