Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Dataflow
The declarative concurrent
   programming model
Larry Diehl

  {:larrytheliquid =>
   %w[.com
        github
        twitter]}
Outline

Purpose of presentation



Gradual explanation of concepts



Helpful tips
Purpose
Lexical Scope


foo = :foo
define_method :foo do
 foo
end
Dynamic Scope

def foo
 @foo
end
Mutability

def initialize
 @foo = :foo
end

def foo
 @foo
end
Mutability


def foo
 @foo = :foo
 @foo
end
Mutability+Concurrency

def initialize
 Thread.new { loop { @foo = :shazbot } }
end

def foo
 @foo = :foo
 @foo
end
The Declarative Model
Declarative Synchronous


my_var = :bound
my_var = :rebind # NOT ALLOWED!
Declarative Synchronous


local do |my_var|
  my_var.object_id # thread sleeps
end
Declarative Synchronous


local do |my_var|
  unify my_var, :bound
  unify my_var, :rebind # =>
  # Dataflow::UnificationE...
Declarative Synchronous

class MyClass
 declare :my_var
 def initialize
  unify my_var, :bound
 end
end
Declarative Concurrent
     (MAGIC)
Declarative Concurrent


local do |my_var|
  Thread.new { unify my_var, :bound }
  my_var.should == :bound
end
Dependency Resolution

local do |sentence, middle, tail|
  Thread.new { unify middle, "base are belong #{tail}" }
  Thread...
Asynchronous Output
def Worker.async(output=nil)
 Thread.new do
  result = # do hard work
  unify output, result if output...
Asynchronous Output
local do |output|
  flow(output) do
    # do hard work
  end
  output.should == # hard work result
end
Anonymous variables

{'google.com' => Dataflow::Variable.new,
 'bing.com' => Dataflow::Variable.new
}.map do |domain,var|
...
need_later

%w[google.com bing.com].map do |domain|
 need_later { open("http://#{domain}").read }
end
Chunked Sequential Processing


 (1..100).each_slice(10).map do |chunk|
  sleep(1)
  chunk.inject(&:+)
 end.inject(&:+) # ...
Chunked Parallel Processing


(1..100).each_slice(10).map do |chunk|
 need_later do
   sleep(1)
   chunk.inject(&:+)
 end
...
Leaving Declarative
    via Async
Ports & Streams

local do |port, stream|
  unify port, Dataflow::Port.new(stream)
  port.send 1
  port.send 2
  stream.tak...
Ports & Streams (async)
local do |port, stream|
  unify port, Dataflow::Port.new(stream)
  Thread.new do
   stream.each do...
FutureQueue
local do |queue, first, second, third|
  unify queue, FutureQueue.new
  queue.pop first
  queue.pop second
  q...
Actors
Ping = Actor.new {                  Pong = Actor.new {
  3.times {                           3.times {
    case rec...
by_need

def baz(num)
 might_get_used = by_need { Factory.gen }
 might_get_used.value if num%2 == 0
end
Tips
Modular

local do |my_var|
  Thread.new { unify my_var, :bound }
  # my_var.wait
  my_var.should == :bound
end
Debugging

local do |my_var|
  my_var.inspect
# => #<Dataflow::Variable:2637860 unbound>
end
Class/Module methods
Dataflow.local do |my_var|
 Dataflow.async do
  Dataflow.unify my_var, :bound
 end
 my_var.should == ...
Use Cases
general purpose
   concurrency for elegant program structure with respect
   to coordination

   concurrency to ...
Ruby Implementations

Pure Ruby library, should work on any implementation


JRuby in particular has a great GC, no GIL, n...
class FutureQueue
 include Dataflow
 declare :push_port, :pop_port

 def initialize
  local do |pushed, popped|
    unify ...
The End
sudo port install dataflow



   http://github.com
   /larrytheliquid
   /dataflow


freenode: #dataflow-gem
Upcoming SlideShare
Loading in …5
×

Dataflow: Declarative concurrency in Ruby

3,084 views

Published on

While Ruby is known for its flexibility due to high mutability and meta-programming capability, these features make writing thread-safe programs using manual locking very error-prone. For this reason some people are switching to languages with easier to manage concurrency paradigms, such as Erlang/Scala’s message passing, or Clojure/Haskell’s Software Transactional Memory (STM).

This talk is about Dataflow, a pure Ruby gem that adds dataflow variables to the Ruby language. Dataflow variables are write-once (or write multiple times with the same value), and suspend execution in the current thread/context if called before being assigned/bound. We will explore how this technique makes writing concurrent but thread-safe code easy, even making it possible to write tests that spawn threads without needing to worry.

Declarative concurrency is a relatively unknown programming model that is an alternative to message passing and STM. Ruby’s malleability makes it an ideal host for this model. Besides performance implications, dataflow variables also have an important impact on declarative program modeling. The talk will also go over the differences in performance and memory of the library in various Ruby implementations.

Published in: Technology, Education
  • Hi there! Get Your Professional Job-Winning Resume Here - Check our website! http://bit.ly/resumpro
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Dataflow: Declarative concurrency in Ruby

  1. 1. Dataflow The declarative concurrent programming model
  2. 2. Larry Diehl {:larrytheliquid => %w[.com github twitter]}
  3. 3. Outline Purpose of presentation Gradual explanation of concepts Helpful tips
  4. 4. Purpose
  5. 5. Lexical Scope foo = :foo define_method :foo do foo end
  6. 6. Dynamic Scope def foo @foo end
  7. 7. Mutability def initialize @foo = :foo end def foo @foo end
  8. 8. Mutability def foo @foo = :foo @foo end
  9. 9. Mutability+Concurrency def initialize Thread.new { loop { @foo = :shazbot } } end def foo @foo = :foo @foo end
  10. 10. The Declarative Model
  11. 11. Declarative Synchronous my_var = :bound my_var = :rebind # NOT ALLOWED!
  12. 12. Declarative Synchronous local do |my_var| my_var.object_id # thread sleeps end
  13. 13. Declarative Synchronous local do |my_var| unify my_var, :bound unify my_var, :rebind # => # Dataflow::UnificationError, # ":bound != :rebind" end
  14. 14. Declarative Synchronous class MyClass declare :my_var def initialize unify my_var, :bound end end
  15. 15. Declarative Concurrent (MAGIC)
  16. 16. Declarative Concurrent local do |my_var| Thread.new { unify my_var, :bound } my_var.should == :bound end
  17. 17. Dependency Resolution local do |sentence, middle, tail| Thread.new { unify middle, "base are belong #{tail}" } Thread.new { unify tail, "to us" } Thread.new { unify sentence, "all your #{middle}" } sentence.should == "all your base are belong to us" end
  18. 18. Asynchronous Output def Worker.async(output=nil) Thread.new do result = # do hard work unify output, result if output end end local do |output| Worker.async(output) output.should == # hard work result end
  19. 19. Asynchronous Output local do |output| flow(output) do # do hard work end output.should == # hard work result end
  20. 20. Anonymous variables {'google.com' => Dataflow::Variable.new, 'bing.com' => Dataflow::Variable.new }.map do |domain,var| Thread.new do unify var, open("http://#{domain}").read end var end
  21. 21. need_later %w[google.com bing.com].map do |domain| need_later { open("http://#{domain}").read } end
  22. 22. Chunked Sequential Processing (1..100).each_slice(10).map do |chunk| sleep(1) chunk.inject(&:+) end.inject(&:+) # => ~10s
  23. 23. Chunked Parallel Processing (1..100).each_slice(10).map do |chunk| need_later do sleep(1) chunk.inject(&:+) end end.inject(&:+) # => ~1s
  24. 24. Leaving Declarative via Async
  25. 25. Ports & Streams local do |port, stream| unify port, Dataflow::Port.new(stream) port.send 1 port.send 2 stream.take(2).should == [1, 2] end
  26. 26. Ports & Streams (async) local do |port, stream| unify port, Dataflow::Port.new(stream) Thread.new do stream.each do |message| puts "received: #{message}" end end %w[x y z].each do |letter| Thread.new{ port.send letter } end stream.take(3).sort.should == %w[x y z] end
  27. 27. FutureQueue local do |queue, first, second, third| unify queue, FutureQueue.new queue.pop first queue.pop second queue.push 1 queue.push 2 queue.push 3 queue.pop third [first, second, third].should == [1, 2, 3] end
  28. 28. Actors Ping = Actor.new { Pong = Actor.new { 3.times { 3.times { case receive case receive when :ping when :pong puts "Ping" puts "Pong" Pong.send :pong Ping.send :ping end end } } } } Ping.send :ping
  29. 29. by_need def baz(num) might_get_used = by_need { Factory.gen } might_get_used.value if num%2 == 0 end
  30. 30. Tips
  31. 31. Modular local do |my_var| Thread.new { unify my_var, :bound } # my_var.wait my_var.should == :bound end
  32. 32. Debugging local do |my_var| my_var.inspect # => #<Dataflow::Variable:2637860 unbound> end
  33. 33. Class/Module methods Dataflow.local do |my_var| Dataflow.async do Dataflow.unify my_var, :bound end my_var.should == :bound end
  34. 34. Use Cases general purpose concurrency for elegant program structure with respect to coordination concurrency to make use of extra processors/cores (depending on Ruby implementation) web development worker daemons concurrently munging together data from various rest api's
  35. 35. Ruby Implementations Pure Ruby library, should work on any implementation JRuby in particular has a great GC, no GIL, native threads, and a tunable threadpool option. Rubinius has more code written in Ruby, so it proxies more method calls (e.g. Array#flatten).
  36. 36. class FutureQueue include Dataflow declare :push_port, :pop_port def initialize local do |pushed, popped| unify push_port, Dataflow::Port.new(pushed) unify pop_port, Dataflow::Port.new(popped) Thread.new { loop do barrier pushed.head, popped.head unify popped.head, pushed.head pushed, popped = pushed.tail, popped.tail end } end end def push(x) push_port.send x end def pop(x) pop_port.send x end end
  37. 37. The End sudo port install dataflow http://github.com /larrytheliquid /dataflow freenode: #dataflow-gem

×