Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Hacking with ruby2ruby
         Marc Chung
           OpenRain
      blog.marcchung.com
       blog.openrain.com
        @...
Hacking with ruby2ruby
         Marc Chung
           OpenRain
      blog.marcchung.com
       blog.openrain.com
        @...
ruby2ruby

   Ruby2Ruby
Walks the Ruby AST
  To emit Ruby
sudo gem install ruby2ruby
Limitations

• Only Ruby 1.8
• Not 1.9, yet
• Built into Rubinius
function rhino_rocks() {
   return “Rhino, rocks!”;
}
js> rhino_rocks();

Rhino, rocks!
js> rhino_rocks.toString();

function rhino_rocks() {
   return “Rhino, rocks!”;
}
js> eval(rhino_rocks.toString()).call();

Rhino, rocks!
ruby_rocks = proc {
  return “Ruby, rocks”;
}
irb> ruby_rocks.call;

Ruby, rocks!
irb> require “ruby2ruby”
irb> require “parse_tree_extensions”
irb> ruby_rocks.to_ruby

proc { return “Ruby, rocks” }
irb> eval(ruby_rocks.to_ruby).call

Ruby, rocks!
define_method
class Voice
  define_method(:shout) do |word|
    word.upcase!
  end
end

puts Ruby2Ruby.translate(Voice)
class Voice < Object
   def yell(word)
      word.upcase!
   end
end
alias
class AliasClass
   def old_busted
     quot;42quot;
   end

  alias_method :new_hotness, :old_busted
end

puts Ruby...
class AliasClass < Object
   def new_hotness
     quot;42quot;
   end

  def old_busted
    quot;42quot;
  end
end
#module_function
module HelperModule
  def help
    puts quot;Help mequot;
  end
  module_function : help
end

puts Ruby2R...
module HelperModule
  def help
   puts(quot;Help mequot;)
  end

  def self. help
    puts(quot;Help mequot;)
  end
end
ActiveRecord Models

 class Doctor < ActiveRecord::Base
    has_many :patients
 end
See, Ruby, Run!

• Drmap
• git clone git://github.com/mchung/drmap.git
• Optionally, install Journeta
Under the hood

• Abstract syntax trees
• S-expressions
• ParseTree
class Book
   def title
      “Ruby programming”
   end
end
case NODE_BREAK:
case NODE_NEXT:
 if (node->nd_stts)
   add_to_parse_tree(self, current, node->nd_stts, locals);

 break;
...
[:class, :Book,
    [:const, :Object],
    [:defn, :title, [:scope, [:block,
       [:args], [:str, quot;Ruby programmingq...
class Book < Object
   def title
    “Ruby programming”
   end
end
/opt/local/usr/bin


• parse_tree_show
• r2r_show
class Book
   def title
      “Ruby programming”
   end
end
$ parse_tree_show book.rb
s(:class, :Book,
   nil,
   s(:scope,
       s(:defn, :title,
       s(:args),
           s(:scope, s(:block,
            ...
$ r2r_show book.rb
class Book
   def title()
      “Ruby programming”
   end
end
Operator Precedence
with Guy Decoux: a b c, d
            [[:fcall, :a,
                [:array,
                   [:fcal...
ruby2java
class JavaClass
   def self.main
     puts “Java, rocks!”
   end
end
[:class, :JavaClass,
    [:const, :Object],
    [:defs, [:self], :main,
       [:scope,
       [:block, [:args],
         ...
class Ruby2Java < SexpProcessor
   def process_class(exp)
      “public class #{exp.shift} #{next_token(exp, true)}”
   en...
public class JavaClass {
  public static void main(String argv[]) {
      return “Hello, Java”;
  }
}
Distributing Ruby


• drmap - multi-machine worker queue
• gisting - map/reduce framework
drmap
irb> fn = proc {|x| x**10}
irb> YAML::dump(fn)
=> quot;--- !ruby/object:Proc {}nnquot;
irb> Marshal.dump(fn)
TypeErr...
Enumerable#drmap
module Enumerable

 def drmap(&block)
  pool = Drmap::BeanstalkPool.new

  jid = rand(100)
  each_with_index do |element, ...
def process
  loop do
      begin
         job_payload = next_job
         fn = job_payload[:proc]
         data = job_pay...
drmap

• Trivial
• Distributed
• Powerful
Demo time!
Gisting


• MapReduce in Ruby
• http://github.com/mchung/gisting
map / reduce

• Functional programming
• Iterating over collections
• (5..10).map {|x| x+1} #=> [6, 7, 8, 9, 10, 11]
• (5....
Google MapReduce

• Modeled after map() and reduce()
• Programming model
• Programs are trivially parallelizable
Demo time!
Gisting::Spec
input.file_pattern = quot;file.txtquot;
input.map do |map_input|
  words = map_input.strip.split(quot;tquot;)
  Emit(words[1...
output.reduce do |reduce_input|
  count = 0
  reduce_input.each do |value|
    count += value.to_i
  end
  Emit(count)
end
inputs = args
spec = Gisting::Spec.new
inputs.each do |file_input|
  input = spec.add_input
  input.file_pattern = file_input...
Gisting

• MapReduce programming model in Ruby
• Uses ruby2ruby to serialize procs
• Uses EventMachine for Map/ReduceServer
•   -: 94978
•   mailbox: 20872
•   google: 6758
•   ebay: 2832
•   yahoo: 2674
•   yahoo.com: 2198
•   myspace.com: 1916
...
self.give(:thanks)
                  OpenRain
               openrain.com

                 My Git Repo
          github.c...
Resources
Code Generation: The Safety Scissors Of Metaprogramming
       mwrc2008.confreaks.com/03bowkett.html

       For...
Hacking with ruby2ruby
Hacking with ruby2ruby
Hacking with ruby2ruby
Upcoming SlideShare
Loading in …5
×

Hacking with ruby2ruby

2,979 views

Published on

Gisting is an implementation of Google\'s MapReduce framework for processing and extracting useful information from very large data sets. At the time of this writing, the code is available for PREVIEW at http://github.com/mchung/gisting. I am currently working to release this framework for general usage.

Published in: Technology, Education
  • Be the first to comment

  • Be the first to like this

Hacking with ruby2ruby

  1. 1. Hacking with ruby2ruby Marc Chung OpenRain blog.marcchung.com blog.openrain.com @heisenthought
  2. 2. Hacking with ruby2ruby Marc Chung OpenRain blog.marcchung.com blog.openrain.com @heisenthought
  3. 3. ruby2ruby Ruby2Ruby Walks the Ruby AST To emit Ruby
  4. 4. sudo gem install ruby2ruby
  5. 5. Limitations • Only Ruby 1.8 • Not 1.9, yet • Built into Rubinius
  6. 6. function rhino_rocks() { return “Rhino, rocks!”; }
  7. 7. js> rhino_rocks(); Rhino, rocks!
  8. 8. js> rhino_rocks.toString(); function rhino_rocks() { return “Rhino, rocks!”; }
  9. 9. js> eval(rhino_rocks.toString()).call(); Rhino, rocks!
  10. 10. ruby_rocks = proc { return “Ruby, rocks”; }
  11. 11. irb> ruby_rocks.call; Ruby, rocks!
  12. 12. irb> require “ruby2ruby” irb> require “parse_tree_extensions” irb> ruby_rocks.to_ruby proc { return “Ruby, rocks” }
  13. 13. irb> eval(ruby_rocks.to_ruby).call Ruby, rocks!
  14. 14. define_method class Voice define_method(:shout) do |word| word.upcase! end end puts Ruby2Ruby.translate(Voice)
  15. 15. class Voice < Object def yell(word) word.upcase! end end
  16. 16. alias class AliasClass def old_busted quot;42quot; end alias_method :new_hotness, :old_busted end puts Ruby2Ruby.translate(AliasClass)
  17. 17. class AliasClass < Object def new_hotness quot;42quot; end def old_busted quot;42quot; end end
  18. 18. #module_function module HelperModule def help puts quot;Help mequot; end module_function : help end puts Ruby2Ruby.translate(HelperModule)
  19. 19. module HelperModule def help puts(quot;Help mequot;) end def self. help puts(quot;Help mequot;) end end
  20. 20. ActiveRecord Models class Doctor < ActiveRecord::Base has_many :patients end
  21. 21. See, Ruby, Run! • Drmap • git clone git://github.com/mchung/drmap.git • Optionally, install Journeta
  22. 22. Under the hood • Abstract syntax trees • S-expressions • ParseTree
  23. 23. class Book def title “Ruby programming” end end
  24. 24. case NODE_BREAK: case NODE_NEXT: if (node->nd_stts) add_to_parse_tree(self, current, node->nd_stts, locals); break; case NODE_YIELD: if (node->nd_stts) add_to_parse_tree(self, current, node->nd_stts, locals); if (node->nd_stts && (nd_type(node->nd_stts) == NODE_ARRAY || nd_type(node->nd_stts) == NODE_ZARRAY) && !node->nd_state) rb_ary_push(current, Qtrue); break;
  25. 25. [:class, :Book, [:const, :Object], [:defn, :title, [:scope, [:block, [:args], [:str, quot;Ruby programmingquot;]]]]]
  26. 26. class Book < Object def title “Ruby programming” end end
  27. 27. /opt/local/usr/bin • parse_tree_show • r2r_show
  28. 28. class Book def title “Ruby programming” end end
  29. 29. $ parse_tree_show book.rb
  30. 30. s(:class, :Book, nil, s(:scope, s(:defn, :title, s(:args), s(:scope, s(:block, s(:str, quot;Ruby programmingquot;))))))
  31. 31. $ r2r_show book.rb
  32. 32. class Book def title() “Ruby programming” end end
  33. 33. Operator Precedence with Guy Decoux: a b c, d [[:fcall, :a, [:array, [:fcall, :b, [:array, [:vcall, :c], [:vcall, :d]]]]]] ParseTree.new.parse_tree_for_string(“a b c, d”)
  34. 34. ruby2java
  35. 35. class JavaClass def self.main puts “Java, rocks!” end end
  36. 36. [:class, :JavaClass, [:const, :Object], [:defs, [:self], :main, [:scope, [:block, [:args], [:fcall, :puts, [:array, [:str, quot;Hello, Javaquot;]]]]]]]
  37. 37. class Ruby2Java < SexpProcessor def process_class(exp) “public class #{exp.shift} #{next_token(exp, true)}” end def process_const(exp) “extends #{exp.shift}” end .... end
  38. 38. public class JavaClass { public static void main(String argv[]) { return “Hello, Java”; } }
  39. 39. Distributing Ruby • drmap - multi-machine worker queue • gisting - map/reduce framework
  40. 40. drmap irb> fn = proc {|x| x**10} irb> YAML::dump(fn) => quot;--- !ruby/object:Proc {}nnquot; irb> Marshal.dump(fn) TypeError: no marshal_dump is defined for class Proc irb> YAML::dump(fn.to_ruby) => quot;--- proc { |x| (x ** 10) }nquot; irb> Marshal.dump(fn.to_ruby) => quot;004bquot;eproc { |x| (x ** 10) }quot;
  41. 41. Enumerable#drmap
  42. 42. module Enumerable def drmap(&block) pool = Drmap::BeanstalkPool.new jid = rand(100) each_with_index do |element, idx| pool.put_job(jid, block.to_ruby, element) end results = [] while results.size < length results << pool.next_result(jid) end results end end
  43. 43. def process loop do begin job_payload = next_job fn = job_payload[:proc] data = job_payload[:data] result = eval(fn).call(data) save(job_payload[:jid], result) success! rescue => e fail!(e) end end end
  44. 44. drmap • Trivial • Distributed • Powerful
  45. 45. Demo time!
  46. 46. Gisting • MapReduce in Ruby • http://github.com/mchung/gisting
  47. 47. map / reduce • Functional programming • Iterating over collections • (5..10).map {|x| x+1} #=> [6, 7, 8, 9, 10, 11] • (5..10).reduce(:+) #=> 45
  48. 48. Google MapReduce • Modeled after map() and reduce() • Programming model • Programs are trivially parallelizable
  49. 49. Demo time!
  50. 50. Gisting::Spec
  51. 51. input.file_pattern = quot;file.txtquot; input.map do |map_input| words = map_input.strip.split(quot;tquot;) Emit(words[1], quot;1quot;) end
  52. 52. output.reduce do |reduce_input| count = 0 reduce_input.each do |value| count += value.to_i end Emit(count) end
  53. 53. inputs = args spec = Gisting::Spec.new inputs.each do |file_input| input = spec.add_input input.file_pattern = file_input input.map do |map_input| # 2722 mailbox 2006-05-23 00:08:39 words = map_input.strip.split(quot;tquot;) Emit(words[1], quot;1quot;) end end output = spec.output output.filebase = quot;/Users/mchung/Public/datasets/outputquot; output.num_tasks = 2 output.reduce do |reduce_input| count = 0 reduce_input.each do |value| count += value.to_i end Emit(count) end result = MapReduce(spec) pp result
  54. 54. Gisting • MapReduce programming model in Ruby • Uses ruby2ruby to serialize procs • Uses EventMachine for Map/ReduceServer
  55. 55. • -: 94978 • mailbox: 20872 • google: 6758 • ebay: 2832 • yahoo: 2674 • yahoo.com: 2198 • myspace.com: 1916 • google.com: 1882
  56. 56. self.give(:thanks) OpenRain openrain.com My Git Repo github.com/mchung Marc Chung blog.marcchung.com Ruby::AZ rubyaz.org
  57. 57. Resources Code Generation: The Safety Scissors Of Metaprogramming mwrc2008.confreaks.com/03bowkett.html Forbidden Fruit: A Taste of Ruby's Parse Tree goruco2008.confreaks.com/03_wanstrath.html Ruby2Ruby seattlerb.rubyforge.org/ruby2ruby/ ParseTree parsetree.rubyforge.org

×