Hacking with ruby2ruby

2,875 views

Published on

Gisting is an implementation of Google\'s MapReduce framework for processing and extracting useful information from very large data sets. At the time of this writing, the code is available for PREVIEW at http://github.com/mchung/gisting. I am currently working to release this framework for general usage.

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,875
On SlideShare
0
From Embeds
0
Number of Embeds
38
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Hacking with ruby2ruby

  1. 1. Hacking with ruby2ruby Marc Chung OpenRain blog.marcchung.com blog.openrain.com @heisenthought
  2. 2. Hacking with ruby2ruby Marc Chung OpenRain blog.marcchung.com blog.openrain.com @heisenthought
  3. 3. ruby2ruby Ruby2Ruby Walks the Ruby AST To emit Ruby
  4. 4. sudo gem install ruby2ruby
  5. 5. Limitations • Only Ruby 1.8 • Not 1.9, yet • Built into Rubinius
  6. 6. function rhino_rocks() { return “Rhino, rocks!”; }
  7. 7. js> rhino_rocks(); Rhino, rocks!
  8. 8. js> rhino_rocks.toString(); function rhino_rocks() { return “Rhino, rocks!”; }
  9. 9. js> eval(rhino_rocks.toString()).call(); Rhino, rocks!
  10. 10. ruby_rocks = proc { return “Ruby, rocks”; }
  11. 11. irb> ruby_rocks.call; Ruby, rocks!
  12. 12. irb> require “ruby2ruby” irb> require “parse_tree_extensions” irb> ruby_rocks.to_ruby proc { return “Ruby, rocks” }
  13. 13. irb> eval(ruby_rocks.to_ruby).call Ruby, rocks!
  14. 14. define_method class Voice define_method(:shout) do |word| word.upcase! end end puts Ruby2Ruby.translate(Voice)
  15. 15. class Voice < Object def yell(word) word.upcase! end end
  16. 16. alias class AliasClass def old_busted quot;42quot; end alias_method :new_hotness, :old_busted end puts Ruby2Ruby.translate(AliasClass)
  17. 17. class AliasClass < Object def new_hotness quot;42quot; end def old_busted quot;42quot; end end
  18. 18. #module_function module HelperModule def help puts quot;Help mequot; end module_function : help end puts Ruby2Ruby.translate(HelperModule)
  19. 19. module HelperModule def help puts(quot;Help mequot;) end def self. help puts(quot;Help mequot;) end end
  20. 20. ActiveRecord Models class Doctor < ActiveRecord::Base has_many :patients end
  21. 21. See, Ruby, Run! • Drmap • git clone git://github.com/mchung/drmap.git • Optionally, install Journeta
  22. 22. Under the hood • Abstract syntax trees • S-expressions • ParseTree
  23. 23. class Book def title “Ruby programming” end end
  24. 24. case NODE_BREAK: case NODE_NEXT: if (node->nd_stts) add_to_parse_tree(self, current, node->nd_stts, locals); break; case NODE_YIELD: if (node->nd_stts) add_to_parse_tree(self, current, node->nd_stts, locals); if (node->nd_stts && (nd_type(node->nd_stts) == NODE_ARRAY || nd_type(node->nd_stts) == NODE_ZARRAY) && !node->nd_state) rb_ary_push(current, Qtrue); break;
  25. 25. [:class, :Book, [:const, :Object], [:defn, :title, [:scope, [:block, [:args], [:str, quot;Ruby programmingquot;]]]]]
  26. 26. class Book < Object def title “Ruby programming” end end
  27. 27. /opt/local/usr/bin • parse_tree_show • r2r_show
  28. 28. class Book def title “Ruby programming” end end
  29. 29. $ parse_tree_show book.rb
  30. 30. s(:class, :Book, nil, s(:scope, s(:defn, :title, s(:args), s(:scope, s(:block, s(:str, quot;Ruby programmingquot;))))))
  31. 31. $ r2r_show book.rb
  32. 32. class Book def title() “Ruby programming” end end
  33. 33. Operator Precedence with Guy Decoux: a b c, d [[:fcall, :a, [:array, [:fcall, :b, [:array, [:vcall, :c], [:vcall, :d]]]]]] ParseTree.new.parse_tree_for_string(“a b c, d”)
  34. 34. ruby2java
  35. 35. class JavaClass def self.main puts “Java, rocks!” end end
  36. 36. [:class, :JavaClass, [:const, :Object], [:defs, [:self], :main, [:scope, [:block, [:args], [:fcall, :puts, [:array, [:str, quot;Hello, Javaquot;]]]]]]]
  37. 37. class Ruby2Java < SexpProcessor def process_class(exp) “public class #{exp.shift} #{next_token(exp, true)}” end def process_const(exp) “extends #{exp.shift}” end .... end
  38. 38. public class JavaClass { public static void main(String argv[]) { return “Hello, Java”; } }
  39. 39. Distributing Ruby • drmap - multi-machine worker queue • gisting - map/reduce framework
  40. 40. drmap irb> fn = proc {|x| x**10} irb> YAML::dump(fn) => quot;--- !ruby/object:Proc {}nnquot; irb> Marshal.dump(fn) TypeError: no marshal_dump is defined for class Proc irb> YAML::dump(fn.to_ruby) => quot;--- proc { |x| (x ** 10) }nquot; irb> Marshal.dump(fn.to_ruby) => quot;004bquot;eproc { |x| (x ** 10) }quot;
  41. 41. Enumerable#drmap
  42. 42. module Enumerable def drmap(&block) pool = Drmap::BeanstalkPool.new jid = rand(100) each_with_index do |element, idx| pool.put_job(jid, block.to_ruby, element) end results = [] while results.size < length results << pool.next_result(jid) end results end end
  43. 43. def process loop do begin job_payload = next_job fn = job_payload[:proc] data = job_payload[:data] result = eval(fn).call(data) save(job_payload[:jid], result) success! rescue => e fail!(e) end end end
  44. 44. drmap • Trivial • Distributed • Powerful
  45. 45. Demo time!
  46. 46. Gisting • MapReduce in Ruby • http://github.com/mchung/gisting
  47. 47. map / reduce • Functional programming • Iterating over collections • (5..10).map {|x| x+1} #=> [6, 7, 8, 9, 10, 11] • (5..10).reduce(:+) #=> 45
  48. 48. Google MapReduce • Modeled after map() and reduce() • Programming model • Programs are trivially parallelizable
  49. 49. Demo time!
  50. 50. Gisting::Spec
  51. 51. input.file_pattern = quot;file.txtquot; input.map do |map_input| words = map_input.strip.split(quot;tquot;) Emit(words[1], quot;1quot;) end
  52. 52. output.reduce do |reduce_input| count = 0 reduce_input.each do |value| count += value.to_i end Emit(count) end
  53. 53. inputs = args spec = Gisting::Spec.new inputs.each do |file_input| input = spec.add_input input.file_pattern = file_input input.map do |map_input| # 2722 mailbox 2006-05-23 00:08:39 words = map_input.strip.split(quot;tquot;) Emit(words[1], quot;1quot;) end end output = spec.output output.filebase = quot;/Users/mchung/Public/datasets/outputquot; output.num_tasks = 2 output.reduce do |reduce_input| count = 0 reduce_input.each do |value| count += value.to_i end Emit(count) end result = MapReduce(spec) pp result
  54. 54. Gisting • MapReduce programming model in Ruby • Uses ruby2ruby to serialize procs • Uses EventMachine for Map/ReduceServer
  55. 55. • -: 94978 • mailbox: 20872 • google: 6758 • ebay: 2832 • yahoo: 2674 • yahoo.com: 2198 • myspace.com: 1916 • google.com: 1882
  56. 56. self.give(:thanks) OpenRain openrain.com My Git Repo github.com/mchung Marc Chung blog.marcchung.com Ruby::AZ rubyaz.org
  57. 57. Resources Code Generation: The Safety Scissors Of Metaprogramming mwrc2008.confreaks.com/03bowkett.html Forbidden Fruit: A Taste of Ruby's Parse Tree goruco2008.confreaks.com/03_wanstrath.html Ruby2Ruby seattlerb.rubyforge.org/ruby2ruby/ ParseTree parsetree.rubyforge.org

×