• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Hacking with ruby2ruby
 

Hacking with ruby2ruby

on

  • 3,461 views

Gisting is an implementation of Google\'s MapReduce framework for processing and extracting useful information from very large data sets. At the time of this writing, the code is available for ...

Gisting is an implementation of Google\'s MapReduce framework for processing and extracting useful information from very large data sets. At the time of this writing, the code is available for PREVIEW at http://github.com/mchung/gisting. I am currently working to release this framework for general usage.

Statistics

Views

Total Views
3,461
Views on SlideShare
3,443
Embed Views
18

Actions

Likes
0
Downloads
0
Comments
0

4 Embeds 18

http://www.slideshare.net 7
http://www.linkedin.com 5
https://www.linkedin.com 5
http://www.docshut.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Hacking with ruby2ruby Hacking with ruby2ruby Presentation Transcript

    • Hacking with ruby2ruby Marc Chung OpenRain blog.marcchung.com blog.openrain.com @heisenthought
    • Hacking with ruby2ruby Marc Chung OpenRain blog.marcchung.com blog.openrain.com @heisenthought
    • ruby2ruby Ruby2Ruby Walks the Ruby AST To emit Ruby
    • sudo gem install ruby2ruby
    • Limitations • Only Ruby 1.8 • Not 1.9, yet • Built into Rubinius
    • function rhino_rocks() { return “Rhino, rocks!”; }
    • js> rhino_rocks(); Rhino, rocks!
    • js> rhino_rocks.toString(); function rhino_rocks() { return “Rhino, rocks!”; }
    • js> eval(rhino_rocks.toString()).call(); Rhino, rocks!
    • ruby_rocks = proc { return “Ruby, rocks”; }
    • irb> ruby_rocks.call; Ruby, rocks!
    • irb> require “ruby2ruby” irb> require “parse_tree_extensions” irb> ruby_rocks.to_ruby proc { return “Ruby, rocks” }
    • irb> eval(ruby_rocks.to_ruby).call Ruby, rocks!
    • define_method class Voice define_method(:shout) do |word| word.upcase! end end puts Ruby2Ruby.translate(Voice)
    • class Voice < Object def yell(word) word.upcase! end end
    • alias class AliasClass def old_busted quot;42quot; end alias_method :new_hotness, :old_busted end puts Ruby2Ruby.translate(AliasClass)
    • class AliasClass < Object def new_hotness quot;42quot; end def old_busted quot;42quot; end end
    • #module_function module HelperModule def help puts quot;Help mequot; end module_function : help end puts Ruby2Ruby.translate(HelperModule)
    • module HelperModule def help puts(quot;Help mequot;) end def self. help puts(quot;Help mequot;) end end
    • ActiveRecord Models class Doctor < ActiveRecord::Base has_many :patients end
    • See, Ruby, Run! • Drmap • git clone git://github.com/mchung/drmap.git • Optionally, install Journeta
    • Under the hood • Abstract syntax trees • S-expressions • ParseTree
    • class Book def title “Ruby programming” end end
    • case NODE_BREAK: case NODE_NEXT: if (node->nd_stts) add_to_parse_tree(self, current, node->nd_stts, locals); break; case NODE_YIELD: if (node->nd_stts) add_to_parse_tree(self, current, node->nd_stts, locals); if (node->nd_stts && (nd_type(node->nd_stts) == NODE_ARRAY || nd_type(node->nd_stts) == NODE_ZARRAY) && !node->nd_state) rb_ary_push(current, Qtrue); break;
    • [:class, :Book, [:const, :Object], [:defn, :title, [:scope, [:block, [:args], [:str, quot;Ruby programmingquot;]]]]]
    • class Book < Object def title “Ruby programming” end end
    • /opt/local/usr/bin • parse_tree_show • r2r_show
    • class Book def title “Ruby programming” end end
    • $ parse_tree_show book.rb
    • s(:class, :Book, nil, s(:scope, s(:defn, :title, s(:args), s(:scope, s(:block, s(:str, quot;Ruby programmingquot;))))))
    • $ r2r_show book.rb
    • class Book def title() “Ruby programming” end end
    • Operator Precedence with Guy Decoux: a b c, d [[:fcall, :a, [:array, [:fcall, :b, [:array, [:vcall, :c], [:vcall, :d]]]]]] ParseTree.new.parse_tree_for_string(“a b c, d”)
    • ruby2java
    • class JavaClass def self.main puts “Java, rocks!” end end
    • [:class, :JavaClass, [:const, :Object], [:defs, [:self], :main, [:scope, [:block, [:args], [:fcall, :puts, [:array, [:str, quot;Hello, Javaquot;]]]]]]]
    • class Ruby2Java < SexpProcessor def process_class(exp) “public class #{exp.shift} #{next_token(exp, true)}” end def process_const(exp) “extends #{exp.shift}” end .... end
    • public class JavaClass { public static void main(String argv[]) { return “Hello, Java”; } }
    • Distributing Ruby • drmap - multi-machine worker queue • gisting - map/reduce framework
    • drmap irb> fn = proc {|x| x**10} irb> YAML::dump(fn) => quot;--- !ruby/object:Proc {}nnquot; irb> Marshal.dump(fn) TypeError: no marshal_dump is defined for class Proc irb> YAML::dump(fn.to_ruby) => quot;--- proc { |x| (x ** 10) }nquot; irb> Marshal.dump(fn.to_ruby) => quot;004bquot;eproc { |x| (x ** 10) }quot;
    • Enumerable#drmap
    • module Enumerable def drmap(&block) pool = Drmap::BeanstalkPool.new jid = rand(100) each_with_index do |element, idx| pool.put_job(jid, block.to_ruby, element) end results = [] while results.size < length results << pool.next_result(jid) end results end end
    • def process loop do begin job_payload = next_job fn = job_payload[:proc] data = job_payload[:data] result = eval(fn).call(data) save(job_payload[:jid], result) success! rescue => e fail!(e) end end end
    • drmap • Trivial • Distributed • Powerful
    • Demo time!
    • Gisting • MapReduce in Ruby • http://github.com/mchung/gisting
    • map / reduce • Functional programming • Iterating over collections • (5..10).map {|x| x+1} #=> [6, 7, 8, 9, 10, 11] • (5..10).reduce(:+) #=> 45
    • Google MapReduce • Modeled after map() and reduce() • Programming model • Programs are trivially parallelizable
    • Demo time!
    • Gisting::Spec
    • input.file_pattern = quot;file.txtquot; input.map do |map_input| words = map_input.strip.split(quot;tquot;) Emit(words[1], quot;1quot;) end
    • output.reduce do |reduce_input| count = 0 reduce_input.each do |value| count += value.to_i end Emit(count) end
    • inputs = args spec = Gisting::Spec.new inputs.each do |file_input| input = spec.add_input input.file_pattern = file_input input.map do |map_input| # 2722 mailbox 2006-05-23 00:08:39 words = map_input.strip.split(quot;tquot;) Emit(words[1], quot;1quot;) end end output = spec.output output.filebase = quot;/Users/mchung/Public/datasets/outputquot; output.num_tasks = 2 output.reduce do |reduce_input| count = 0 reduce_input.each do |value| count += value.to_i end Emit(count) end result = MapReduce(spec) pp result
    • Gisting • MapReduce programming model in Ruby • Uses ruby2ruby to serialize procs • Uses EventMachine for Map/ReduceServer
    • • -: 94978 • mailbox: 20872 • google: 6758 • ebay: 2832 • yahoo: 2674 • yahoo.com: 2198 • myspace.com: 1916 • google.com: 1882
    • self.give(:thanks) OpenRain openrain.com My Git Repo github.com/mchung Marc Chung blog.marcchung.com Ruby::AZ rubyaz.org
    • Resources Code Generation: The Safety Scissors Of Metaprogramming mwrc2008.confreaks.com/03bowkett.html Forbidden Fruit: A Taste of Ruby's Parse Tree goruco2008.confreaks.com/03_wanstrath.html Ruby2Ruby seattlerb.rubyforge.org/ruby2ruby/ ParseTree parsetree.rubyforge.org