Ruby AST Tools
Upcoming SlideShare
Loading in...5
×
 

Ruby AST Tools

on

  • 5,692 views

 

Statistics

Views

Total Views
5,692
Views on SlideShare
5,470
Embed Views
222

Actions

Likes
9
Downloads
68
Comments
0

6 Embeds 222

http://websideattractions.com 121
http://www.websideattractions.com 84
http://www.slideshare.net 14
http://www.09h15.com 1
http://web.archive.org 1
http://websideattractions.posthaven.com 1

Accessibility

Categories

Upload Details

Uploaded via as Apple Keynote

Usage Rights

CC Attribution-NonCommercial-NoDerivs LicenseCC Attribution-NonCommercial-NoDerivs LicenseCC Attribution-NonCommercial-NoDerivs License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • <br />
  • Ruby 1.8 uses a Bison Yacc-like parser <br /> Many different styles lead to the same Abstract Syntax tree or set of instructions. <br />
  • Best to thing of the AST not as a tree but a large container composed of inner nodes <br /> <br />
  • themselves composed of smaller nodes. until you get to an individual “token” <br /> <br />
  • Both by Ryan Davis <br />
  • <br />
  • <br />
  • Pure ruby implementation using racc. For those that don’t know RACC is a ruby based YACC parser. <br /> Slowness not a huge issue in my experience. <br />
  • Pure ruby implementation using racc. For those that don’t know RACC is a ruby based YACC parser. <br /> Slowness not a huge issue in my experience. <br />
  • Pure ruby implementation using racc. For those that don’t know RACC is a ruby based YACC parser. <br /> Slowness not a huge issue in my experience. <br />
  • Pure ruby implementation using racc. For those that don’t know RACC is a ruby based YACC parser. <br /> Slowness not a huge issue in my experience. <br />
  • Obtains the Parse tree and represents it as nested s-expresions. <br />
  • Obtains the Parse tree and represents it as nested s-expresions. <br />
  • composed of array’s, symbol’s and literals <br /> `structure` just returns the node type structure not the contents of the nodes. this is very handy for code structure analysis. <br />
  • Takes the sexp objects returned by ParseTree and processes returning output. <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • Essentially turns the parse tree into an AST. It does this via a set of rewrite methods. <br /> By default ParseTree does this. To get the un-unified version you have to use RawParseTree. <br /> ParseTree does this by processing with a CompositeSexpProcessor called Unifier <br />
  • A Module that can be included in a SexpProcessor. <br />
  • Another good example of a SexpProcessor. <br /> Take a class or method and translates it into a string of valid Ruby code that would create that Class/method exactly. <br /> Useful for metaprogramming and dissecting dynamically created classes, methods, and procs <br />
  • Convert Methods, UnboundMethods and Procs to sexps and strings of ruby code. Helpful in debuging mysterious Procs. <br />
  • Often times I find libraries that use ParseTree but don't take advantage of SexpProcessor and often rebuild similar functionality. very sad. <br />
  • 2 Types of Libraries based on ParseTree <br />
  • Good at finding problem areas that may need refactoring <br />
  • <br />
  • Also adds to varying amounts to a score multiplier for specific types of nestings. <br /> <br />
  • <br />
  • with no options <br /> Ryan Davis has said to remember it's more about relative scores. Particularly bad scores I generally consider is those over 40 or if I'm being very rough those over 25. <br />
  • with group option, groups by class <br />
  • with detail version (old output) gives why each method earned it's score. <br />
  • Converted over by Ben Scofield <br /> Working on a site that will allow you to share and download those weightings. <br />
  • by Marty Andrews <br />
  • by Marty Andrews <br />
  • by Marty Andrews <br />
  • by Marty Andrews <br />
  • by Marty Andrews <br />
  • by Marty Andrews <br />
  • Once again, Very good at finding problem areas that need refactoring. <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • by github’s Chris Wanstrath <br /> Allows you to express queries in clear and concise ruby code. <br /> Think of it as a Framework for specific API adapters, e.g. ActiveRecord <br /> Includes a Rubigen based adapter generator for any data-source API you want <br /> <br />
  • by github’s Chris Wanstrath <br /> Allows you to express queries in clear and concise ruby code. <br /> Think of it as a Framework for specific API adapters, e.g. ActiveRecord <br /> Includes a Rubigen based adapter generator for any data-source API you want <br /> <br />
  • by github’s Chris Wanstrath <br /> Allows you to express queries in clear and concise ruby code. <br /> Think of it as a Framework for specific API adapters, e.g. ActiveRecord <br /> Includes a Rubigen based adapter generator for any data-source API you want <br /> <br />
  • Just gives us another syntax for doing finds, the syntax may be preferable to some. Still needs lots of work to do everything you can do with find or SQL. <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • No composite SexpProcessor equivalent yet. <br />
  • line and column, node/event type, “code” that produced it. <br />
  • Very similar output to ParseTree, but does include line and column info for inner most nodes. <br />
  • Uses Lexer and goes thru each node. <br />
  • <br />

Ruby AST Tools Ruby AST Tools Presentation Transcript

  • Ruby AST Tools The ParseTree Family Brian Landau Wednesday, March 25, 2009
  • The Ruby AST Code Code Style 1 Style 2 AST Code Code Style 3 Style 4 Wednesday, March 25, 2009
  • The Ruby AST Nodes Wednesday, March 25, 2009
  • The Ruby AST Nodes def (a,b) def (a,b) c= a.to_s c= a.to_s return end return end def (a,b) def (a,b) c= a.to_s c= a.to_s Wednesday, March 25, 2009
  • Parsing Ruby in Ruby ParseTree ruby_parser Wednesday, March 25, 2009
  • ParseTree Wednesday, March 25, 2009
  • ParseTree RubyInline based Wednesday, March 25, 2009
  • ParseTree RubyInline based Input sources: Class Method Proc String Wednesday, March 25, 2009
  • ruby_parser Wednesday, March 25, 2009
  • ruby_parser Uses Racc Wednesday, March 25, 2009
  • ruby_parser Uses Racc Retrives line numbers Wednesday, March 25, 2009
  • ruby_parser Uses Racc Retrives line numbers Slower than ParseTree Wednesday, March 25, 2009
  • ruby_parser Uses Racc Retrives line numbers Slower than ParseTree Sexp output is ParseTree compatible Wednesday, March 25, 2009
  • Sexp Nodes Wednesday, March 25, 2009
  • Sexp Nodes class Simple def say(*args) puts args * ' ' end end Wednesday, March 25, 2009
  • Sexp Nodes class Simple def say(*args) puts args * ' ' end end s(:class, :Simple, nil, s(:scope, s(:defn, :say, s(:args, :quot;*argsquot;), s(:scope, s(:block, s(:call, nil, :puts, s(:arglist, s(:call, s(:lvar, :args), :*, s(:arglist, s(:str, quot; quot;)))))))))) Wednesday, March 25, 2009
  • Sexp Objects Inherits from Array `structure` method `each_of_type` method `sexp_type` method `sexp_body` method Wednesday, March 25, 2009
  • SexpProcessors Or How I Learned to Love ParseTree Wednesday, March 25, 2009
  • Making a SexpProcessors Wednesday, March 25, 2009
  • Making a SexpProcessors Subclass `SexpProcessor` Wednesday, March 25, 2009
  • Making a SexpProcessors Subclass `SexpProcessor` Call `super` from initialize Wednesday, March 25, 2009
  • Making a SexpProcessors Subclass `SexpProcessor` Call `super` from initialize Define `process_xxx` methods Wednesday, March 25, 2009
  • Making a SexpProcessors Subclass `SexpProcessor` Call `super` from initialize Define `process_xxx` methods Shift off Sexp nodes Wednesday, March 25, 2009
  • Making a SexpProcessors Subclass `SexpProcessor` Call `super` from initialize Define `process_xxx` methods Shift off Sexp nodes Default process method Wednesday, March 25, 2009
  • Making a SexpProcessors Subclass `SexpProcessor` Call `super` from initialize Define `process_xxx` methods Shift off Sexp nodes Default process method `rewrite_xxx` methods Wednesday, March 25, 2009
  • Making a SexpProcessors Subclass SexpProcessor class MyProcessor < SexpProcessor end Wednesday, March 25, 2009
  • Making a SexpProcessors Call “super” in “initialize” class MyProcessor < SexpProcessor def initialize super self.warn_on_default = false self.strict = false end end Wednesday, March 25, 2009
  • Making a SexpProcessors class MyProcessor < SexpProcessor # ... Process “call” node def process_call(exp) recv = process exp.shift name = exp.shift args = process exp.shift return s() end end Wednesday, March 25, 2009
  • Making a SexpProcessors class MyProcessor < SexpProcessor # ... def process_call(exp) recv = process exp.shift name = exp.shift args = process exp.shift return s() end Must shift o all Sexp Nodes end Wednesday, March 25, 2009
  • Making a SexpProcessors class MyProcessor < SexpProcessor # ... def process_call(exp) recv = process exp.shift name = exp.shift args = process exp.shift return s() end Must return a Sexp end Wednesday, March 25, 2009
  • Making a SexpProcessors class MyProcessor < SexpProcessor # ... def default_process(exp) until exp.size == 0 exp.shift end return s() A default process method end end Wednesday, March 25, 2009
  • UnifiedRuby pt = ParseTree.new parse_tree = pt.parse_tree(Simple) ast = pt.process(parse_tree) # OR ast = pt.process(File.read('simple.rb')) Wednesday, March 25, 2009
  • UnifiedRuby class MyProcessor < SexpProcessor include UnifiedRuby # ... end Wednesday, March 25, 2009
  • Ruby2Ruby Ruby2Ruby.translate(Simple, :say) # => quot;def say(*args)...quot; Wednesday, March 25, 2009
  • ParseTree extensions require 'parse_tree' require 'ruby2ruby' require 'parse_tree_extensions' bomb = proc do quot;BOOM!quot; end puts bomb.to_ruby # => quot;proc { quot;BOOM!quot; }quot; Wednesday, March 25, 2009
  • ParseTree Family Flog ambition Parse Tree roodi Wednesday, March 25, 2009
  • ParseTree Family Code Analytics Custom DSL Wednesday, March 25, 2009
  • Flog Takes a set of files as input Processes for “bad code” Wednesday, March 25, 2009
  • Flog Decends from SexpProcessor Includes UnifiedRuby Wednesday, March 25, 2009
  • Flog Assigns weighted scores to: Node Types Node Constructions Method calls Wednesday, March 25, 2009
  • Flog Reports scores by: Class Method Wednesday, March 25, 2009
  • Flog > flog lib 377.6: flog total 11.1: flog/method average 150.2: ClassMethods#acts_as_markup 24.7: HTML#array_to_html 21.5: ClassMethods#get_markdown_class 18.4: main#none 17.9: ActsAsMarkup#none Wednesday, March 25, 2009
  • Flog > flog -g lib 377.6: flog total 11.1: flog/method average 171.7: ClassMethods total 150.2: ClassMethods#acts_as_markup 21.5: ClassMethods#get_markdown_class 24.7: HTML total 24.7: HTML#array_to_html ... Wednesday, March 25, 2009
  • Flog > flog -d lib 377.6: flog total 11.1: flog/method average 150.2: ClassMethods#acts_as_markup 50.9: [] 42.6: branch 24.0: send 16.5: define_method 11.9: to_s 11.8: assignment ... Wednesday, March 25, 2009
  • self-flagellation Choose your own weightings Wednesday, March 25, 2009
  • roodi Ruby Object Oriented Design Inferometer Wednesday, March 25, 2009
  • roodi Ruby Object Oriented Design Inferometer Cyclomatic complexity Wednesday, March 25, 2009
  • roodi Ruby Object Oriented Design Inferometer Cyclomatic complexity Assignment in conditional Wednesday, March 25, 2009
  • roodi Ruby Object Oriented Design Inferometer Cyclomatic complexity Assignment in conditional Class/Method line count Wednesday, March 25, 2009
  • roodi Ruby Object Oriented Design Inferometer Cyclomatic complexity Assignment in conditional Class/Method line count Empty rescue body Wednesday, March 25, 2009
  • roodi Ruby Object Oriented Design Inferometer Cyclomatic complexity Assignment in conditional Class/Method line count Empty rescue body No for loops Wednesday, March 25, 2009
  • roodi Ruby Object Oriented Design Inferometer Cyclomatic complexity Assignment in conditional Class/Method line count Empty rescue body No for loops Number of parameters Wednesday, March 25, 2009
  • roodi > sudo gem install roodi > roodi “lib/**/*.rb” Wednesday, March 25, 2009
  • roodi: How it works Wednesday, March 25, 2009
  • roodi: How it works Parse files Wednesday, March 25, 2009
  • roodi: How it works Parse files Move node-by-node Wednesday, March 25, 2009
  • roodi: How it works Parse files Move node-by-node Evaluate each “Check” Wednesday, March 25, 2009
  • roodi: How it works Parse files Move node-by-node Evaluate each “Check” Do Recursively Wednesday, March 25, 2009
  • Ambition Wednesday, March 25, 2009
  • Ambition DSL for Querying as Ruby Wednesday, March 25, 2009
  • Ambition DSL for Querying as Ruby Framework for API Adapters Wednesday, March 25, 2009
  • Ambition DSL for Querying as Ruby Framework for API Adapters Generator for new Adapters Wednesday, March 25, 2009
  • Ambition ActiveRecord Adapter Queries only run when needed to_s = SQL to_hash = find-compatible hash Wednesday, March 25, 2009
  • Ambition: How it works Processors parameters: Context/Owner Block Wednesday, March 25, 2009
  • Ambition: How it works Context/Owner User.select { |m| m.name == 'jon' } Processor Block Wednesday, March 25, 2009
  • Ambition: How it works Parse the block Process nodes recursively Translate “clauses” to query string Wednesday, March 25, 2009
  • Ambition: How it works Adapters: Translator for Processors Make API calls with a Query Class Wednesday, March 25, 2009
  • The Future: Ruby 1.9 YARV A virtual machine based stack architecture implementation Ripper Offers similar functionality as ParseTree and SexpProcessor Wednesday, March 25, 2009
  • Ripper Effects YARV behavior Returns array of arrays & literals Line and column for each token Ripper::Filter Wednesday, March 25, 2009
  • Ripper: Lexer require 'ripper' require 'pp' p Ripper.lex(quot;def m(a) nil endquot;) #=> [[[1, 0], :on_kw, quot;defquot;], # [[1, 3], :on_sp, quot; quot; ], # [[1, 4], :on_ident, quot;mquot; ], # [[1, 5], :on_lparen, quot;(quot; ], # [[1, 6], :on_ident, quot;aquot; ], # [[1, 7], :on_rparen, quot;)quot; ], # [[1, 8], :on_sp, quot; quot; ], # [[1, 9], :on_kw, quot;nilquot;], # [[1, 12], :on_sp, quot; quot; ], # [[1, 13], :on_kw, quot;endquot;]] Wednesday, March 25, 2009
  • Ripper: Sexp Tree require 'ripper' require 'pp' pp Ripper.sexp(quot;def m(a) nil endquot;) #=> [:program, # [:stmts_add, # [:stmts_new], # [:def, # [:@ident, quot;mquot;, [1, 4]], # [:paren, [:params, [[:@ident, quot;aquot;, [1, 6]]], nil, nil, nil]], # [:bodystmt, # [:stmts_add, [:stmts_new], [:var_ref, [:@kw, quot;nilquot;, [1, 9]]]], # nil, # nil, # nil]]]] Wednesday, March 25, 2009
  • Ripper: Filter require 'ripper/filter' class CommentStripper < Ripper::Filter def self.strip(src) new(src).parse end def on_default(event, token, data) print token end def on_comment(token, data) puts end end CommentStripper.strip(ARGF) Wednesday, March 25, 2009
  • Q&A Rate at: http://speakerrate.com/talks/594-ruby-ast-tools brian.landau@viget.com http://www.viget.com/extend http://www.websideattractions.com/ Wednesday, March 25, 2009