Ruby AST Tools
       The ParseTree Family




       Brian Landau
Wednesday, March 25, 2009
The Ruby AST

            Code                  Code
            Style 1               Style 2


                         ...
The Ruby AST Nodes




Wednesday, March 25, 2009
The Ruby AST Nodes

                  def       (a,b)     def     (a,b)


                  c=        a.to_s    c=      a....
Parsing Ruby in Ruby



            ParseTree
            ruby_parser

Wednesday, March 25, 2009
ParseTree




Wednesday, March 25, 2009
ParseTree

            RubyInline based




Wednesday, March 25, 2009
ParseTree

            RubyInline based
            Input sources:
                 Class
                 Method
        ...
ruby_parser




Wednesday, March 25, 2009
ruby_parser

            Uses Racc




Wednesday, March 25, 2009
ruby_parser

            Uses Racc
            Retrives line numbers




Wednesday, March 25, 2009
ruby_parser

            Uses Racc
            Retrives line numbers
            Slower than ParseTree



Wednesday, March...
ruby_parser

            Uses Racc
            Retrives line numbers
            Slower than ParseTree
            Sexp ou...
Sexp Nodes




Wednesday, March 25, 2009
Sexp Nodes
       class Simple
        def say(*args)
          puts args * ' '
        end
       end




Wednesday, Marc...
Sexp Nodes
       class Simple
        def say(*args)
          puts args * ' '
        end
       end




       s(:class...
Sexp Objects


                 Inherits from Array
                 `structure` method
                 `each_of_type` me...
SexpProcessors
       Or How I Learned to Love ParseTree




Wednesday, March 25, 2009
Making a SexpProcessors




Wednesday, March 25, 2009
Making a SexpProcessors


            Subclass `SexpProcessor`




Wednesday, March 25, 2009
Making a SexpProcessors


            Subclass `SexpProcessor`
            Call `super` from initialize




Wednesday, Mar...
Making a SexpProcessors


            Subclass `SexpProcessor`
            Call `super` from initialize
            Define ...
Making a SexpProcessors


            Subclass `SexpProcessor`
            Call `super` from initialize
            Define ...
Making a SexpProcessors


            Subclass `SexpProcessor`
            Call `super` from initialize
            Define ...
Making a SexpProcessors


            Subclass `SexpProcessor`
            Call `super` from initialize
            Define ...
Making a SexpProcessors


                            Subclass SexpProcessor

       class MyProcessor < SexpProcessor

  ...
Making a SexpProcessors


                            Call “super” in “initialize”

       class MyProcessor < SexpProcess...
Making a SexpProcessors


       class MyProcessor < SexpProcessor
        # ...
                                     Proc...
Making a SexpProcessors


       class MyProcessor < SexpProcessor
        # ...

         def process_call(exp)
         ...
Making a SexpProcessors


       class MyProcessor  SexpProcessor
        # ...

         def process_call(exp)
          ...
Making a SexpProcessors


       class MyProcessor  SexpProcessor
        # ...

        def default_process(exp)
        ...
UnifiedRuby


     pt = ParseTree.new
     parse_tree = pt.parse_tree(Simple)
     ast = pt.process(parse_tree)

     # OR
...
UnifiedRuby


      class MyProcessor  SexpProcessor
       include UnifiedRuby

         # ...

      end




Wednesday, Ma...
Ruby2Ruby



        Ruby2Ruby.translate(Simple, :say)
        # = quot;def say(*args)...quot;




Wednesday, March 25, 20...
ParseTree extensions

  require 'parse_tree'
  require 'ruby2ruby'
  require 'parse_tree_extensions'

  bomb = proc do
   ...
ParseTree Family

                            Flog           ambition

                                   Parse
          ...
ParseTree Family




            Code Analytics
            Custom DSL


Wednesday, March 25, 2009
Flog




            Takes a set of files as input
            Processes for “bad code”



Wednesday, March 25, 2009
Flog




            Decends from SexpProcessor
            Includes UnifiedRuby



Wednesday, March 25, 2009
Flog


            Assigns weighted scores to:
                 Node Types
                 Node Constructions
           ...
Flog



            Reports scores by:
                 Class
                 Method


Wednesday, March 25, 2009
Flog
     flog lib
         377.6: flog total
         11.1: flog/method average

         150.2: ClassMethods#acts_as_markup...
Flog
     flog -g lib
         377.6: flog total
         11.1: flog/method average

         171.7: ClassMethods total
     ...
Flog
     flog -d lib
         377.6: flog total
         11.1: flog/method average

         150.2: ClassMethods#acts_as_mar...
self-flagellation




            Choose your own weightings




Wednesday, March 25, 2009
roodi
       Ruby Object Oriented Design Inferometer




Wednesday, March 25, 2009
roodi
       Ruby Object Oriented Design Inferometer


            Cyclomatic complexity




Wednesday, March 25, 2009
roodi
       Ruby Object Oriented Design Inferometer


            Cyclomatic complexity
            Assignment in conditi...
roodi
       Ruby Object Oriented Design Inferometer


            Cyclomatic complexity
            Assignment in conditi...
roodi
       Ruby Object Oriented Design Inferometer


            Cyclomatic complexity
            Assignment in conditi...
roodi
       Ruby Object Oriented Design Inferometer


            Cyclomatic complexity
            Assignment in conditi...
roodi
       Ruby Object Oriented Design Inferometer


            Cyclomatic complexity
            Assignment in conditi...
roodi


     sudo gem install roodi
     roodi “lib/**/*.rb”




Wednesday, March 25, 2009
roodi:
       How it works




Wednesday, March 25, 2009
roodi:
       How it works

            Parse files




Wednesday, March 25, 2009
roodi:
       How it works

            Parse files
            Move node-by-node




Wednesday, March 25, 2009
roodi:
       How it works

            Parse files
            Move node-by-node
            Evaluate each “Check”



Wedn...
roodi:
       How it works

            Parse files
            Move node-by-node
            Evaluate each “Check”
       ...
Ambition




Wednesday, March 25, 2009
Ambition



            DSL for Querying as Ruby




Wednesday, March 25, 2009
Ambition



            DSL for Querying as Ruby
            Framework for API Adapters




Wednesday, March 25, 2009
Ambition



            DSL for Querying as Ruby
            Framework for API Adapters
            Generator for new Adap...
Ambition ActiveRecord Adapter




            Queries only run when needed
            to_s = SQL
            to_hash = fin...
Ambition:
       How it works


            Processors parameters:
                 Context/Owner
                 Block

...
Ambition:
       How it works

           Context/Owner


       User.select { |m| m.name == 'jon' }




             Proc...
Ambition:
       How it works


            Parse the block
            Process nodes recursively
            Translate “c...
Ambition:
       How it works


            Adapters:
                 Translator for Processors
                 Make API...
The Future:
       Ruby 1.9


            YARV
                 A virtual machine based stack architecture
               ...
Ripper


            Effects YARV behavior
            Returns array of arrays  literals
            Line and column for e...
Ripper:
       Lexer
      require 'ripper'
      require 'pp'

      p Ripper.lex(quot;def m(a) nil endquot;)
       #= [...
Ripper:
       Sexp Tree

 require 'ripper'
 require 'pp'

 pp Ripper.sexp(quot;def m(a) nil endquot;)
  #= [:program,
  #...
Ripper:
       Filter
       require 'ripper/filter'

       class CommentStripper  Ripper::Filter
        def self.strip(s...
QA
       Rate at:
       http://speakerrate.com/talks/594-ruby-ast-tools


       brian.landau@viget.com
       http://ww...
Upcoming SlideShare
Loading in …5
×

Ruby AST Tools

4,328 views
4,184 views

Published on

Published in: Technology, Education
0 Comments
9 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
4,328
On SlideShare
0
From Embeds
0
Number of Embeds
223
Actions
Shares
0
Downloads
75
Comments
0
Likes
9
Embeds 0
No embeds

No notes for slide

  • Ruby 1.8 uses a Bison Yacc-like parser
    Many different styles lead to the same Abstract Syntax tree or set of instructions.
  • Best to thing of the AST not as a tree but a large container composed of inner nodes

  • themselves composed of smaller nodes. until you get to an individual &#x201C;token&#x201D;

  • Both by Ryan Davis


  • Pure ruby implementation using racc. For those that don&#x2019;t know RACC is a ruby based YACC parser.
    Slowness not a huge issue in my experience.
  • Pure ruby implementation using racc. For those that don&#x2019;t know RACC is a ruby based YACC parser.
    Slowness not a huge issue in my experience.
  • Pure ruby implementation using racc. For those that don&#x2019;t know RACC is a ruby based YACC parser.
    Slowness not a huge issue in my experience.
  • Pure ruby implementation using racc. For those that don&#x2019;t know RACC is a ruby based YACC parser.
    Slowness not a huge issue in my experience.
  • Obtains the Parse tree and represents it as nested s-expresions.
  • Obtains the Parse tree and represents it as nested s-expresions.
  • composed of array&#x2019;s, symbol&#x2019;s and literals
    `structure` just returns the node type structure not the contents of the nodes. this is very handy for code structure analysis.
  • Takes the sexp objects returned by ParseTree and processes returning output.












  • Essentially turns the parse tree into an AST. It does this via a set of rewrite methods.
    By default ParseTree does this. To get the un-unified version you have to use RawParseTree.
    ParseTree does this by processing with a CompositeSexpProcessor called Unifier
  • A Module that can be included in a SexpProcessor.
  • Another good example of a SexpProcessor.
    Take a class or method and translates it into a string of valid Ruby code that would create that Class/method exactly.
    Useful for metaprogramming and dissecting dynamically created classes, methods, and procs
  • Convert Methods, UnboundMethods and Procs to sexps and strings of ruby code. Helpful in debuging mysterious Procs.
  • Often times I find libraries that use ParseTree but don't take advantage of SexpProcessor and often rebuild similar functionality. very sad.
  • 2 Types of Libraries based on ParseTree
  • Good at finding problem areas that may need refactoring

  • Also adds to varying amounts to a score multiplier for specific types of nestings.


  • with no options
    Ryan Davis has said to remember it's more about relative scores. Particularly bad scores I generally consider is those over 40 or if I'm being very rough those over 25.
  • with group option, groups by class
  • with detail version (old output) gives why each method earned it's score.
  • Converted over by Ben Scofield
    Working on a site that will allow you to share and download those weightings.
  • by Marty Andrews
  • by Marty Andrews
  • by Marty Andrews
  • by Marty Andrews
  • by Marty Andrews
  • by Marty Andrews
  • Once again, Very good at finding problem areas that need refactoring.




  • by github&#x2019;s Chris Wanstrath
    Allows you to express queries in clear and concise ruby code.
    Think of it as a Framework for specific API adapters, e.g. ActiveRecord
    Includes a Rubigen based adapter generator for any data-source API you want

  • by github&#x2019;s Chris Wanstrath
    Allows you to express queries in clear and concise ruby code.
    Think of it as a Framework for specific API adapters, e.g. ActiveRecord
    Includes a Rubigen based adapter generator for any data-source API you want

  • by github&#x2019;s Chris Wanstrath
    Allows you to express queries in clear and concise ruby code.
    Think of it as a Framework for specific API adapters, e.g. ActiveRecord
    Includes a Rubigen based adapter generator for any data-source API you want

  • Just gives us another syntax for doing finds, the syntax may be preferable to some. Still needs lots of work to do everything you can do with find or SQL.





  • No composite SexpProcessor equivalent yet.
  • line and column, node/event type, &#x201C;code&#x201D; that produced it.
  • Very similar output to ParseTree, but does include line and column info for inner most nodes.
  • Uses Lexer and goes thru each node.

  • Ruby AST Tools

    1. 1. Ruby AST Tools The ParseTree Family Brian Landau Wednesday, March 25, 2009
    2. 2. The Ruby AST Code Code Style 1 Style 2 AST Code Code Style 3 Style 4 Wednesday, March 25, 2009
    3. 3. The Ruby AST Nodes Wednesday, March 25, 2009
    4. 4. The Ruby AST Nodes def (a,b) def (a,b) c= a.to_s c= a.to_s return end return end def (a,b) def (a,b) c= a.to_s c= a.to_s Wednesday, March 25, 2009
    5. 5. Parsing Ruby in Ruby ParseTree ruby_parser Wednesday, March 25, 2009
    6. 6. ParseTree Wednesday, March 25, 2009
    7. 7. ParseTree RubyInline based Wednesday, March 25, 2009
    8. 8. ParseTree RubyInline based Input sources: Class Method Proc String Wednesday, March 25, 2009
    9. 9. ruby_parser Wednesday, March 25, 2009
    10. 10. ruby_parser Uses Racc Wednesday, March 25, 2009
    11. 11. ruby_parser Uses Racc Retrives line numbers Wednesday, March 25, 2009
    12. 12. ruby_parser Uses Racc Retrives line numbers Slower than ParseTree Wednesday, March 25, 2009
    13. 13. ruby_parser Uses Racc Retrives line numbers Slower than ParseTree Sexp output is ParseTree compatible Wednesday, March 25, 2009
    14. 14. Sexp Nodes Wednesday, March 25, 2009
    15. 15. Sexp Nodes class Simple def say(*args) puts args * ' ' end end Wednesday, March 25, 2009
    16. 16. Sexp Nodes class Simple def say(*args) puts args * ' ' end end s(:class, :Simple, nil, s(:scope, s(:defn, :say, s(:args, :quot;*argsquot;), s(:scope, s(:block, s(:call, nil, :puts, s(:arglist, s(:call, s(:lvar, :args), :*, s(:arglist, s(:str, quot; quot;)))))))))) Wednesday, March 25, 2009
    17. 17. Sexp Objects Inherits from Array `structure` method `each_of_type` method `sexp_type` method `sexp_body` method Wednesday, March 25, 2009
    18. 18. SexpProcessors Or How I Learned to Love ParseTree Wednesday, March 25, 2009
    19. 19. Making a SexpProcessors Wednesday, March 25, 2009
    20. 20. Making a SexpProcessors Subclass `SexpProcessor` Wednesday, March 25, 2009
    21. 21. Making a SexpProcessors Subclass `SexpProcessor` Call `super` from initialize Wednesday, March 25, 2009
    22. 22. Making a SexpProcessors Subclass `SexpProcessor` Call `super` from initialize Define `process_xxx` methods Wednesday, March 25, 2009
    23. 23. Making a SexpProcessors Subclass `SexpProcessor` Call `super` from initialize Define `process_xxx` methods Shift off Sexp nodes Wednesday, March 25, 2009
    24. 24. Making a SexpProcessors Subclass `SexpProcessor` Call `super` from initialize Define `process_xxx` methods Shift off Sexp nodes Default process method Wednesday, March 25, 2009
    25. 25. Making a SexpProcessors Subclass `SexpProcessor` Call `super` from initialize Define `process_xxx` methods Shift off Sexp nodes Default process method `rewrite_xxx` methods Wednesday, March 25, 2009
    26. 26. Making a SexpProcessors Subclass SexpProcessor class MyProcessor < SexpProcessor end Wednesday, March 25, 2009
    27. 27. Making a SexpProcessors Call “super” in “initialize” class MyProcessor < SexpProcessor def initialize super self.warn_on_default = false self.strict = false end end Wednesday, March 25, 2009
    28. 28. Making a SexpProcessors class MyProcessor < SexpProcessor # ... Process “call” node def process_call(exp) recv = process exp.shift name = exp.shift args = process exp.shift return s() end end Wednesday, March 25, 2009
    29. 29. Making a SexpProcessors class MyProcessor < SexpProcessor # ... def process_call(exp) recv = process exp.shift name = exp.shift args = process exp.shift return s() end Must shift o all Sexp Nodes end Wednesday, March 25, 2009
    30. 30. Making a SexpProcessors class MyProcessor SexpProcessor # ... def process_call(exp) recv = process exp.shift name = exp.shift args = process exp.shift return s() end Must return a Sexp end Wednesday, March 25, 2009
    31. 31. Making a SexpProcessors class MyProcessor SexpProcessor # ... def default_process(exp) until exp.size == 0 exp.shift end return s() A default process method end end Wednesday, March 25, 2009
    32. 32. UnifiedRuby pt = ParseTree.new parse_tree = pt.parse_tree(Simple) ast = pt.process(parse_tree) # OR ast = pt.process(File.read('simple.rb')) Wednesday, March 25, 2009
    33. 33. UnifiedRuby class MyProcessor SexpProcessor include UnifiedRuby # ... end Wednesday, March 25, 2009
    34. 34. Ruby2Ruby Ruby2Ruby.translate(Simple, :say) # = quot;def say(*args)...quot; Wednesday, March 25, 2009
    35. 35. ParseTree extensions require 'parse_tree' require 'ruby2ruby' require 'parse_tree_extensions' bomb = proc do quot;BOOM!quot; end puts bomb.to_ruby # = quot;proc { quot;BOOM!quot; }quot; Wednesday, March 25, 2009
    36. 36. ParseTree Family Flog ambition Parse Tree roodi Wednesday, March 25, 2009
    37. 37. ParseTree Family Code Analytics Custom DSL Wednesday, March 25, 2009
    38. 38. Flog Takes a set of files as input Processes for “bad code” Wednesday, March 25, 2009
    39. 39. Flog Decends from SexpProcessor Includes UnifiedRuby Wednesday, March 25, 2009
    40. 40. Flog Assigns weighted scores to: Node Types Node Constructions Method calls Wednesday, March 25, 2009
    41. 41. Flog Reports scores by: Class Method Wednesday, March 25, 2009
    42. 42. Flog flog lib 377.6: flog total 11.1: flog/method average 150.2: ClassMethods#acts_as_markup 24.7: HTML#array_to_html 21.5: ClassMethods#get_markdown_class 18.4: main#none 17.9: ActsAsMarkup#none Wednesday, March 25, 2009
    43. 43. Flog flog -g lib 377.6: flog total 11.1: flog/method average 171.7: ClassMethods total 150.2: ClassMethods#acts_as_markup 21.5: ClassMethods#get_markdown_class 24.7: HTML total 24.7: HTML#array_to_html ... Wednesday, March 25, 2009
    44. 44. Flog flog -d lib 377.6: flog total 11.1: flog/method average 150.2: ClassMethods#acts_as_markup 50.9: [] 42.6: branch 24.0: send 16.5: define_method 11.9: to_s 11.8: assignment ... Wednesday, March 25, 2009
    45. 45. self-flagellation Choose your own weightings Wednesday, March 25, 2009
    46. 46. roodi Ruby Object Oriented Design Inferometer Wednesday, March 25, 2009
    47. 47. roodi Ruby Object Oriented Design Inferometer Cyclomatic complexity Wednesday, March 25, 2009
    48. 48. roodi Ruby Object Oriented Design Inferometer Cyclomatic complexity Assignment in conditional Wednesday, March 25, 2009
    49. 49. roodi Ruby Object Oriented Design Inferometer Cyclomatic complexity Assignment in conditional Class/Method line count Wednesday, March 25, 2009
    50. 50. roodi Ruby Object Oriented Design Inferometer Cyclomatic complexity Assignment in conditional Class/Method line count Empty rescue body Wednesday, March 25, 2009
    51. 51. roodi Ruby Object Oriented Design Inferometer Cyclomatic complexity Assignment in conditional Class/Method line count Empty rescue body No for loops Wednesday, March 25, 2009
    52. 52. roodi Ruby Object Oriented Design Inferometer Cyclomatic complexity Assignment in conditional Class/Method line count Empty rescue body No for loops Number of parameters Wednesday, March 25, 2009
    53. 53. roodi sudo gem install roodi roodi “lib/**/*.rb” Wednesday, March 25, 2009
    54. 54. roodi: How it works Wednesday, March 25, 2009
    55. 55. roodi: How it works Parse files Wednesday, March 25, 2009
    56. 56. roodi: How it works Parse files Move node-by-node Wednesday, March 25, 2009
    57. 57. roodi: How it works Parse files Move node-by-node Evaluate each “Check” Wednesday, March 25, 2009
    58. 58. roodi: How it works Parse files Move node-by-node Evaluate each “Check” Do Recursively Wednesday, March 25, 2009
    59. 59. Ambition Wednesday, March 25, 2009
    60. 60. Ambition DSL for Querying as Ruby Wednesday, March 25, 2009
    61. 61. Ambition DSL for Querying as Ruby Framework for API Adapters Wednesday, March 25, 2009
    62. 62. Ambition DSL for Querying as Ruby Framework for API Adapters Generator for new Adapters Wednesday, March 25, 2009
    63. 63. Ambition ActiveRecord Adapter Queries only run when needed to_s = SQL to_hash = find-compatible hash Wednesday, March 25, 2009
    64. 64. Ambition: How it works Processors parameters: Context/Owner Block Wednesday, March 25, 2009
    65. 65. Ambition: How it works Context/Owner User.select { |m| m.name == 'jon' } Processor Block Wednesday, March 25, 2009
    66. 66. Ambition: How it works Parse the block Process nodes recursively Translate “clauses” to query string Wednesday, March 25, 2009
    67. 67. Ambition: How it works Adapters: Translator for Processors Make API calls with a Query Class Wednesday, March 25, 2009
    68. 68. The Future: Ruby 1.9 YARV A virtual machine based stack architecture implementation Ripper Offers similar functionality as ParseTree and SexpProcessor Wednesday, March 25, 2009
    69. 69. Ripper Effects YARV behavior Returns array of arrays literals Line and column for each token Ripper::Filter Wednesday, March 25, 2009
    70. 70. Ripper: Lexer require 'ripper' require 'pp' p Ripper.lex(quot;def m(a) nil endquot;) #= [[[1, 0], :on_kw, quot;defquot;], # [[1, 3], :on_sp, quot; quot; ], # [[1, 4], :on_ident, quot;mquot; ], # [[1, 5], :on_lparen, quot;(quot; ], # [[1, 6], :on_ident, quot;aquot; ], # [[1, 7], :on_rparen, quot;)quot; ], # [[1, 8], :on_sp, quot; quot; ], # [[1, 9], :on_kw, quot;nilquot;], # [[1, 12], :on_sp, quot; quot; ], # [[1, 13], :on_kw, quot;endquot;]] Wednesday, March 25, 2009
    71. 71. Ripper: Sexp Tree require 'ripper' require 'pp' pp Ripper.sexp(quot;def m(a) nil endquot;) #= [:program, # [:stmts_add, # [:stmts_new], # [:def, # [:@ident, quot;mquot;, [1, 4]], # [:paren, [:params, [[:@ident, quot;aquot;, [1, 6]]], nil, nil, nil]], # [:bodystmt, # [:stmts_add, [:stmts_new], [:var_ref, [:@kw, quot;nilquot;, [1, 9]]]], # nil, # nil, # nil]]]] Wednesday, March 25, 2009
    72. 72. Ripper: Filter require 'ripper/filter' class CommentStripper Ripper::Filter def self.strip(src) new(src).parse end def on_default(event, token, data) print token end def on_comment(token, data) puts end end CommentStripper.strip(ARGF) Wednesday, March 25, 2009
    73. 73. QA Rate at: http://speakerrate.com/talks/594-ruby-ast-tools brian.landau@viget.com http://www.viget.com/extend http://www.websideattractions.com/ Wednesday, March 25, 2009

    ×