Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Ruby AST Tools
       The ParseTree Family




       Brian Landau
Wednesday, March 25, 2009
The Ruby AST

            Code                  Code
            Style 1               Style 2


                         ...
The Ruby AST Nodes




Wednesday, March 25, 2009
The Ruby AST Nodes

                  def       (a,b)     def     (a,b)


                  c=        a.to_s    c=      a....
Parsing Ruby in Ruby



            ParseTree
            ruby_parser

Wednesday, March 25, 2009
ParseTree




Wednesday, March 25, 2009
ParseTree

            RubyInline based




Wednesday, March 25, 2009
ParseTree

            RubyInline based
            Input sources:
                 Class
                 Method
        ...
ruby_parser




Wednesday, March 25, 2009
ruby_parser

            Uses Racc




Wednesday, March 25, 2009
ruby_parser

            Uses Racc
            Retrives line numbers




Wednesday, March 25, 2009
ruby_parser

            Uses Racc
            Retrives line numbers
            Slower than ParseTree



Wednesday, March...
ruby_parser

            Uses Racc
            Retrives line numbers
            Slower than ParseTree
            Sexp ou...
Sexp Nodes




Wednesday, March 25, 2009
Sexp Nodes
       class Simple
        def say(*args)
          puts args * ' '
        end
       end




Wednesday, Marc...
Sexp Nodes
       class Simple
        def say(*args)
          puts args * ' '
        end
       end




       s(:class...
Sexp Objects


                 Inherits from Array
                 `structure` method
                 `each_of_type` me...
SexpProcessors
       Or How I Learned to Love ParseTree




Wednesday, March 25, 2009
Making a SexpProcessors




Wednesday, March 25, 2009
Making a SexpProcessors


            Subclass `SexpProcessor`




Wednesday, March 25, 2009
Making a SexpProcessors


            Subclass `SexpProcessor`
            Call `super` from initialize




Wednesday, Mar...
Making a SexpProcessors


            Subclass `SexpProcessor`
            Call `super` from initialize
            Define ...
Making a SexpProcessors


            Subclass `SexpProcessor`
            Call `super` from initialize
            Define ...
Making a SexpProcessors


            Subclass `SexpProcessor`
            Call `super` from initialize
            Define ...
Making a SexpProcessors


            Subclass `SexpProcessor`
            Call `super` from initialize
            Define ...
Making a SexpProcessors


                            Subclass SexpProcessor

       class MyProcessor < SexpProcessor

  ...
Making a SexpProcessors


                            Call “super” in “initialize”

       class MyProcessor < SexpProcess...
Making a SexpProcessors


       class MyProcessor < SexpProcessor
        # ...
                                     Proc...
Making a SexpProcessors


       class MyProcessor < SexpProcessor
        # ...

         def process_call(exp)
         ...
Making a SexpProcessors


       class MyProcessor  SexpProcessor
        # ...

         def process_call(exp)
          ...
Making a SexpProcessors


       class MyProcessor  SexpProcessor
        # ...

        def default_process(exp)
        ...
UnifiedRuby


     pt = ParseTree.new
     parse_tree = pt.parse_tree(Simple)
     ast = pt.process(parse_tree)

     # OR
...
UnifiedRuby


      class MyProcessor  SexpProcessor
       include UnifiedRuby

         # ...

      end




Wednesday, Ma...
Ruby2Ruby



        Ruby2Ruby.translate(Simple, :say)
        # = quot;def say(*args)...quot;




Wednesday, March 25, 20...
ParseTree extensions

  require 'parse_tree'
  require 'ruby2ruby'
  require 'parse_tree_extensions'

  bomb = proc do
   ...
ParseTree Family

                            Flog           ambition

                                   Parse
          ...
ParseTree Family




            Code Analytics
            Custom DSL


Wednesday, March 25, 2009
Flog




            Takes a set of files as input
            Processes for “bad code”



Wednesday, March 25, 2009
Flog




            Decends from SexpProcessor
            Includes UnifiedRuby



Wednesday, March 25, 2009
Flog


            Assigns weighted scores to:
                 Node Types
                 Node Constructions
           ...
Flog



            Reports scores by:
                 Class
                 Method


Wednesday, March 25, 2009
Flog
     flog lib
         377.6: flog total
         11.1: flog/method average

         150.2: ClassMethods#acts_as_markup...
Flog
     flog -g lib
         377.6: flog total
         11.1: flog/method average

         171.7: ClassMethods total
     ...
Flog
     flog -d lib
         377.6: flog total
         11.1: flog/method average

         150.2: ClassMethods#acts_as_mar...
self-flagellation




            Choose your own weightings




Wednesday, March 25, 2009
roodi
       Ruby Object Oriented Design Inferometer




Wednesday, March 25, 2009
roodi
       Ruby Object Oriented Design Inferometer


            Cyclomatic complexity




Wednesday, March 25, 2009
roodi
       Ruby Object Oriented Design Inferometer


            Cyclomatic complexity
            Assignment in conditi...
roodi
       Ruby Object Oriented Design Inferometer


            Cyclomatic complexity
            Assignment in conditi...
roodi
       Ruby Object Oriented Design Inferometer


            Cyclomatic complexity
            Assignment in conditi...
roodi
       Ruby Object Oriented Design Inferometer


            Cyclomatic complexity
            Assignment in conditi...
roodi
       Ruby Object Oriented Design Inferometer


            Cyclomatic complexity
            Assignment in conditi...
roodi


     sudo gem install roodi
     roodi “lib/**/*.rb”




Wednesday, March 25, 2009
roodi:
       How it works




Wednesday, March 25, 2009
roodi:
       How it works

            Parse files




Wednesday, March 25, 2009
roodi:
       How it works

            Parse files
            Move node-by-node




Wednesday, March 25, 2009
roodi:
       How it works

            Parse files
            Move node-by-node
            Evaluate each “Check”



Wedn...
roodi:
       How it works

            Parse files
            Move node-by-node
            Evaluate each “Check”
       ...
Ambition




Wednesday, March 25, 2009
Ambition



            DSL for Querying as Ruby




Wednesday, March 25, 2009
Ambition



            DSL for Querying as Ruby
            Framework for API Adapters




Wednesday, March 25, 2009
Ambition



            DSL for Querying as Ruby
            Framework for API Adapters
            Generator for new Adap...
Ambition ActiveRecord Adapter




            Queries only run when needed
            to_s = SQL
            to_hash = fin...
Ambition:
       How it works


            Processors parameters:
                 Context/Owner
                 Block

...
Ambition:
       How it works

           Context/Owner


       User.select { |m| m.name == 'jon' }




             Proc...
Ambition:
       How it works


            Parse the block
            Process nodes recursively
            Translate “c...
Ambition:
       How it works


            Adapters:
                 Translator for Processors
                 Make API...
The Future:
       Ruby 1.9


            YARV
                 A virtual machine based stack architecture
               ...
Ripper


            Effects YARV behavior
            Returns array of arrays  literals
            Line and column for e...
Ripper:
       Lexer
      require 'ripper'
      require 'pp'

      p Ripper.lex(quot;def m(a) nil endquot;)
       #= [...
Ripper:
       Sexp Tree

 require 'ripper'
 require 'pp'

 pp Ripper.sexp(quot;def m(a) nil endquot;)
  #= [:program,
  #...
Ripper:
       Filter
       require 'ripper/filter'

       class CommentStripper  Ripper::Filter
        def self.strip(s...
QA
       Rate at:
       http://speakerrate.com/talks/594-ruby-ast-tools


       brian.landau@viget.com
       http://ww...
Upcoming SlideShare
Loading in …5
×

Ruby AST Tools

4,619 views

Published on

Published in: Technology, Education
  • Be the first to comment

Ruby AST Tools

  1. 1. Ruby AST Tools The ParseTree Family Brian Landau Wednesday, March 25, 2009
  2. 2. The Ruby AST Code Code Style 1 Style 2 AST Code Code Style 3 Style 4 Wednesday, March 25, 2009
  3. 3. The Ruby AST Nodes Wednesday, March 25, 2009
  4. 4. The Ruby AST Nodes def (a,b) def (a,b) c= a.to_s c= a.to_s return end return end def (a,b) def (a,b) c= a.to_s c= a.to_s Wednesday, March 25, 2009
  5. 5. Parsing Ruby in Ruby ParseTree ruby_parser Wednesday, March 25, 2009
  6. 6. ParseTree Wednesday, March 25, 2009
  7. 7. ParseTree RubyInline based Wednesday, March 25, 2009
  8. 8. ParseTree RubyInline based Input sources: Class Method Proc String Wednesday, March 25, 2009
  9. 9. ruby_parser Wednesday, March 25, 2009
  10. 10. ruby_parser Uses Racc Wednesday, March 25, 2009
  11. 11. ruby_parser Uses Racc Retrives line numbers Wednesday, March 25, 2009
  12. 12. ruby_parser Uses Racc Retrives line numbers Slower than ParseTree Wednesday, March 25, 2009
  13. 13. ruby_parser Uses Racc Retrives line numbers Slower than ParseTree Sexp output is ParseTree compatible Wednesday, March 25, 2009
  14. 14. Sexp Nodes Wednesday, March 25, 2009
  15. 15. Sexp Nodes class Simple def say(*args) puts args * ' ' end end Wednesday, March 25, 2009
  16. 16. Sexp Nodes class Simple def say(*args) puts args * ' ' end end s(:class, :Simple, nil, s(:scope, s(:defn, :say, s(:args, :quot;*argsquot;), s(:scope, s(:block, s(:call, nil, :puts, s(:arglist, s(:call, s(:lvar, :args), :*, s(:arglist, s(:str, quot; quot;)))))))))) Wednesday, March 25, 2009
  17. 17. Sexp Objects Inherits from Array `structure` method `each_of_type` method `sexp_type` method `sexp_body` method Wednesday, March 25, 2009
  18. 18. SexpProcessors Or How I Learned to Love ParseTree Wednesday, March 25, 2009
  19. 19. Making a SexpProcessors Wednesday, March 25, 2009
  20. 20. Making a SexpProcessors Subclass `SexpProcessor` Wednesday, March 25, 2009
  21. 21. Making a SexpProcessors Subclass `SexpProcessor` Call `super` from initialize Wednesday, March 25, 2009
  22. 22. Making a SexpProcessors Subclass `SexpProcessor` Call `super` from initialize Define `process_xxx` methods Wednesday, March 25, 2009
  23. 23. Making a SexpProcessors Subclass `SexpProcessor` Call `super` from initialize Define `process_xxx` methods Shift off Sexp nodes Wednesday, March 25, 2009
  24. 24. Making a SexpProcessors Subclass `SexpProcessor` Call `super` from initialize Define `process_xxx` methods Shift off Sexp nodes Default process method Wednesday, March 25, 2009
  25. 25. Making a SexpProcessors Subclass `SexpProcessor` Call `super` from initialize Define `process_xxx` methods Shift off Sexp nodes Default process method `rewrite_xxx` methods Wednesday, March 25, 2009
  26. 26. Making a SexpProcessors Subclass SexpProcessor class MyProcessor < SexpProcessor end Wednesday, March 25, 2009
  27. 27. Making a SexpProcessors Call “super” in “initialize” class MyProcessor < SexpProcessor def initialize super self.warn_on_default = false self.strict = false end end Wednesday, March 25, 2009
  28. 28. Making a SexpProcessors class MyProcessor < SexpProcessor # ... Process “call” node def process_call(exp) recv = process exp.shift name = exp.shift args = process exp.shift return s() end end Wednesday, March 25, 2009
  29. 29. Making a SexpProcessors class MyProcessor < SexpProcessor # ... def process_call(exp) recv = process exp.shift name = exp.shift args = process exp.shift return s() end Must shift o all Sexp Nodes end Wednesday, March 25, 2009
  30. 30. Making a SexpProcessors class MyProcessor SexpProcessor # ... def process_call(exp) recv = process exp.shift name = exp.shift args = process exp.shift return s() end Must return a Sexp end Wednesday, March 25, 2009
  31. 31. Making a SexpProcessors class MyProcessor SexpProcessor # ... def default_process(exp) until exp.size == 0 exp.shift end return s() A default process method end end Wednesday, March 25, 2009
  32. 32. UnifiedRuby pt = ParseTree.new parse_tree = pt.parse_tree(Simple) ast = pt.process(parse_tree) # OR ast = pt.process(File.read('simple.rb')) Wednesday, March 25, 2009
  33. 33. UnifiedRuby class MyProcessor SexpProcessor include UnifiedRuby # ... end Wednesday, March 25, 2009
  34. 34. Ruby2Ruby Ruby2Ruby.translate(Simple, :say) # = quot;def say(*args)...quot; Wednesday, March 25, 2009
  35. 35. ParseTree extensions require 'parse_tree' require 'ruby2ruby' require 'parse_tree_extensions' bomb = proc do quot;BOOM!quot; end puts bomb.to_ruby # = quot;proc { quot;BOOM!quot; }quot; Wednesday, March 25, 2009
  36. 36. ParseTree Family Flog ambition Parse Tree roodi Wednesday, March 25, 2009
  37. 37. ParseTree Family Code Analytics Custom DSL Wednesday, March 25, 2009
  38. 38. Flog Takes a set of files as input Processes for “bad code” Wednesday, March 25, 2009
  39. 39. Flog Decends from SexpProcessor Includes UnifiedRuby Wednesday, March 25, 2009
  40. 40. Flog Assigns weighted scores to: Node Types Node Constructions Method calls Wednesday, March 25, 2009
  41. 41. Flog Reports scores by: Class Method Wednesday, March 25, 2009
  42. 42. Flog flog lib 377.6: flog total 11.1: flog/method average 150.2: ClassMethods#acts_as_markup 24.7: HTML#array_to_html 21.5: ClassMethods#get_markdown_class 18.4: main#none 17.9: ActsAsMarkup#none Wednesday, March 25, 2009
  43. 43. Flog flog -g lib 377.6: flog total 11.1: flog/method average 171.7: ClassMethods total 150.2: ClassMethods#acts_as_markup 21.5: ClassMethods#get_markdown_class 24.7: HTML total 24.7: HTML#array_to_html ... Wednesday, March 25, 2009
  44. 44. Flog flog -d lib 377.6: flog total 11.1: flog/method average 150.2: ClassMethods#acts_as_markup 50.9: [] 42.6: branch 24.0: send 16.5: define_method 11.9: to_s 11.8: assignment ... Wednesday, March 25, 2009
  45. 45. self-flagellation Choose your own weightings Wednesday, March 25, 2009
  46. 46. roodi Ruby Object Oriented Design Inferometer Wednesday, March 25, 2009
  47. 47. roodi Ruby Object Oriented Design Inferometer Cyclomatic complexity Wednesday, March 25, 2009
  48. 48. roodi Ruby Object Oriented Design Inferometer Cyclomatic complexity Assignment in conditional Wednesday, March 25, 2009
  49. 49. roodi Ruby Object Oriented Design Inferometer Cyclomatic complexity Assignment in conditional Class/Method line count Wednesday, March 25, 2009
  50. 50. roodi Ruby Object Oriented Design Inferometer Cyclomatic complexity Assignment in conditional Class/Method line count Empty rescue body Wednesday, March 25, 2009
  51. 51. roodi Ruby Object Oriented Design Inferometer Cyclomatic complexity Assignment in conditional Class/Method line count Empty rescue body No for loops Wednesday, March 25, 2009
  52. 52. roodi Ruby Object Oriented Design Inferometer Cyclomatic complexity Assignment in conditional Class/Method line count Empty rescue body No for loops Number of parameters Wednesday, March 25, 2009
  53. 53. roodi sudo gem install roodi roodi “lib/**/*.rb” Wednesday, March 25, 2009
  54. 54. roodi: How it works Wednesday, March 25, 2009
  55. 55. roodi: How it works Parse files Wednesday, March 25, 2009
  56. 56. roodi: How it works Parse files Move node-by-node Wednesday, March 25, 2009
  57. 57. roodi: How it works Parse files Move node-by-node Evaluate each “Check” Wednesday, March 25, 2009
  58. 58. roodi: How it works Parse files Move node-by-node Evaluate each “Check” Do Recursively Wednesday, March 25, 2009
  59. 59. Ambition Wednesday, March 25, 2009
  60. 60. Ambition DSL for Querying as Ruby Wednesday, March 25, 2009
  61. 61. Ambition DSL for Querying as Ruby Framework for API Adapters Wednesday, March 25, 2009
  62. 62. Ambition DSL for Querying as Ruby Framework for API Adapters Generator for new Adapters Wednesday, March 25, 2009
  63. 63. Ambition ActiveRecord Adapter Queries only run when needed to_s = SQL to_hash = find-compatible hash Wednesday, March 25, 2009
  64. 64. Ambition: How it works Processors parameters: Context/Owner Block Wednesday, March 25, 2009
  65. 65. Ambition: How it works Context/Owner User.select { |m| m.name == 'jon' } Processor Block Wednesday, March 25, 2009
  66. 66. Ambition: How it works Parse the block Process nodes recursively Translate “clauses” to query string Wednesday, March 25, 2009
  67. 67. Ambition: How it works Adapters: Translator for Processors Make API calls with a Query Class Wednesday, March 25, 2009
  68. 68. The Future: Ruby 1.9 YARV A virtual machine based stack architecture implementation Ripper Offers similar functionality as ParseTree and SexpProcessor Wednesday, March 25, 2009
  69. 69. Ripper Effects YARV behavior Returns array of arrays literals Line and column for each token Ripper::Filter Wednesday, March 25, 2009
  70. 70. Ripper: Lexer require 'ripper' require 'pp' p Ripper.lex(quot;def m(a) nil endquot;) #= [[[1, 0], :on_kw, quot;defquot;], # [[1, 3], :on_sp, quot; quot; ], # [[1, 4], :on_ident, quot;mquot; ], # [[1, 5], :on_lparen, quot;(quot; ], # [[1, 6], :on_ident, quot;aquot; ], # [[1, 7], :on_rparen, quot;)quot; ], # [[1, 8], :on_sp, quot; quot; ], # [[1, 9], :on_kw, quot;nilquot;], # [[1, 12], :on_sp, quot; quot; ], # [[1, 13], :on_kw, quot;endquot;]] Wednesday, March 25, 2009
  71. 71. Ripper: Sexp Tree require 'ripper' require 'pp' pp Ripper.sexp(quot;def m(a) nil endquot;) #= [:program, # [:stmts_add, # [:stmts_new], # [:def, # [:@ident, quot;mquot;, [1, 4]], # [:paren, [:params, [[:@ident, quot;aquot;, [1, 6]]], nil, nil, nil]], # [:bodystmt, # [:stmts_add, [:stmts_new], [:var_ref, [:@kw, quot;nilquot;, [1, 9]]]], # nil, # nil, # nil]]]] Wednesday, March 25, 2009
  72. 72. Ripper: Filter require 'ripper/filter' class CommentStripper Ripper::Filter def self.strip(src) new(src).parse end def on_default(event, token, data) print token end def on_comment(token, data) puts end end CommentStripper.strip(ARGF) Wednesday, March 25, 2009
  73. 73. QA Rate at: http://speakerrate.com/talks/594-ruby-ast-tools brian.landau@viget.com http://www.viget.com/extend http://www.websideattractions.com/ Wednesday, March 25, 2009

×