Search

Ruby AST Tools

Loading...

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

0 comments

Post a comment

    Post a comment
    Embed Video
    Edit your comment Cancel

    Notes on slide 1


    Ruby 1.8 uses a Bison Yacc-like parser
    Many different styles lead to the same Abstract Syntax tree or set of instructions.

    Best to thing of the AST not as a tree but a large container composed of inner nodes

    themselves composed of smaller nodes. until you get to an individual “token”

    Both by Ryan Davis



    Pure ruby implementation using racc. For those that don’t know RACC is a ruby based YACC parser.
    Slowness not a huge issue in my experience.

    Pure ruby implementation using racc. For those that don’t know RACC is a ruby based YACC parser.
    Slowness not a huge issue in my experience.

    Pure ruby implementation using racc. For those that don’t know RACC is a ruby based YACC parser.
    Slowness not a huge issue in my experience.

    Pure ruby implementation using racc. For those that don’t know RACC is a ruby based YACC parser.
    Slowness not a huge issue in my experience.

    Obtains the Parse tree and represents it as nested s-expresions.

    Obtains the Parse tree and represents it as nested s-expresions.

    composed of array’s, symbol’s and literals
    `structure` just returns the node type structure not the contents of the nodes. this is very handy for code structure analysis.

    Takes the sexp objects returned by ParseTree and processes returning output.













    Essentially turns the parse tree into an AST. It does this via a set of rewrite methods.
    By default ParseTree does this. To get the un-unified version you have to use RawParseTree.
    ParseTree does this by processing with a CompositeSexpProcessor called Unifier

    A Module that can be included in a SexpProcessor.

    Another good example of a SexpProcessor.
    Take a class or method and translates it into a string of valid Ruby code that would create that Class/method exactly.
    Useful for metaprogramming and dissecting dynamically created classes, methods, and procs

    Convert Methods, UnboundMethods and Procs to sexps and strings of ruby code. Helpful in debuging mysterious Procs.

    Often times I find libraries that use ParseTree but don't take advantage of SexpProcessor and often rebuild similar functionality. very sad.

    2 Types of Libraries based on ParseTree

    Good at finding problem areas that may need refactoring


    Also adds to varying amounts to a score multiplier for specific types of nestings.


    with no options
    Ryan Davis has said to remember it's more about relative scores. Particularly bad scores I generally consider is those over 40 or if I'm being very rough those over 25.

    with group option, groups by class

    with detail version (old output) gives why each method earned it's score.

    Converted over by Ben Scofield
    Working on a site that will allow you to share and download those weightings.

    by Marty Andrews

    by Marty Andrews

    by Marty Andrews

    by Marty Andrews

    by Marty Andrews

    by Marty Andrews

    Once again, Very good at finding problem areas that need refactoring.





    by github’s Chris Wanstrath
    Allows you to express queries in clear and concise ruby code.
    Think of it as a Framework for specific API adapters, e.g. ActiveRecord
    Includes a Rubigen based adapter generator for any data-source API you want

    by github’s Chris Wanstrath
    Allows you to express queries in clear and concise ruby code.
    Think of it as a Framework for specific API adapters, e.g. ActiveRecord
    Includes a Rubigen based adapter generator for any data-source API you want

    by github’s Chris Wanstrath
    Allows you to express queries in clear and concise ruby code.
    Think of it as a Framework for specific API adapters, e.g. ActiveRecord
    Includes a Rubigen based adapter generator for any data-source API you want

    Just gives us another syntax for doing finds, the syntax may be preferable to some. Still needs lots of work to do everything you can do with find or SQL.






    No composite SexpProcessor equivalent yet.

    line and column, node/event type, “code” that produced it.

    Very similar output to ParseTree, but does include line and column info for inner most nodes.

    Uses Lexer and goes thru each node.


    6 Favorites

    Ruby AST Tools - Presentation Transcript

    1. Ruby AST Tools The ParseTree Family Brian Landau Wednesday, March 25, 2009
    2. The Ruby AST Code Code Style 1 Style 2 AST Code Code Style 3 Style 4 Wednesday, March 25, 2009
    3. The Ruby AST Nodes Wednesday, March 25, 2009
    4. The Ruby AST Nodes def (a,b) def (a,b) c= a.to_s c= a.to_s return end return end def (a,b) def (a,b) c= a.to_s c= a.to_s Wednesday, March 25, 2009
    5. Parsing Ruby in Ruby ParseTree ruby_parser Wednesday, March 25, 2009
    6. ParseTree Wednesday, March 25, 2009
    7. ParseTree RubyInline based Wednesday, March 25, 2009
    8. ParseTree RubyInline based Input sources: Class Method Proc String Wednesday, March 25, 2009
    9. ruby_parser Wednesday, March 25, 2009
    10. ruby_parser Uses Racc Wednesday, March 25, 2009
    11. ruby_parser Uses Racc Retrives line numbers Wednesday, March 25, 2009
    12. ruby_parser Uses Racc Retrives line numbers Slower than ParseTree Wednesday, March 25, 2009
    13. ruby_parser Uses Racc Retrives line numbers Slower than ParseTree Sexp output is ParseTree compatible Wednesday, March 25, 2009
    14. Sexp Nodes Wednesday, March 25, 2009
    15. Sexp Nodes class Simple def say(*args) puts args * ' ' end end Wednesday, March 25, 2009
    16. Sexp Nodes class Simple def say(*args) puts args * ' ' end end s(:class, :Simple, nil, s(:scope, s(:defn, :say, s(:args, :\"*args\"), s(:scope, s(:block, s(:call, nil, :puts, s(:arglist, s(:call, s(:lvar, :args), :*, s(:arglist, s(:str, \" \")))))))))) Wednesday, March 25, 2009
    17. Sexp Objects Inherits from Array `structure` method `each_of_type` method `sexp_type` method `sexp_body` method Wednesday, March 25, 2009
    18. SexpProcessors Or How I Learned to Love ParseTree Wednesday, March 25, 2009
    19. Making a SexpProcessors Wednesday, March 25, 2009
    20. Making a SexpProcessors Subclass `SexpProcessor` Wednesday, March 25, 2009
    21. Making a SexpProcessors Subclass `SexpProcessor` Call `super` from initialize Wednesday, March 25, 2009
    22. Making a SexpProcessors Subclass `SexpProcessor` Call `super` from initialize Define `process_xxx` methods Wednesday, March 25, 2009
    23. Making a SexpProcessors Subclass `SexpProcessor` Call `super` from initialize Define `process_xxx` methods Shift off Sexp nodes Wednesday, March 25, 2009
    24. Making a SexpProcessors Subclass `SexpProcessor` Call `super` from initialize Define `process_xxx` methods Shift off Sexp nodes Default process method Wednesday, March 25, 2009
    25. Making a SexpProcessors Subclass `SexpProcessor` Call `super` from initialize Define `process_xxx` methods Shift off Sexp nodes Default process method `rewrite_xxx` methods Wednesday, March 25, 2009
    26. Making a SexpProcessors Subclass SexpProcessor class MyProcessor < SexpProcessor end Wednesday, March 25, 2009
    27. Making a SexpProcessors Call “super” in “initialize” class MyProcessor < SexpProcessor def initialize super self.warn_on_default = false self.strict = false end end Wednesday, March 25, 2009
    28. Making a SexpProcessors class MyProcessor < SexpProcessor # ... Process “call” node def process_call(exp) recv = process exp.shift name = exp.shift args = process exp.shift return s() end end Wednesday, March 25, 2009
    29. Making a SexpProcessors class MyProcessor < SexpProcessor # ... def process_call(exp) recv = process exp.shift name = exp.shift args = process exp.shift return s() end Must shift o all Sexp Nodes end Wednesday, March 25, 2009
    30. Making a SexpProcessors class MyProcessor < SexpProcessor # ... def process_call(exp) recv = process exp.shift name = exp.shift args = process exp.shift return s() end Must return a Sexp end Wednesday, March 25, 2009
    31. Making a SexpProcessors class MyProcessor < SexpProcessor # ... def default_process(exp) until exp.size == 0 exp.shift end return s() A default process method end end Wednesday, March 25, 2009
    32. UnifiedRuby pt = ParseTree.new parse_tree = pt.parse_tree(Simple) ast = pt.process(parse_tree) # OR ast = pt.process(File.read('simple.rb')) Wednesday, March 25, 2009
    33. UnifiedRuby class MyProcessor < SexpProcessor include UnifiedRuby # ... end Wednesday, March 25, 2009
    34. Ruby2Ruby Ruby2Ruby.translate(Simple, :say) # => \"def say(*args)...\" Wednesday, March 25, 2009
    35. ParseTree extensions require 'parse_tree' require 'ruby2ruby' require 'parse_tree_extensions' bomb = proc do \"BOOM!\" end puts bomb.to_ruby # => \"proc { \\\"BOOM!\\\" }\" Wednesday, March 25, 2009
    36. ParseTree Family Flog ambition Parse Tree roodi Wednesday, March 25, 2009
    37. ParseTree Family Code Analytics Custom DSL Wednesday, March 25, 2009
    38. Flog Takes a set of files as input Processes for “bad code” Wednesday, March 25, 2009
    39. Flog Decends from SexpProcessor Includes UnifiedRuby Wednesday, March 25, 2009
    40. Flog Assigns weighted scores to: Node Types Node Constructions Method calls Wednesday, March 25, 2009
    41. Flog Reports scores by: Class Method Wednesday, March 25, 2009
    42. Flog > flog lib 377.6: flog total 11.1: flog/method average 150.2: ClassMethods#acts_as_markup 24.7: HTML#array_to_html 21.5: ClassMethods#get_markdown_class 18.4: main#none 17.9: ActsAsMarkup#none Wednesday, March 25, 2009
    43. Flog > flog -g lib 377.6: flog total 11.1: flog/method average 171.7: ClassMethods total 150.2: ClassMethods#acts_as_markup 21.5: ClassMethods#get_markdown_class 24.7: HTML total 24.7: HTML#array_to_html ... Wednesday, March 25, 2009
    44. Flog > flog -d lib 377.6: flog total 11.1: flog/method average 150.2: ClassMethods#acts_as_markup 50.9: [] 42.6: branch 24.0: send 16.5: define_method 11.9: to_s 11.8: assignment ... Wednesday, March 25, 2009
    45. self-flagellation Choose your own weightings Wednesday, March 25, 2009
    46. roodi Ruby Object Oriented Design Inferometer Wednesday, March 25, 2009
    47. roodi Ruby Object Oriented Design Inferometer Cyclomatic complexity Wednesday, March 25, 2009
    48. roodi Ruby Object Oriented Design Inferometer Cyclomatic complexity Assignment in conditional Wednesday, March 25, 2009
    49. roodi Ruby Object Oriented Design Inferometer Cyclomatic complexity Assignment in conditional Class/Method line count Wednesday, March 25, 2009
    50. roodi Ruby Object Oriented Design Inferometer Cyclomatic complexity Assignment in conditional Class/Method line count Empty rescue body Wednesday, March 25, 2009
    51. roodi Ruby Object Oriented Design Inferometer Cyclomatic complexity Assignment in conditional Class/Method line count Empty rescue body No for loops Wednesday, March 25, 2009
    52. roodi Ruby Object Oriented Design Inferometer Cyclomatic complexity Assignment in conditional Class/Method line count Empty rescue body No for loops Number of parameters Wednesday, March 25, 2009
    53. roodi > sudo gem install roodi > roodi “lib/**/*.rb” Wednesday, March 25, 2009
    54. roodi: How it works Wednesday, March 25, 2009
    55. roodi: How it works Parse files Wednesday, March 25, 2009
    56. roodi: How it works Parse files Move node-by-node Wednesday, March 25, 2009
    57. roodi: How it works Parse files Move node-by-node Evaluate each “Check” Wednesday, March 25, 2009
    58. roodi: How it works Parse files Move node-by-node Evaluate each “Check” Do Recursively Wednesday, March 25, 2009
    59. Ambition Wednesday, March 25, 2009
    60. Ambition DSL for Querying as Ruby Wednesday, March 25, 2009
    61. Ambition DSL for Querying as Ruby Framework for API Adapters Wednesday, March 25, 2009
    62. Ambition DSL for Querying as Ruby Framework for API Adapters Generator for new Adapters Wednesday, March 25, 2009
    63. Ambition ActiveRecord Adapter Queries only run when needed to_s = SQL to_hash = find-compatible hash Wednesday, March 25, 2009
    64. Ambition: How it works Processors parameters: Context/Owner Block Wednesday, March 25, 2009
    65. Ambition: How it works Context/Owner User.select { |m| m.name == 'jon' } Processor Block Wednesday, March 25, 2009
    66. Ambition: How it works Parse the block Process nodes recursively Translate “clauses” to query string Wednesday, March 25, 2009
    67. Ambition: How it works Adapters: Translator for Processors Make API calls with a Query Class Wednesday, March 25, 2009
    68. The Future: Ruby 1.9 YARV A virtual machine based stack architecture implementation Ripper Offers similar functionality as ParseTree and SexpProcessor Wednesday, March 25, 2009
    69. Ripper Effects YARV behavior Returns array of arrays & literals Line and column for each token Ripper::Filter Wednesday, March 25, 2009
    70. Ripper: Lexer require 'ripper' require 'pp' p Ripper.lex(\"def m(a) nil end\") #=> [[[1, 0], :on_kw, \"def\"], # [[1, 3], :on_sp, \" \" ], # [[1, 4], :on_ident, \"m\" ], # [[1, 5], :on_lparen, \"(\" ], # [[1, 6], :on_ident, \"a\" ], # [[1, 7], :on_rparen, \")\" ], # [[1, 8], :on_sp, \" \" ], # [[1, 9], :on_kw, \"nil\"], # [[1, 12], :on_sp, \" \" ], # [[1, 13], :on_kw, \"end\"]] Wednesday, March 25, 2009
    71. Ripper: Sexp Tree require 'ripper' require 'pp' pp Ripper.sexp(\"def m(a) nil end\") #=> [:program, # [:stmts_add, # [:stmts_new], # [:def, # [:@ident, \"m\", [1, 4]], # [:paren, [:params, [[:@ident, \"a\", [1, 6]]], nil, nil, nil]], # [:bodystmt, # [:stmts_add, [:stmts_new], [:var_ref, [:@kw, \"nil\", [1, 9]]]], # nil, # nil, # nil]]]] Wednesday, March 25, 2009
    72. Ripper: Filter require 'ripper/filter' class CommentStripper < Ripper::Filter def self.strip(src) new(src).parse end def on_default(event, token, data) print token end def on_comment(token, data) puts end end CommentStripper.strip(ARGF) Wednesday, March 25, 2009
    73. Q&A Rate at: http://speakerrate.com/talks/594-ruby-ast-tools brian.landau@viget.com http://www.viget.com/extend http://www.websideattractions.com/ Wednesday, March 25, 2009

    + Brian LandauBrian Landau, 7 months ago

    custom

    950 views, 6 favs, 1 embeds more stats

    More info about this document

    CC Attribution-NonCommercial-NoDerivs LicenseCC Attribution-NonCommercial-NoDerivs LicenseCC Attribution-NonCommercial-NoDerivs License

    Go to text version

    • Total Views 950
      • 906 on SlideShare
      • 44 from embeds
    • Comments 0
    • Favorites 6
    • Downloads 12
    Most viewed embeds
    • 44 views on http://www.websideattractions.com

    more

    All embeds
    • 44 views on http://www.websideattractions.com

    less

    Flagged as inappropriate Flag as inappropriate
    Flag as inappropriate

    Select your reason for flagging this presentation as inappropriate. If needed, use the feedback form to let us know more details.

    Cancel
    File a copyright complaint
    Having problems? Go to our helpdesk?

    Categories

    Tags

    more
    -->
    Search
    RSS Feed
    What's new?

    © 2009 SlideShare Inc. All Rights Reserved