Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Tork03 LT

965 views

Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Tork03 LT

  1. 1. Ruby Masaya TARUI
  2. 2. ••• tokyu.rb #ruby-ja
  3. 3.
  4. 4. • Regexp• Rex• PEG ( Treetop , Citrus , Parslet )• Ripper• Bootstrap• M17N• ERB• Racc
  5. 5. ••••
  6. 6. • Racc strscan Ruby C → Rex
  7. 7. Rex• C• Regexp!• Ruby /(match1)|(match2)|.../ Marshal capture [ match1,match1’s capture1, match1’s capture2...• Cature Capture 1.9 Lost Ruby parsec
  8. 8. • tork05 ( tork03 )• @kawabata PEG : Parsing Expression Grammar https://github.com/kawabata/aozora-proc•• Ruby M17N Treetop, Citrus →(Rule Class )• M17N
  9. 9. PEG•• Packrat Parser O(n) Packrat Parser =•
  10. 10. # -*- coding: utf-8 -*-require pprequire ./peg.rbexpr_peg = <<EOTparser expr(top) top ← Expr !. ->$1 Expr ← Sum Sum ← Product ((+ / -) Product)* -> { restruct(r) } Product ← Value ((* / /) Value)* -> { restruct(r) } Value ← Sp n:[0-9]+ Sp -> {n.join.to_i} / Sp ( e:Expr ) Sp -> e Sp ← ( " " / "t" / "n" )*EOTdef restruct(r) r[1].inject(r[0]){|a,b| [b[0],a,b[1]] }endexpr_parser = PEG::ParserGenerator.parse(expr_peg)require ppexpr = " 1+ 2 * (3 - 4) / 5 + 6 "pp exprpp expr_parser.parse(expr)=>" 1+ 2 * (3 - 4) / 5 + 6 "["+", ["+", 1, ["/", ["*", 2, ["-", 3, 4]], 5]], 6]StackOverFlow orz
  11. 11. PEG• PEG• M17N OK
  12. 12. -> { restruct(r) }• Ruby → Ruby
  13. 13. parse.y$ wc parse.y 10745 27247 234392 parse.y•
  14. 14. Ruby Scanner parse “a <%= ‘%%>’ %>b”
  15. 15. -> { restruct(r) }• Ruby → Ruby• 8 10
  16. 16. ERB Ruby Scanner class TrimScanner < Scanner # :nodoc: def initialize(src, trim_mode, percent) super @trim_mode = trim_mode @percent = percent if @trim_mode == > @scan_line = self.method(:trim_line1) elsif @trim_mode == <> @scan_line = self.method(:trim_line2) elsif @trim_mode == - @scan_line = self.method(:explicit_trim_line) else @scan_line = self.method(:scan_line) end end attr_accessor :stag def scan(&block) @stag = nil if @percent @src.each_line do |line| percent_line(line, &block) end else @scan_line.call(@src, &block) end nil end def percent_line(line, &block) if @stag || line[0] != ?% return @scan_line.call(line, &block) end line[0] = if line[0] == ?% @scan_line.call(line, &block) else yield(PercentLine.new(line.chomp)) end end
  17. 17. def trim_line2(line) head = nil line.scan(/(.*?)(<%%|%%>|<%=|<%#|<%|%>n|%>|n|z)/m) do |tokens| tokens.each do |token| next if token.empty? head = token unless head ERB Ruby Scanner if token == "%>n" yield(%>) if is_erb_stag?(head) yield(:cr) else yield("n") end head = nil else yield(token) head = nil if token == "n" end end end end def explicit_trim_line(line) line.scan(/(.*?)(^[ t]*<%-|<%-|<%%|%%>|<%=|<%#|<%|-%>n|-%>|%>|z)/m) do |tokens| tokens.each do |token| next if token.empty? if @stag.nil? && /[ t]*<%-/ =~ token yield(<%) elsif @stag && token == "-%>n" yield(%>) yield(:cr) elsif @stag && token == -%> yield(%>) else yield(token) end end end end ERB_STAG = %w(<%= <%# <%) def is_erb_stag?(s) ERB_STAG.member?(s) end end Scanner.default_scanner = TrimScanner Scanner
  18. 18. Racc Ruby Scannerdef scan_action buf = nest = 1 pre = nil @in_block = action begin pre = nil if s = reads(/As+/) # does not set pre buf << s end until @line.empty? if s = reads(/A[^"`{}%#/$]+/) buf << (pre = s) next end case ch = read(1) when { nest += 1 buf << (pre = ch) when } nest -= 1 if nest == 0 @in_block = nil return buf end buf << (pre = ch) when # # comment buf << ch << @line break when "", ", ` buf << (pre = scan_quoted(ch)) when % if literal_head? pre, @line # % string, regexp, array buf << ch case ch = read(1) when /[qQx]/n buf << ch << (pre = scan_quoted(read(1), %string)) when /wW/n
  19. 19. # % string, regexp, array buf << ch case ch = read(1) when /[qQx]/n buf << ch << (pre = scan_quoted(read(1), %string))Racc Ruby Scanner when /wW/n buf << ch << (pre = scan_quoted(read(1), %array)) when /s/n buf << ch << (pre = scan_quoted(read(1), %symbol)) when /r/n buf << ch << (pre = scan_quoted(read(1), %regexp)) when /[a-zA-Z0-9= ]/n # does not include "_" scan_error! "unknown type of % literal %#{ch}" else buf << (pre = scan_quoted(ch, %string)) end else # operator buf << ||op-> if $raccs_print_type buf << (pre = ch) end when / if literal_head? pre, @line # regexp buf << (pre = scan_quoted(ch, regexp)) else # operator buf << ||op-> if $raccs_print_type buf << (pre = ch) end when $ # gvar buf << ch << (pre = read(1)) else raise racc: fatal: must not happen end end buf << "n" end while next_line() raise racc: fatal: scan finished before parser finishedend
  20. 20.
  21. 21. Ripper!• parse.y 1.9 Ruby•irb(main):001:0> Ripper.parse("a = b + 1")=> nil
  22. 22. Ripper• test/ripper/test_files.rbclass Parser < Ripper PARSER_EVENTS.each {|n| eval "def on_#{n}(*args) r = [:#{n}, *args]; r.inspect; Object.new end" } SCANNER_EVENTS.each {|n| eval "def on_#{n}(*args) r = [:#{n}, *args]; r.inspect; Object.new end" }end•
  23. 23. Ripper Scanner class RubyCodeDetector < Ripper class ParseError < Exception attr_reader :message,:result def initialize(r,msg) @result = r @message = msg end end def on_parse_error(msg) l = @lines[lineno-1][0,column-1] src = (@lines[0,lineno-1]+[l])*"n" src.force_encoding(@src.encoding) raise ParseError.new(src,msg) end def initialize(*args) @src=args[0] @lines=@src.split(/n/).map{|i| i.force_encoding("ASCII-8BIT") } super end def parse super raise ParseError.new(@src,"syntax error, unexpected $end, expecting }") nil rescue ParseError => e if e.message =~ /G syntax error, unexpected }/ e.result else raise $! nil end end end
  24. 24. Ripper Scanner class RubyCodeDetector < Ripper class ParseError < Exception attr_reader :message,:result def initialize(r,msg) @result = r @message = msg end end def on_parse_error(msg) l = @lines[lineno-1][0,column-1] src = (@lines[0,lineno-1]+[l])*"n" src.force_encoding(@src.encoding) raise ParseError.new(src,msg) end def initialize(*args) @src=args[0] @lines=@src.split(/n/).map{|i| i.force_encoding("ASCII-8BIT") } super end def parse super raise ParseError.new(@src,"syntax error, unexpected $end, expecting }") nil rescue ParseError => e if e.message =~ /G syntax error, unexpected }/ e.result else raise $! nil end end end
  25. 25. • Regexp• Rex• PEG ( Treetop , Citrus , Parslet )• Ripper• Bootstrap• M17N• ERB• Racc
  26. 26. • Ripper 1.9• Ripper

×