A brand new way of
Perl product, and it‟s future
Masaaki Goshima (@goccy54)
mixi Inc.
Me
• Name : Masaaki Goshima (@goccy54)
• Job : mixi (Tanpopo Group)
– Development for Developers
– Developing a platform f...
gperl
• Fastest perl-like language
– Aiming perl5 compatible syntax
– Written with C++
• The last commit for repository wa...
NO! ITS STILL ALIVE!!!
• However, gperl itself has departed
• Modules(lexical analyzer, parser, code
generator, interprete...
This presentation‟s Goal
• Today, we‟re going to present “Spinout
projects” for each modules
gperl
Compiler::Lexer Compile...
Agenda
• Section 1
– Introduction of Compiler::* Modules
• Section 2
– (Application1) Running on multi platforms
• Section...
Section1
• Introduction of Compiler::* Modules
1. Compiler::Lexer
– Lexical Analyzer for Perl5
2. Compiler::Parser
– Creat...
Compiler::Lexer
• Lexical Analyzer for Perl5
• Features
– Simple (return Array Reference of Token Object)
– Fast (faster 1...
Example
1: my$v = sprintf(<<'TEMPLATE', 'yapc');
2: %s::Asia
3: TEMPLATE
4: $v =~ s#yapc#YAPC#;
%s::Asia : HereDocument
TE...
Tips
• Cannot tokenize pattern
1. func*v
• 「*」is Glob or Mul
2. func/ ..
• 「/」is RegexDelimiter or Div
3. func<<FLAG
• 「FL...
Future plan
• Supporting recursively tokenizing
– More wide range of parsing application
– Recursive parsing will take tim...
Compiler::Parser
• Create Abstract Syntax Tree for Perl5
• Features
– Fast (faster than PPI)
– Readable Code
• Simple desi...
Example
my$v= sin$v + $v * $v / $v - $v&&$v
my($v= (sin((($v+ (($v* $v) / $v)) - $v))&&$v))
=
$v &&
left right
rightleft
T...
Example2) Can parse BlackPerl
BEFOREHAND: close door, each window &exit;
waituntiltime;
open spell book; study;
read (spel...
Future plan
• Replace PPI !!!
– Supporting some expressions
• ThreeTermOperator, Glob, given/when, goto etc..
– Supporting...
Compiler::CodeGenerator::LLVM
• Create LLVM IR for Perl5
->
$a->{b}->[0]->c(@args)
->
->
$a {}
b
[]
c
0
@args
left
left
le...
Native Code
Other Language
LLVM(Low Level Virtual Machine)
• Compiler Infrastructure
• Better than GNU GCC
LLVM
X86
ARM
Po...
How to use
• Dependency: clang/llvm version 3.2 or 3.3
useCompiler::Lexer;
useCompiler::Parser;
useCompiler::CodeGenerator...
Benchmark(Fibonacci)
Language
time(real)[
sec]
X
gperl 0.10 X81.2
LuaJIT
(2.0.0-beta10)
0.12 X67.6
v8(0.8.0) 0.18 X45.1
Co...
Section 2
• Application1)
– Running on multi platforms
1. Running on Web Browser
• Recently, way of writing code that runs on web
browser is not onlyJavaScript
– e.g.) CoffeScri...
Compiler::Tools::Transpiler
• Translate Perl5 codes to JavaScript codes
– (Step1) translate Perl5 codes to LLVM-IR with
Co...
How to use
useCompiler::Tools::Transpiler;
my$transpiler= Compiler::Tools::Transpiler->new({
engine=>'JavaScript‟,
library...
ROADMAP/Repository
• Fix some emscripten‟s bugs
• Supporting accessors for HTML objects
• Supporting Canvas API
• etc..
• ...
2. Running on iOS and OSX
• Recently, software that can develop iOS or OSX
applications using light weight language has
be...
PerlMotion
iOS and OS X development
using the Perl5 programming language
PerlMotion’s Architecture
UIKit
Cocoa Touch Frameworks
Core Animation Core Audio Core Data
Perl5 x Cocoa Touch Frameworks
...
DEMO
• HelloWorldby PerlMotion
Current Status
Still not support functions
Symbol Management System : Glob
Garbage Collection
Regexp
Three Term Operator
e...
ROADMAP
• Supporting deployment on device
• Preparing debugging environment
– (stdout/stderr/gdb..etc)
• Supporting existi...
Welcome your Contribution!
• I want discuss design or objective
– Especially, I need advice on “What Perl
community likes?...
Conclusion at Section 1 and 2
• Perl5 codes Run Anywhere!!!
– iOS,OSX,Web Browser and others
• Welcome your Contribution
–...
• Application2)
– Static analysis tool
Section3
Compiler::Tools::CopyPasteDetector
• Detect Copy and Paste for Perl5
• Features
– Fast (detecting Engine written by C++ wi...
DEMO
• Detect Copy & Paste of Catalyst and Mojolicious
Another Modules
• Perl::MinimumVersion::Fast(@tokuhirom)
– Find a minimum required version of perl for Perl code
• Test::L...
Future plan
• Perl::Metrics::Simple::Fast
– Will release faster module than
Perl::Metrics::Simple by using
Compiler::Lexer...
Conclusion at Section 3
• Introduction of copy and paste detecting
tool for Perl5
– Compiler::Tools::CopyPasteDetector
• C...
Finally
Compiler::Lexer Compiler::Parser
Compiler::CodeGenerator
::LLVM
Type Inference
Engine
Static Typing
- Coming soon ...
これからのPerlプロダクトのかたち(YAPC::Asia 2013)
Upcoming SlideShare
Loading in...5
×

これからのPerlプロダクトのかたち(YAPC::Asia 2013)

5,511

Published on

Published in: Technology
0 Comments
26 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
5,511
On Slideshare
0
From Embeds
0
Number of Embeds
10
Actions
Shares
0
Downloads
16
Comments
0
Likes
26
Embeds 0
No embeds

No notes for slide

これからのPerlプロダクトのかたち(YAPC::Asia 2013)

  1. 1. A brand new way of Perl product, and it‟s future Masaaki Goshima (@goccy54) mixi Inc.
  2. 2. Me • Name : Masaaki Goshima (@goccy54) • Job : mixi (Tanpopo Group) – Development for Developers – Developing a platform for refining legacy software – Managing Jenkins • Personally, interested in Perl and developing yet another Perl, gperl – (YAPC::Asia 2012) Perlと出会いPerlを作る
  3. 3. gperl • Fastest perl-like language – Aiming perl5 compatible syntax – Written with C++ • The last commit for repository was 11 months ago... – “Is this already departed !?”
  4. 4. NO! ITS STILL ALIVE!!! • However, gperl itself has departed • Modules(lexical analyzer, parser, code generator, interpreter) are useful – Lexical analyzer: implementing perl5 syntax highlighter – Parser: static analysis tool (used as refinement of legacy perl5 code) Input ->Lexer -> Parser ->CodeGenerator -> VM (JIT) -> output
  5. 5. This presentation‟s Goal • Today, we‟re going to present “Spinout projects” for each modules gperl Compiler::Lexer Compiler::Parser Compiler::CodeGenerator ::LLVM Lexer Parser CodeGenerator ???::???::??? ????? Next Module
  6. 6. Agenda • Section 1 – Introduction of Compiler::* Modules • Section 2 – (Application1) Running on multi platforms • Section 3 – (Application2) Static analysis tool
  7. 7. Section1 • Introduction of Compiler::* Modules 1. Compiler::Lexer – Lexical Analyzer for Perl5 2. Compiler::Parser – Create Abstract Syntax Tree for Perl5 3. Compiler::CodeGenerator::LLVM – Create LLVM IR for Perl5
  8. 8. Compiler::Lexer • Lexical Analyzer for Perl5 • Features – Simple (return Array Reference of Token Object) – Fast (faster 10 ~ 100 times than PPI::Tokenize) • Wirtten with C++ • Perfect hashing for reserved keywords • memory pool for token objects – Readable Code • Nothing use parser generator like yacc/bison
  9. 9. Example 1: my$v = sprintf(<<'TEMPLATE', 'yapc'); 2: %s::Asia 3: TEMPLATE 4: $v =~ s#yapc#YAPC#; %s::Asia : HereDocument TEMPLATE : HereDocumentEnd $v : Var =~ : RegOK s : RegReplace # : RegDelim yapc : RegReplaceFrom # : RegMiddleDelim YAPC : RegReplaceTo # : RegDelim ; : SemiColon my : VarDecl $v : LocalVar = : Assign sprintf : BuiltinFunc ( : LeftParenthesis << : LeftShift TEMPLATE : HereDocumentRawTag , : Comma yapc : RawString ) : RightParenthesis ; : SemiColon tokenize
  10. 10. Tips • Cannot tokenize pattern 1. func*v • 「*」is Glob or Mul 2. func/ .. • 「/」is RegexDelimiter or Div 3. func<<FLAG • 「FLAG」is HereDocumentTag or Constant It may make a mistake
  11. 11. Future plan • Supporting recursively tokenizing – More wide range of parsing application – Recursive parsing will take time, optimizing will be next future work func*v =>func(*v) or func() * v func/ … =>func(/../) or func() / v func<<FLAG =>func(<<FLAG) or func() << FLAG sub func($) {} or sub func() {}
  12. 12. Compiler::Parser • Create Abstract Syntax Tree for Perl5 • Features – Fast (faster than PPI) – Readable Code • Simple design • Nothing use generator Can generate Virtual Machine code as walking tree by post order -> $a->{b}->[0]->c(@args) -> -> $a {} b [] c 0 @args left left left right argsright right data data
  13. 13. Example my$v= sin$v + $v * $v / $v - $v&&$v my($v= (sin((($v+ (($v* $v) / $v)) - $v))&&$v)) = $v && left right rightleft The most difficult part of Perl5 parser: hidden(optional) parenthesis
  14. 14. Example2) Can parse BlackPerl BEFOREHAND: close door, each window &exit; waituntiltime; open spell book; study; read (spell, $scan, select); tell us; write it, print(thehex) whileeach watches, reverse length,write again; kill spiders, pop them, chop, split, kill them. unlink arms, shift, waitand listen (listening, wait). sort the flock (then, warn"the goats", kill"the sheep"); kill them, dump qualms, shift moralities, values aside, each one; die sheep; die (to, reversethe => system you accept (reject, respect)); next step, killnext sacrifice, each sacrifice, wait, redo ritual until"all the spirits are pleased"; do it ("as they say"). do it(*everyone***must***participate***in***forbidden**s*e*x*). return last victim; package body; exitcrypt (time, times&“half a time”) &close it. select (quickly) and warnnext victim; AFTERWARDS: tell nobody. wait, waituntiltime; wait until next year, next decade; sleep, sleep, die yourself, die@last BlackPerl for Perl5
  15. 15. Future plan • Replace PPI !!! – Supporting some expressions • ThreeTermOperator, Glob, given/when, goto etc.. – Supporting compatible methods PPI provides • PPI::Document::find • PPI::Document::prune
  16. 16. Compiler::CodeGenerator::LLVM • Create LLVM IR for Perl5 -> $a->{b}->[0]->c(@args) -> -> $a {} b [] c 0 @args left left left right argsright right data data 1 2 3 4 5 6 7 8 9 10 Compiler::Parser Compiler::CodeGenerator::LLVM LLVM IR Running with JIT
  17. 17. Native Code Other Language LLVM(Low Level Virtual Machine) • Compiler Infrastructure • Better than GNU GCC LLVM X86 ARM Power C/C++/Object ive-C JavaScript py2llvm MacRuby clang
  18. 18. How to use • Dependency: clang/llvm version 3.2 or 3.3 useCompiler::Lexer; useCompiler::Parser; useCompiler::CodeGenerator::LLVM; my$tokens = Compiler::Lexer->new('')->tokenize($code); my $ast= Compiler::Parser->new->parse($tokens); my$generator = Compiler::CodeGenerator::LLVM->new(); my$llvm_ir= $generator->generate($ast); # generate LLVM IR $generator->debug_run($ast); # run with JIT
  19. 19. Benchmark(Fibonacci) Language time(real)[ sec] X gperl 0.10 X81.2 LuaJIT (2.0.0-beta10) 0.12 X67.6 v8(0.8.0) 0.18 X45.1 Compiler::CodeGenerator:: LLVM 0.72 X11.2 Perl(5.16.0) 8.12 X1.0 Environment MacOSX(10.8.4) 2.2GHz Intel Core i7 Memory: 8GB N=35
  20. 20. Section 2 • Application1) – Running on multi platforms
  21. 21. 1. Running on Web Browser • Recently, way of writing code that runs on web browser is not onlyJavaScript – e.g.) CoffeScript, JSX, TypeScript, Dart • I wrote module for Perl5! SEE ALSO :perl.js (@gfx)
  22. 22. Compiler::Tools::Transpiler • Translate Perl5 codes to JavaScript codes – (Step1) translate Perl5 codes to LLVM-IR with Compiler::* modules – (Step2) translate LLVM-IR to JavaScript codes with emscripten Perl5 LLVM emscripten
  23. 23. How to use useCompiler::Tools::Transpiler; my$transpiler= Compiler::Tools::Transpiler->new({ engine=>'JavaScript‟, library_path=> [„lib‟] }); openmy $fh, '<', 'target_application.pl'; my$perl5_code = do { local$/; <$fh> }; my$javascript_code= $transpiler->transpile($perl5_code); open $fh, '>', 'target_application.js'; print$fh $javascript_code; close$fh; ※ Needs clang-3.2 and LLVM-3.2 (Not ver. 3.3 or later) because emscripten has still not support LLVM 3.3 or later
  24. 24. ROADMAP/Repository • Fix some emscripten‟s bugs • Supporting accessors for HTML objects • Supporting Canvas API • etc.. • Repository – https://github.com/goccy/p5-Compiler-Tools-Transpiler.git
  25. 25. 2. Running on iOS and OSX • Recently, software that can develop iOS or OSX applications using light weight language has being released – RubyMotion – mocl – Titanium • I wrote module for Perl5!
  26. 26. PerlMotion iOS and OS X development using the Perl5 programming language
  27. 27. PerlMotion’s Architecture UIKit Cocoa Touch Frameworks Core Animation Core Audio Core Data Perl5 x Cocoa Touch Frameworks Bindings Library LLVM IR (Links Perl5 codes and Cocoa Touch Frameworks) Perl5 Codes Compiler::Lexer Compiler::Parser Compiler::CodeGenerator::LLVM iOS Simulator iOS DeviceMacOSX Generates Native Code for each target Architecture
  28. 28. DEMO • HelloWorldby PerlMotion
  29. 29. Current Status Still not support functions Symbol Management System : Glob Garbage Collection Regexp Three Term Operator etc.. Not support functions Dynamic evaluation like eval, s/../../e, require Supported functions Primitive Types : Int, double, String, Array, Hash, ArrayRefence, HashReference, IOHandler, BlessedObject … Operators : binary operator, single term operator … BuiltinFunction : print, say, shift, push, sin, open … Variable Definition, Function Definition OOP System : package, bless, @ISA
  30. 30. ROADMAP • Supporting deployment on device • Preparing debugging environment – (stdout/stderr/gdb..etc) • Supporting existing framework (e.g. UIKit) • Supporting CPAN modules written by Pure Perl 2014/4 Will release at April, 2014!
  31. 31. Welcome your Contribution! • I want discuss design or objective – Especially, I need advice on “What Perl community likes?” (Naming rule, etc…) https://github.com/goccy/perl-motion.git
  32. 32. Conclusion at Section 1 and 2 • Perl5 codes Run Anywhere!!! – iOS,OSX,Web Browser and others • Welcome your Contribution – Compiler::Lexer • https://github.com/goccy/p5-Compiler-Lexer.git – Compiler::Parser • https://github.com/goccy/p5-Compiler-Parser.git – Compiler::CodeGenerator::LLVM • https://github.com/goccy/p5-Compiler-CodeGenerator.git – Compiler::Tools::Transpiler • https://github.com/goccy/p5-Compiler-Tools-Transpiler.git – PerlMotion • https://github.com/goccy/perl-motion.git
  33. 33. • Application2) – Static analysis tool Section3
  34. 34. Compiler::Tools::CopyPasteDetector • Detect Copy and Paste for Perl5 • Features – Fast (detecting Engine written by C++ with POSIX Thread) – Accuracy (based on Compiler::Lexer and B::Deparse) – High functionality Visualizer (some metrics or scattergram etc…)
  35. 35. DEMO • Detect Copy & Paste of Catalyst and Mojolicious
  36. 36. Another Modules • Perl::MinimumVersion::Fast(@tokuhirom) – Find a minimum required version of perl for Perl code • Test::LocalFunctions::Fast(@papix) – Detect unused local function Compiler::Lexer or Compiler::Parser provide that you can write faster modules than existing module that uses PPI
  37. 37. Future plan • Perl::Metrics::Simple::Fast – Will release faster module than Perl::Metrics::Simple by using Compiler::Lexer and Compiler::Parser • Next future, I will CPANize of static alnalysis module that has been used at mixi – It includes rich Visualizer
  38. 38. Conclusion at Section 3 • Introduction of copy and paste detecting tool for Perl5 – Compiler::Tools::CopyPasteDetector • Compiler::Lexer or Compiler::Parser provide that you can write faster modules than existing module that uses PPI
  39. 39. Finally Compiler::Lexer Compiler::Parser Compiler::CodeGenerator ::LLVM Type Inference Engine Static Typing - Coming soon - gperl will assemble these brand new modules...
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×