Perl at SkyCon'12
Upcoming SlideShare
Loading in...5
×
 

Perl at SkyCon'12

on

  • 2,821 views

Slides for my talk at SkyCon'12 in Limerick.

Slides for my talk at SkyCon'12 in Limerick.
Here I've squeezed four talks into one, covering a lot of ground quickly, so I've included links to more detailed presentations and other resources.

Statistics

Views

Total Views
2,821
Views on SlideShare
2,763
Embed Views
58

Actions

Likes
4
Downloads
7
Comments
0

12 Embeds 58

http://pesome.com 25
https://twitter.com 7
http://fr.slideshare.net 6
http://lanyrd.com 6
http://prueba765476.wordpress.com 4
http://de.slideshare.net 2
http://haverstockmedia.com 2
http://www.onlydoo.com 2
http://vrindavijayan.blogspot.de 1
http://vrindavijayan.blogspot.in 1
https://si0.twimg.com 1
http://www.linkedin.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution-NonCommercial-ShareAlike LicenseCC Attribution-NonCommercial-ShareAlike LicenseCC Attribution-NonCommercial-ShareAlike License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Perl at SkyCon'12 Perl at SkyCon'12 Presentation Transcript

  • http://xkcd.com/224/
  • Perl Code ProfilingMemory Profiling Perl 5 Perl 6 Tim Bunce - SkyCon’12
  • Devel::NYTProfPerl Source Code Profiler Tim Bunce - SkyCon’12
  • Devel::DProf Is Broken$ perl -we print "sub s$_ { sqrt(42) for 1..100 }; s$_({});n" for 1..1000 > x.pl$ perl -d:DProf x.pl$ dprofpp -r Total Elapsed Time = 0.108 Seconds Real Time = 0.108 Seconds Exclusive Times %Time ExclSec CumulS #Calls sec/call Csec/c Name 9.26 0.010 0.010 1 0.0100 0.0100 main::s76 9.26 0.010 0.010 1 0.0100 0.0100 main::s323 9.26 0.010 0.010 1 0.0100 0.0100 main::s626 9.26 0.010 0.010 1 0.0100 0.0100 main::s936 0.00 - -0.000 1 - - main::s77 0.00 - -0.000 1 - - main::s82
  • Profiling 101 The Basics
  • What To Measure? CPU Time Real TimeSubroutines ? ?Statements ? ?
  • Devel::NYTProf Does it all
  • Running NYTProf perl -d:NYTProf ... perl -MDevel::NYTProf ...Configure profiler via the NYTPROF env var perldoc Devel::NYTProf for the detailsTo profile code that’s invoked elsewhere: PERL5OPT=-d:NYTProf NYTPROF=file=/tmp/nytprof.out:addpid=1:...
  • Reporting: KCachegrind• KCachegrind call graph - new and cool - contributed by C. L. Kao. - requires KCachegrind $ nytprofcg # generates nytprof.callgraph $ kcachegrind # load the file via the gui
  • KCachegrind
  • Reporting: HTML• HTML report - page per source file, annotated with times and links - subroutine index table with sortable columns - interactive Treemap of subroutine times - generates Graphviz dot file of call graph - -m (--minimal) faster generation but less detailed$ nytprofhtml # writes HTML report in ./nytprof/...$ nytprofhtml --file=/tmp/nytprof.out.793 --open
  • Summary Links to annotated source codeLink to sortable table of all subs Timings for perl builtins
  • Color coding based on Overall time spent in and below this subMedian Average Deviationrelative to rest of this file (in + below)Time between starting this perlstatement and starting the next.So includes overhead of calls to perl subs. Timings for each location calling into, or out of, the subroutine
  • Treemap showing relative proportions of exclusive time Boxes represent subroutines Colors only used to show packages (and aren’t pretty yet)Hover over box to see details Click to drill-down one level in package hierarchy
  • Calls between packages
  • Calls to/from/within package
  • Questions? For more details seeSlides: http://www.slideshare.net/Tim.Bunce/develnytprof-v4-at-yapceu-201008-4906467 Screencast: http://blip.tv/timbunce/devel-nytprof-yapc-asia-2012-6376582
  • Perl Memory Use Tim Bunce @ SkyCon’12
  • Ouch!$ perl some_script.plOut of memory!$$ perl some_script.plKilled.$$ perl some_script.pl$Someone shouts: "Hey! My process has been killed!"$ perl some_script.pl[...later...] "Umm, whats taking so long?"
  • Process Memory
  • $ perl -e system("cat /proc/$$/stat") # $$ = pid4752 (perl) S 4686 4752 4686 34816 4752 4202496 536 0 0 0 0 0 0 0 20 0 1 0 62673440 123121664 44018446744073709551615 4194304 4198212 140735314078128 140735314077056 140645336670206 0 0 134 018446744071579305831 0 0 17 10 0 0 0 0 0 0 0 0 0 0 4752 111 111 111$ perl -e system("cat /proc/$$/statm")30059 441 346 1 0 160 0$ perl -e system("ps -p $$ -o vsz,rsz,sz,size") VSZ RSZ SZ SZ120236 1764 30059 640$ perl -e system("top -b -n1 -p $$")... PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND13063 tim 20 0 117m 1764 1384 S 0.0 0.1 0:00.00 perl$ perl -e system("cat /proc/$$/status")...VmPeak:! 120236 kBVmSize:! 120236 kB <- total (code, libs, stack, heap etc.)VmHWM:! 1760 kBVmRSS:! 1760 kB <- how much of the total is resident in physical memoryVmData:! 548 kB <- data (heap)VmStk:! 92 kB <- stackVmExe:! 4 kB <- codeVmLib:! 4220 kB <- libs, including libperl.soVmPTE:! 84 kBVmPTD:! 28 kBVmSwap:! 0 kB... Further info on unix.stackexchange.com
  • C Program Code int main(...) { ... } Read-only Data eg “String constants” Read-write Data un/initialized variables Heap (not to scale!) Shared Lib Code Shared Lib R/O Data repeated for each libShared Lib R/W Data // C Stack (not the perl stack) System
  • $ perl -e system("cat /proc/$$/maps")address perms ... pathname00400000-00401000 r-xp ... /.../perl-5.NN.N/bin/perl00601000-00602000 rw-p ... /.../perl-5.NN.N/bin/perl0087f000-008c1000 rw-p ... [heap]7f858cba1000-7f8592a32000 r--p ... /usr/lib/locale/locale-archive-rpm7f8592c94000-7f8592e1a000 r-xp ... /lib64/libc-2.12.so7f8592e1a000-7f859301a000 ---p ... /lib64/libc-2.12.so7f859301a000-7f859301e000 r--p ... /lib64/libc-2.12.so7f859301e000-7f859301f000 rw-p ... /lib64/libc-2.12.so7f859301f000-7f8593024000 rw-p ......other libs...7f8593d1b000-7f8593e7c000 r-xp ... /.../lib/5.NN.N/x86_64-linux/CORE/libperl.so7f8593e7c000-7f859407c000 ---p ... /.../lib/5.NN.N/x86_64-linux/CORE/libperl.so7f859407c000-7f8594085000 rw-p ... /.../lib/5.NN.N/x86_64-linux/CORE/libperl.so7f85942a6000-7f85942a7000 rw-p ...7fff61284000-7fff6129a000 rw-p ... [stack]7fff613fe000-7fff61400000 r-xp ... [vdso]ffffffffff600000-ffffffffff601000 r-xp ... [vsyscall]
  • $ perl -e system("cat /proc/$$/smaps") # note ‘smaps’ not ‘maps’address perms ... pathname...7fb00fbc1000-7fb00fd22000 r-xp ... /.../5.10.1/x86_64-linux/CORE/libperl.soSize: 1412 kB <- size of executable code in libperl.soRss: 720 kB <- amount thats currently in physical memoryPss: 364 kBShared_Clean: 712 kBShared_Dirty: 0 kBPrivate_Clean: 8 kBPrivate_Dirty: 0 kBReferenced: 720 kBAnonymous: 0 kBAnonHugePages: 0 kBSwap: 0 kBKernelPageSize: 4 kBMMUPageSize: 4 kB... repeated for every segment ...... repeated for every segment ...
  • Memory Pages✦ Process view: ✦ Single large memory space. Simple.✦ Operating System view: ✦ Memory is divided into pages ✦ Pages are loaded to physical memory on demand ✦ Mapping can change without the process knowing
  • C Program Code Read-only Data Memory is divided into pages Page size is typically 4KB Read-write Data Heap ← Page ‘resident’ in physical memory ← Page not resident RSS “Resident Set Size” is how much process memory is Shared Lib Code currently in physical memoryShared Lib R/O DataShared Lib R/W Data C Stack System
  • Key Point✦ Don’t use Resident Set Size (RSS) ✦ It can shrink even while the process size grows.✦ Heap size or Total memory size is a good indicator.
  • The Heap
  • Heap ← Your perl stuff goes here • Heap is managed by malloc() • Memory freed is rarely returned to the operating system • Heap grows but rarely shrinks
  • Perl Data AnatomyInteger (IV)String (PV)Numberwith astring Head Body Data Illustrations from illguts
  • Array(IV)Hash(HV)
  • Glob (GV) Symbol Table (Stash)Sub (CV) lots of tiny chunks!
  • Devel::Peek• Gives you a textual view of data $ perl -MDevel::Peek -e %a = (42 => "Hello World!"); Dump(%a) SV = IV(0x1332fd0) at 0x1332fe0 REFCNT = 1 FLAGS = (TEMP,ROK) RV = 0x1346730 SV = PVHV(0x1339090) at 0x1346730 REFCNT = 2 FLAGS = (SHAREKEYS) ARRAY = 0x1378750 (0:7, 1:1) KEYS = 1 FILL = 1 MAX = 7 Elt "42" HASH = 0x73caace8 SV = PV(0x1331090) at 0x1332de8 REFCNT = 1 FLAGS = (POK,pPOK) PV = 0x133f960 "Hello World!"0 CUR = 12 <= length in use LEN = 16 <= amount allocated
  • Devel::Size• Gives you a measure of the size of a data structure $ perl -MDevel::Size=total_size -le print total_size( 0 ) 24 $ perl -MDevel::Size=total_size -le print total_size( [] ) 64 $ perl -MDevel::Size=total_size -le print total_size( {} ) 120 $ perl -MDevel::Size=total_size -le print total_size( [ 1..100 ] ) 3264• Is very fast, and accurate for most simple data types.• Has limitations and bugs, but is the best tool we have.
  • Memory Profiling
  • What?✦ Track memory size over time?✦ See where memory is allocated and freed?✦ Experiments with Devel::NYTProf✦ Turned out to not seem useful
  • Space in Hiding✦ Perl tends to consume extra memory to save time✦ This can lead to surprises, for example: ✦ sub foo { my $var = "X" x 10_000_000; } foo(); # ~20MB still used after return! ✦ sub bar{ my $var = "X" x 10_000_000; bar($_[0]-1) if $_[0]; # recurse } bar(50); # ~1GB still used after return!
  • My Plan
  • The Plan✦ Extend Devel::Size✦ Add a C-level callback hook✦ Add some kind of "data path name" for the callback to use✦ Add a function to Devel::Size to return the size of everything✦ Stream the data to disk✦ Write tools to visualize the data✦ Add multi-phase scan 1. scan symbol tables, skip where ref count > 1 2. process the skipped items 3. scan arenas for other values (e.g. leaks)✦ Write tool to compare two sets of data
  • The Status✓ Add a C-level callback hook✓ Add some kind of "data path name" for the callback to use✓ Add a function to Devel::Size to return the size of everything.✓ Stream the data to disk✓ Write tools to visualize the data • Will become a separate distribution • “Devel::SizeMeGraph”? • Source repo available by Sunday • Ready to demonstrate
  • Demonstration
  • Questions? For more details seeFull slides: http://www.slideshare.net/Tim.Bunce/perl-memory-use-yapcasia2012Screencast: http://blip.tv/timbunce/perl-memory-use-and-devel-sizeme-at-yapc- asia-2012-6381282 Tim.Bunce@pobox.com http://blog.timbunce.org @timbunce
  • PerlA Summary of the Onions Tim Bunce – SkyCon’12
  • 2007
  • - Perl 5 isn’t the new kid on the block - Perl is 25 years old - Perl 5 is 18 years old- A mature language with a mature culture
  • From “The State of the Onion 10” by Larry Wall, 2006
  • - Perl is 25 years old- Perl 5 is 18 years old- Perl 6 was conceived 12 years ago- Perl 6 “Rakudo Star” is 2 years old
  • “CPAN is my language, Perl is my VM”- Vast library of free code modules on “CPAN”- Over 5,100 active authors (making releases)- Over 26,000 distributions (110,000 modules)- ~1,000 uploads per month (by ~500 authors)- ~250 new distributions per month- Automated smoke testing for all uploads
  • search.cpan.org / metacpan.org
  • Dependency Analysis available for all Modules http://deps.cpantesters.org/?module=Moose;perl=latest
  • Automated Smoke Testinghttp://matrix.cpantesters.org/?dist=Moose%202.0603
  • Perl 5 Development- 5.10 – 2007- 5.12 – 2010- 5.14 – 2011- 5.16 – 2012- 5.17.4 latest monthly development release- 5.18 – due May 2013
  • Perl 5 New Features- Refactored internals many fixes, more speed, less memory- New language features (state, say, //, autodie, ...)- Language feature management- Unicode 6.1 and many new Unicode features- Many powerful new regex features See http://www.slideshare.net/rjbs/whats-new-in-perl-v510-v516
  • http://xkcd.com/208/
  • Demo Regexp::DebugerSee http://www.youtube.com/watch?v=zcSFIUiMgAs
  • A Culture of Testing-2002: Perl 5.8.0 had 26,725 core tests +41,666 more for bundled libraries etc.-2007: Perl 5.10.0 has 78,883 core tests +109,427-2010: Perl 5.11.5 has 191,008 core tests +167,015-2012: Perl 5.16.0 has 262,370 core tests +261,776
  • Perl 6Another member of the Perl language family
  • Learn it once, use it many times. Learn asyou go. Many acceptable levels ofcompetence. Multiple ways to say the samething. No shame in borrowing. Indeterminatedimensionality. Local ambiguity is okay. Natural LanguagePunctuation by prosody and inflection. Principles in PerlDisambiguation by number, case and wordorder. Topicalization. Discourse structure.Pronominalization. No theoretical axes togrind. Style not enforced except by peerpressure. Cooperative design. “Inevitable”Divergence. http://www.wall.org/~larry/natural.html
  • Timeline2000: Perl 6 conceived (after a smashed coffee mug)2001-2004: Initial design docs2005: First prototype, pugs (in Haskell)2005+ Continual radical evolution (the whirlpool)2010: 1st Rakudo Star release2012: 17th Rakudo Star release Mostly implemented in Perl6 (>60% and rising)
  • “If wed done Perl 6 on a schedule, youd have it bynow. And it would be crap.” —Larry Wall, 2002“Do it right.” and “Its ready when its ready.” Freedom to explore deeply and change“Truly radical and far-reaching improvements overthe past few years.”
  • “Weve spent a decade stealing the very best ideasfrom the best programming languages, and makingthem simple and practical for mortal developers touse.” —Damian Conway
  • “Should be called Perl 8 or Perl 9”“feedback at many levels from multiple implementations”
  • Multiple Implementations Two main implementations: Rakudo - built on Parrot Compiler Toolchain Niecza - targeting the CLR (.NET and Mono) Plus: STD, viv, Perlito, Pugs and others Work on a JVM implementation is starting All sharing a common test suite See http://perl6.org/compilers
  • Some Perl 6 FeaturesRich set of operators and meta-operatorsRich type systemRich object system, including roles/traitsA full Meta Object ProtocolMultiple dispatch using types and expressive signaturesGradual typingLazy lists and iterators, with controllable eagernessDeep introspectionNative Call InterfaceRepresentational polymorphismPowerful matching and parsing with subclassable grammars
  • Series and Reduction‣ say 1, 2, 4 ... 1024; 1 2 4 8 16 32 64 128 256 512 1024‣ my @fib = 1, 1, *+* ... *; # infinite say @fib[^10]; # 0..9 1 1 2 3 5 8 13 21 34 55 89‣ say [*] 1..10; # reduction, use any operator 3628800‣ sub postfix:<!>($n) { [*] 1..$n } say 10! 3628800
  • Multiple Dispatch‣ multi fact(0) { 1 } multi fact($n) { $n * fact($n – 1) }‣ multi fib(0) { 0 } multi fib(1) { 1 } multi fib($n) { fib($n – 1) + fib($n – 2) }‣ multi quicksort([]) { () } multi quicksort([$pivot, *@rest]) { my @before = @rest.grep(* < $pivot); my @after = @rest.grep(* >= $pivot); (quicksort(@before), $pivot, quicksort(@after)) }
  • Native Call Interfacesub PQexecPrepared( - OpaquePointer $conn, Str $statement_name, Int $n_params, CArray[Str] $param_values, CArray[int] $param_length, CArray[int] $param_formats, Int $resultFormat) - returns OpaquePointer is native(libpq) { ... }Also supports structures and callbacks.
  • my @suits = < ♣ ♢ ♡ ♠ >;my @ranks = 2..10, < J Q K A >;# concatenate each rank with each suitmy @deck = @ranks X~ @suits;# create hash of card to points valuemy %points = @deck Z ( (2..10, 10, 10, 10, 11) X+ (0,0,0,0) );# grab five cards from the deckmy @hand = @deck.pick(5);# display my handsay ~@hand;# tell me how many points its worthsay [+] %points{@hand};
  • @xyz»++ # increment all elements of @xyz@x = @a »min« @b # @x is smallest of @a and @b$mean = ([+] @a) / @a # calculate mean of @a$sumsq = [+] (@x »**» 2) # sum of squares of @x$fact = [*] 1..$n # $n factorialfor %hash.kv -> $k, $v { say "$k: $v" }
  • Example Module JSON::Tiny https://github.com/moritz/json
  • module JSON::Tiny;proto to-json($) is export {*}multi to-json(Real:D $d) { ~$d }multi to-json(Bool:D $d) { $d ?? true !! false; }multi to-json(Str:D $d) {    "    ~ $d.trans([", , "b", "f", "n", "r", "t"]            => [", , b, f, n, r, t])            .subst(/<-[c32..c126]>/, { ord(~$_).fmt(u%04x) }, :g)    ~ "}multi to-json(Positional:D $d) {  return [ ~ $d.map(&to-json).join(, ) ~ ];}multi to-json(Hash:D $d) { return { ~ $d.map({ to-json(.key)~ : ~to-json(.value) }).join(, )~ };}multi to-json(Any:U $) { null }multi to-json(Any:D $s) { die "Cant serialize an object of type "~$s.WHAT.perl}use JSON::Tiny::Actions;use JSON::Tiny::Grammar;sub from-json($text) is export {    my $a = JSON::Tiny::Actions.new();    my $o = JSON::Tiny::Grammar.parse($text, :actions($a));    return $o.ast;}
  • grammar JSON::Tiny::Grammar;rule TOP { ^ [ <object> | <array> ] $ }rule object { { ~ } <pairlist> }rule pairlist { <?> <pair> * % , }rule pair { <?> <string> : <value> }rule array { [ ~ ] <arraylist> }rule arraylist { <?> <value>* % [ , ] }proto token value {*};token value:sym<number> {    -?    [ 0 | <[1..9]> <[0..9]>* ]    [ . <[0..9]>+ ]?    [ <[eE]> [+|-]? <[0..9]>+ ]?}token value:sym<true> { <sym> };token value:sym<false> { <sym> };token value:sym<null> { <sym> };token value:sym<object> { <object> };token value:sym<array> { <array> };token value:sym<string> { <string> }token string { " ~ " ( <str> | <str_escape> )* }token str { <-["tn]>+ }token str_escape { <["/bfnrt]> | u <xdigit>**4 }
  • class JSON::Tiny::Actions;method TOP($/) { make $/.values.[0].ast };method object($/) { make $<pairlist>.ast.hash }method pairlist($/) { make $<pair>>>.ast.flat }method pair($/) { make $<string>.ast => $<value>.ast }method array($/) { make $<arraylist>.ast }method arraylist($/) { make [$<value>>>.ast] }method string($/) {    make $0.elems == 1        ?? ($0[0].<str> || $0[0].<str_escape>).ast        !! join , $0.list.map({ (.<str> || .<str_escape>).ast });}method value:sym<number>($/) { make +$/.Str }method value:sym<string>($/) { make $<string>.ast }method value:sym<true>($/) { make Bool::True }method value:sym<false>($/) { make Bool::False }method value:sym<null>($/) { make Any }method value:sym<object>($/) { make $<object>.ast }method value:sym<array>($/) { make $<array>.ast }method str($/) { make ~$/ }method str_escape($/) {    if $<xdigit> {        make chr(:16($<xdigit>.join));    } else {        my %h = => "", / => "/", b => "b", n => "n", t => "t", f => "f", r => "r", " => """;        make %h{~$/};    }}
  • Perl 6Already full of awesomeHundreds of examples on http://rosettacode.orgDevelopers happy to stay in stealth mode for nowGet ahead of the revolution: http://perl6.org
  • In Summary...
  • Perlhas a massive library of reusable codehas a culture of best practice and testinghas a happy welcoming growing communityhas a great future in Perl 5 and Perl 6is a great language for getting your job done for the last 25 years, and the next 25!
  • Questions?See also http://www.slideshare.net/Tim.Bunce/perl-myths-200909 http://perlmonks.org http://search.cpan.org http://rakudo.org/how-to-help http://blog.timbunce.org @timbunce
  • http://xkcd.com/519/