Perl at SkyCon'12

Perl
Code Proﬁling
Memory Proﬁling
Perl 5
Perl 6

Tim Bunce - SkyCon’12

Devel::NYTProf
Perl Source Code Proﬁler

Tim Bunce - SkyCon’12

Devel::DProf Is Broken
$ perl -we 'print "sub s$_ { sqrt(42) for 1..100 };
s$_({});n" for 1..1000' > x.pl

$ perl -d:DProf x.pl

$ dprofpp -r
Total Elapsed Time = 0.108 Seconds
Real Time = 0.108 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c Name
9.26 0.010 0.010 1 0.0100 0.0100 main::s76
9.26 0.010 0.010 1 0.0100 0.0100 main::s323
9.26 0.010 0.010 1 0.0100 0.0100 main::s626
9.26 0.010 0.010 1 0.0100 0.0100 main::s936
0.00 - -0.000 1 - - main::s77
0.00 - -0.000 1 - - main::s82

What To Measure?

CPU Time Real Time

Subroutines
? ?
Statements
? ?

Running NYTProf
perl -d:NYTProf ...
perl -MDevel::NYTProf ...

Configure profiler via the NYTPROF env var
perldoc Devel::NYTProf for the details

To profile code that’s invoked elsewhere:
PERL5OPT=-d:NYTProf
NYTPROF=file=/tmp/nytprof.out:addpid=1:...

Reporting: KCachegrind
• KCachegrind call graph - new and cool
- contributed by C. L. Kao.
- requires KCachegrind

$ nytprofcg # generates nytprof.callgraph
$ kcachegrind # load the file via the gui

Reporting: HTML
• HTML report
- page per source ﬁle, annotated with times and links
- subroutine index table with sortable columns
- interactive Treemap of subroutine times
- generates Graphviz dot ﬁle of call graph
- -m (--minimal) faster generation but less detailed

$ nytprofhtml # writes HTML report in ./nytprof/...
$ nytprofhtml --file=/tmp/nytprof.out.793 --open

Summary

Links to annotated
source code

Link to sortable table
of all subs
Timings for perl builtins

Color coding based on Overall time spent in and below this sub
Median Average Deviation
relative to rest of this ﬁle (in + below)

Time between starting this perl
statement and starting the next.
So includes overhead of calls to
perl subs. Timings for each location calling into,
or out of, the subroutine

Treemap showing relative
proportions of exclusive time

Boxes represent subroutines
Colors only used to show
packages (and aren’t pretty yet)

Hover over box to see details
Click to drill-down one level
in package hierarchy

Questions?
For more details see
Slides: http://www.slideshare.net/Tim.Bunce/develnytprof-v4-at-yapceu-201008-4906467
Screencast: http://blip.tv/timbunce/devel-nytprof-yapc-asia-2012-6376582

Perl Memory Use
Tim Bunce @ SkyCon’12

Ouch!
$ perl some_script.pl
Out of memory!
$

Killed.
$

$
Someone shouts: "Hey! My process has been killed!"

[...later...] "Umm, what's taking so long?"

$ perl -e 'system("cat /proc/$$/stat")' # $$ = pid
4752 (perl) S 4686 4752 4686 34816 4752 4202496 536 0 0 0 0 0 0 0 20 0 1 0 62673440 123121664 440
18446744073709551615 4194304 4198212 140735314078128 140735314077056 140645336670206 0 0 134 0
18446744071579305831 0 0 17 10 0 0 0 0 0 0 0 0 0 0 4752 111 111 111

$ perl -e 'system("cat /proc/$$/statm")'
30059 441 346 1 0 160 0

$ perl -e 'system("ps -p $$ -o vsz,rsz,sz,size")'
VSZ RSZ SZ SZ
120236 1764 30059 640

$ perl -e 'system("top -b -n1 -p $$")'
...
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13063 tim 20 0 117m 1764 1384 S 0.0 0.1 0:00.00 perl

$ perl -e 'system("cat /proc/$$/status")'
...
VmPeak:! 120236 kB
VmSize:! 120236 kB <- total (code, libs, stack, heap etc.)
VmHWM:! 1760 kB
VmRSS:! 1760 kB <- how much of the total is resident in physical memory
VmData:! 548 kB <- data (heap)
VmStk:! 92 kB <- stack
VmExe:! 4 kB <- code
VmLib:! 4220 kB <- libs, including libperl.so
VmPTE:! 84 kB
VmPTD:! 28 kB
VmSwap:! 0 kB
...
Further info on unix.stackexchange.com

C Program Code int main(...) { ... }
Read-only Data eg “String constants”
Read-write Data un/initialized variables
Heap

(not to scale!)

Shared Lib Code
Shared Lib R/O Data repeated for each lib
Shared Lib R/W Data //

C Stack (not the perl stack)
System

$ perl -e 'system("cat /proc/$$/maps")'
address perms ... pathname
00400000-00401000 r-xp ... /.../perl-5.NN.N/bin/perl
00601000-00602000 rw-p ... /.../perl-5.NN.N/bin/perl

0087f000-008c1000 rw-p ... [heap]

7f858cba1000-7f8592a32000 r--p ... /usr/lib/locale/locale-archive-rpm

7f8592c94000-7f8592e1a000 r-xp ... /lib64/libc-2.12.so
7f8592e1a000-7f859301a000 ---p ... /lib64/libc-2.12.so
7f859301a000-7f859301e000 r--p ... /lib64/libc-2.12.so
7f859301e000-7f859301f000 rw-p ... /lib64/libc-2.12.so
7f859301f000-7f8593024000 rw-p ...

...other libs...

7f8593d1b000-7f8593e7c000 r-xp ... /.../lib/5.NN.N/x86_64-linux/CORE/libperl.so
7f8593e7c000-7f859407c000 ---p ... /.../lib/5.NN.N/x86_64-linux/CORE/libperl.so
7f859407c000-7f8594085000 rw-p ... /.../lib/5.NN.N/x86_64-linux/CORE/libperl.so
7f85942a6000-7f85942a7000 rw-p ...

7fff61284000-7fff6129a000 rw-p ... [stack]

7fff613fe000-7fff61400000 r-xp ... [vdso]
ffffffffff600000-ffffffffff601000 r-xp ... [vsyscall]

$ perl -e 'system("cat /proc/$$/smaps")' # note ‘smaps’ not ‘maps’

address perms ... pathname
...

7fb00fbc1000-7fb00fd22000 r-xp ... /.../5.10.1/x86_64-linux/CORE/libperl.so
Size: 1412 kB <- size of executable code in libperl.so
Rss: 720 kB <- amount that's currently in physical memory
Pss: 364 kB
Shared_Clean: 712 kB
Shared_Dirty: 0 kB
Private_Clean: 8 kB
Private_Dirty: 0 kB
Referenced: 720 kB
Anonymous: 0 kB
AnonHugePages: 0 kB
Swap: 0 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB

... repeated for every segment ...
... repeated for every segment ...

Memory Pages
✦ Process view:
✦ Single large memory space. Simple.
✦ Operating System view:
✦ Memory is divided into pages
✦ Pages are loaded to physical memory on demand
✦ Mapping can change without the process knowing

C Program Code
Read-only Data Memory is divided into pages
Page size is typically 4KB
Read-write Data
Heap

← Page ‘resident’ in physical
memory
← Page not resident

RSS “Resident Set Size”
is how much process memory is
Shared Lib Code currently in physical memory
Shared Lib R/O Data
Shared Lib R/W Data

C Stack
System

Key Point

✦ Don’t use Resident Set Size (RSS)
✦ It can shrink even while the process size grows.
✦ Heap size or Total memory size is a good indicator.

Heap
← Your perl stuff goes here

• Heap is managed by malloc()
• Memory freed is rarely returned to
the operating system
• Heap grows but rarely shrinks

Perl Data Anatomy
Integer
(IV)

String
(PV)

Number
with a
string

Head Body Data Illustrations from illguts

Glob (GV) Symbol Table (Stash)

Sub (CV)

lots of tiny chunks!

Devel::Peek
• Gives you a textual view of data
$ perl -MDevel::Peek -e '%a = (42 => "Hello World!"); Dump(%a)'
SV = IV(0x1332fd0) at 0x1332fe0
REFCNT = 1
FLAGS = (TEMP,ROK)
RV = 0x1346730
SV = PVHV(0x1339090) at 0x1346730
REFCNT = 2
FLAGS = (SHAREKEYS)
ARRAY = 0x1378750 (0:7, 1:1)
KEYS = 1
FILL = 1
MAX = 7
Elt "42" HASH = 0x73caace8
SV = PV(0x1331090) at 0x1332de8
REFCNT = 1
FLAGS = (POK,pPOK)
PV = 0x133f960 "Hello World!"0
CUR = 12 <= length in use
LEN = 16 <= amount allocated

Devel::Size
• Gives you a measure of the size of a data structure
$ perl -MDevel::Size=total_size -le 'print total_size( 0 )'
24

$ perl -MDevel::Size=total_size -le 'print total_size( [] )'
64

$ perl -MDevel::Size=total_size -le 'print total_size( {} )'
120

$ perl -MDevel::Size=total_size -le 'print total_size( [ 1..100 ] )'
3264

• Is very fast, and accurate for most simple data types.
• Has limitations and bugs, but is the best tool we have.

What?

✦ Track memory size over time?
✦ See where memory is allocated and freed?
✦ Experiments with Devel::NYTProf
✦ Turned out to not seem useful

Space in Hiding
✦ Perl tends to consume extra memory to save time
✦ This can lead to surprises, for example:
✦ sub foo {
my $var = "X" x 10_000_000;
}
foo(); # ~20MB still used after return!

✦ sub bar{
my $var = "X" x 10_000_000;
bar($_[0]-1) if $_[0]; # recurse
}
bar(50); # ~1GB still used after return!

The Plan
✦ Extend Devel::Size
✦ Add a C-level callback hook
✦ Add some kind of "data path name" for the callback to use
✦ Add a function to Devel::Size to return the size of everything
✦ Stream the data to disk
✦ Write tools to visualize the data
✦ Add multi-phase scan
1. scan symbol tables, skip where ref count > 1
2. process the skipped items
3. scan arenas for other values (e.g. leaks)
✦ Write tool to compare two sets of data

The Status
✓ Add a C-level callback hook
✓ Add some kind of "data path name" for the callback to use
✓ Add a function to Devel::Size to return the size of everything.
✓ Stream the data to disk
✓ Write tools to visualize the data

• Will become a separate distribution
• “Devel::SizeMeGraph”?
• Source repo available by Sunday
• Ready to demonstrate

Questions?
For more details see
Full slides: http://www.slideshare.net/Tim.Bunce/perl-memory-use-yapcasia2012
Screencast: http://blip.tv/timbunce/perl-memory-use-and-devel-sizeme-at-yapc-
asia-2012-6381282

Tim.Bunce@pobox.com
http://blog.timbunce.org
@timbunce

Perl
A Summary of the Onions

Tim Bunce – SkyCon’12

- Perl 5 isn’t the new kid on the block
- Perl is 25 years old
- Perl 5 is 18 years old
- A mature language with a mature culture

From “The State of the Onion 10” by Larry Wall, 2006

- Perl is 25 years old
- Perl 5 is 18 years old
- Perl 6 was conceived 12 years ago
- Perl 6 “Rakudo Star” is 2 years old

“CPAN is my language,
Perl is my VM”
- Vast library of free code modules on “CPAN”
- Over 5,100 active authors (making releases)
- Over 26,000 distributions (110,000 modules)
- ~1,000 uploads per month (by ~500 authors)
- ~250 new distributions per month
- Automated smoke testing for all uploads

search.cpan.org / metacpan.org

Dependency Analysis available for all Modules
http://deps.cpantesters.org/?module=Moose;perl=latest

Automated Smoke Testing

http://matrix.cpantesters.org/?dist=Moose%202.0603

Perl 5 Development
- 5.10 – 2007
- 5.12 – 2010
- 5.14 – 2011
- 5.16 – 2012
- 5.17.4 latest monthly development release
- 5.18 – due May 2013

Perl 5 New Features

- Refactored internals

many ﬁxes, more speed, less memory
- New language features (state, say, //, autodie, ...)
- Language feature management
- Unicode 6.1 and many new Unicode features
- Many powerful new regex features
See http://www.slideshare.net/rjbs/whats-new-in-perl-v510-v516

Demo
Regexp::Debuger
See http://www.youtube.com/watch?v=zcSFIUiMgAs

A Culture of Testing
-2002: Perl 5.8.0 had 26,725 core tests

+41,666 more for bundled libraries etc.
-2007: Perl 5.10.0 has 78,883 core tests

+109,427

+167,015
+261,776

Perl 6
Another member of the Perl language family

Learn it once, use it many times. Learn as
you go. Many acceptable levels of
competence. Multiple ways to say the same
thing. No shame in borrowing. Indeterminate
dimensionality. Local ambiguity is okay.
Natural Language
Punctuation by prosody and inﬂection.
Principles in Perl
Disambiguation by number, case and word
order. Topicalization. Discourse structure.
Pronominalization. No theoretical axes to
grind. Style not enforced except by peer
pressure. Cooperative design. “Inevitable”
Divergence. http://www.wall.org/~larry/natural.html

Timeline
2000: Perl 6 conceived (after a smashed coffee mug)
2001-2004: Initial design docs
2005: First prototype, pugs (in Haskell)
2005+ Continual radical evolution (the whirlpool)
2010: 1st Rakudo Star release
2012: 17th Rakudo Star release
Mostly implemented in Perl6 (>60% and rising)

“If we'd done Perl 6 on a schedule, you'd have it by
now. And it would be crap.”
—Larry Wall, 2002

“Do it right.” and “It's ready when it's ready.”
Freedom to explore deeply and change

“Truly radical and far-reaching improvements over
the past few years.”

“We've spent a decade stealing the very best ideas
from the best programming languages, and making
them simple and practical for mortal developers to
use.”
—Damian Conway

“Should be called Perl 8 or Perl 9”

“feedback at many levels from multiple implementations”

Multiple Implementations
Two main implementations:
Rakudo - built on Parrot Compiler Toolchain
Niecza - targeting the CLR (.NET and Mono)
Plus: STD, viv, Perlito, Pugs and others
Work on a JVM implementation is starting

All sharing a common test suite

See http://perl6.org/compilers

Some Perl 6 Features
Rich set of operators and meta-operators
Rich type system
Rich object system, including roles/traits
A full Meta Object Protocol
Multiple dispatch using types and expressive signatures
Gradual typing
Lazy lists and iterators, with controllable eagerness
Deep introspection
Native Call Interface
Representational polymorphism
Powerful matching and parsing with subclassable grammars

Series and Reduction
‣ say 1, 2, 4 ... 1024;
1 2 4 8 16 32 64 128 256 512 1024

‣ my @fib = 1, 1, *+* ... *; # infinite
say @fib[^10]; # 0..9
1 1 2 3 5 8 13 21 34 55 89

‣ say [*] 1..10; # reduction, use any operator
3628800

‣ sub postfix:<!>($n) { [*] 1..$n }
say 10!
3628800

Multiple Dispatch
‣ multi fact(0) { 1 }
multi fact($n) { $n * fact($n – 1) }

‣ multi fib(0) { 0 }
multi fib(1) { 1 }
multi fib($n) { fib($n – 1) + fib($n – 2) }

‣ multi quicksort([]) { () }
multi quicksort([$pivot, *@rest]) {
my @before = @rest.grep(* < $pivot);
my @after = @rest.grep(* >= $pivot);
(quicksort(@before), $pivot, quicksort(@after))
}

Native Call Interface
sub PQexecPrepared(
- OpaquePointer $conn,
Str $statement_name,
Int $n_params,
CArray[Str] $param_values,
CArray[int] $param_length,
CArray[int] $param_formats,
Int $resultFormat)

- returns OpaquePointer
is native('libpq')
{ ... }

Also supports structures and callbacks.

my @suits = < ♣ ♢ ♡ ♠ >;
my @ranks = 2..10, < J Q K A >;

# concatenate each rank with each suit
my @deck = @ranks X~ @suits;

# create hash of card to points value
my %points = @deck Z ( (2..10, 10, 10, 10, 11)
X+ (0,0,0,0) );

# grab five cards from the deck
my @hand = @deck.pick(5);

# display my hand
say ~@hand;

# tell me how many points it's worth
say [+] %points{@hand};

@xyz»++ # increment all elements of @xyz

@x = @a »min« @b # @x is smallest of @a and @b

$mean = ([+] @a) / @a # calculate mean of @a

$sumsq = [+] (@x »**» 2) # sum of squares of @x

$fact = [*] 1..$n # $n factorial

for %hash.kv -> $k, $v { say "$k: $v" }

Example Module
JSON::Tiny

https://github.com/moritz/json

module JSON::Tiny;

proto to-json($) is export {*}
multi to-json(Real:D $d) { ~$d }
multi to-json(Bool:D $d) { $d ?? 'true' !! 'false'; }
multi to-json(Str:D $d) {
    '"'
    ~ $d.trans(['"', '', "b", "f", "n", "r", "t"]
            => ['"', '', 'b', 'f', 'n', 'r', 't'])
            .subst(/<-[c32..c126]>/, { ord(~$_).fmt('u%04x') }, :g)
    ~ '"'
}
multi to-json(Positional:D $d) {
  return '[ ' ~ $d.map(&to-json).join(', ') ~ ' ]';
}
multi to-json(Hash:D $d) {
return '{ '~ $d.map({ to-json(.key)~' : '~to-json(.value) }).join(', ')~ ' }';
}
multi to-json(Any:U $) { 'null' }
multi to-json(Any:D $s) { die "Can't serialize an object of type "~$s.WHAT.perl}

use JSON::Tiny::Actions;
use JSON::Tiny::Grammar;

sub from-json($text) is export {
    my $a = JSON::Tiny::Actions.new();
    my $o = JSON::Tiny::Grammar.parse($text, :actions($a));
    return $o.ast;
}

grammar JSON::Tiny::Grammar;

rule TOP { ^ [ <object> | <array> ] $ }
rule object { '{' ~ '}' <pairlist> }
rule pairlist { <?> <pair> * % , }
rule pair { <?> <string> ':' <value> }
rule array { '[' ~ ']' <arraylist> }
rule arraylist { <?> <value>* % [ , ] }

proto token value {*};
token value:sym<number> {
    '-'?
    [ 0 | <[1..9]> <[0..9]>* ]
    [ . <[0..9]>+ ]?
    [ <[eE]> [+|-]? <[0..9]>+ ]?
}
token value:sym<true> { <sym> };
token value:sym<false> { <sym> };
token value:sym<null> { <sym> };
token value:sym<object> { <object> };
token value:sym<array> { <array> };
token value:sym<string> { <string> }

token string { " ~ " ( <str> | <str_escape> )* }
token str { <-["tn]>+ }
token str_escape { <["/bfnrt]> | u <xdigit>**4 }

class JSON::Tiny::Actions;

method TOP($/) { make $/.values.[0].ast };

method object($/) { make $<pairlist>.ast.hash }
method pairlist($/) { make $<pair>>>.ast.flat }
method pair($/) { make $<string>.ast => $<value>.ast }
method array($/) { make $<arraylist>.ast }
method arraylist($/) { make [$<value>>>.ast] }
method string($/) {
    make $0.elems == 1
        ?? ($0[0].<str> || $0[0].<str_escape>).ast
        !! join '', $0.list.map({ (.<str> || .<str_escape>).ast });
}
method value:sym<number>($/) { make +$/.Str }
method value:sym<string>($/) { make $<string>.ast }
method value:sym<true>($/) { make Bool::True }
method value:sym<false>($/) { make Bool::False }
method value:sym<null>($/) { make Any }
method value:sym<object>($/) { make $<object>.ast }
method value:sym<array>($/) { make $<array>.ast }
method str($/) { make ~$/ }
method str_escape($/) {
    if $<xdigit> {
        make chr(:16($<xdigit>.join));
    } else {
        my %h = '' => "", '/' => "/", 'b' => "b", 'n' => "n",
't' => "t", 'f' => "f", 'r' => "r", '"' => """;
        make %h{~$/};
    }
}

Perl 6

Already full of awesome
Hundreds of examples on http://rosettacode.org
Developers happy to stay in stealth mode for now
Get ahead of the revolution: http://perl6.org

Perl
has a massive library of reusable code
has a culture of best practice and testing
has a happy welcoming growing community
has a great future in Perl 5 and Perl 6
is a great language for getting your job done
for the last 25 years, and the next 25!

Questions?
See also http://www.slideshare.net/Tim.Bunce/perl-myths-200909

http://perlmonks.org
http://search.cpan.org
http://rakudo.org/how-to-help
http://blog.timbunce.org
@timbunce

Perl at SkyCon'12

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Perl at SkyCon'12

Similar to Perl at SkyCon'12 (20)

More from Tim Bunce

More from Tim Bunce (8)

Recently uploaded

Recently uploaded (20)

Perl at SkyCon'12