PHP 5 memory
Understand and master

Hello everybody
 Julien PAULI
 Programming with PHP since early 2000s
 Programming in C
 PHP Internals programmer/reviewer
 PHP 5.5 & 5.6 Release Manager
 @julienpauli
 Tech blog at jpauli.github.io
 jpauli@php.net

What about you ?
 Got some Unix/Linux understandings ?
 Have already experienced C programming ?
 Have already experienced PHP programming ?

What we'll cover together
 Memory , what's that ?
 bytes, stack, heap, etc.
 Measuring a process memory consumption
 memory image map analysis
 Understanding PHP memory consumption
 Zend Memory Manager coming
 Measuring PHP memory consumption
 from PHP land or from system land

Memory from hardware

Memory from software

Memory in a user process
 The Virtual Memory image is divided in segments
 text
 data
 heap
 stack

Memory usage can grow
 Stack will grow as functions will get called
 And will narrow as the calls stop and return
 Heap will grow as the programmer will decide
 Using dynamic allocation functions (malloc, mmap)
 Programmer has to free memory by hand
 If not : memory leak

Monitoring memory

Linux memory monitoring
 'top' or /proc FS
> cat /proc/28754/status
VmPeak: 20452 kB
VmSize: 20324 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 316 kB
VmRSS: 316 kB
VmData: 16440 kB
VmStk: 136 kB
VmExe: 4 kB
VmLib: 1664 kB
VmPTE: 28 kB
VmSwap: 0 kB
Size of the VM map (out of total mem)
Resident Set Size : Size actually in PM
Size of the data segment in VM
Size of the stack segment in VM
Size of the text segment in VM
pid

Going even deeper
 Let's show the detailed process memory map :
> cat /proc/28754/smaps
shared segment
private mem
shared mem

PHP !

PHP, just a process
 PHP is just a process like any other
 You then can monitor its memory usage by asking
the OS
<?php
passthru(sprintf('cat /proc/%d/status', getmypid()));
<?php
function heap() {
return shell_exec(sprintf('grep "VmRSS:" /proc/%d/status', getmypid()));
}

The PHP model
Startup
Shutdown
Request #1
Request #2
Request #3
Startup
Shutdown
Request #1
Request #2
Request #3
php-fpm worker#1 php-fpm worker#2

The PHP model
 One same process may treat many requests
 If the process leaks memory, you'll suffer from that
 Request n+1 must know nothing about request n
 Need to "flush" the request-allocated memory
 Need to track request-bound memory claims
 ZendMM is the layer that does the job
 Share-nothing architecture : by design.

Inside PHP
 PHP internal request-bound heap allocator :
Zend Memory Manager

Why a custom layer ?
 ZendMM
 Allows monitoring of request-bound heap usage by
basically counting malloc/free calls
 Allows the PHP user to limit the heap mem usage
 Allows caching of alloced blocks to prevent memory
fragmentation and syscalls
 Allows preallocating blocks of well-known-sizes for
PHP internal structures to fit-in in an aligned way
 Ease memory leaks debugging in core and exts

ZendMM guards and leak tracker
 Only works in debug mode PHP
 report_memleaks=1 in php.ini
 Does NOT replace valgrind

Zend MM test script
<?php
ini_set('memory_limit', -1); /* unlimited ZendMM heap */
function heap() {
return shell_exec(sprintf('grep "VmRSS:" /proc/%d/status', getmypid()));
}
echo heap();
$a = range(1, 1024*1024); /* Stress memory by allocating */
echo heap();
unset($a); /* Stress memory by freeing */
echo heap();

Zend MM launch test
> time USE_ZEND_ALLOC=1 php zendmm.php
VmRSS: 9640 kB
VmRSS: 159296 kB
VmRSS: 10068 kB
real 0m0.237s
user 0m0.148s
sys 0m0.080s
> time USE_ZEND_ALLOC=0 php zendmm.php
VmRSS: 9608 kB
VmRSS: 148988 kB
VmRSS: 140804 kB
real 0m0.288s
user 0m0.176s
sys 0m0.108s

Valgrind tests for Symfony2 command
 app/console runs lots of PHP code under SF2
>USE_ZEND_ALLOC=1 valgrind php app/console debug:container
total heap usage: 84,000 allocs, 84,000 frees, 25,966,154 bytes allocated
>USE_ZEND_ALLOC=0 valgrind php app/console debug:container
total heap usage: 813,570 allocs, 813,570 frees, 74,579,732 bytes allocated

ZendMM internals

ZendMM benefits
 Better memory management
 More memory efficient
 Far less malloc/free calls
 Less context switches, less Kernel stress
 Less CPU usage
 Less heap fragmentation / compaction
 A PHP ~10% faster with ZendMM enabled
 Really depends on use-case

Memory manager internals
 Layer on top of the heap
 Will allocate memory from the heap by chunks of
customizable size (segments)
 Will use a customizable low level heap (malloc /
mmap)

A quick word on ZendMM internals
 ZEND_MM_SEG_SIZE env variable to customize
segment size
 Must be power-of-two aligned
 Default value is 256Kb

ZendMM in PHP user land
 memory_limit (INI setting)
 memory_get_usage(true)
 Returns the size of all the allocated segments
 memory_get_usage()
 Returns the occupied size in all the allocated
segments
 memory_get_[peak]_usage([real])
 Returns the max memory that has been
allocated/used. Could have been freed meantime

ZendMM memory monitoring

ini_set('memory_limit', -1); // unlimited memory
function get_mem_stats() {
printf("Memory usage %.2f Kbn", memory_get_usage() / 1024);
if ($segSize = getenv('ZEND_MM_SEG_SIZE')) {
printf("Heap segmentation : %d segments of %d bytes (%d Kb used)n",
memory_get_usage(1)/$segSize, $segSize, memory_get_usage(1)/1024);
}
}
get_mem_stats();
$a = str_repeat('a', 1024*1024*10); // 10 Mb
get_mem_stats();
Adjusting heap segment size

Adjusting heap segment size
 Segment size = 256Kb (default value)
 Segment size = 4Mb
$> ZEND_MM_SEG_SIZE=262144 php /tmp/mem.php
Memory usage 221.26 Kb
Heap segmentation : 1 segments of 262144 bytes (256 Kb used)
Memory usage 10461.47 Kb
Heap segmentation : 42 segments of 262144 bytes (10752 Kb used)
get_mem_stats();
$a = str_repeat('a', 1024*1024*10); // 10 Mb
get_mem_stats();
$> ZEND_MM_SEG_SIZE=4194304 php /tmp/mem.php
Memory usage 221.26 Kb
Heap segmentation : 1 segments of 4194304 bytes (4096 Kb used)
Memory usage 10461.47 Kb
Heap segmentation : 4 segments of 4194304 bytes (16384 Kb used)

memory_get_usage() ?
 memory_get_usage() tells you how much your
allocated blocs consume
 They usually don't fill their segment entirely
 Thus memory_get_usage(true) shows more
 This doesn't count stack
 This does only count request-bound memory
 This doesn't count linked libraries present in the
process memory map
 This doesn't show non-request-bound memory

Real memory image
 valgrind massif

Massif memory monitoring
--------------------------------------------------------------------------------
n time(ms) total(B) useful-heap(B) extra-heap(B) stacks(B)
--------------------------------------------------------------------------------
0 0 0 0 0 0
1 309 56 37 19 0
2 341 384 357 27 0
3 376 1,343,992 1,340,371 3,621 0
4 408 1,360,480 1,355,289 5,191 0
5 443 1,376,512 1,367,265 9,247 0
6 471 1,753,976 1,735,671 18,305 0
7 505 2,345,152 2,275,927 69,225 0
8 535 2,405,224 2,329,798 75,426 0
9 570 2,433,408 2,348,849 84,559 0
10 613 2,592,912 2,461,650 131,262 0
11 657 2,681,240 2,534,247 146,993 0

Massif details
94.52% (2,534,247B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->39.11% (1,048,576B) 0x87A7BB: zend_interned_strings_init (zend_string.c:48)
|
->17.70% (474,464B) 0x866449: _zend_hash_quick_add_or_update (zend_alloc.h:95)
| ->16.48% (441,936B) 0x85F66D: zend_register_functions (zend_API.c:2138)
|
->11.93% (320,000B) 0x878625: gc_init (zend_gc.c:124)
| ->11.93% (320,000B) 0x8585DF: OnUpdateGCEnabled (zend.c:82)
| ->11.93% (320,000B) 0x86E353: zend_register_ini_entries (zend_ini.c:208)

Recommendations / statements
 Understand memory_get_usage()
 It only shows request-bound allocations, not
persistent allocations (that reside through requests)
 PHP extensions may allocate persistent memory
 Do NOT activate extensions you will not use
 Libraries used by PHP may also allocate persistent
memory
 Use your OS to monitor your process memory
accurately

Playing with memory in PHP user land

Master your PHP mem usage
 In PHP land ...
 all variable types consume memory
 every script asked for compilation will eat memory
 This memory will be allocated using ZendMM
 The memory for parsed script is freed when the
request ends
 The memory for user variable is freed when the
data is not used any more
 And here comes the challenge
 When isn't the data needed any more ??

Compilation eats memory
 Compiling a script eats request-bound memory
 If you compile a class, that eats lots of memory
 You'd better use that class at runtime
 Use an autoloader to be sure
<?php
$mem = memory_get_usage();
require __DIR__ . '/../vendor/autoload.php';
require __DIR__ . '/../src/Symfony/Component/DependencyInjection/ContainerBuilder.php'
echo memory_get_usage() - $mem . "n";
php app/bar.php
246,692

PHP Variables
 Variables are internally zval structs

PHP Variables
 What eats memory is what is stored into the zval,
not really the zval itself :
 A huge string
 Tip : a file_get_contents('huge-file') is a huge string
 A complex array or object
 Resources don't really consume mem in zval
<?php
/* This consumes sizeof(zval) + 1024 bytes */
$a = str_repeat('a', 1024);

PHP Variables
 What you want to avoid is have PHP duplicate the
zval
 But PHP is kind about that
 What you want to happen in PHP freeing the
memory ASAP
 Should you know when PHP duplicates or frees zval
, that's the most important !

zvals and refcount
 PHP simply counts how many symbols (PHP
vars) point to a zval
 This is called the refcount
<?php
$a = "foo";
$b = $a;

zvals and refcount
 PHP uses a CopyOnWrite (COW) system for
zvals
 Memory is saved
 Memory gets allocated only on changes
<?php
$a = "foo";
$b = $a;
$a = 17;

zvals and refcount
 PHP frees memory for a zval
when its refcount reaches 0
 Yes, unset() just refcount-- ,
that's all
<?php
$a = "foo";
$b = $a;
$c = $b;
$b = "bar";
unset($a);

No references needed
 You see how smart PHP is with memory ?
 It's been designed with that in mind
 No references needed to hint PHP !
 Don't try to hint PHP with references
 References can lead to adverse effects
 Force PHP to copy a zval
 Prevents PHP from freeing memory of a zval
&

Tracking refcount
 xdebug_debug_zval()
 symfony_zval_info()
namespace Foo;
class A
{
public $var = 'varA';
}
$a = new A();
xdebug_debug_zval('a');
a: (refcount=1, is_ref=0)=class FooA { public $var =
(refcount=2, is_ref=0)='varA'; }

Tracking refcount
namespace Foo;
class C
{
public $b;
public function __construct(B $b) {
$this->b = $b;
}
}
$c = new C($b = new B);
xdebug_debug_zval('c');
c: (refcount=1, is_ref=0)=class FooC { public $b = (refcount=2, is_ref=0)=class FooB { } }
namespace Foo;
class B
{
}
unset($b);
xdebug_debug_zval('c');
c: (refcount=1, is_ref=0)=class FooC { }
c: (refcount=1, is_ref=0)=class FooC { public $b = (refcount=1, is_ref=0)=class FooB { } }
unset($c->b);
xdebug_debug_zval('c');

Garbage collector ?
 As of PHP5.3 , a garbage collector exists
 Used to free circular references
 And that's all !
 PHP already frees itself your vars as their
refcount reaches 0
 And it's always been like that

Circular references
$a = new A;
$b = new B
(object) 'A'
refcount = 1
$a (object) 'B'
refcount = 1
$b

Circular references
$a->b = $b;
$b->a = $a;
(object) 'A'
refcount = 2
$a (object) 'B'
refcount = 2
$b
$b->a $a->b

Circular references
 Objects are still in memory but no more PHP var
point to them
 We can call that a "PHP Userland memory leak"
unset($a, $b);
(object) 'A'
refcount = 1
$b->a (object) 'B'
refcount = 1
$a->b

Garbage collector
$a = new A;
$b = new B;
$a->b = $b;
$b->a = $a;
unset($a, $b);
echo gc_collect_cycles(); // 2

PHP references

PHP references main line
 Using references (&) in PHP can really fool you
 They usually force the engine to duplicate memory
containers
 Which is bad for performance
 Especially when the var container is huge

References mismatch
<?php
function foo($data)
{
echo "in function : " . memory_get_usage() . "n";
}
echo "Initial memory : " . memory_get_usage() . "n";
$r = range(1, 1024);
echo "Array created : " . memory_get_usage() . "n";
foo($r);
echo "End of function " . memory_get_usage() . "n";
Initial memory : 227.136
Array created : 374.912
in function : 374.944
End of function 374.944

References mismatch
<?php
function foo($data)
{
echo "in function : " . memory_get_usage() . "n";
}
echo "Initial memory : " . memory_get_usage() . "n";
$r = range(1, 1024);
$r2 = &$r;
echo "Array created : " . memory_get_usage() . "n";
foo($r);
echo "End of function " . memory_get_usage() . "n";
Initial memory : 227.208
Array created : 375.096
in function : 473.584
End of function 375.128

When does the engine separate ?
 In any mismatch case :
zval passed to function arginfo decl. zval received in function separated by engine?
is_ref=0
refcount = 1
pass_by_ref=0 is_ref=0
refcount = 2
NO
is_ref=1
refcount > 1
pass_by_ref=0 is_ref=1
refcount =1
YES
is_ref=0
refcount > 1
pass_by_ref=0 is_ref=0
refcount > 1 ++
NO
is_ref=0
refcount = 1
pass_by_ref=1 is_ref=1
refcount = 2
YES
is_ref=1
refcount > 1
pass_by_ref=1 is_ref=1
refcount > 1 ++
NO
is_ref=0
refcount > 1
pass_by_ref=1 is_ref=1
refcount = 2
YES

Symfony_debug to notice mismatches
 https://github.com/symfony/debug
function bar($var) { }
$a = "foo";
$b = &$a;
bar($a);
Notice: Separating zval for call to function 'bar' in /tmp/memory.php on line 20

foreach behavior

Foreach separation behavior
 It happens sometimes foreach() duplicates your
variable for iteration
 This will eat performances in case of big or
complex arrays being iterated
 There is nothing special to say about objects
implementing Traversable
 The behavior is then just yours

foreach iterating array #1
 If the variable has a refcount of 1, no duplication is
performed by foreach()
$a = range(1,1024);
echo memory_get_usage() . "n" ;
foreach ($a as $v) {
if ($v == 1) {
echo memory_get_usage() . "n" ;
}
}
echo memory_get_usage() . "n" ;
373936
374056
374056

foreach iterating array #2
 If the variable has a refcount >1, foreach() will
duplicate it fully for iteration
$a = range(1,1024);
$b = $a;
echo memory_get_usage() . "n" ;
foreach ($a as $v) {
if ($v == 1) {
echo memory_get_usage() . "n" ;
}
}
echo memory_get_usage() . "n" ;
373936
472512
374056

foreach iterating array #3
 If the variable is a reference, foreach() will work
onto that array and won't perform duplication
$a = range(1,1024);
$b = &$a;
echo memory_get_usage() . "n" ;
foreach ($a as $v) {
if ($v == 1) {
echo memory_get_usage() . "n" ;
}
}
echo memory_get_usage() . "n" ;
373936
374056
374056

Monitoring memory usage
 Not much tools exist (for PHP)
 memory_get_usage()
 OS' help (/proc , pmap , etc...)
 Valgrind with massif tool
 PHP Extensions
 Xdebug
 memprof
 memtrack

Quick intro to memprof
 https://github.com/arnaud-lb/php-memory-profiler
$b = range(1, 1024 * 1024); /* a lot of memory */
$b[] = foo();
loader('/Zend/Date.php'); /* a lot of PHP source code */
memprof_dump_callgrind(fopen('/tmp/trace.out', 'w'));

Thank you for listening