Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Understanding PHP memory

51,650 views

Published on

Published in: Technology

Understanding PHP memory

  1. PHP 5 memory Understand and master
  2. Hello everybody  Julien PAULI  Programming with PHP since early 2000s  Programming in C  PHP Internals programmer/reviewer  PHP 5.5 & 5.6 Release Manager  @julienpauli  Tech blog at jpauli.github.io  jpauli@php.net
  3. What about you ?  Got some Unix/Linux understandings ?  Have already experienced C programming ?  Have already experienced PHP programming ?
  4. What we'll cover together  Memory , what's that ?  bytes, stack, heap, etc.  Measuring a process memory consumption  memory image map analysis  Understanding PHP memory consumption  Zend Memory Manager coming  Measuring PHP memory consumption  from PHP land or from system land
  5. Memory from hardware
  6. Memory from software
  7. Memory in a user process  The Virtual Memory image is divided in segments  text  data  heap  stack
  8. Memory usage can grow  Stack will grow as functions will get called  And will narrow as the calls stop and return  Heap will grow as the programmer will decide  Using dynamic allocation functions (malloc, mmap)  Programmer has to free memory by hand  If not : memory leak
  9. Monitoring memory
  10. Linux memory monitoring  'top' or /proc FS > cat /proc/28754/status VmPeak: 20452 kB VmSize: 20324 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 316 kB VmRSS: 316 kB VmData: 16440 kB VmStk: 136 kB VmExe: 4 kB VmLib: 1664 kB VmPTE: 28 kB VmSwap: 0 kB Size of the VM map (out of total mem) Resident Set Size : Size actually in PM Size of the data segment in VM Size of the stack segment in VM Size of the text segment in VM pid
  11. Going even deeper  Let's show the detailed process memory map : > cat /proc/28754/smaps shared segment private mem shared mem
  12. PHP !
  13. PHP, just a process  PHP is just a process like any other  You then can monitor its memory usage by asking the OS <?php passthru(sprintf('cat /proc/%d/status', getmypid())); <?php function heap() { return shell_exec(sprintf('grep "VmRSS:" /proc/%d/status', getmypid())); }
  14. The PHP model Startup Shutdown Request #1 Request #2 Request #3 Startup Shutdown Request #1 Request #2 Request #3 php-fpm worker#1 php-fpm worker#2
  15. The PHP model  One same process may treat many requests  If the process leaks memory, you'll suffer from that  Request n+1 must know nothing about request n  Need to "flush" the request-allocated memory  Need to track request-bound memory claims  ZendMM is the layer that does the job  Share-nothing architecture : by design.
  16. Inside PHP  PHP internal request-bound heap allocator : Zend Memory Manager
  17. Why a custom layer ?  ZendMM  Allows monitoring of request-bound heap usage by basically counting malloc/free calls  Allows the PHP user to limit the heap mem usage  Allows caching of alloced blocks to prevent memory fragmentation and syscalls  Allows preallocating blocks of well-known-sizes for PHP internal structures to fit-in in an aligned way  Ease memory leaks debugging in core and exts
  18. ZendMM guards and leak tracker  Only works in debug mode PHP  report_memleaks=1 in php.ini  Does NOT replace valgrind
  19. Zend MM test script <?php ini_set('memory_limit', -1); /* unlimited ZendMM heap */ function heap() { return shell_exec(sprintf('grep "VmRSS:" /proc/%d/status', getmypid())); } echo heap(); $a = range(1, 1024*1024); /* Stress memory by allocating */ echo heap(); unset($a); /* Stress memory by freeing */ echo heap();
  20. Zend MM launch test > time USE_ZEND_ALLOC=1 php zendmm.php VmRSS: 9640 kB VmRSS: 159296 kB VmRSS: 10068 kB real 0m0.237s user 0m0.148s sys 0m0.080s > time USE_ZEND_ALLOC=0 php zendmm.php VmRSS: 9608 kB VmRSS: 148988 kB VmRSS: 140804 kB real 0m0.288s user 0m0.176s sys 0m0.108s
  21. Valgrind tests for Symfony2 command  app/console runs lots of PHP code under SF2 >USE_ZEND_ALLOC=1 valgrind php app/console debug:container total heap usage: 84,000 allocs, 84,000 frees, 25,966,154 bytes allocated >USE_ZEND_ALLOC=0 valgrind php app/console debug:container total heap usage: 813,570 allocs, 813,570 frees, 74,579,732 bytes allocated
  22. ZendMM internals
  23. ZendMM benefits  Better memory management  More memory efficient  Far less malloc/free calls  Less context switches, less Kernel stress  Less CPU usage  Less heap fragmentation / compaction  A PHP ~10% faster with ZendMM enabled  Really depends on use-case
  24. Memory manager internals  Layer on top of the heap  Will allocate memory from the heap by chunks of customizable size (segments)  Will use a customizable low level heap (malloc / mmap)
  25. A quick word on ZendMM internals  ZEND_MM_SEG_SIZE env variable to customize segment size  Must be power-of-two aligned  Default value is 256Kb
  26. ZendMM in PHP user land  memory_limit (INI setting)  memory_get_usage(true)  Returns the size of all the allocated segments  memory_get_usage()  Returns the occupied size in all the allocated segments  memory_get_[peak]_usage([real])  Returns the max memory that has been allocated/used. Could have been freed meantime
  27. ZendMM memory monitoring
  28. ini_set('memory_limit', -1); // unlimited memory function get_mem_stats() { printf("Memory usage %.2f Kbn", memory_get_usage() / 1024); if ($segSize = getenv('ZEND_MM_SEG_SIZE')) { printf("Heap segmentation : %d segments of %d bytes (%d Kb used)n", memory_get_usage(1)/$segSize, $segSize, memory_get_usage(1)/1024); } } get_mem_stats(); $a = str_repeat('a', 1024*1024*10); // 10 Mb get_mem_stats(); Adjusting heap segment size
  29. Adjusting heap segment size  Segment size = 256Kb (default value)  Segment size = 4Mb $> ZEND_MM_SEG_SIZE=262144 php /tmp/mem.php Memory usage 221.26 Kb Heap segmentation : 1 segments of 262144 bytes (256 Kb used) Memory usage 10461.47 Kb Heap segmentation : 42 segments of 262144 bytes (10752 Kb used) get_mem_stats(); $a = str_repeat('a', 1024*1024*10); // 10 Mb get_mem_stats(); $> ZEND_MM_SEG_SIZE=4194304 php /tmp/mem.php Memory usage 221.26 Kb Heap segmentation : 1 segments of 4194304 bytes (4096 Kb used) Memory usage 10461.47 Kb Heap segmentation : 4 segments of 4194304 bytes (16384 Kb used)
  30. memory_get_usage() ?  memory_get_usage() tells you how much your allocated blocs consume  They usually don't fill their segment entirely  Thus memory_get_usage(true) shows more  This doesn't count stack  This does only count request-bound memory  This doesn't count linked libraries present in the process memory map  This doesn't show non-request-bound memory
  31. Real memory image  valgrind massif
  32. Massif memory monitoring -------------------------------------------------------------------------------- n time(ms) total(B) useful-heap(B) extra-heap(B) stacks(B) -------------------------------------------------------------------------------- 0 0 0 0 0 0 1 309 56 37 19 0 2 341 384 357 27 0 3 376 1,343,992 1,340,371 3,621 0 4 408 1,360,480 1,355,289 5,191 0 5 443 1,376,512 1,367,265 9,247 0 6 471 1,753,976 1,735,671 18,305 0 7 505 2,345,152 2,275,927 69,225 0 8 535 2,405,224 2,329,798 75,426 0 9 570 2,433,408 2,348,849 84,559 0 10 613 2,592,912 2,461,650 131,262 0 11 657 2,681,240 2,534,247 146,993 0
  33. Massif details 94.52% (2,534,247B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc. ->39.11% (1,048,576B) 0x87A7BB: zend_interned_strings_init (zend_string.c:48) | ->17.70% (474,464B) 0x866449: _zend_hash_quick_add_or_update (zend_alloc.h:95) | ->16.48% (441,936B) 0x85F66D: zend_register_functions (zend_API.c:2138) | ->11.93% (320,000B) 0x878625: gc_init (zend_gc.c:124) | ->11.93% (320,000B) 0x8585DF: OnUpdateGCEnabled (zend.c:82) | ->11.93% (320,000B) 0x86E353: zend_register_ini_entries (zend_ini.c:208)
  34. Recommendations / statements  Understand memory_get_usage()  It only shows request-bound allocations, not persistent allocations (that reside through requests)  PHP extensions may allocate persistent memory  Do NOT activate extensions you will not use  Libraries used by PHP may also allocate persistent memory  Use your OS to monitor your process memory accurately
  35. Playing with memory in PHP user land
  36. Master your PHP mem usage  In PHP land ...  all variable types consume memory  every script asked for compilation will eat memory  This memory will be allocated using ZendMM  The memory for parsed script is freed when the request ends  The memory for user variable is freed when the data is not used any more  And here comes the challenge  When isn't the data needed any more ??
  37. Compilation eats memory  Compiling a script eats request-bound memory  If you compile a class, that eats lots of memory  You'd better use that class at runtime  Use an autoloader to be sure <?php $mem = memory_get_usage(); require __DIR__ . '/../vendor/autoload.php'; require __DIR__ . '/../src/Symfony/Component/DependencyInjection/ContainerBuilder.php' echo memory_get_usage() - $mem . "n"; php app/bar.php 246,692
  38. PHP Variables  Variables are internally zval structs
  39. PHP Variables  What eats memory is what is stored into the zval, not really the zval itself :  A huge string  Tip : a file_get_contents('huge-file') is a huge string  A complex array or object  Resources don't really consume mem in zval <?php /* This consumes sizeof(zval) + 1024 bytes */ $a = str_repeat('a', 1024);
  40. PHP Variables  What you want to avoid is have PHP duplicate the zval  But PHP is kind about that  What you want to happen in PHP freeing the memory ASAP  Should you know when PHP duplicates or frees zval , that's the most important !
  41. zvals and refcount  PHP simply counts how many symbols (PHP vars) point to a zval  This is called the refcount <?php $a = "foo"; $b = $a;
  42. zvals and refcount  PHP uses a CopyOnWrite (COW) system for zvals  Memory is saved  Memory gets allocated only on changes <?php $a = "foo"; $b = $a; $a = 17;
  43. zvals and refcount  PHP frees memory for a zval when its refcount reaches 0  Yes, unset() just refcount-- , that's all <?php $a = "foo"; $b = $a; $c = $b; $b = "bar"; unset($a);
  44. No references needed  You see how smart PHP is with memory ?  It's been designed with that in mind  No references needed to hint PHP !  Don't try to hint PHP with references  References can lead to adverse effects  Force PHP to copy a zval  Prevents PHP from freeing memory of a zval &
  45. Tracking refcount  xdebug_debug_zval()  symfony_zval_info() namespace Foo; class A { public $var = 'varA'; } $a = new A(); xdebug_debug_zval('a'); a: (refcount=1, is_ref=0)=class FooA { public $var = (refcount=2, is_ref=0)='varA'; }
  46. Tracking refcount namespace Foo; class C { public $b; public function __construct(B $b) { $this->b = $b; } } $c = new C($b = new B); xdebug_debug_zval('c'); c: (refcount=1, is_ref=0)=class FooC { public $b = (refcount=2, is_ref=0)=class FooB { } } namespace Foo; class B { } unset($b); xdebug_debug_zval('c'); c: (refcount=1, is_ref=0)=class FooC { } c: (refcount=1, is_ref=0)=class FooC { public $b = (refcount=1, is_ref=0)=class FooB { } } unset($c->b); xdebug_debug_zval('c');
  47. Garbage collector ?  As of PHP5.3 , a garbage collector exists  Used to free circular references  And that's all !  PHP already frees itself your vars as their refcount reaches 0  And it's always been like that
  48. Circular references $a = new A; $b = new B (object) 'A' refcount = 1 $a (object) 'B' refcount = 1 $b
  49. Circular references $a->b = $b; $b->a = $a; (object) 'A' refcount = 2 $a (object) 'B' refcount = 2 $b $b->a $a->b
  50. Circular references  Objects are still in memory but no more PHP var point to them  We can call that a "PHP Userland memory leak" unset($a, $b); (object) 'A' refcount = 1 $b->a (object) 'B' refcount = 1 $a->b
  51. Garbage collector $a = new A; $b = new B; $a->b = $b; $b->a = $a; unset($a, $b); echo gc_collect_cycles(); // 2
  52. PHP references
  53. PHP references main line  Using references (&) in PHP can really fool you  They usually force the engine to duplicate memory containers  Which is bad for performance  Especially when the var container is huge
  54. References mismatch <?php function foo($data) { echo "in function : " . memory_get_usage() . "n"; } echo "Initial memory : " . memory_get_usage() . "n"; $r = range(1, 1024); echo "Array created : " . memory_get_usage() . "n"; foo($r); echo "End of function " . memory_get_usage() . "n"; Initial memory : 227.136 Array created : 374.912 in function : 374.944 End of function 374.944
  55. References mismatch <?php function foo($data) { echo "in function : " . memory_get_usage() . "n"; } echo "Initial memory : " . memory_get_usage() . "n"; $r = range(1, 1024); $r2 = &$r; echo "Array created : " . memory_get_usage() . "n"; foo($r); echo "End of function " . memory_get_usage() . "n"; Initial memory : 227.208 Array created : 375.096 in function : 473.584 End of function 375.128
  56. When does the engine separate ?  In any mismatch case : zval passed to function arginfo decl. zval received in function separated by engine? is_ref=0 refcount = 1 pass_by_ref=0 is_ref=0 refcount = 2 NO is_ref=1 refcount > 1 pass_by_ref=0 is_ref=1 refcount =1 YES is_ref=0 refcount > 1 pass_by_ref=0 is_ref=0 refcount > 1 ++ NO is_ref=0 refcount = 1 pass_by_ref=1 is_ref=1 refcount = 2 YES is_ref=1 refcount > 1 pass_by_ref=1 is_ref=1 refcount > 1 ++ NO is_ref=0 refcount > 1 pass_by_ref=1 is_ref=1 refcount = 2 YES
  57. Symfony_debug to notice mismatches  https://github.com/symfony/debug function bar($var) { } $a = "foo"; $b = &$a; bar($a); Notice: Separating zval for call to function 'bar' in /tmp/memory.php on line 20
  58. foreach behavior
  59. Foreach separation behavior  It happens sometimes foreach() duplicates your variable for iteration  This will eat performances in case of big or complex arrays being iterated  There is nothing special to say about objects implementing Traversable  The behavior is then just yours
  60. foreach iterating array #1  If the variable has a refcount of 1, no duplication is performed by foreach() $a = range(1,1024); echo memory_get_usage() . "n" ; foreach ($a as $v) { if ($v == 1) { echo memory_get_usage() . "n" ; } } echo memory_get_usage() . "n" ; 373936 374056 374056
  61. foreach iterating array #2  If the variable has a refcount >1, foreach() will duplicate it fully for iteration $a = range(1,1024); $b = $a; echo memory_get_usage() . "n" ; foreach ($a as $v) { if ($v == 1) { echo memory_get_usage() . "n" ; } } echo memory_get_usage() . "n" ; 373936 472512 374056
  62. foreach iterating array #3  If the variable is a reference, foreach() will work onto that array and won't perform duplication $a = range(1,1024); $b = &$a; echo memory_get_usage() . "n" ; foreach ($a as $v) { if ($v == 1) { echo memory_get_usage() . "n" ; } } echo memory_get_usage() . "n" ; 373936 374056 374056
  63. Monitoring memory usage  Not much tools exist (for PHP)  memory_get_usage()  OS' help (/proc , pmap , etc...)  Valgrind with massif tool  PHP Extensions  Xdebug  memprof  memtrack
  64. Quick intro to memprof  https://github.com/arnaud-lb/php-memory-profiler $b = range(1, 1024 * 1024); /* a lot of memory */ $b[] = foo(); loader('/Zend/Date.php'); /* a lot of PHP source code */ memprof_dump_callgrind(fopen('/tmp/trace.out', 'w'));
  65. Thank you for listening

×