Php in the Dark (old version)

who am i JEROEN KEPPENS• Freelance IT Consultant• 10+ years professional PHP/MySQL experience• Senior Developer / Software Architect• Scrum Master• Team Leader• Gambling & Telecom Industry

who am i AMAZIUM• My Company - Amazium bvba• IT services on medium to long term projects• Development, Team Leading, Agile Coaching• Website - http://www.amazium.com• Blog - http://blog.amazium.com

SHEBANGshell & cron scripts, locking, shebang & input/output

shell & cron scripts WHAT & WHY?• Shell scripts are executed on the command line• Use of PHP-CLI instead of apache module (CGI)• Cronjob is a script scheduled to run at a certain time• Always use php-cli, don’t use wget in crons• Example uses: • Calculate reports, both scheduled as manual • Create monthly invoices • Perform maintenance tasks • Cache management (pre-caching, reloading,...) • Data acquisition (cfr. load articles from a webservice) • Data aggregation

shell & cron scripts $_SERVER• No HTTP_*, REMOTE_* & SERVER_* information• Extra information available: Array ( [TERM_PROGRAM] => Apple_Terminal [TERM] => xterm-color [SHELL] => /bin/bash [TERM_PROGRAM_VERSION] => 273 [USER] => jkeppens [COMMAND_MODE] => unix2003 [PWD] => /Users/jkeppens [HOME] => /Users/jkeppens [LOGNAME] => jkeppens [LC_CTYPE] => UTF-8 [DISPLAY] => /tmp/launch-hr4Dxt/org.x:0 [argv] => Array ( [0] => /tmp/test.php ) [argc] => 1 )

shell & cron scripts SHEBANG• A shebang is a character sequence consisting of the hash sign and exclamation mark, followed by a command at the start of a ﬁle.• Example in PHP script #!/usr/bin/php <?php // your code• The PHP script needs to be executable to be run this way. $ chmod +x invoices.php $ export APPLICATION_ENV=development $ ./invoices.php -monthly [development] Report has been generated and mailed.

shell & cron scripts WHO’S CALLING?• Some scripts can - or should - only run on the command line.• Check on the type of the “interface” using php_sapi_name()• php_sapi_name() returns a lowercase string that describes the type of interface (Server API - SAPI) that PHP is using.• In case of PHP-CLI the function returns the value “cli”.• Possible values: aolserver, apache, apache2filter, apache2handler, caudium, cgi (until PHP 5.3), cgi-fcgi, cli, continuity, embed, isapi, litespeed, milter, nsapi, phttpd, pi3web, roxen, thttpd, tux, and webjames

shell & cron scripts INPUT• $_SERVER[“argc”] - number of arguments• $_SERVER[“argv”] - array of arguments• First element = script name $ php -q input.php -monthly -test=yes -v Our input is: $_SERVER[argc] = int(4) $_SERVER[argv] = array(4) { [0]=> string(9) "input.php" [1]=> string(8) "-monthly" [2]=> string(9) "-test=yes" [3]=> string(2) "-v" }

shell & cron scripts GETOPT• getopt() parses arguments passed to a script• No validation• Short options passed as string: • Individual characters (no values) • Character followed by colon (value required) • Character followed by double colon (optional value)• Long options since php5.3. Passed as array, not as string.• Optional values attached to argument.• Returns array of argument / value (or false) pairs• PEAR - Console_Getopt• Zend Framework - Zend_Console_Getopt

shell & cron scripts GETOPT - NATIVE // Define short options $ ./test.php -h -u jeroen -psecret$shortopts = "";$shortopts .= "hv"; // No values Array$shortopts .= "u:"; // Required value ($shortopts .= "p::"; // Optional value [h] => [u] => jeroen// Define long options [p] => secret$longopts = array( ) "user:", // Required value "password::", // Optional value $ ./test.php -u jeroen -p secret "help", // No value "verbose", // No value Array); ( [u] => jeroen// Get the options (5.3 style) [p] =>$options = getopt( ) $shortopts, $longopts $ ./test.php -u -p secret); Array// Show it (print_r($options); [u] => -p )

shell & cron scripts GETOPT - PEAR // Define long & short options $ ./test.php -h -u jeroen -psecret xxx// ... Array (// Get Console_Getopt object [0] => Array (require_once Console/Getopt.php; [0] => Array ($con = new Console_Getopt(); [0] => h [1] =>// Get arguments )$args = $con->readPHPArgv(); [1] => Array (array_shift($args); [0] => u [1] => jeroen// Parse options )$options = $con->getopt2( [2] => Array ( $args, $shortOptions, $longOptions [0] => p); [1] => secret )// Check if we got an error [1] => Array(if ($options instanceof PEAR_Error) { [0] => xxx echo $options; )} else { ) print_r($options[0]);} $ ./test.php -u Console_Getopt: option requires an argument -- u

shell & cron scripts GETOPT - ZF// Define options$opts = array( help|h => Show help, verbose|v => Verbose mode, user|u=s => Username (string), pass|p-s => Password (string) (optional));try { $options = new Zend_Console_Getopt($opts); $options->parse(); if (!empty($options->help)) { echo $options->getUsageMessage(); } else { $username = $options->user; // or $options->u or $options->getOption(user); $password = $options->pass; $verbose = !empty($options->verbose); }} catch (Zend_Console_Getopt_Exception $e) { echo $options->getUsageMessage();}http://framework.zend.com/manual/en/zend.console.getopt.html

interactive input READLINE• Some scripts require user input cfr. “forms” in web apps• Basic version: read from STDIN // Write out text to the output stream & capture name as input fwrite(STDOUT, PHP_EOL . Please enter your name: ); $name = fgets(STDIN); // Write out file content fwrite(STDOUT, PHP_EOL . Hello . trim($name) . PHP_EOL . PHP_EOL);• Advanced version: use the readline() function. This allows you to interact with the shell user and ask for input.• Use the power of the shell in your script: • history (up/down arrows) • auto completion • callback handlers

interactive input HISTORY$historyFile = ./readline.hist;if (is_file($historyFile)) { readline_read_history($historyFile);}while (true) { $line = strtolower(trim(readline("Command: "))); echo Received [ . $line . ] . PHP_EOL; readline_add_history($line); // add command to history // check special actions switch ($line) { // check special actions case clear: // clear the history readline_clear_history(); break; case history: // print out the history print_r(readline_list_history()); break; case quit: case exit: // quit / exit break 2; default: break; }}readline_write_history($historyFile);

interactive input AUTO-COMPLETEreadline_completion_function(readlineCompletion);function readlineCompletion($string, $index) { $completion = array( history, quit, exit, clear, clean, cls ); $matches = array(); foreach ($completion as $c) { if (strpos($c, $string) === 0) { $matches[] = $c; } } return $matches;}$line = strtolower(trim(readline("Command: ")));

shell & cron scripts SCHEDULED JOBS• Deﬁne environment variables NAME = VARIABLE• Deﬁne schedule in crontab * * * * * command to be executed | | | | | | | | | +----- day of week (0 - 6) (Sunday=0) | | | +------- month (1 - 12) | | +--------- day of month (1 - 31) | +----------- hour (0 - 23) +------------- min (0 - 59)• Example APPLICATION_ENV = development 30 4 * * * /opt/scripts/invoices.php -monthly

shell & cron scripts CRON & WGETDO NOT USE WGET IN CRONJOBS

shell & cron scripts LOCKING• Some script should only run once at a time.• Locking mechanism using PID files in /var/run: • On starting the script, check if the pid file exists • If it does, check if a process with pid id specified is running • If it is running, exit your script • If it is not running, replace the pid in the pidfile by the current script’s pid • If it does not exist, create the file in lock mode and put the current script’s pid in it. Release the lock. • When done with the script, remove the pid file.

shell & cron scripts LOCKING$pidfile = /var/run/amazium/locktest.pid;// Check if the file exists$otherIsRunning = false;if (file_exists($pidfile)) { $pid = (int)file_get_contents($pidfile); if (!empty($pid)) { if (posix_kill($pid, 0)) { // check if process with pid is running die(Another instance of this script exists.); } }}// Get a pointer to the PID file for writing & try to get a lock$fp = fopen($pidfile, w+);if (!flock($fp, LOCK_EX | LOCK_NB)) { die(Could not get exclusive lock on pidfile);}fwrite($fp, getmypid()); // We have a lock, write my pid file// ... do stuff ...// At the end, we clean up after ourselves@unlink($pidfile);

shell & cron scripts EXTRAS• Some nice classes exist to spice up your scripts:• Pear Console_Color : Add colored text to your script ➡ http://pear.php.net/package/Console_Color• Pear Console_ProgressBar : Add a progressbar to track progress of your script while in interactive mode. ➡ http://pear.php.net/package/Console_ProgressBar• Pear Console_Table : Show data in tables ➡ http://pear.php.net/package/Console_Table

DAEMONScreating daemons the right way

daemons WHAT• A daemon or service is a background process designed to run autonomously, with little or none user interaction.• Origins in Greek mythology: neither good or evil, they were little spirits that did useful things for mankind.• Examples: • apache server • mysql server • proftpd daemon

daemons HOW• Creating a daemon in linux uses a specific set of rules in a given order: • Fork off the parent process • Change file mode mask (umask) • Open any logs for writing • Create a new session id and detach from current session • Change the current working directory to a safe place • Close standard file descriptors (stdin, stdout & stderr) • Actual daemon code• Make sure the code is written securily and in a way that it fails gracefully : no user interaction, so the user might not notice until too late.

daemons STEP 1: FORK• We fork to detach the process from any shell that started it. This way it keeps running when the shell exits.• PHP: pcntl_fork function creates a child process that differs from the parent process only in its PID and PPID.• On success, the PID of the child process is returned in the parent’s and 0 in the child’s thread of execution.• On failure, -1 is returned in the parent’s context, no child process will be created and a php error is raised.• All variables initiated in the parent before the fork, are also available as is in the child’s thread of execution.

daemons STEP 1: FORK// Everything defined and set here is available // in both parent as child// ...// We fork the script$pid = pcntl_fork();// We add our daemon and child codeif ($pid == -1) { die(Could not fork!);} elseif ($pid) { // we are the parent} else { // we are the child}

daemons STEP 2: UMASK• umask or user file creation mask is a function that sets the default permissions of newly created files and folders.• In order to write any files (including logs) created by the daemon, the file mode mask (umask) must be changed to ensure that they can be written to or read from properly.• As a child we are a clone of our parent, so we also inherited the umask of the parent. The child has no idea what the umask is set to, so it’s always good to reset it by using umask(0), even if we don’t plan to use it.• umask() sets the php’s umask to the system’s umask & mask. If your system umask is 0666, your new umask is 0666 & mask.

daemons STEP 2: UMASK• In order to find out what to set your umask to, you have to think about the octal notation you want the files to be when they are created. Folders start with 777, and files with 666.• You then subtract your umask value from these numbers. Example, if we set our umask to 0022, permissions of newly created folders will be 755 and of files 644.• Octal notations: • 1 = execute • 2 = write • 3 = execute + write • 4 = read • 5 = write + execute • 6 = read + write • 7 = read + write + execute

daemons STEP 2: UMASKfunction fperms($file) { return substr(sprintf(%o, fileperms($file)), -4);}$testFile = "/tmp/umask_test.txt";if (file_exists($testFile)) unlink($testFile);// Touch file (no umask) and write out file permissionstouch($testFile);echo fperms($testFile); // 0644 (my system umask is set to 0022)unlink($testFile);// Reset umaskumask(0027);// Touch file (with umask) and write out file permissionstouch($testFile);echo fperms($testFile); // 0640 (0666 - 0027)unlink($testFile);

daemons STEP 3: LOG FILES• Open log files any way you want / are used to: • Text file • Database • Syslog • etc...• Syslog sends your log messages to a system wide logger, where they can be configured to be written to a file, send to a network server or filtered away entirely.

daemons STEP 3: LOG FILES// open syslog, include the process ID and also send// the log to standard error, and use a user defined// logging mechanismopenlog(SyslogTest, LOG_PID | LOG_PERROR, LOG_LOCAL0);// some code// ...$access = date(Y-m-d H:i:s);syslog(LOG_WARNING, Data was access @ . $access);// Some more code// ...closelog();// In the log file:// Feb 21 15:36:12 amazium SyslogTest[11785]: Data was access

daemons STEP 4: DETACH SID• Each process on a *nix system is a member of a process group (a.k.a. session). The id of each group is the process id of it’s owner.• After forking, the child inherits the process group of the parent and its parent process id (PPID) is equal to the process id (PID) of the parent.• We want to create our own process group and become our own process leader, otherwise we will look like an orphan.• In php we detach our session using posix_setsid. Returns the new session_id on success or -1 on errors.

daemons STEP 4: DETACH SID// We fork the script$pid = pcntl_fork();// We add our daemon and child codeif ($pid == -1) { die(Could not fork!);} elseif ($pid) { // we are the parent} else { // lets detach $sid = posix_setsid(); if ($sid < 0) { die(Could not detach session id.); } // child code}

daemons STEP 5: WORKING DIR• The child inherits the working directory from the parent.• This working directory could be a network mount, a removable drive or somewhere the administrator may want to unmount at some point.• To unmount any of these the system will have to kill any processes still using them, which would be unfortunate for our daemon.• For this reason we set our working directory to the root directory, which we are sure will always exist and can’t be unmounted.

daemons STEP 5: WORKING DIR// Current working direcho Current working dir: . getcwd() . PHP_EOL;// Outputs: Current working dir: /tmp// Set the working dir to the scripts root$workingDir = dirname(__FILE__);chdir($workingDir);// New working direcho New working dir: . getcwd() . PHP_EOL;// Outputs: New working dir: /data/scripts/

daemons STEP 6: IN/OUT/ERROR• A daemon doesn’t interact with the user directly, so it has no use for the standard file descriptors STDIN, STDOUT and STDERR.• File descriptors are inherited from the parent and the child has no idea what they are connected to.• We close the file descriptors. We can reconnect the STDOUT & STDERR to logfiles.

daemons STEP 6: IN/OUT/ERROR// Log file to write to$logfile = /tmp/some-log-file.log;// Close standard I/O descriptorsfclose(STDIN);fclose(STDOUT);fclose(STDERR);// First 3 descriptors fill up the blanks$fdIN = fopen(/dev/null, r);$fdOUT = fopen($logfile, a);$fdERR = fopen(php://stdout, a); // set to stdout// STDIN reads from /dev/null// STDOUT writes to $logfile// STDERR writes to STDOUT, so also to $logfile

daemons STEP 7: PAYLOAD• Last but not least, we add our payload.• Parent: we exit(0) and let our child live alone• Child: contains the active part of the script which contains our business logic.

daemons PUTTING IT TOGETHER$pid = pcntl_fork();if ($pid == -1) { // we could not fork die(Could not fork!);} elseif ($pid) { exit(0); // we exit and let our child live on its own} else { umask(0); // Reset uumask openlog(MyLog, LOG_PID | LOG_PERROR, LOG_LOCAL0); // open log $sid = posix_setsid(); // Try to detach session if ($sid < 0) { syslog(LOG_ERROR, Could not detach session id.); exit(1); } chdir(dirname(__FILE__)); // Change working dir // Close standard I/O descriptors & redefine fclose(STDIN); fclose(STDOUT); fclose(STDERR); $fdIN = fopen(/dev/null, r); $fdOUT = fopen($logfile, a); $fdERR = fopen(php://stdout, a); // set to stdout // Payload in endless loop while (true) { // Daemon code } // Close the log closelog(); }

ANGELSmonitoring, signals, zombies & performance

angels ANGEL SCRIPT• Angel scripts watch over daemons using a combination of the pid file and the *nix ps command and restart them when necessary.• If the PID file exists, we check if the PID is running. If not, we remove the PID file and restart the script.• If the PID file does not exist, we restart the script.• Angel script runs every minute scheduled via cron

angels ANGEL SCRIPTrequire_once 01-angel-config.php;// $daemons = array();// $daemons[] = array(// command => /opt/scripts/my-daemon.php,// params => array(-daemon),// pidfile => /var/run/amazium/my-daemon.pid// );foreach ($daemons as $daemon) { if (file_exists($daemon[pidfile])) { $pid = trim(file_get_contents($daemon[pidfile])); if (!empty($pid)) { if (posix_kill($pid, 0)) { // fake 0 signal to check if alive continue; // Next check } } @unlink($daemon[pidfile]); } $pid = pcntl_fork(); if ($pid === 0) { $sid = posix_setsid(); if ($sid != -1) { pcntl_exec($daemon[command], $daemon[params]); } }}

angels SIGNALS• Sometimes you will need to communicate with a daemon process. One way to do so is by sending “signals”.• There are a number of different signals you can send, some with a speciﬁc meaning, others interpreted by the application.• To stop a process you can use SIGTERM and SIGKILL. Sigterm is the polite way to kill a script: you can catch it and end your daemon gracefully.You can’t catch sigkill.• SIGHUP is typically a signal you send if you want the daemon to reinitialize (cfr. reloading logs).• SIGINT typically gets triggered on system shutdown• SIGUSR1 is typically a request to dump states to syslog• Send signals from your script using posix_kill( $pid, $signal )

angels SENDING SIGNALS// Send a signal 9 to self$pid = posix_getpid();posix_kill($pid, 9);// Send a signal SIGUSR1 to the whole process group$pgrp = posix_getpgrp();posix_kill($pgrp, SIGUSR1);// send a SIGHUP to processes in pids$pids = array(/* .. */);foreach ($pids as $pid) { posix_kill($pid, SIGHUP);}// Use kill with pseudo-signal 0 to see if process is aliveif (posix_kill($somepid, 0)) { echo "$somepid is alive!n";} else { echo "$somepid is deceased.n";}

angels HANDLING SIGNALSdeclare(ticks = 1);// signal handler functionfunction signalHandler ($signal) { switch ($signal) { case SIGTERM: // handle shutdown tasks exit(); break; case SIGHUP: // handle restart tasks break; case SIGUSR1: echo "Caught SIGUSR1...n"; break; default: // handle all other signals }}// Setup Signal Handlerspcntl_signal(SIGTERM, "signalHandler");pcntl_signal(SIGHUP, "signalHandler");pcntl_signal(SIGUSR1, "signalHandler");// Send SIGUSR1 to current process idposix_kill(posix_getpid(), SIGUSR1);

angels ZOMBIES• A zombie process or defunct process is a process that has completed execution but still has an entry in the process table. This entry is still needed to allow the parent process to read the child’s exit status.• When a process ends, all of the memory and resources associated with it are deallocated so they can be used by other processes.• The parent can read the childs exit status by executing the wait system call, at which stage the zombie is removed. It is commonly executed in a SIGCHLD signal handler on the parent (SIGCHLD is received when a child has died).• If the parent explicitly ignores SIGCHLD by setting its handler to SIG_IGN, all child exit status information will be discarded and no zombie processes will be left

angels ZOMBIES// Reaper to clean up zombiesfunction reaper($signal) { $pid = pcntl_waitpid(-1, $status, WNOHANG); if ($pid == -1) { // No child waiting. Ignore it. } else { if (pcntl_wifexited($signal)) { // Check if regular exit echo "Process $pid exited.n"; } else { echo "False alarm on $pidn"; } reaper($signal); // Check if more children ended } pcntl_signal(SIGCHLD, reaper); // Reinstall signal handler}// Install signal handler on SIGCHLD, the signal sent // to a process when a child process terminates.pcntl_signal(SIGCHLD, reaper);// OR: pcntl_signal(SIGCHLD, SIG_IGN);

angels PERFORMANCE• Tuning your scripts is KEY when working with daemons• PHP s*cks at garbage collection: not necessary in web context• Since PHP 5.3 it got a little better due to implementation of circular reference garbage collection. • gc_enable() - enable garbage collect if disabled in php.ini • gc_collect_cycles() - forces collection of any existing garbage cycles. Run this in your while loop.• Clean up after you in each run cycle by unsetting or nullifying objects. This will decrease refcount and make them available for garbage collection. Avoid variables in global scope.• Clear your stat cache regularly using clearstatcache()• Track memory leaking using memory_get_usage()

DAEMON SPAWNchildren, communication & shared variables

daemon spawn CHILDREN• When processing queues, you sometimes want multiple concurrent workers.• A daemon can spawn multiple children to do the work while he plays project manager handing out tasks.• Sometimes referred to as “multi-threading” in PHP, but it’s actually “concurrent processing”. PHP has no multi-threading.• Typically you “daemonize” your parent and then fork your children.• You don’t change the session id for your children, you want them to stay in the same process group• You put your “payload” after the loop that creates your forks

angels DAEMONIZE// Daemonize the parent$pid = pcntl_fork();if ($pid == -1) { writeToLog(Could not daemonize!); exit(1);} elseif ($pid) { writeToLog(Daemonized to . $pid); exit(0);} else { // Daemonized umask(0); $sid = posix_setsid(); if ($sid < 0) { writeToLog(Could not detach session id.); exit(1); }}// HERE WE HAVE THE PARENT DAEMON ALIVE$parentPid = getmypid(); // value same for parent as children// Create our childrenfor ($i = 0; $i < MAX_CHILDREN; $i++) { if (getmypid() == $parentPid) { $pids[] = $pid = pcntl_fork(); // SPAWN NEW CHILD if ($pid < 0) { writeToLog(Could not fork!); exit(1); } }}if (getmypid() == $parentPid) { // Parent Code} else { // Child Code}

daemon spawn IPC• IPC or inter-process communication is a series of techniques to exchange data between one or more concurrent processes.• IPC techniques are usually divided into methods for message passing, synchronization, shared memory and remote procedure calls. Some techniques (but not limited to) : • File • Signal • Socket • Semaphore • Shared Memory • Message Queues

daemon spawn IPC : SOCKETS• In PHP we typically use socket pairs to create one socket for the parent and one socket for the children.• The socket_create_pair() function creates two connected and indistinguishable sockets.• Read from the socket using socket_read()• Write to the socket using socket_write()• Use socket_last_error() to get the last error and pass it to socket_strerror() to get a string representation of the error.

daemon spawn IPC : SEMAPHORE• Semaphores can be thought of as simple counters that indicate the status of a resource. This counter is a protected variable and cannot be accessed by the user directly.• The usage of this semaphore variable is simple. If counter is greater that 0, then the resource is available, and if the counter is 0 or less, then that resource is busy or being used by someone else.• This simple mechanism helps in synchronizing multithreaded and multiprocess based applications.• Get a semaphore key using sem_get()• Block a semaphore while using it sem_acquire()• Release a semaphore when done using sem_release()• Remove a semaphore using sem_remove()

daemon spawn IPC : SHM• SHM_* functions create System V compatible shared memory, accessible from outside of PHP• Create a integer KEY identifying the shared memory segment. Typically ftok() is used for this.• Attach script to the memory segment using shm_attach()• Read from the segment using shm_put_var()• Write to the segment using shm_get_var()• Check if a variable exists using shm_has_var()• Detach script from the segment using shm_detach()• Remove the shared memory segment using shm_remove()• Disadvantage : integer keys

daemon spawn IPC : APC• APC or the Alternative PHP Cache provide functions to use shared memory. APC is available via PECL.• Read from the segment using apc_fetch()• Write to the segment using apc_store()• Check if a variable exists using apc_exists()• Increment/decrement using apc_inc() & apc_dec()• Delete a variable from apc using apc_delete()• Advantage: Key-Value pairs, auto-serialization• Disadvantage : PHP speciﬁc• Other alternative: Memcache

daemon spawnIPC : MESSAGE QUEUE• PHP implements System V message queues.• Create/attach to a message queue using msg_get_queue()• Check if a message queue exists using msg_queue_exists()• Remove a message queue using msg_remove_queue()• Put a message on the queue using msg_send()• Read a message from the queue using msg_receive()• Get queue stats using msg_stat_queue()• Message queues can also be implemented using MySQL or external queue systems.

http://www.amazium.com	395
http://it-republik.de	251
http://blog.jensihnow.de	125
http://blog.max-walter.net	41
http://entwickler.com	40
http://blog.jensihnow.de.	27
http://entwickler.de	21
http://a0.twimg.com	6
http://blog.jensihnow.he-webpack.de	5
http://paper.li	5
https://twimg0-a.akamaihd.net	3
https://blog.max-walter.net	3
http://entwickler-magazin.de	2
http://twitter.com	2
http://us-w1.rockmelt.com	2
http://phpmagazin.de	1
http://drizzlin.com	1
http://www.cottica.net	1

Php in the Dark (old version)

by Jeroen Keppens on Feb 22, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

18 Embeds 931

Statistics

Php in the Dark (old version) — Presentation Transcript