It Works On DevByMarcel EsserCode Samurai, CROSCONContact+1 202 730 9728+49 176 785 69729marcel.esser@croscon.com

The Many Faces of BugsWhat are bugs, really?On dev, a bug is an oversight or human error. We’ve used x or y incorrectly, or our flow control is incorrect.On test, a bug is a failed assertion; x does not equal y, although it’s supposed to.On stage, a bug is handbrake. You can recover gracefully and roll back.On live, any problem is a bug. It prevents the normal course of business operations.

PreventionAn apple a day…

Prevention: OverviewHow do I prevent live bugs?Practice pre-emptive warfare.Encourage behavior and practices that detect bugs early.Automate processes to prevent human error.If an action can be automated with reasonable effort, do so.Monitor the eco-system.Your application is not just PHP; it’s the entire platform.Work smarter.Identify where investing more time yields greater rewards.

Prevention: Pre-emptive WarfareThings you should do:Write unit tests (and preferably, follow TDD).K.I.S.S. Complicated code is hard to maintain.Develop functional tests (for example, Selenium IDE).Re-factor often.Maintain automatic documentation (for example, phpdoc).Ask someone stupid.Know your tools.

Prevention: AutomationAutomation ExamplesWrite scripts to push your configuration and application files to your staging and production machines. rsync and ssh work great.Test upgrades and reconfigurations on dev->test and script them for dev->stage. This is your test for pushing to production.Run customized functional test suites against live after pushes to ensure that your code works as expected.

Prevention: AutomationProduction Push from SVN#!/bin/shsshroot@prod "/etc/init.d/apache2 stop"rm -rf /tmp/x; mkdir -p /tmp/xsvn export https://devs/svn/repo/tags/STABLE /tmp/x/rsync -avz /tmp/x/ root@prod:/var/www/app/sshroot@prod "/etc/init.d/apache2 start"

Prevention: MonitoringMonitoring: Best effort to lazy time ratioSometimes bugs aren’t bugs; monitor your application servers to keep track of resource usage and predict load situations. For example, Munin or Zabbix.Use a service-monitoring tool such as Nagios or Zabbix to script actions that ensure your application is available.If you are doing background processing, develop scripts to monitor the status of your custom daemons.

Prevention: Be SmarterWorking a little smarterDon’t fix problems twice.Keep a log.When you think of a potential problem, make a note.Don’t assume.Create incentives for bug killers.Schedule time for preventative work.

Debugging on live?Surely, you jest.

Debugging on live?Where do live bugs come from?Very common causes:Differences between dev/test/stage/live platforms.Differences in application state (for example, the database).Differences in configuration.Differences in hardware.Poor quality assurance.Incorrect assumptions.

Debugging on live?How to Gather Intelligence on LiveYour objectives:Bug reportsProfilingTracingRemote debuggingIdentifying bugs in your application server

Debugging on live: Bug reportsWhat’s a good bug report?Ideally, you have:The account of the user that experienced the error.The page that generated the error.What the user was doing before the error occurred.The user’s client software.Competent support personnel.

Debugging on live: ProfilingWhat is Xdebug?PHP debugging extension written by DerickRethans.It profiles.It traces.It breakpoints.It has good client support (Eclipse/NetBeans).Zend Debugger is also good – but not everyone has it.

Debugging on live: Profiling with Xdebug# load as a ZE extensionzend_extension=/path/xdebug.so# enable debugging trigger# i.e. XDEBUG_PROFILE=1 in GET, POST,# or cookiexdebug.profiler_enable_trigger = 1# where to save profile outputxdebug.profiler_output_dir = /tmp# filename specifier; <path>.<time>.outxdebug.profiler_output_name = %s.%t.out

Debugging on live: Profiling with XdebugVisit your URL:http://127.0.0.1/cake_1.2.5/?XDEBUG_PROFILE=12) Grab the profile dump from the server.3) Fire up KCacheGrind.4) Profit.

Debugging on live: Tracing with Xdebug# load as a ZE extensionzend_extension=/path/xdebug.so# turn on automatic tracingxdebug.auto_trace=On# where to store the tracedumpsxdebug.trace_output_dir=/tmp/# more memory reporting, pleasexdebug.show_mem_delta=On

Debugging on live: Tracing with XdebugVisit your URL:http://127.0.0.1/worksondev/index.php2) Grab the trace dump from the server.3) Fire up a text editor.4) Profit.

Debugging on live: Debugging with Xdebug# load as a ZE extensionzend_extension=/path/xdebug.so# turn on automatic tracingxdebug.remote_enable = on# debugger protocolxdebug.remote_handler=dbgp# debugger client hostxdebug.remote_host=localhost# debugger client portxdebug.remote_port=9000

Debugging on live: A Word of CautionBefore you get ahead of yourself…A couple of notes about remote debugging:There is a performance penalty. Enable it only when you need to, and turn it off after you get your data.Remember that debugging information is extremely sensitive and should be kept confidential.Use automatic scripts to turn debug extensions/settings on/off automatically so you don’t accidentally leave “leavings”.

Debugging on live: Identifying Server BugsWhen All Else FailsRemember that you are computer programmers and:Take the problem machine out of rotation.Setup Apache/Yourwebserver to spawn 1 worker process.Attach a tracer to that process, like strace or dtrace.Figure out where it dies.Map that to the appropriate PHP code or extension.Don’t do whatever you did.

Oh, and btw.In case you forgot…

Shameless PlugHi, my name is Marcel Esser. I work for CROSCON.We do: Bespoke application developmentCustomized service monitoringMyCourt productivity softwareInfo?marcel.esser@croscon.com202.730.9728