It Works On Dev

It Works On Dev
ByMarcel Esser
Code Samurai, CROSCON
Contact
+1 202 730 9728
+49 176 785 69729
marcel.esser@croscon.com

The Many Faces of Bugs
What are bugs, really?
On dev, a bug is an oversight or human error. We’ve used x or y incorrectly, or our flow control is incorrect.
On test, a bug is a failed assertion; x does not equal y, although it’s supposed to.
On stage, a bug is handbrake. You can recover gracefully and roll back.
On live, any problem is a bug. It prevents the normal course of business operations.

Prevention
An apple a day…

Prevention: Overview
How do I prevent live bugs?
Practice pre-emptive warfare.Encourage behavior and practices that detect bugs early.
Automate processes to prevent human error.If an action can be automated with reasonable effort, do so.
Monitor the eco-system.Your application is not just PHP; it’s the entire platform.
Work smarter.Identify where investing more time yields greater rewards.

Prevention: Pre-emptive Warfare
Things you should do:
Write unit tests (and preferably, follow TDD).
K.I.S.S. Complicated code is hard to maintain.
Develop functional tests (for example, Selenium IDE).
Re-factor often.
Maintain automatic documentation (for example, phpdoc).
Ask someone stupid.
Know your tools.

Prevention: Automation
Automation Examples
Write scripts to push your configuration and application files to your staging and production machines. rsync and ssh work great.
Test upgrades and reconfigurations on dev->test and script them for dev->stage. This is your test for pushing to production.
Run customized functional test suites against live after pushes to ensure that your code works as expected.

Prevention: Automation
Production Push from SVN
#!/bin/sh
sshroot@prod "/etc/init.d/apache2 stop"
rm -rf /tmp/x; mkdir -p /tmp/x
svn export https://devs/svn/repo/tags/STABLE /tmp/x/
rsync -avz /tmp/x/ root@prod:/var/www/app/
sshroot@prod "/etc/init.d/apache2 start"

Prevention: Monitoring
Monitoring: Best effort to lazy time ratio
Sometimes bugs aren’t bugs; monitor your application servers to keep track of resource usage and predict load situations. For example, Munin or Zabbix.
Use a service-monitoring tool such as Nagios or Zabbix to script actions that ensure your application is available.
If you are doing background processing, develop scripts to monitor the status of your custom daemons.

Prevention: Be Smarter
Working a little smarter
Don’t fix problems twice.
Keep a log.
When you think of a potential problem, make a note.
Don’t assume.
Create incentives for bug killers.
Schedule time for preventative work.

Debugging on live?
Surely, you jest.

Debugging on live?
Where do live bugs come from?
Very common causes:
Differences between dev/test/stage/live platforms.
Differences in application state (for example, the database).
Differences in configuration.
Differences in hardware.
Poor quality assurance.
Incorrect assumptions.

Debugging on live?
How to Gather Intelligence on Live
Your objectives:
Bug reports
Profiling
Tracing
Remote debugging
Identifying bugs in your application server

Debugging on live: Bug reports
What’s a good bug report?
Ideally, you have:
The account of the user that experienced the error.
The page that generated the error.
What the user was doing before the error occurred.
The user’s client software.
Competent support personnel.

Debugging on live: Profiling
What is Xdebug?
PHP debugging extension written by DerickRethans.
It profiles.
It traces.
It breakpoints.
It has good client support (Eclipse/NetBeans).
Zend Debugger is also good – but not everyone has it.

Debugging on live: Profiling with Xdebug
# load as a ZE extensionzend_extension=/path/xdebug.so# enable debugging trigger
# i.e. XDEBUG_PROFILE=1 in GET, POST,# or cookie
xdebug.profiler_enable_trigger = 1
# where to save profile output
xdebug.profiler_output_dir = /tmp
# filename specifier; <path>.<time>.out
xdebug.profiler_output_name = %s.%t.out

Debugging on live: Profiling with Xdebug
Visit your URL:http://127.0.0.1/cake_1.2.5/?XDEBUG_PROFILE=1
2) Grab the profile dump from the server.
3) Fire up KCacheGrind.
4) Profit.

Debugging on live: Tracing with Xdebug
# load as a ZE extensionzend_extension=/path/xdebug.so
# turn on automatic tracing
xdebug.auto_trace=On
# where to store the tracedumps
xdebug.trace_output_dir=/tmp/
# more memory reporting, please
xdebug.show_mem_delta=On

Debugging on live: Tracing with Xdebug
Visit your URL:http://127.0.0.1/worksondev/index.php
2) Grab the trace dump from the server.
3) Fire up a text editor.
4) Profit.

Debugging on live: Debugging with Xdebug
# load as a ZE extensionzend_extension=/path/xdebug.so
# turn on automatic tracing
xdebug.remote_enable = on
# debugger protocolxdebug.remote_handler=dbgp
# debugger client hostxdebug.remote_host=localhost
# debugger client portxdebug.remote_port=9000

Debugging on live: A Word of Caution
Before you get ahead of yourself…
A couple of notes about remote debugging:
There is a performance penalty. Enable it only when you need to, and turn it off after you get your data.
Remember that debugging information is extremely sensitive and should be kept confidential.
Use automatic scripts to turn debug extensions/settings on/off automatically so you don’t accidentally leave “leavings”.

Debugging on live: Identifying Server Bugs
When All Else Fails
Remember that you are computer programmers and:
Take the problem machine out of rotation.
Setup Apache/Yourwebserver to spawn 1 worker process.
Attach a tracer to that process, like strace or dtrace.
Figure out where it dies.
Map that to the appropriate PHP code or extension.
Don’t do whatever you did.

Oh, and btw.
In case you forgot…

Shameless Plug
Hi, my name is Marcel Esser. I work for CROSCON.
We do:
Bespoke application development
Customized service monitoring
MyCourt productivity software
Info?
marcel.esser@croscon.com
202.730.9728

It Works On Dev

by marcelesser on Oct 28, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

2 Embeds 2

Statistics

It Works On Dev — Presentation Transcript

http://www.slideshare.net	1
http://www.lmodules.com	1