It Works On Dev


Published on

Slides for talk "It Works on Dev" at ZendCon 2009.

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

It Works On Dev

  1. 1. It Works On Dev<br />ByMarcel Esser<br />Code Samurai, CROSCON<br />Contact<br />+1 202 730 9728<br />+49 176 785 69729<br /><br />
  2. 2. The Many Faces of Bugs<br />What are bugs, really?<br />On dev, a bug is an oversight or human error. We’ve used x or y incorrectly, or our flow control is incorrect.<br />On test, a bug is a failed assertion; x does not equal y, although it’s supposed to.<br />On stage, a bug is handbrake. You can recover gracefully and roll back.<br />On live, any problem is a bug. It prevents the normal course of business operations.<br />
  3. 3. Prevention<br />An apple a day…<br />
  4. 4. Prevention: Overview<br />How do I prevent live bugs?<br />Practice pre-emptive warfare.Encourage behavior and practices that detect bugs early.<br />Automate processes to prevent human error.If an action can be automated with reasonable effort, do so.<br />Monitor the eco-system.Your application is not just PHP; it’s the entire platform.<br />Work smarter.Identify where investing more time yields greater rewards.<br />
  5. 5. Prevention: Pre-emptive Warfare<br />Things you should do:<br />Write unit tests (and preferably, follow TDD).<br />K.I.S.S. Complicated code is hard to maintain.<br />Develop functional tests (for example, Selenium IDE).<br />Re-factor often.<br />Maintain automatic documentation (for example, phpdoc).<br />Ask someone stupid.<br />Know your tools.<br />
  6. 6. Prevention: Automation<br />Automation Examples<br />Write scripts to push your configuration and application files to your staging and production machines. rsync and ssh work great.<br />Test upgrades and reconfigurations on dev-&gt;test and script them for dev-&gt;stage. This is your test for pushing to production.<br />Run customized functional test suites against live after pushes to ensure that your code works as expected.<br />
  7. 7. Prevention: Automation<br />Production Push from SVN<br />#!/bin/sh<br />sshroot@prod &quot;/etc/init.d/apache2 stop&quot;<br />rm -rf /tmp/x; mkdir -p /tmp/x<br />svn export https://devs/svn/repo/tags/STABLE /tmp/x/<br />rsync -avz /tmp/x/ root@prod:/var/www/app/<br />sshroot@prod &quot;/etc/init.d/apache2 start&quot;<br />
  8. 8. Prevention: Monitoring<br />Monitoring: Best effort to lazy time ratio<br />Sometimes bugs aren’t bugs; monitor your application servers to keep track of resource usage and predict load situations. For example, Munin or Zabbix.<br />Use a service-monitoring tool such as Nagios or Zabbix to script actions that ensure your application is available.<br />If you are doing background processing, develop scripts to monitor the status of your custom daemons.<br />
  9. 9. Prevention: Be Smarter<br />Working a little smarter<br />Don’t fix problems twice.<br />Keep a log.<br />When you think of a potential problem, make a note.<br />Don’t assume.<br />Create incentives for bug killers.<br />Schedule time for preventative work.<br />
  10. 10. Debugging on live?<br />Surely, you jest.<br />
  11. 11. Debugging on live?<br />Where do live bugs come from?<br />Very common causes:<br />Differences between dev/test/stage/live platforms.<br />Differences in application state (for example, the database).<br />Differences in configuration.<br />Differences in hardware.<br />Poor quality assurance.<br />Incorrect assumptions.<br />
  12. 12. Debugging on live?<br />How to Gather Intelligence on Live<br />Your objectives:<br />Bug reports<br />Profiling<br />Tracing<br />Remote debugging<br />Identifying bugs in your application server<br />
  13. 13. Debugging on live: Bug reports<br />What’s a good bug report?<br />Ideally, you have:<br />The account of the user that experienced the error.<br />The page that generated the error.<br />What the user was doing before the error occurred.<br />The user’s client software.<br />Competent support personnel.<br />
  14. 14. Debugging on live: Profiling<br />What is Xdebug?<br />PHP debugging extension written by DerickRethans.<br />It profiles.<br />It traces.<br />It breakpoints.<br />It has good client support (Eclipse/NetBeans).<br />Zend Debugger is also good – but not everyone has it.<br />
  15. 15. Debugging on live: Profiling with Xdebug<br /># load as a ZE extensionzend_extension=/path/ enable debugging trigger<br /># i.e. XDEBUG_PROFILE=1 in GET, POST,# or cookie<br />xdebug.profiler_enable_trigger = 1<br /># where to save profile output<br />xdebug.profiler_output_dir = /tmp<br /># filename specifier; &lt;path&gt;.&lt;time&gt;.out<br />xdebug.profiler_output_name = %s.%t.out<br />
  16. 16. Debugging on live: Profiling with Xdebug<br />Visit your URL:<br />2) Grab the profile dump from the server.<br />3) Fire up KCacheGrind.<br />4) Profit.<br />
  17. 17. Debugging on live: Tracing with Xdebug<br /># load as a ZE extensionzend_extension=/path/<br /># turn on automatic tracing<br />xdebug.auto_trace=On<br /># where to store the tracedumps<br />xdebug.trace_output_dir=/tmp/<br /># more memory reporting, please<br />xdebug.show_mem_delta=On<br />
  18. 18. Debugging on live: Tracing with Xdebug<br />Visit your URL:<br />2) Grab the trace dump from the server.<br />3) Fire up a text editor.<br />4) Profit.<br />
  19. 19. Debugging on live: Debugging with Xdebug<br /># load as a ZE extensionzend_extension=/path/<br /># turn on automatic tracing<br />xdebug.remote_enable = on<br /># debugger protocolxdebug.remote_handler=dbgp<br /># debugger client hostxdebug.remote_host=localhost<br /># debugger client portxdebug.remote_port=9000<br />
  20. 20. Debugging on live: A Word of Caution<br />Before you get ahead of yourself…<br />A couple of notes about remote debugging:<br />There is a performance penalty. Enable it only when you need to, and turn it off after you get your data.<br />Remember that debugging information is extremely sensitive and should be kept confidential.<br />Use automatic scripts to turn debug extensions/settings on/off automatically so you don’t accidentally leave “leavings”.<br />
  21. 21. Debugging on live: Identifying Server Bugs<br />When All Else Fails<br />Remember that you are computer programmers and:<br />Take the problem machine out of rotation.<br />Setup Apache/Yourwebserver to spawn 1 worker process.<br />Attach a tracer to that process, like strace or dtrace.<br />Figure out where it dies.<br />Map that to the appropriate PHP code or extension.<br />Don’t do whatever you did.<br />
  22. 22. Oh, and btw.<br />In case you forgot…<br />
  23. 23. Shameless Plug<br />Hi, my name is Marcel Esser. I work for CROSCON.<br />We do: <br />Bespoke application development<br />Customized service monitoring<br />MyCourt productivity software<br />Info?<br /><br />202.730.9728<br />