Oscon2008 Qa Leak Testing Latest Slides


Published on

This is a talk that Carsten Book and I gave at OSCON about fixing memory leaks in Firefox 3.

Published in: Business, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Oscon2008 Qa Leak Testing Latest Slides

  1. 1. <ul><ul><li>OSCON 2008 - Fixing Hard Problems Through Iterative QA and Development </li></ul></ul>Clint Talbert - Carsten Book Mozilla Corporation 24.July 2008
  2. 2. Introduction <ul><li>Process Overview </li></ul><ul><li>Leak Testing Overview </li></ul><ul><li>Tools </li></ul><ul><li>Memory Leak Debug Test Builds </li></ul><ul><li>Leak Gauge </li></ul><ul><li>Manual Tests (Litmus) </li></ul><ul><li>Add-on Testing </li></ul><ul><li>Milestones so far </li></ul>
  3. 3. The Approach <ul><li>Development and Test Development collaborate on tests and tools </li></ul><ul><li>Create widely applicable, reproducible tests </li></ul><ul><li>Test Tools are treated as first rate products </li></ul><ul><li>Implement a reporting structure for results </li></ul>
  4. 4. Collaboration <ul><li>Developers and Test developers bring different perspectives to the table </li></ul><ul><li>Developers concentrate on the deep tools to measure behavior of code (i.e. Ref-Counting Analysis etc) </li></ul><ul><li>Test Developers work on harnesses and reporting infrastructure, lowering barrier of entry </li></ul><ul><li>Having reliable test tools that are run and developed on a regular basis help bring a sense of testing into the development culture. </li></ul>
  5. 5. Reproducible <ul><li>Use VM's where you can </li></ul><ul><li>Keep tests and tools simple </li></ul><ul><li>By taking reliable test tools into “impure” systems, you can get results that you can trust. </li></ul>
  6. 6. Test Tools Are Products <ul><li>Test Tools must grow with the product </li></ul><ul><li>Test Tools must be simple and maintainable </li></ul><ul><li>Test Tools must be developed and QA'd just like the product they are testing </li></ul><ul><li>Test Tools should be very easy to run </li></ul><ul><li>Test Tools should require only the most basic level of infrastructure possible to run. (But should also fit into a high infrastructure scenario for reporting) </li></ul>
  7. 7. Reporting Results <ul><li>Use results to further a culture of testing </li></ul><ul><li>Use results to understand if the test is reliable, and if so broaden your testing scenarios </li></ul><ul><li>Easy to understand results empower community members </li></ul>
  8. 8. Reporting Results <ul><li>What our results reporting looks like </li></ul>
  9. 9. Leak Testing Overview <ul><li>Lots of Memory Improvements for Firefox 3: </li></ul><ul><li>The XPCOM cycle collector continuously cleans up unused memory. Plus, hundreds of memory leaks are now remedied </li></ul><ul><li>Memory Leak Tests after every Code change to make sure we introduce no new Memory Leaks </li></ul><ul><li>QA also started with a lot of Tests for Memory Leaks </li></ul><ul><li>Mozilla QA runs manual tests using the Litmus Testcase Suite (using Tools such as Debug Builds and Leak Gauge)‏ </li></ul><ul><li>Regression Memory Leak Testing of Firefox </li></ul><ul><li>Memory Leak Tests of new and popular Addon's </li></ul><ul><li>Future: More Automated Tests run by Mozilla QA </li></ul>
  10. 10. Tools <ul><li>Various Tools for Performance and Leak Testing ‏ </li></ul><ul><li>QA uses Debug Builds and Leak Gauge </li></ul><ul><li>Debug Builds: Debug Builds can be build with Trace-Malloc Support to search for Memory Leak Builds – every time up to date build with latest checkins and overview over Leaking Components (but Debug Builds need to be built individually and you need a Development Environment (Xcode, Visual Studio Express, etc) for most platforms)‏ </li></ul><ul><li>Leak Gauge: Developed by David Baron - It is designed to assist in detecting what leaks of large object graphs occur during normal browsing activity. The logging can be run during normal browsing without significant overhead . Log taken by setting environment variables in a release build. </li></ul>
  11. 11. Debug Builds <ul><li>Using Firefox Debug Builds with Trace Malloc enabled for all Plattforms (Windows XP/Vista, Linux and Mac 10.4/10.5)‏ </li></ul><ul><li>Trace Malloc allows searching for Memory Leaks </li></ul><ul><li>Can provide a Log File Output </li></ul><ul><li>Identify Leaking Components, this information helps Developers to debug and fix the Memory Leak ! </li></ul>
  12. 12. Leak Gauge <ul><li>Easy to use Memory Leak Test Tool developed by David Baron </li></ul><ul><li>Only Requirement is a environment variable and works on all platforms “c:NSPR_LOG_MODULES=DOMLeak:5,DocumentLeak:5,nsDocShellLeak:5,NodeInfoManagerLeak:5” </li></ul><ul><li>And “set NSPR_L OG_FILE= c:leak1.log “ to d efine a Leak Log </li></ul><ul><li>Works on Firefox 2 and Firefox 3+ Release Builds </li></ul><ul><li>Leak Log Upload Form on http://mxr.mozilla.org/mozilla/source/tools/footprint/leak-gauge.html for analysis </li></ul><ul><li>Upload Form indicates if a Memory Leak was found </li></ul>
  13. 13. Leak Gauge Log Output <ul><li>W hen a Memory Leak was found the output of the Log analysis look like this : </li></ul>Memory Leak
  14. 14. Litmus based Tests <ul><li>To make sure we have no hidden Memory Leaks in Firefox Mozilla QA was running a FFT (Full Functional Tests) Testrun in Litmus ( http://litmus.mozilla.org ) with Debug Builds on Linux, Mac and Windows. Memory Leaks were Reported to the Bugzilla DB. </li></ul><ul><li>Litmus is a Open Source Test Case Management Tool. Litmus contains manual testcases and a Full Functional Test contains around 750 manual testcases from all Firefox areas. </li></ul>
  15. 15. Add-ons Testing <ul><li>Addon Memory Leak Testing is a big focus of the QA Memory Leak Testing Work </li></ul><ul><li>We test new Extensions uploaded to Addons.mozilla.org and also the Top-Downloaded and Recommend Addon's </li></ul><ul><li>Leak Testing is done with Debug Builds and also with Leak Gauge </li></ul><ul><li>We provide also a Leak Gauge “How-To” for Addon Developers and AMO-Editors </li></ul><ul><li>In the Future – A successful Leak Test is a requirement for new Addons on Addons.mozilla.org </li></ul><ul><li>A best practices Document is in progress to give Addon Developers a hand in avoiding Memory Leak </li></ul>
  16. 16. Addon Testing Practice <ul><li>We use new Profiles to avoid any false-positive Results from other Extensions </li></ul><ul><li>We test all aspects of the Extension (Install, Uninstall, Features of the Extension) </li></ul><ul><li>Provide Detailed Steps to reproduce so that this Memory Leak is reproducible by Developers and also to be able to verify the fix of this Leak. </li></ul><ul><li>In case a Memory Leak was found we file a Bug in the Mozilla Bug Database and inform the Extension Developer </li></ul><ul><li>Future Plan: We will use more Automation to do this testing ! </li></ul>
  17. 17. Success so far <ul><li>Successfully identified various types of Leaks in Extensions and Firefox Components – also in specific Scenarios – beyond automated Tests </li></ul><ul><li>Memory Leak Logs from Debug Builds and Leak Gauge Help Developers to identify the cause of the Leak </li></ul><ul><li>Memory Leak Testing of Extensions help to maintain the great Firefox 3 Performance </li></ul>
  18. 18. Questions ? <ul><li>References: </li></ul><ul><li>http://wiki.mozilla.org/Performance:Leak_Tools Overview over Performance/Leak Tools </li></ul><ul><li>http://blog.pavlov.net/2008/03/11/firefox-3-memory-usage/ Blog Post from Stuart Parmenter </li></ul><ul><li>http://blog.mozilla.com/tomcat/2008/03/21/extension-memory-leak-testing/ Blog Post about the QA Extension Testing </li></ul><ul><li>Contact: </li></ul><ul><li>Clint Talbert (ctalbert on irc.mozilla.org) [email_address] </li></ul><ul><li>Carsten Book (tomcat on irc.mozilla.org) [email_address] </li></ul>