The document discusses Mozilla's approach to fixing memory leaks through iterative testing and development, including developing test tools as products, collaborating between developers and testers, using reproducible tests, and reporting results to further a culture of testing. Key areas discussed include leak testing Firefox and add-ons using tools like debug builds, Leak Gauge, and Litmus tests. Their process has successfully identified and helped developers fix various memory leaks.

1. OSCON 2008 - Fixing Hard Problems Through Iterative QA and Development Clint Talbert - Carsten Book Mozilla Corporation 24.July 2008

2. Introduction Process Overview Leak Testing Overview Tools Memory Leak Debug Test Builds Leak Gauge Manual Tests (Litmus) Add-on Testing Milestones so far

3. The Approach Development and Test Development collaborate on tests and tools Create widely applicable, reproducible tests Test Tools are treated as first rate products Implement a reporting structure for results

4. Collaboration Developers and Test developers bring different perspectives to the table Developers concentrate on the deep tools to measure behavior of code (i.e. Ref-Counting Analysis etc) Test Developers work on harnesses and reporting infrastructure, lowering barrier of entry Having reliable test tools that are run and developed on a regular basis help bring a sense of testing into the development culture.

5. Reproducible Use VM's where you can Keep tests and tools simple By taking reliable test tools into “impure” systems, you can get results that you can trust.

6. Test Tools Are Products Test Tools must grow with the product Test Tools must be simple and maintainable Test Tools must be developed and QA'd just like the product they are testing Test Tools should be very easy to run Test Tools should require only the most basic level of infrastructure possible to run. (But should also fit into a high infrastructure scenario for reporting)

7. Reporting Results Use results to further a culture of testing Use results to understand if the test is reliable, and if so broaden your testing scenarios Easy to understand results empower community members

8. Reporting Results What our results reporting looks like

9. Leak Testing Overview Lots of Memory Improvements for Firefox 3: The XPCOM cycle collector continuously cleans up unused memory. Plus, hundreds of memory leaks are now remedied Memory Leak Tests after every Code change to make sure we introduce no new Memory Leaks QA also started with a lot of Tests for Memory Leaks Mozilla QA runs manual tests using the Litmus Testcase Suite (using Tools such as Debug Builds and Leak Gauge)‏ Regression Memory Leak Testing of Firefox Memory Leak Tests of new and popular Addon's Future: More Automated Tests run by Mozilla QA

10. Tools Various Tools for Performance and Leak Testing ‏ QA uses Debug Builds and Leak Gauge Debug Builds: Debug Builds can be build with Trace-Malloc Support to search for Memory Leak Builds – every time up to date build with latest checkins and overview over Leaking Components (but Debug Builds need to be built individually and you need a Development Environment (Xcode, Visual Studio Express, etc) for most platforms)‏ Leak Gauge: Developed by David Baron - It is designed to assist in detecting what leaks of large object graphs occur during normal browsing activity. The logging can be run during normal browsing without significant overhead . Log taken by setting environment variables in a release build.

11. Debug Builds Using Firefox Debug Builds with Trace Malloc enabled for all Plattforms (Windows XP/Vista, Linux and Mac 10.4/10.5)‏ Trace Malloc allows searching for Memory Leaks Can provide a Log File Output Identify Leaking Components, this information helps Developers to debug and fix the Memory Leak !

12. Leak Gauge Easy to use Memory Leak Test Tool developed by David Baron Only Requirement is a environment variable and works on all platforms “c:\NSPR_LOG_MODULES=DOMLeak:5,DocumentLeak:5,nsDocShellLeak:5,NodeInfoManagerLeak:5” And “set NSPR_L OG_FILE= c:\leak1.log “ to d efine a Leak Log Works on Firefox 2 and Firefox 3+ Release Builds Leak Log Upload Form on http://mxr.mozilla.org/mozilla/source/tools/footprint/leak-gauge.html for analysis Upload Form indicates if a Memory Leak was found

13. Leak Gauge Log Output W hen a Memory Leak was found the output of the Log analysis look like this : Memory Leak

14. Litmus based Tests To make sure we have no hidden Memory Leaks in Firefox Mozilla QA was running a FFT (Full Functional Tests) Testrun in Litmus ( http://litmus.mozilla.org ) with Debug Builds on Linux, Mac and Windows. Memory Leaks were Reported to the Bugzilla DB. Litmus is a Open Source Test Case Management Tool. Litmus contains manual testcases and a Full Functional Test contains around 750 manual testcases from all Firefox areas.

15. Add-ons Testing Addon Memory Leak Testing is a big focus of the QA Memory Leak Testing Work We test new Extensions uploaded to Addons.mozilla.org and also the Top-Downloaded and Recommend Addon's Leak Testing is done with Debug Builds and also with Leak Gauge We provide also a Leak Gauge “How-To” for Addon Developers and AMO-Editors In the Future – A successful Leak Test is a requirement for new Addons on Addons.mozilla.org A best practices Document is in progress to give Addon Developers a hand in avoiding Memory Leak

16. Addon Testing Practice We use new Profiles to avoid any false-positive Results from other Extensions We test all aspects of the Extension (Install, Uninstall, Features of the Extension) Provide Detailed Steps to reproduce so that this Memory Leak is reproducible by Developers and also to be able to verify the fix of this Leak. In case a Memory Leak was found we file a Bug in the Mozilla Bug Database and inform the Extension Developer Future Plan: We will use more Automation to do this testing !

17. Success so far Successfully identified various types of Leaks in Extensions and Firefox Components – also in specific Scenarios – beyond automated Tests Memory Leak Logs from Debug Builds and Leak Gauge Help Developers to identify the cause of the Leak Memory Leak Testing of Extensions help to maintain the great Firefox 3 Performance

18. Questions ? References: http://wiki.mozilla.org/Performance:Leak_Tools Overview over Performance/Leak Tools http://blog.pavlov.net/2008/03/11/firefox-3-memory-usage/ Blog Post from Stuart Parmenter http://blog.mozilla.com/tomcat/2008/03/21/extension-memory-leak-testing/ Blog Post about the QA Extension Testing Contact: Clint Talbert (ctalbert on irc.mozilla.org) [email_address] Carsten Book (tomcat on irc.mozilla.org) [email_address]