Stuffing a 900lb Gorilla
    into a SmartPhone
     Joel Maher & Clint Talbert

          Mozilla Corporation




         ...
What is Fennec?




                  2
How are they going to fit
     me in here?           3
How are they going to fit
     me in here?           3
•XPCShell - low level XPCOM testing




     How are they going to fit
          me in here?                 3
•XPCShell - low level XPCOM testing
•Reftest - rendering testing




     How are they going to fit
          me in here?  ...
•XPCShell - low level XPCOM testing
•Reftest - rendering testing
•Crashtest - regression testing for crash bugs




     H...
•XPCShell - low level XPCOM testing
•Reftest - rendering testing
•Crashtest - regression testing for crash bugs
•Mochitest...
•XPCShell - low level XPCOM testing
•Reftest - rendering testing
•Crashtest - regression testing for crash bugs
•Mochitest...
•XPCShell - low level XPCOM testing
•Reftest - rendering testing
•Crashtest - regression testing for crash bugs
•Mochitest...
•XPCShell - low level XPCOM testing
•Reftest - rendering testing
•Crashtest - regression testing for crash bugs
•Mochitest...
Platforms
            4
Platforms
            4
Platforms
            4
Platforms
            4
Platforms
            4
Platforms
            4
Platforms
            4
Platforms
            5
Roadmap
          6
Fennec Test Automation




                         7
Test Reporting
                 8
Current Approach
• Running tests on desktop build
• Decoupling the tests
• Making tools mobile
• Breaking tests into small...
Decoupling the Tests

• Historically, tests integrated into build
  system
• Required directory structure and tools
• Crea...
Making Tools Mobile

• Python requirements for each platform
• Networking test tools (ssl, http server)
• Test harness scr...
Breaking Tests into
        Chunks

• Cannot run full test harness at once
• Chunks based on directories
• Managing test d...
Stabilizing End to End

• Modifying test harness to look for a hang
• Collecting log files
• Getting the same results over ...
Reducing Total Runtime
• Initial runtime for all tests end to end was
  26 hours
• Test on release and optimized builds on...
Tracking Results

• Instead of one log file per test suite, we
  now have hundreds
• Initially we would concatenate the log...
Get the source:
http://wiki.mozilla.org/mobile/QAOSCON09
                             Questions?
                         ...
Photo Credits
•   Gorilla Photo: http://www.wrybaby.com/blog/wp-content/uploads/2008/09/
    gorilla_phone_1.jpg

•   HTC ...
Upcoming SlideShare
Loading in …5
×

How to stuff a 900 pound gorilla into a smartphone

1,688 views

Published on

Migrating the 100,000+ unittests for Firefox to run on mobile phones to test the new mobile product from Mozilla.

Presented at OSCON09, July 23, 2:35pm.

Published in: Technology, News & Politics
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,688
On SlideShare
0
From Embeds
0
Number of Embeds
10
Actions
Shares
0
Downloads
18
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

How to stuff a 900 pound gorilla into a smartphone

  1. Stuffing a 900lb Gorilla into a SmartPhone Joel Maher & Clint Talbert Mozilla Corporation 1
  2. What is Fennec? 2
  3. How are they going to fit me in here? 3
  4. How are they going to fit me in here? 3
  5. •XPCShell - low level XPCOM testing How are they going to fit me in here? 3
  6. •XPCShell - low level XPCOM testing •Reftest - rendering testing How are they going to fit me in here? 3
  7. •XPCShell - low level XPCOM testing •Reftest - rendering testing •Crashtest - regression testing for crash bugs How are they going to fit me in here? 3
  8. •XPCShell - low level XPCOM testing •Reftest - rendering testing •Crashtest - regression testing for crash bugs •Mochitest - JS test framework heavily modified to do general purpose content testing How are they going to fit me in here? 3
  9. •XPCShell - low level XPCOM testing •Reftest - rendering testing •Crashtest - regression testing for crash bugs •Mochitest - JS test framework heavily modified to do general purpose content testing •Mochitest-Chrome - Chrome UI interaction testing How are they going to fit me in here? 3
  10. •XPCShell - low level XPCOM testing •Reftest - rendering testing •Crashtest - regression testing for crash bugs •Mochitest - JS test framework heavily modified to do general purpose content testing •Mochitest-Chrome - Chrome UI interaction testing •Mochitest-Browser - White box UI automation testing How are they going to fit me in here? 3
  11. •XPCShell - low level XPCOM testing •Reftest - rendering testing •Crashtest - regression testing for crash bugs •Mochitest - JS test framework heavily modified to do general purpose content testing •Mochitest-Chrome - Chrome UI interaction testing •Mochitest-Browser - White box UI automation testing •Mozmill - Black box UI automation (not ready for Fennec yet) How are they going to fit me in here? 3
  12. Platforms 4
  13. Platforms 4
  14. Platforms 4
  15. Platforms 4
  16. Platforms 4
  17. Platforms 4
  18. Platforms 4
  19. Platforms 5
  20. Roadmap 6
  21. Fennec Test Automation 7
  22. Test Reporting 8
  23. Current Approach • Running tests on desktop build • Decoupling the tests • Making tools mobile • Breaking tests into small pieces • Stabilizing end to end • Reducing the runtime • Tracking results 9
  24. Decoupling the Tests • Historically, tests integrated into build system • Required directory structure and tools • Created makefile target to package tests into a tarball for easy distribution 10
  25. Making Tools Mobile • Python requirements for each platform • Networking test tools (ssl, http server) • Test harness scripts had hard coded paths • Limited memory and storage 11
  26. Breaking Tests into Chunks • Cannot run full test harness at once • Chunks based on directories • Managing test data, manifest and log files 12
  27. Stabilizing End to End • Modifying test harness to look for a hang • Collecting log files • Getting the same results over and over again • Device stabilization 13
  28. Reducing Total Runtime • Initial runtime for all tests end to end was 26 hours • Test on release and optimized builds only, no debug (13 hours) • Running on 4 devices in parallel (4 hours) • Increasing pool to 8 devices balances parallel overhead with long tests 14
  29. Tracking Results • Instead of one log file per test suite, we now have hundreds • Initially we would concatenate the logs and parse them for a summary • Upload log chunks to CouchDB server 15
  30. Get the source: http://wiki.mozilla.org/mobile/QAOSCON09 Questions? Thank you jmaher@mozilla.com ctalbert@mozilla.com 16
  31. Photo Credits • Gorilla Photo: http://www.wrybaby.com/blog/wp-content/uploads/2008/09/ gorilla_phone_1.jpg • HTC Touch Pro: http://www.letsgodigital.org/en/19174/htc-touch-pro/ • Nokia e71: http://www.flickr.com/photos/normanjohn/2645312857/ • Nokia n810: http://168hours.wordpress.com/2008/08/10/10-coolest-devices- running-linux/ • Samsung Omnia: http://www.gsmarena.com/samsung_i900_omnia- pictures-2422.php • Our Mobile Tinderbox: http://www.flickr.com/photos/drkscrtlv/sets/ 72157619162182878/ • Signpost: http://www.flickr.com/photos/jmcphotos/2131206015/ 17

×