How to stuff a 900 pound gorilla into a smartphone

•

1 like•928 views

Migrating the 100,000+ unittests for Firefox to run on mobile phones to test the new mobile product from Mozilla. Presented at OSCON09, July 23, 2:35pm.

Technology News & Politics

Stufﬁng a 900lb Gorilla
into a SmartPhone
Joel Maher & Clint Talbert

Mozilla Corporation

1

•XPCShell - low level XPCOM testing

How are they going to ﬁt
me in here? 3

•XPCShell - low level XPCOM testing
•Reftest - rendering testing

How are they going to ﬁt
me in here? 3

•XPCShell - low level XPCOM testing
•Reftest - rendering testing
•Crashtest - regression testing for crash bugs

How are they going to ﬁt
me in here? 3

•XPCShell - low level XPCOM testing
•Reftest - rendering testing
•Crashtest - regression testing for crash bugs
•Mochitest - JS test framework heavily modiﬁed
to do general purpose content testing

How are they going to ﬁt
me in here? 3

Current Approach
• Running tests on desktop build
• Decoupling the tests
• Making tools mobile
• Breaking tests into small pieces
• Stabilizing end to end
• Reducing the runtime
• Tracking results
9

Decoupling the Tests

• Historically, tests integrated into build
system
• Required directory structure and tools
• Created makeﬁle target to package tests
into a tarball for easy distribution

10

Making Tools Mobile

• Python requirements for each platform
• Networking test tools (ssl, http server)
• Test harness scripts had hard coded paths
• Limited memory and storage

11

Breaking Tests into
Chunks

• Cannot run full test harness at once
• Chunks based on directories
• Managing test data, manifest and log ﬁles

12

Stabilizing End to End

• Modifying test harness to look for a hang
• Collecting log ﬁles
• Getting the same results over and over
again
• Device stabilization

13

Reducing Total Runtime
• Initial runtime for all tests end to end was
26 hours
• Test on release and optimized builds only,
no debug (13 hours)
• Running on 4 devices in parallel (4 hours)
• Increasing pool to 8 devices balances
parallel overhead with long tests

14

Tracking Results

• Instead of one log ﬁle per test suite, we
now have hundreds
• Initially we would concatenate the logs and
parse them for a summary
• Upload log chunks to CouchDB server

15

Get the source:
http://wiki.mozilla.org/mobile/QAOSCON09
Questions?
Thank you
jmaher@mozilla.com
ctalbert@mozilla.com

16

Photo Credits
• Gorilla Photo: http://www.wrybaby.com/blog/wp-content/uploads/2008/09/
gorilla_phone_1.jpg

• HTC Touch Pro: http://www.letsgodigital.org/en/19174/htc-touch-pro/

• Nokia e71: http://www.ﬂickr.com/photos/normanjohn/2645312857/

• Nokia n810: http://168hours.wordpress.com/2008/08/10/10-coolest-devices-
running-linux/

• Samsung Omnia: http://www.gsmarena.com/samsung_i900_omnia-
pictures-2422.php

• Our Mobile Tinderbox: http://www.ﬂickr.com/photos/drkscrtlv/sets/
72157619162182878/

• Signpost: http://www.ﬂickr.com/photos/jmcphotos/2131206015/

17

What's hot

Ansible E2E Testing

Roberto Perez-Rodriguez

EhTrace -- RoP Hooks

Shane Macaulay

Presentation given by Błażej Kasperczyk at Pykonik meetup in Kraków. How many applications, and where do we put them? Why is our system so bad at keeping up with what the users want? What to do in case of a noisy neighbour? When you're aiming to provide a platform where the developers could easily launch an application without worrying about configuring the system, you will have to code it sooner or later. As with most very simple concepts, it presents a plethora of challenges to deal with.

Let's build a PaaS platform, how hard could it be?

DreamLab

PostgreSQL: meet your queue

Theo Schlossnagle

From Test to Live with Rex

Jan Gehring

Openvz booth

OpenVZ

XS Boston 2008 Quantitative

The Linux Foundation

This 52min training session summarises what you need to know with regards to Tungsten Clustering architecture and topologies; and then dives into its installation process. TOPICS COVERED - Re-cap of the Architecture And Topology - Plan The Installation - Documentation - Software - Environmental Decisions - Review Pre-Requisites - Review Installation Types And Methods - Walk Through: Simple 3 Node Cluster Installation

Training Slides: 103 - Basics - Simple Tungsten Clustering Installation

Continuent

Devel::NYTProf v3 - 200908 (OUTDATED, see 201008)

Tim Bunce

So often infosec pros learn programming and scripting using things like python, bash, ruby, and perl, and using computers that aren't... well, Windows. Consequently, a large number ignore the awesomeness that is Powershell. In this presentation we'll look at why PowerShell should be in your toolkit, whether you are charged with defending (or attacking) Windows computers. Presented at BSides Jax 2014.

Who Should Use Powershell? You Should Use Powershell!

Ben Finke

Understanding salt modular sub-systems and customization

jasondenning

Rabbits, indians and... Symfony meets queueing brokers

Gaetano Giunta

Asynchronous Programming in Kotlin with Coroutines

Tobias Schürg

Making Symfony Services async with RabbitMq (and more Symfony)

Gaetano Giunta

OSv is the revolutionary new open source technology that combines the power of virtualization and micro-services architecture. This combination allows unmodified applications deployed in a virtualized environment to outperform bare-metal deployments. Yes. You've heard it right: for the first time ever we can stop asking the question of how much performance would I lose if I virtualize. OSv lets you ask a different question: how much would my application gain in performance if I virtualize it. This talk will start by looking into the architecture of OSv and the kind of optimizations it makes possible for native, unmodified applications. We will then focus on JVM-specific optimizations and specifically on speedups available to big data management distributed applications. Finally, we will look into the relationship between OSv and Docker and how that layering can help make OSv a secret sauce for turbo-charging Cloud Foundry application deployments.

OSv: probably the best OS for cloud workloads you've never hear of

rhatr

Distributing OSGi

Bram de Kruijff

RabbitMQ

Lenz Gschwendtner

DjangoZoom lightning talk

noderabbit

Parallel Running Automation Solution with Docker, Jenkins and Zalenium

Evozon Test Lab

How to Make Your Code OSGi Friendly Without Depending on OSGi - Neil Bartlett

mfrancis

What's hot (20)

Ansible E2E Testing

EhTrace -- RoP Hooks

Let's build a PaaS platform, how hard could it be?

PostgreSQL: meet your queue

From Test to Live with Rex

Openvz booth

XS Boston 2008 Quantitative

Training Slides: 103 - Basics - Simple Tungsten Clustering Installation

Devel::NYTProf v3 - 200908 (OUTDATED, see 201008)

Who Should Use Powershell? You Should Use Powershell!

Understanding salt modular sub-systems and customization

Rabbits, indians and... Symfony meets queueing brokers

Asynchronous Programming in Kotlin with Coroutines

Making Symfony Services async with RabbitMq (and more Symfony)

OSv: probably the best OS for cloud workloads you've never hear of

Distributing OSGi

RabbitMQ

DjangoZoom lightning talk

Parallel Running Automation Solution with Docker, Jenkins and Zalenium

How to Make Your Code OSGi Friendly Without Depending on OSGi - Neil Bartlett

Similar to How to stuff a 900 pound gorilla into a smartphone

Performance Benchmarking: Tips, Tricks, and Lessons Learned

Tim Callaghan

Presented at STPCon 2016. With the extensive amount of testing performed nightly on large software projects, test and verification teams often experience lengthy wait times for the availability of test results of the latest build. As we strive to identify and resolve issues as fast as possible, alternative methods of test execution have to be found. Learn how to use Jenkins to launch tests in parallel across a number of Virtual Machines, monitor execution health, and process results. Learn about various Jenkins plugins and how they contributed to the solution. Learn how to trigger downstream jobs, even if they are on separate Jenkins instances.

Test parallelization using Jenkins

Rogue Wave Software

Mozilla QA has created a new feature called the Mozmill Crowd Extension that allows any Firefox user to automatically and effortlessly run our set of basic Mozmill functional test cases against their installed Firefox browser and report the results to our central reporting server. Any user in any locale can now participate in giving us valuable feedback on how their browser is operating in their environment. Mainly targeting the thousands of Firefox nightly testers, this extension will give Mozilla unprecedented insights on how a particular nightly build is working across the globe. In this session we will describe and demonstrate how the extension works, what data is collected from the tests and how the results are reported and analyzed.

Mozmill Crowd

Henrik Skupin

Presentation about Overthere for J-Fall 2011

Vincent Partington

In this session, we will look at how Apache Flink can be used to stream anonymized API request and response data from a production environment to make sure staging environments are up-to-date and reflect the most recent features (and bugs) that comprise a service. The talk will also examine how to deal with issues of data retention, throttling, and persistence, finishing with recommendations for how to use these sandbox environments to rapidly prototype and test new features and fixes.

Virtual Flink Forward 2020: How Streaming Helps Your Staging Environment and ...

Flink Forward

Testing sync engine

Ilya Puchka

Automated testing is important. We all know that we should do it. We also know that this can be painful, for many reasons. One of the most agonizing aspects of automated testing is the handling of the data. In order to run even the simplest of tests against the user interface, a service or API or even a PL/SQL unit typically requires that a proper starting point needs to be established in the database with respect to the data. Complex set up steps need to prepare various records to ensure the test can even start and afterwards in similarly complex tear down scripts we have to clean up after the test. This session demonstrates how this hardship can be a thing of the past. Using snapshots of a test database in a Docker container with a managed test data set that supports all tests, we can create automated tests without any set up or tear down effort. These tests can run very fast, concurrently, and whenever and wherever you like them to run. This way of working opens up much higher test coverage and much increased productivity for developers and testers. This quickie demonstrates a new approach to [managing the data required for] test automation. Using Docker Containers with databases including a shared test data set and starting a fresh container for each automated test, the individual test cases become much simpler. With this approach, achieving a much higher coverage with automated testing comes within reach. Additionally, tests can be much more sophisticated - as a much richer data set is guaranteed to be available. Developing the test cases is much more productive and much more focused on the actual test, rather than the setup and tear down actions. Through the use of the experimental checkoint mechanism in Docker, restarting a database can be done in mere seconds.

Automated testing on steroids – Trick for managing test data using Docker sna...

Lucas Jellema

Getting your mobile test automation process in place - using Cucumber and Cal...

Niels Frydenholm

PHPUnit & Continuous Integration: An Introduction

alexmace

Amazon Elastic Kubernetes Service (EKS)는 표준 Kubernetes 환경에서 실행되는 어플리케이션과 완벽히 호환됩니다. AWS상에서 Kubernetes 클러스터를 생성하고, 컨테이너 어플리케이션을 배포, 관리, 확장 및 로깅, 모니터링에 대한 실습과 함께, 최근 릴리즈된 AWS IAM 권한을 Pod에 할당하는 방법 등을 Amazon EKS에서 구현하는 과정을 진행합니다.

[AWS Dev Day] 실습워크샵 | Amazon EKS 핸즈온 워크샵

Amazon Web Services Korea

Automated Testing with Docker on Steroids - nlOUG TechExperience 2018 (Amersf...

Lucas Jellema

Automated Visual Regression Testing by Dave Sadlon

QA or the Highway

Design for Scale / Surge 2010

Christopher Brown

ONOS System Test - ONS2016

Suibin Zhang

Ansible top 10 - 2018

Viresh Doshi

FIWARE Tech Summit - Docker Swarm Secrets for Creating Great FIWARE Platforms

FIWARE

The Attached slide was presented at Null Open Security/OWAP/G4H combined community event, the document shared here is a representation of Independent study on usage of Metasploit on purpose built vulnerable machine Metasploitable3. With New attack vectors such as Elastic Search API and Jenkins servers -21/01/2017 Contains 1. Introduction to Metasploit (why metasploit?) 2. Demo Setup and talked on how to- Using Metasploitable3 3. Networking with VirtualBox for personal lab 4. Auxiliary Modules (Scanners and Servers ) - Demo of snmp_enum 5. Exploit Module (searching exploits) 6. Payload types 7. Exploit Demo 1 - /exploit/multi/elasticsearch/script_mvel_rce 8. Exploit Demo 2 - /exploit/multi/http/jenkins_script_console

Metasploit For Beginners

Ramnath Shenoy

Verifying your Ansible Roles using Docker, Test Kitchen and Serverspec

Edmund Dipple

The venerable Servlet Container still has some performance tricks up its sleeve - this talk will demonstrate Apache Tomcat's stability under high load, describe some do's (and some don'ts!), explain how to performance test a Servlet-based application, troubleshoot and tune the container and your application and compare the performance characteristics of the different Tomcat connectors. The presenters will share their combined experience supporting real Tomcat applications for over 20 years and show how a few small changes can make a big, big difference.

Fastest Servlets in the West

Stuart (Pid) Williams

Version Control and Continuous Integration

Geff Henderson Chang

Similar to How to stuff a 900 pound gorilla into a smartphone (20)

Performance Benchmarking: Tips, Tricks, and Lessons Learned

Test parallelization using Jenkins

Mozmill Crowd

Presentation about Overthere for J-Fall 2011

Virtual Flink Forward 2020: How Streaming Helps Your Staging Environment and ...

Testing sync engine

Automated testing on steroids – Trick for managing test data using Docker sna...

Getting your mobile test automation process in place - using Cucumber and Cal...

PHPUnit & Continuous Integration: An Introduction

[AWS Dev Day] 실습워크샵 | Amazon EKS 핸즈온 워크샵

Automated Testing with Docker on Steroids - nlOUG TechExperience 2018 (Amersf...

Automated Visual Regression Testing by Dave Sadlon

Design for Scale / Surge 2010

ONOS System Test - ONS2016

Ansible top 10 - 2018

FIWARE Tech Summit - Docker Swarm Secrets for Creating Great FIWARE Platforms

Metasploit For Beginners

Verifying your Ansible Roles using Docker, Test Kitchen and Serverspec

Fastest Servlets in the West

Version Control and Continuous Integration

Recently uploaded

Discord is a free app offering voice, video, and text chat functionalities, primarily catering to the gaming community. It serves as a hub for users to create and join servers tailored to their interests. Discord’s ecosystem comprises servers, each functioning as a distinct online community with its own channels dedicated to specific topics or activities. Users can engage in text-based discussions, voice calls, or video chats within these channels. Understanding Discord Servers Discord servers are virtual spaces where users congregate to interact, share content, and build communities. Servers may revolve around gaming, hobbies, interests, or fandoms, providing a platform for like-minded individuals to connect. Communication Features Discord offers a range of communication tools, including text channels for messaging, voice channels for real-time audio conversations, and video channels for face-to-face interactions. These features facilitate seamless communication and collaboration. What Does NSFW Mean? The acronym NSFW stands for “Not Safe For Work,” indicating content that may be inappropriate for professional or public settings. NSFW Content NSFW content encompasses material that is sexually explicit, violent, or otherwise graphic in nature. It often includes nudity, profanity, or depictions of sensitive topics.

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

UK Journal

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

GenCyber Cyber Security Day Presentation

Michael W. Hawkins

What is a good lead in your organisation? Which leads are priority? What happens to leads? When sales and marketing give different answers to these questions, or perhaps aren't sure of the answers at all, frustrations build and opportunities are left on the table. Join us for an illuminating session with Cian McLoughlin, HubSpot Principal Customer Success Manager, as we look at that crucial piece of the customer journey in which leads are transferred from marketing to sales.

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

HampshireHUG

CNv6 Instructor Chapter 6 Quality of Service

giselly40

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

With more memory available, system performance of three Dell devices increased, which can translate to a better user experience Conclusion When your system has plenty of RAM to meet your needs, you can efficiently access the applications and data you need to finish projects and to-do lists without sacrificing time and focus. Our test results show that with more memory available, three Dell PCs delivered better performance and took less time to complete the Procyon Office Productivity benchmark. These advantages translate to users being able to complete workflows more quickly and multitask more easily. Whether you need the mobility of the Latitude 5440, the creative capabilities of the Precision 3470, or the high performance of the OptiPlex Tower Plus 7010, configuring your system with more RAM can help keep processes running smoothly, enabling you to do more without compromising performance.

Boost PC performance: How more available memory can improve productivity

Principled Technologies

Data Cloud, More than a CDP by Matt Robison

Anna Loughnan Colquhoun

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

In an era where artificial intelligence (AI) stands at the forefront of business innovation, Information Architecture (IA) is at the core of functionality. See “There’s No AI Without IA” – (from 2016 but even more relevant today) Understanding and leveraging how Information Architecture (IA) supports AI synergies between knowledge engineering and prompt engineering is critical for senior leaders looking to successfully deploy AI for internal and externally facing knowledge processes. This webinar be a high-level overview of the methodologies that can elevate AI-driven knowledge processes supporting both employees and customers. Core Insights Include: Strategic Knowledge Engineering: Delve into how structuring AI's knowledge base is required to prevent hallucinations, enable contextual retrieval of accurate information. This will include discussion of gold standard libraries of use cases support testing various LLMs and structures and configurations of knowledge base. Precision in Prompt Engineering: Learn the art of crafting prompts that direct AI to deliver targeted, relevant responses, thereby optimizing customer experiences and business outcomes. Unified Approach for Enhanced AI Performance: Explore the intersection of knowledge and prompt engineering to develop AI systems that are not only more responsive but also aligned with overarching business strategies. Guiding Principles for Implementation: Equip yourself with best practices, ethical guidelines, and strategic considerations for embedding these technologies into your business ecosystem effectively. This webinar is designed to empower business and technology leaders with the knowledge to harness the full potential of AI, ensuring their organizations not only keep pace with digital transformation but lead the charge. Join us to map a roadmap to fully leverage Information Architecture (IA) and AI chart a course towards a future where AI is a key pillar of strategic innovation and business success.

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx

Earley Information Science

Handwritten Text Recognition for manuscripts and early printed texts

Maria Levchenko

Automating Google Workspace (GWS) & more with Apps Script

wesley chun

BooK Now Call us at +918448380779 to hire a gorgeous and seductive call girl for sex. Take a Delhi Escort Service. The help of our escort agency is mostly meant for men who want sexual Indian Escorts In Delhi NCR. It should be noted that any impersonator will get 100 attention from our Young Girls Escorts in Delhi. They will assume the position of reliable allies. VIP Call Girl With Original Photos Book Tonight +918448380779 Our Cheap Price 1 Hour not available 2 Hours 5000 Full Night 8000 TAG: Call Girls in Delhi, Noida, Gurgaon, Ghaziabad, Connaught Place, Greater Kailash Delhi, Lajpat Nagar Delhi, Mayur Vihar Delhi, Chanakyapuri Delhi, New Friends Colony Delhi, Majnu Ka Tilla, Karol Bagh, Malviya Nagar, Saket, Khan Market, Noida Sector 18, Noida Sector 76, Noida Sector 51, Gurgaon Mg Road, Iffco Chowk Gurgaon, Rajiv Chowk Gurgaon All Delhi Ncr Free Home Deliver

08448380779 Call Girls In Friends Colony Women Seeking Men

Delhi Call girls

A Domino Admins Adventures (Engage 2024)

Gabriella Davis

Presentation on how to chat with PDF using ChatGPT code interpreter

naman860154

Evaluating the top large language models.pdf

ChristopherTHyatt

The presentation explores the development and application of artificial intelligence (AI) from its inception to its current status in the modern world. The term "artificial intelligence" was first coined by John McCarthy in 1956 to describe efforts to develop computer programs capable of performing tasks that typically require human intelligence. This concept was first introduced at a conference held at Dartmouth College, where programs demonstrated capabilities such as playing chess, proving theorems, and interpreting texts. In the early stages, Alan Turing contributed to the field by defining intelligence as the ability of a being to respond to certain questions intelligently, proposing what is now known as the Turing Test to evaluate the presence of intelligent behavior in machines. As the decades progressed, AI evolved significantly. The 1980s focused on machine learning, teaching computers to learn from data, leading to the development of models that could improve their performance based on their experiences. The 1990s and 2000s saw further advances in algorithms and computational power, which allowed for more sophisticated data analysis techniques, including data mining. By the 2010s, the proliferation of big data and the refinement of deep learning techniques enabled AI to become mainstream. Notable milestones included the success of Google's AlphaGo and advancements in autonomous vehicles by companies like Tesla and Waymo. A major theme of the presentation is the application of generative AI, which has been used for tasks such as natural language text generation, translation, and question answering. Generative AI uses large datasets to train models that can then produce new, coherent pieces of text or other media. The presentation also discusses the ethical implications and the need for regulation in AI, highlighting issues such as privacy, bias, and the potential for misuse. These concerns have prompted calls for comprehensive regulations to ensure the safe and equitable use of AI technologies. Artificial intelligence has also played a significant role in healthcare, particularly highlighted during the COVID-19 pandemic, where it was used in drug discovery, vaccine development, and analyzing the spread of the virus. The capabilities of AI in healthcare are vast, ranging from medical diagnostics to personalized medicine, demonstrating the technology's potential to revolutionize fields beyond just technical or consumer applications. In conclusion, AI continues to be a rapidly evolving field with significant implications for various aspects of society. The development from theoretical concepts to real-world applications illustrates both the potential benefits and the challenges that come with integrating advanced technologies into everyday life. The ongoing discussion about AI ethics and regulation underscores the importance of managing these technologies responsibly to maximize their their benefits while minimizing potential harms.

Artificial Intelligence: Facts and Myths

Joaquim Jorge

What are drone anti-jamming systems? The drone anti-jamming systems and anti-spoof technology protect against interference, jamming, and spoofing of the UAVs. To protect their security, countries are beginning to research drone anti-jamming systems, also known as drone strike weapons. The anti-jam and anti-spoof technology protects against interference, jamming and spoofing. A drone strike weapon is a drone attack weapon that can attack and destroy enemy drones. So what is so unique about this amazing system?

What Are The Drone Anti-jamming Systems Technology?

Antenna Manufacturer Coco

Scaling API-first – The story of a global engineering organization

Radu Cotescu

Recently uploaded (20)

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

GenCyber Cyber Security Day Presentation

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

CNv6 Instructor Chapter 6 Quality of Service

How to Troubleshoot Apps for the Modern Connected Worker

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Boost PC performance: How more available memory can improve productivity

Data Cloud, More than a CDP by Matt Robison

Strategies for Landing an Oracle DBA Job as a Fresher

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx

Handwritten Text Recognition for manuscripts and early printed texts

Automating Google Workspace (GWS) & more with Apps Script

08448380779 Call Girls In Friends Colony Women Seeking Men

A Domino Admins Adventures (Engage 2024)

Presentation on how to chat with PDF using ChatGPT code interpreter

Evaluating the top large language models.pdf

Artificial Intelligence: Facts and Myths

What Are The Drone Anti-jamming Systems Technology?

Scaling API-first – The story of a global engineering organization

How to stuff a 900 pound gorilla into a smartphone

1. Stufﬁng a 900lb Gorilla into a SmartPhone Joel Maher & Clint Talbert Mozilla Corporation 1

2. What is Fennec? 2

3. How are they going to ﬁt me in here? 3

4. How are they going to ﬁt me in here? 3

5. •XPCShell - low level XPCOM testing How are they going to ﬁt me in here? 3

6. •XPCShell - low level XPCOM testing •Reftest - rendering testing How are they going to ﬁt me in here? 3

7. •XPCShell - low level XPCOM testing •Reftest - rendering testing •Crashtest - regression testing for crash bugs How are they going to ﬁt me in here? 3

8. •XPCShell - low level XPCOM testing •Reftest - rendering testing •Crashtest - regression testing for crash bugs •Mochitest - JS test framework heavily modiﬁed to do general purpose content testing How are they going to ﬁt me in here? 3

9. •XPCShell - low level XPCOM testing •Reftest - rendering testing •Crashtest - regression testing for crash bugs •Mochitest - JS test framework heavily modiﬁed to do general purpose content testing •Mochitest-Chrome - Chrome UI interaction testing How are they going to ﬁt me in here? 3

10. •XPCShell - low level XPCOM testing •Reftest - rendering testing •Crashtest - regression testing for crash bugs •Mochitest - JS test framework heavily modiﬁed to do general purpose content testing •Mochitest-Chrome - Chrome UI interaction testing •Mochitest-Browser - White box UI automation testing How are they going to ﬁt me in here? 3

11. •XPCShell - low level XPCOM testing •Reftest - rendering testing •Crashtest - regression testing for crash bugs •Mochitest - JS test framework heavily modiﬁed to do general purpose content testing •Mochitest-Chrome - Chrome UI interaction testing •Mochitest-Browser - White box UI automation testing •Mozmill - Black box UI automation (not ready for Fennec yet) How are they going to ﬁt me in here? 3

12. Platforms 4

13. Platforms 4

14. Platforms 4

15. Platforms 4

16. Platforms 4

17. Platforms 4

18. Platforms 4

19. Platforms 5

20. Roadmap 6

21. Fennec Test Automation 7

22. Test Reporting 8

23. Current Approach • Running tests on desktop build • Decoupling the tests • Making tools mobile • Breaking tests into small pieces • Stabilizing end to end • Reducing the runtime • Tracking results 9

24. Decoupling the Tests • Historically, tests integrated into build system • Required directory structure and tools • Created makeﬁle target to package tests into a tarball for easy distribution 10

25. Making Tools Mobile • Python requirements for each platform • Networking test tools (ssl, http server) • Test harness scripts had hard coded paths • Limited memory and storage 11

26. Breaking Tests into Chunks • Cannot run full test harness at once • Chunks based on directories • Managing test data, manifest and log ﬁles 12

27. Stabilizing End to End • Modifying test harness to look for a hang • Collecting log ﬁles • Getting the same results over and over again • Device stabilization 13

28. Reducing Total Runtime • Initial runtime for all tests end to end was 26 hours • Test on release and optimized builds only, no debug (13 hours) • Running on 4 devices in parallel (4 hours) • Increasing pool to 8 devices balances parallel overhead with long tests 14

29. Tracking Results • Instead of one log ﬁle per test suite, we now have hundreds • Initially we would concatenate the logs and parse them for a summary • Upload log chunks to CouchDB server 15

30. Get the source: http://wiki.mozilla.org/mobile/QAOSCON09 Questions? Thank you jmaher@mozilla.com ctalbert@mozilla.com 16

31. Photo Credits • Gorilla Photo: http://www.wrybaby.com/blog/wp-content/uploads/2008/09/ gorilla_phone_1.jpg • HTC Touch Pro: http://www.letsgodigital.org/en/19174/htc-touch-pro/ • Nokia e71: http://www.flickr.com/photos/normanjohn/2645312857/ • Nokia n810: http://168hours.wordpress.com/2008/08/10/10-coolest-devices- running-linux/ • Samsung Omnia: http://www.gsmarena.com/samsung_i900_omnia- pictures-2422.php • Our Mobile Tinderbox: http://www.flickr.com/photos/drkscrtlv/sets/ 72157619162182878/ • Signpost: http://www.flickr.com/photos/jmcphotos/2131206015/ 17

How to stuff a 900 pound gorilla into a smartphone

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to How to stuff a 900 pound gorilla into a smartphone

Similar to How to stuff a 900 pound gorilla into a smartphone (20)

Recently uploaded

Recently uploaded (20)

How to stuff a 900 pound gorilla into a smartphone