Outsourcing test services are all the rage today. But are they really faster, better, and cheaper? Shelley Rueger shares how you can improve the efficiency and effectiveness of your test process using a third-party test service. She provides guidance on how to determine if your product is a good candidate for testing services, how to select the right vendor, and how to avoid common pitfalls. Shelley discusses her team's experience as they made the transition from in-house testing to using external testing services. She addresses questions including: When should you outsource testing? When should you not? What questions should you ask a test services company before engaging them? What issues should you keep an eye out for during the transition? Leave with an actionable plan for implementing successful third-party testing within your organization—or come away with the knowledge that it will not be right for you.

1. W18
Special Topics
10/15/2014 3:00:00 PM
Implementing Outsourced
Testing Services with a Third
Party
Presented by:
Shelley Rueger
Moxie Software
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Shelley Rueger
Moxie Software
Vice president of quality assurance for The Neat Company, Shelley Rueger has more than
twenty years of experience in software development, primarily in the field of software quality
assurance. Shelley has worked at companies both large (HP, Motorola) and small (Bazaarvoice,
Moxie Software) across all types of software (desktop to mobile to SaaS, and enterprise to small
business to consumer). During her career she has specialized in building QA organizations from
the ground up by using the right tool for the right job and selecting from a menu of test
strategies that include using in-house personnel, offshore contract personnel, and third-party
services.

3. Implementing Effective
Third Party Testing
Shelley Rueger
Senior Director of Quality Assurance
Moxie

4. Agenda
 Why might you want to use a third party?
 Will it work for your product?
 Will it work for your team and company?
 How do you select a vendor?
 How do you implement third party testing successfully?
 What were our results?
 Time for questions

5. Why might you want to use a
third party?
 Team augmentation – insufficient resources available
 Reasonable cost
 Access to a wide range of
 Operating systems
 Browsers
 Mobile devices
 Access to “clean” testers who do not have previous
experience with product
 Quick turn-around time

6. Will it work for your product?
 Product is probably a good candidate if
 Product is not super-highly confidential
 Subject-matter expertise is not necessary
 Additional scarce hardware or software is not necessary
 Requirements or user story are documented fairly well
 Exploratory testing is acceptable
 Users can tolerate a (possibly) lower standard of quality

7. Will it work for your team and
company?
 Will still require some investment internally – this isn’t
free
 Requires support from QA, development, and product
management

8. How do you select a vendor?
 What types of testing can they do?
 How much do they cost? How do they bill?
 How many testers will they provide?
 Can you select testers?
 How responsive are they to issues? (What is their SLA?)
 How robust is their infrastructure?
 What assistance do they provide during the test cycle?
 Will their internal issue tracking tool integrate with yours?
 What is the process for defect verification?

9. How do you implement third
party testing successfully?
 Pick a pilot project
 Gather an internal team
 Select a vendor
 Educate the team on new process
 Try it out on a build (suggest a “free for all” test run)
 What did you learn?
 Iterate and modify

10. What were our results?
 Bugs found per build increased from 2 to 13 on average
 Mobile platforms covered increased from 8 to 40
 Test cycle duration decreased from 5 days to 2 days
 Bug verification cycle increased from ½ day to 3 days
 Internal QA staff for project decreased from 2 to 0.5
 Reduction in cost of QA for project decreased from
$180K to $36K

11. Questions?
 Contact Shelley Rueger at shelley@shelleyrueger.net