This report discusses whether automated testing or developer-testing can effectively replace manual testing. The report is available for download here: http://www.softwarevalue.com/insights/publications/ta-archives/automate-testers-teach-developers-test/ To access more Trusted Advisor articles, visit: http://www.softwarevalue.com/insights/publications/#trustedadvisor

1. ©2014 David Consulting Group Page 1 of 4
Shouldn't we be able to get rid of testers by automation
and teaching developers to test?
August 2014
Scope of this Report
To answer this question with a yes or no answer we will need to look beyond the hyperbole of the question and address three separate questions. The questions that must be address begin with whether all types of testing can be automated followed by whether automated testing is sufficient and finally whether developers can replace testers.
Can automated testing can cover needed testing?
The American Software Testing Qualification Board defines four testing levels:
1. Component (unit) testing
2. Integration (including regression) testing
3. System testing
4. Acceptance (including user, contract, operations, alpha/beta or field tests) testing
The marketplace would suggest that the answer to the question of whether all four levels of testing can be automated is “yes” albeit time, effort and resources are required. A review of software test automation tools shows that there are products available to support all four levels of testing when using structured white and black box testing techniques. There are numerous websites available that provide lists and comparisons of different testing tool sets including QAtestingtools.com and Smartbear.com. While all testing types can be automated, automation is not typically advertised as a plug and play activity. For example, in the book "Implementing Automated Software Testing,” Thom Garrett detailed how to implement test automation “implementing testing automation requires skill, resources (time and money) and a combination of commercial and home grown tools.” Most test automation tools expect those authoring tests to understand a scripting language (scripting is a form of programming). There are some testing tool manufactures such as Selenium that advertise that the tool provides a method of authoring tests without learning a scripting language.
Is automated testing sufficient?
If all types of testing can be automated then the next question that needs to be asked is whether automated testing (whether one tool or a suite) can deliver sufficiently complete testing to avoid manual testing.

2. ©2014 David Consulting Group Page 2 of 4
Most test professionals AND automation tool manufacturers unequivocally state that complete code coverage using test automation is unrealistic. Given that complete testing code coverage (and by extension, functional coverage in large projects) is not possible when implementing automation, it is recommend to prioritize which tests to automate. Prioritization is typically based on attributes such as value compared to effort, risk, and repeatability. If automated testing typically can’t yield complete coverage then experience-based testing will be required to fill the gaps so that a level of perceived sufficiency can be established. According to the ITQSB, experience based testing techniques include error guessing and exploratory testing both of which are derived from the tester's skill and intuition and their experience with similar applications and technologies.
While all types of testing can be automated, humans are needed to script the tests and when automation is not sufficient humans must perform manual testing to close the sufficiency gap.
Is there evidence that developers can replace testers?
Finally, in order to answer the question, “Shouldn't we be able to get rid of testers by automation and teaching developers to test?” we need to determine if developers can learn enough about testing to replace the need for specific, siloed testing personnel (or that testers can learn enough about development to be indistinguishable from developers). The data suggests that the answer is a definite “maybe”.
Agile methods (methods that support all of the principles identified in the Agile Manifesto) are built on the presumption of cross functional teams. Here team members are expected to perform or support activities outside of their specialties in order to maximize the performance of the team. The concept is known as T-shaped skills or T-shaped people. The horizontal bar on the T represents breath of skills each person has while the vertical bar represents their deepest area of skill and capability. The breadth of skills facilitates collaboration and swarming within the team. This concept is also known as generalizing specialists. Stated in terms of testing and development; all team members may develop and all team members may test.
Mature agile teams implement drop role designations even though some team members have more experience in coding, business analysis or testing.
Data shows that on average, teams without differentiated testers had the best overall performance of quality, productivity and responsiveness. At Agile 2014, Larry Maccherone of Rally Software presented findings mined from several thousand Agile projects. The projects were drawn from the base of projects that use Rally Software. Quality was measured in terms of post release defect density, productivity in terms of delivered stories and responsiveness in terms in duration. Further he stated that there was evidence that teams without identified testers tended to be more mature agile teams albeit the statistical relationship of undifferentiated teams to agile maturity was not something he was willing to publish. The data suggests that in agile teams there is no need to have differentiated testing personnel. What this does not necessarily mean is that knowledge of professional testing techniques does not enhance delivery quality.

3. ©2014 David Consulting Group Page 3 of 4
DCG’s data and statistical estimation models show that including team’s members with testing knowledge and expertise increase delivered quality. Supporting the need for testing expertise, in an as yet unpublished interview with Dr. David Rico, Dr Rico was asked whether his research showed that testing as a role was required on an agile project. Dr. Rico indicated that his research and observations showed that testing was being dispersed across the team and that professional testers were playing the role of testing consultant or testing lead role on the team. While testing is being dispersed across the whole team including developers, testers still have an important role in delivery. Supporting Dr. Rico’s research and comments, Tom and Mary Poppendieck’s published a case study from Rally Software in their 2006 book, Implementing Lean Software Development: From Concept to Cash. The case study showed that test first methods could be used to automate and have developers perform much of the required testing but not all.
Conclusions
Can automation and training developers to test replace the need for testers on projects? The answer requires peeling the onion one layer at a time. The three steps to deriving an answer are:
1. There is evidence that automated testing can cover all required testing types. All of the types of testing defined by the ITQSB for the Foundation level of certification can be automated. Software manufacturers offer testing suites that support automated testing ranging from component to acceptance testing.
2. At present automated testing is insufficient to completely replace manual testing. Full code and functional coverage is generally thought to be unrealistic on sizeable projects therefore some amount of manual testing is required to achieve sufficient testing.
3. Some data suggest that developers can replace testers, but that may be a perception rather than fact. Other data suggests that knowledge and testing skills improves quality and that testers provide support all other team members as testing is diffused across the team therefore are necessary. Mature agile software teams are cross-functional with team members that can support each other swarming to tasks when necessary. Data presented at Agile 2014 by Larry Maccherone of Rally Software shows that teams without differentiated testers have higher overall performance.
The performance of agile teams are building a substantial amount of evidence that they work. Don Riefer recently noted that “agile quality, as measured in terms of defect density post-release averages six percent better than experienced in plan-driven projects.” He further notes that as teams and organizations mature in their use of agile, quality improves. Shouldn't we be able to get rid of testers by automation and teaching developers to test?” If we are using cross functional teams to deliver value the answer to the question is “sort of”.

4. ©2014 David Consulting Group Page 4 of 4
Sources
 http://www.astqb.org/educational-resources/syllabi-lifecycle2.php, referenced August 20, 2014
 http://www.methodsandtools.com/archive/archive.php?id=94, referenced August 24, 2014
 For example see http://www.sencha.com/blog/automating-unit-tests, referenced August 25. Note a Google search will return a long list of similar references.
 http://www.astqb.org/educational-resources/syllabi-test4.php, referenced August 23, 2014.
 Maccherone, Larry. “Impact of Agile Quantified.” Agile 2014. Orlando, FL. July 30, 2014. Track Presentation (Note there are similar presentations by Dr. Maccherone on Slide Share.)
 Rico, David, Software Process and Measurement Podcast interview. August 21, 2014. Currently unpublished.
 Poppendieck, Mary. Implementing lean software development : from concept to cash / Mary Poppendieck, Tom Poppendieck. 2007. pp 149 - 152
 Reifer, Don. “Eleven Agile Method ‘Trends and Take Aways’ from the Reifer 2014 Quantitative Analysis of Agile Methods Study.”August 1, 2014. PDF