This document summarizes the results of an annual QA benchmark study conducted by Pyramid in October 2019. Some key findings include: - Only 18% of companies report delivering their desired customer experience while 86% of CEOs believe they do. - Automation is having a positive impact but adoption rates are still low, with an average of 30% automation across teams. - Top challenges for QA teams include keeping up with rapid timelines, sourcing quality resources, and testing complex systems. - Companies seeking higher automation levels (75%+) tend to have more experienced QA professionals focusing on efficiency.

1. The future is bright!
October 10, 2019

2. Antony Edwards
COO, Eggplant
Carl Johanson
QA Practice Director,
Pyramid

3. 86%
18%
of testing teams are
achieving their objectives.
of companies are delivering their
desired CX.
11%of teams are able to keep up with
the pace of DevOps
30%is the average level of test
automation in SW teams§
“80% of CEOs believe they deliver a
superior customer experience, but
only 8% of their customers agree”

4. Annual QA Benchmark Study
October 2019

5. QA Benchmark Survey
“Thinking that we can
quickly move from
immature Agile to fully
implemented DevOps
with all the benefits.
Existing Agile processes
aren't even flushed out
and bringing benefits
yet.”
“Thinking that just
because you can swing a
bat, you are ready for the
World Series”
Enterprise QA Survey Responder

6. 13%
13%
37%
37%37%
13%13%
37%
Who did we survey - Respondents
Years of Experience
1-3
4-10
11-20
~18 yrs
average
20+
C-LevelOther
VP/Dir Mgr/Lead
Job Title
86%
59%
59%
46%
DevOpsDev
Ops Testing
Responsibility
(multiple allowed)
2.3
roles per
respondent
50%
executives
QA Benchmark Survey

7. 30%
70%
Who did we survey – Companies
Team Size
ISV
IT
Internal vs ISV
40%
23%
10%
27%
51-100
100-500
500+ 1-50
Developer/QA Ratio
3%
23%
30%
43%
3:1
2:1
3+
1:1
~2.8:1
average
~85
average
70%
internal
QA Benchmark Survey

8. How impactful is QA Automation?
“100s of employees
with QA in their title
that do not do QA”
80%
Positive
Impact of
QA Automation
Neutral
QA Benchmark Survey

9. How has Automation been Applied?
80%
Positive
Impact of
QA Automation
Neutral
“We need to
abandon the
old QA
mindset that
you still need
QA to
manually
test.”
25% or less
Automation
48%
75% or Greater
Automation
11%
QA Benchmark Survey

10. Where is Automation today and were is it Headed?
Performance
37%
API / Integration
34%
UI Layer
33%
Services Layer
44%
Data Validation
23%
QA Automation over 25%
85% 93% 86% 85% 74%
Now and in the next 2 years…
QA Benchmark Survey

11. “time to create test
scripts given our rapidtimelines”
Enterprise QA Survey Responders
“Sourcing qualityresources”
“Proper Strategy andDirection”
“Communication of
results to business roles ”
“Testing complex matrix”
“End to end testing for a
highly interconnectedsystem”
“automation skills andexperience ”
Top Challenges
QA Benchmark Survey

12. • More Experienced QA
Professionals (> 20 yrs.)
• More likely to be used by
customers
• More likely focusing on
efficiency, and optimization
(i.e. performance,
outsourcing, automation)
• More likely using Agile
development methodology
• More likely to have a
professional QA testing
function, rather than internal
user testing
• Top Tools are Selenium,
Eggplant, Microsoft, and
Smart Bear
For Companies
seeking 75%
Automation in
next 2 years…
QA Benchmark Survey

13. What type of impact has automated
testing had on your organization?
Respondents who value
automated testing also believe a
dedicated testing function is
important to the success of their
QA initiatives.
85%
60%
80%
80%
81%
82%
ETL / Data Validation
Services Layer
Performance
UI Layer
Integration
AxisTitle
Positive impact: Importance:
QA Benchmark Survey

14. Be a participant
(if you are a QA Professional)
To take part, just complete a 10 to15-
minute survey by linking through the QR
code at right or going to:
bit.ly/pyramidqa
(Note: link is case sensitive)
Participants will receive study results
with the ability to compare your
organization with similar organizations.
QA Benchmark 2019

15. % of Test Execution that is Automated
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019
Test automation rates are
not increasing!
18%
of companies are delivering their desired
CX.
“80% of CEOs believe they deliver a superior
customer experience, but only 8% of their
customers agree”

16. Testing
Verification and Validation
© Copyright 2018 Eggplant
Specification Software
Business
Outcomes
Users
✓ Verification Validation

17. Testing
Verification and Validation
© Copyright 2018 Eggplant
Specification Software
Business
Outcomes
Users
✓ Verification Validation

18. “The Business”Testing
© Copyright 2018 Eggplant
Software
Business
Outcomes
Users
Specification
✓ Verification ValidationVerification
Verification and Validation

19. “The Business”Testing
Specification
✓
© Copyright 2018 Eggplant
Business
Outcomes
Users
Software
86% 18%
ValidationVerification
Verification and Validation

20. “The Business”Testing
Product
Specification
✓
Business
Outcomes
Users
Software
86% 18%
Validation
Gap
Verification
Verification and Validation

21. Product
Mgmt
Exploratory
Testing
Scripted
Testing
Specification
✓
Business
Outcomes
Users
Software
ValidationVerification
Verification and Validation

22. Manual Exploratory Testing

23. Manual Exploratory Testing
⇢Focused on the USER not the specification.
⇢Uses a vast amount of knowledge about users and
risks to improve Customer Experience.
⇢Explores functionality, performance, and usability.
⇢Considers the entire universe of tests.
⇢Expensive and difficult to scale-up.
⇢Can’t keep up with continuous delivery & DevOps.
⇢Not transparent, quantitative, or replicable.

24. Customer Driven Testing
© Copyright 2018 Eggplant
Model
Selection Engine
Product Users
Validation Engine

25. Customer Driven Testing
Model
Selection Engine
Product Users
Validation Engine

26. The End