QA Club Kiev 18 - Test Management and Approaches

1. QA CLUB KIEV #18
TEST MANAGEMENT AND APPROACHES
B Y O L E K S A N D R M A I D A N I U K

2. BACKGROUND AND EXPERIENCE
Head of Quality Assurance Solutions at Ciklum
Co-founder at QA Club Kiev
Advisory Board Member at BrainBasket Foundation
Consultant at GoIT
Co-founder at QAExperts.pro
Co-founder at TestathonUA

3. AGENDA
1. Test Management Dependencies
2. Test Management and Agile
3. Agile alternative to Test Management
4. Test Reports
5. Test Management: When to use what?
6. Test Management comparison
7. Q&A

4. TEST MANAGEMENT DEPENDENCIES
 Domain-dependent (e-Commence, Medicine, Nuclear
Energy)
 SDLC-dependent (Waterfall, Agile)
 TeamStructure-dependent (2-5, 6-15.. 16-30 size of the
team)
 Phase-dependent (Proof of Concept, Active Development,
Maintenance)
 Environment-dependent (1 environment for testing or 10,
or 25?)
 CorporatePolicy-dependent (Microsoft, Google, licenses
purchased)
 Services-dependent (Manual, Automation, Performance,
Security etc.)

5. TEST MANAGEMENT AND AGILE
Stop use a heavy weight
specialized tool for test
management
Just stop.

6. THE AGILE ALTERNATIVE TO TEST MANAGEMENT
You need to manage the test effort for
the:
 Backlog
 Source Control Management
 Continuous Integration
 Automated Regression Tests

7. WHERE DO THE TEST LIVES?
 High-level acceptance criteria, test ideas,
Exploratory testing charters belong to the
Backlog with the associated Story;
 Technical artifacts including test automation
and manual regression test scripts belong to the
Source Control System versioned with the
associated code.

8. WHERE DO WE CAPTURE TESTING ESTIMATES?
In Agile, we ultimately care about Done
Stories.
Coded but not Tested means Not Done.
We don’t need a separate place to put
estimates.

9. HOW DO I PRIORITIZE TESTS?
Agile teams work with a prioritized backlog.
Instead of prioritizing tests, they prioritize Stories.
And Stories are either Done or not.
Given that context, it does not make sense to talk
about prioritizing the tests in isolation.

10. THERE IS NEVER ENOUGH TIME TO TEST
If the Story is important enough to code,
it’s important enough to test.
If you’re working in an Agile context it is
absolutely critical that everyone on the
team understands this.

11. TRADITIONAL: WHAT ABOUT THE TEST REPORTS?
Traditional test management systems
provide all kinds of reports: pass/fail
statistics, execution time actuals vs
estimated, planned vs executed tests, etc.
Much of this information is irrelevant in an
Agile context.
Continuous Integration results
95
%
5%

12. PROPOSED: WHAT ABOUT THE TEST REPORTS?
The CI system provides the information
that remains relevant: the automated
test execution results. And those
results should be 100% Green (passed)
most of the time.
Traditional test management
80
%
20
%

13. WHAT ABOUT HISTORICAL TEST RESULTS DATA?
Most teams find that the current CI reports are more
interesting than the historic results. If the CI build goes Red
for any reason, Agile teams stop and fix it. Thus Agile teams
don’t have the same kind of progression of pass/fail ratios
that traditional teams see during a synch and stabilize
phase. And that means historic trends usually are not all
that interesting.
However, if the team really wants to keep historic test
execution results (or are compelled to do so as a matter of
regulatory compliance), the test results can be stored in the
source control system with the code.

14. REGULATORY COMPLIANCE AND TEST MANAGEMENT
In that context, specialized test management solutions may
be the defacto standard, but they’re not the best answer. If
I’m working on a system where we have to be clear,
concrete, and explicit about requirements, tests, and
execution results, then I would much rather do Acceptance
Test Driven Development.
ATDD provides the added value of executable requirements.
Instead of the tests and requirements just saying what the
system should do, they can be executed to demonstrate that
it does.

15. ATDD REQUIRES EFFORT
Certainly, doing ATDD requires effort. But so does
maintaining a separate test management system and all the
corresponding traceability matrices and overhead
documentation.

16. TEST MANAGEMENT SYSTEMS: COMPARISON
GuRock
TestRail
HP Quality
Center
MicroFocus
QA Director
TestLodge PassMark
TestLog
Zephyr Oracle Test
Manager
IBM Rational
Quality
Manager
QASymphon
y qTest
Tricentis
Tosca
Testsuite
GoogleDocs TechExcel
DevTest
SmartBear
QaComplete
ApTest
Manager
Bugzilla
Testopia
SpiraTest MicroFocus
SilkCentral
Testuff QaTraq
Professional
TestLink

17. TEST MANAGEMENT SHOULD HELP YOU
Store test scripts in one place and CRUDs
Requirements Management
Easy and quick access to the test scripts
Calculate and show test coverage
Easy create test suites (e.g. Smoke, Regression)
Support Versioning
Release Management
Dashboards and Reporting

18. TEST MANAGEMENT: WHAT USE WHEN?
GoogleDocs – 1-5 QA team size projects, Agile, Active
Development, No Security and Corporate Policy, Any non-
critical domain for life
TestRail – 5+ QA team size, Agile, Active Development and
Maintenance, Security Policy, Automation is implemented in
the project Domain-dependent (e-Commence, Medicine, Nuclear Energy)
SDLC-dependent (Waterfall, Agile)
TeamStructure-dependent (2-5, 6-15.., 16-30 size of the team)
Phase-dependent (Proof of Concept, Active Development, Maintenance)
Environment-dependent (1 environment for testing or 10 or 25?)
CorporatePolicy-dependent (Microsoft, Google, licenses purchased)
Services-dependent (Manual, Automation, Performance, Security etc)

19. TEST MANAGEMENT SYSTEM AS A RULE
Send them the URL to this presentation. Ask them to read it.
Then ask them what additional value they’re getting out a
test management system that they wouldn’t get from
leveraging SCM, CI, the Backlog, and the automated
regression tests.

20. SO, HAVE I CONVINCED YOU?
If not, please tell me why 