EuroSTAR Software Testing Conference 2013 presentation on "OMG What Have We Done" by Gerlof Hoekstra. See more at: http://conference.eurostarsoftwaretesting.com/past-presentations/

1. OMG, What Have We Done !?
Gerlof Hoekstra, Atos
www.eurostarconferences.com
@esconfs
#esconfs

2.  Gerlof Hoekstra (NL)
 Started as software engineer
 Early 90’s: Testing!
 Sr Test Consultant/-Manager at Atos
 From software testing to large scale
integration testing
◦ Complex business processes
◦ Many stakeholders
◦ Many (software) suppliers

3.  I had a dream…
 Implemented standard test
procedures
 10 years later: everything still the
same
 Dream came true: but does this
make me happy?

4.  Assembling my E2E test team
 Certified test professionals or ….?
 What has happened?
 Somewhere, something must
have gone wrong….

5.  The Test Plan
 Independent Testing
 Test Reporting & The Go / No Go Advice
 Entry/exit criteria
 Defect databases
 Defect handling workflows
 Test case specifications
 Defect severity definitions
 …
 …

6. Raise a
defect
On hold, postpone defect N
Accept
new Open Assigned
fix defect
in progress
Deliver fix
to test
analyse defect Y fixed Deployed
Retest
defect
Postponed
Retest OK Y Validated Closed
N Rejected Accept
rejection
Y
N

7. The creation of the test plan involves the following activities:
1. Establishing the assignment
2. Understanding the assignment
3. Determining the test basis
4. Analysing the product risks
5. Determining the test strategy
6. Estimating the effort
7. Determining the planning
8. Allocating test units and test techniques
9. Defining the test products
10.Defining the organisation
11.Defining the infrastructure
12.Organising the management
13.Determining the test project risks and countermeasures
14.Feedback and consolidation of the plan

8. The creation of the test plan involves the following activities:
1. Establishing the assignment
2. Understanding the assignment
3. Determining the test basis
4. Analysing the product risks
5. Determining the test strategy
6. Estimating the effort
7. Determining the planning
8. Allocating test units and test techniques
9. Defining the test products
10.Defining the organisation
11.Defining the infrastructure
12.Organising the management
13.Determining the test project risks and countermeasures
14.Feedback and consolidation of the plan
Now look at this !?!

9.  Shall we strip the Test Plan to the bone and never write those
big, boring and useless documents anymore?
 Shall we strip the test planning process to the bone and never
do it alone at the office but always in heavy interaction with
our stakeholders?

10.  “Testing requires skills that developers generally don’t have”
 “Testing should be done by independent professionals, specialists”
 “You can’t let a butcher test his own meat”
 Really??
Testing is part of our life …

11.  “We strictly separate development and
testing…”
 “We pay the developing party a fixed
price for the product they deliver…”
 “We pay the testing provider a fixed
price for each defect they discover…”
 “But we subtract that from the fee for
the developing party!”

12.  Nail down entry/exit & acceptance criteria
 Hire independent test supplier for acceptance testing, preferably
fixed price based on:
◦ function points
◦ number of test cases
◦ number of requirements
◦ ---
 Test supplier executes acceptance tests
 The customer accepts

13.  Is this the way to get a product accepted by its stakeholders?
 Don’t we reduce testing to just a series of contractual checks?
 Does this customer realize what he is (not) getting?
 If we need to ‘dig somewhat deeper’: how about the contract?
 Shouldn’t we (the test providers) at least warn our customer?

14.  Numbers/Graphs/KPI’s
 # test cases executed
 # test cases passed
 $ test cases failed
 # defects found/fixed/open
 Coverage %
 Defect density
 Defect Detection %
 Lists of open defects
 Traffic lights
 GO/NO GO ADVICE

15. We have reached 95%
multiple condition
coverage
Does this mean:
 we have a quality product now?
 the product complies to the design?
 we have a usable product?
What on earth does that
mean??
Must be good, 95% is
almost everything
Multiple condition coverage

16.  If my customer asks me (the test manager) for a go/no go
advice, I consider that as a personal failure. I did not properly
do my job!
 If I did, what would my advice be worth anyway?

17.  We covered 95% of the area
 We found 60 mines
 We disarmed 58 mines
 The 2 mines we did not disarm are here
 These are MSM MKII mines containing 4Kg of
Composition B explosives
IS THIS USEFUL INFORMATION ?
WOULD YOU APPRECIATE MY GO/NO GO ADVISE?

19.  Do you consider your ‘mature’ test process as a bless or a
curse?
 Shouldn’t I complain and just be glad with what we have
accomplished?

20.  Consider you own test plan template. If you would strip it to
the bone:
◦ What parts would you immediately throw out?
◦ What should definitely be in there?
 Consider your own test planning process. How could you
make it
◦ Lighter?
◦ More effective?
◦ More interactive?
◦ Less boring?

21.  To test suppliers
◦ (Ethical question): Do you join the rat race of test commoditization/
should we warn customers for the consequences of excessive
separation of testing? How can we?
 To customers
◦ Does your purchase department really know what they are doing when
they define all those fancy KPI’s.
◦ How can you build positive incentives into test SLA’s to stimulate testers
to use their brains in stead of just following procedures and recipes?

22.  Suggestions for alternative ways of reporting?
 Can you imagine why I never use the traffic lights symbols in
test reporting?
 What would be a good alternative?