Why do customers every now and then find issues we didn’t find, despite all the efforts we put in testing? Is it maybe because we don’t test the system as the customer uses it? Using real-life tests based upon operational profiles will help you find the bugs before your customer does. And in case your customer still finds these nasty intermittent, hard-to-reproduce, problems, the very same technique helps you to pin-point its root cause. This presentation was held at the Dutch Testing Conference 2014, Ede

1. |
Real-life test; Real-life
situations
A practical example of the Statistical Usage Test
Dutch Testing Conference 2014
André Verschelling; Sr. test Consultant High Tech
1Real-life test; Real-life situations

2. |
Customer satisfaction
 Number one need: Involvement
 Customers want to feel like
you care about their needs,
and that you’re emotionally
invested in helping them
solve their problems
(Source: http://www.businesspundit.com/
top-10-things-that-all-customers-want/)
2Real-life test; Real-life situations

3. |
Therefore, we test
 Because testing …
… is a process that provides insight into, and advice on,
quality and the related risks
 And quality …
… is the totality of features and characteristics of a
product or service that bear on its ability to satisfy
stated or implied needs
 Thus, …
… the customer gets what he really needs, right?
3Real-life test; Real-life situations

4. |
What is happening
4Real-life test; Real-life situations
Focus
shift

5. |
Does that satisfy the customer?
 “On system level we do exploratory
testing”
 Does that really address the customer
needs …
… and does it solves his/her problems?
5Real-life test; Real-life situations

6. |
What about test design techniques?
6Real-life test; Real-life situations
Test Design
Technique
Useful for Test Design
Technique
Useful for
Decision Table Verification Evolutionary
Algorithms
Verification
Classification Tree
Method
Verification State Transition
Test
Verification
Elementary
Comparison Test
Verification Real-life test:
Statistical usage
Test
Verification &
Validation
Error Guessing Verification Rare Event Test Verification
Exploratory Testing Verification &
Validation
Semantic Test Verification
Data Cycle Test Verification Syntactic Test Verification
Process Cycle test Verification Use Case Test Verification &
Validation

7. |
 Exploratory Testing
 Only explores what the testers has interest in
 Tester in center stage, instead of user
 Actually verifying, not validating
 Use Case Testing
 Fixed and pre-defined workflows to represent user
behavior
 Based upon defined actors/users
 Real-life test: Statistical Usage Test
 Statistically correct user behavior randomly applied
 Based upon operational usage of all users
7Real-life test; Real-life situations
Validation Techniques

8. |
Statistical Usage Test
 A truly real-life test
 Operational Usage is captured in Operational Profiles
 Which operation is most likely to be performed
under which circumstances?
• Events
• ‘States’ (history classes)
• Probability of state/event combination
8Real-life test; Real-life situations
Event 1 Event 2 … Event m
History Class 1 P1,1 P1,2 .. P1,m
History Class 2 P2,1 P2,2 .. P2,m
… .. .. .. ..
History Class n Pn,1 Pn,2 .. Pn,m

9. |
Statistical Usage Test
 Example of an operational profile (PVR/VCR)
9Real-life test; Real-life situations
Events ►
History classes ▼
evStop evRewind
(X tics)
evPlay
(X tics)
evFastForward
(X tics)
evRecord
(X tics)
Standby 0 0.2 0.5 0.2 0.1
Rewind 0.3 0 0.6 0.1 0
Play
(counter < 15000)
0.5 0.3 0 0.2 0
Fast forward 0.5 0.1 0.4 0 0
Record 1 0 0 0 0
Play
(counter ≥ 15000)
0.8 0.15 0 0.05 0

10. |
Gather the Operational Profile (1)
 Formal analysis / specification
 Differentiate customers
 Differentiate users
 Differentiate system modes
 Differentiate functions
 Specify operational profile
for each [customer, user, system mode, function]
 Learn by example
 Watch and record
 Interviews
10Real-life test; Real-life situations

11. |
Gather the Operational Profile (2)
 Better: Use production data
 Log files
 Transaction data
 Usage data
 Most systems are evolutions
instead of revolutions
 Evaluate predecessors
 Evaluate comparable systems
11Real-life test; Real-life situations

12. |
Statistical Usage Test
 Test case starts with initial state (i.e. History Class 1)
 Randomly choose next event
 Example Test Case for the given Operational Profile
(= sequence of transitions):
 Test case must continue until execution profile
corresponds with Operational Profile
12Real-life test; Real-life situations
Standby(current count = 15000) | evRewind(10000 Tics) | Rewind(5000) |
evPlay(6000) | Play(11000) | evStop | Standby(11000) | evPlay(5000)
| Play(16000) | evRewind (8000) | Rewind(8000) | evPlay(3000) |
Play(11000) | etc., etc., etc.

13. |
 Everything your users do is done
 prove it
 No fixed sequence, but
random selection at runtime
 statistically correct
 Thus, caring about the users’
needs
 But what if the real user finds an anomaly?
13Real-life test; Real-life situations
Run the test

14. |
OK, there’s more to it
 Given a system (mobile X-ray system) with:
 Intermittent failures
 Unclear problem descriptions like
• “Sometimes the connection between stand and Mobile
Viewing Station is lost”
• “Sometimes images are lost”
 Problems could not be reproduced by functional
tests
 Root cause of system failure not yet found
 Log file contains not enough information on problem
14Real-life test; Real-life situations

15. |
Gather the Operational Profile
 Log file contains
information on usage
 Define most important
parameters:
 Switch on / switch off
system
 Start / stop low dose
fluoroscopy
 Start / stop high
definition fluoroscopy
 Store images
 Log file excerpt
10-02-10 14:05:34 S INIT > > > > INIT_Main:
Starting application < < < <
10-02-10 14:05:34 S INIT Build by nlybstqvp4-
sys24 on NLYBSTQVP4DT037,
10-02-10 14:05:34 M IPDR DFI Unit Serial
Number: [00-00-10-6B-22-03-01]
10-02-10 14:05:34 M IPDR DFI Unit Versions:
IN [0x1] PROC [0x1]
CTRL [0x5] DISP [0x1]
10-02-10 14:05:34 S IPDR Starting DFI selftest
10-02-10 14:38:55 M SYS USR switch on RIGHT
10-02-10 14:38:55 M SYS Xray mode continuous
fluoro
10-02-10 14:38:55 M USR START FLUORO RUN RIGHT
10-02-10 14:38:55 M SYS USR switch off at
187683
10-02-10 14:38:56 M USR STOP FLUORO RUN
10-02-10 14:38:56 M FILE Stored [1] new
image(s) on disk
10-02-10 14:38:56 M FILE Deleted [0] image(s)
from disk
15Real-life test; Real-life situations

16. |
Gather the Operational Profile
 Data from log file
16Real-life test; Real-life situations
Case
#
Start
date
dd-mm-
yy
Start Time
hh:mm:ss
End
Date
dd-mm-
yy
End Time
hh:mm:ss
Run Time
hh:mm:ss
Left
Fluo
#
Right
Fluo
#
Images
Stored
#
Total
Fluo
time
(sec)
Avg Fluo
time (sec)
1 10-02-10 14:05:34 10-02-10 14:08:16 0:02:42 0 0 0 0 0
2 10-02-10 14:08:16 10-02-10 14:57:08 0:48:52 2 24 24 15 0.576923077
3 12-02-10 7:12:11 12-02-10 10:27:44 3:15:33 9 74 75 63 0.759036145
38 9-03-10 7:15:20 09-03-10 13:57:47 6:42:27 60 1 18 58 0.950819672
39 10-03-10 7:37:01 10-03-10 7:37:37 0:00:36 0 0 0
Totals 61:27:51 565 480 521 1065
Average 1:34:34 14 12 13 28

17. |
Gather the Operational Profile
 Interpreting the data  determine history classes
 Basic profile
 A ‘case’ is from system start to system stop
 Sometimes no fluoroscopy is done at all
 Most of the times a case consist of fluoroscopy
(LDF, HDF or both)
17Real-life test; Real-life situations
History class Start
system
Stop
system
Make fluoroscopy
run(s)
System switched off 1 0 0
System started 0 0.08 0.92
Fluoro Runs completed 0 1 0

18. |
Gather the Operational Profile
 Add more details for fluoroscopy history classes
18Real-life test; Real-life situations
History class Start
system
Stop
system
Make fluoroscopy
run(s)
System switched off 1 0 0
System started 0 0.08 0.92
Fluoro Runs completed 0 1 0
1 – 7 * LDF 8 – 16 * LDF 17 – 45 * LDF 46 – 270 * LDF
Standby 45% 29% 24% 2%
+
1 – 7 * HDF 8 – 16 * HDF 17 – 78 * HDF
Standby 40% 40% 20%

19. |
Executing the test: Automate
Switch on the system
Decide on the number of fluoroscopy
runs
@nr_fluoL = 0
@nr_fluoR = 0
# start throwing the dice for the
left fluoro runs
@leftSel = rand()
case @leftSel
when 0 .. 0.41
@nr_fluoL = 0
when 0.42 .. 0.67
@nr_fluoL = 1 + rand(7)
when 0.68 .. 0.84
@nr_fluoL = 8 + rand(9)
when 0.85 .. 0.99
@nr_fluoL = 17 + rand(28)
else
@nr_fluoL = 46 + rand(225)
end
# then for the right fluoro runs (=
images being stored)
@rightSel = rand()
case @rightSel
when 0 .. 0.20
@nr_fluoR = 0
when 0.21 .. 0.59
@nr_fluoR = 1 + rand(7)
when 0.60 .. 0.78
@nr_fluoR = 8 + rand(9)
else
@nr_fluoR = 17 + rand(62)
end
Perform the selected number of runs
Switch off the system
And start all over again (endless
loop)
19Real-life test; Real-life situations

20. |
Statistical Usage Test results
 System starts showing intermittent failures
 Finally, test results in a Fatal Error (hard disk broken)
 13 days of test versus 1 month of real-life usage
 Images stored: 80719 versus 521
 Total Fluoroscopy time: 135995 sec versus 1065 sec
20Real-life test; Real-life situations
Run Time
hh:mm:ss # Left Fluo # Right Fluo
# Images
Stored
Total Fluo
time (sec)
Customer system 61:27:51 565 480 521 1065
Test system 198:19:31 71873 89548 80719 135995

21. |
Statistical Usage Test results
 Problem reproduced
 Test results have proven the data analysis
 Possible improvements on the Operational Profile
and/or the test
 Combine more user data (e.g. more log files, more
customers)
 Define actual contribution on available data (no
assumption of normal distribution)
 Automation is mandatory, e.g. using FitNesse
21Real-life test; Real-life situations

22. |
Conclusion
 Real-life testing is doing what your customer wants to
do
 Paying attention to the
customer needs
 Real-life testing helps finding
issues
 Help solving the
customers’ problems
22Real-life test; Real-life situations

23. Is your customer satisfied?
Unless you have 100%
customer satisfaction…you
must improve.
Horst Schulze