It’s one thing to be exposed to new techniques from conferences and training courses, but it’s quite another thing to apply them in real life. A major reason is that people tend to focus on learning the technique without first grasping the underlying principles. Basic testing principles, such as the pesticide paradox of software defects and defect clustering, have been known for many years. Other principles, such as “Test automation is not automatic” and “Not every software failure is a defect,” are learned by experience. Once you grasp the principle, particular techniques become more applicable and extensible. However, principles take time to learn and much practice to apply well. Randy Rice explains why true learning and application are not instant and what it takes to really absorb what we learn. Randy shows how two specific techniques—pairwise testing and risk-based testing—can be misapplied unless the key concepts are first understood. Leave knowing how to build your own set of software testing principles that can be applied in many contexts

1. K1
Keynote
5/7/2014 8:30:00 AM
Principles Before Practices:
Transform Your Testing by
Understanding Key Concepts
Presented by:
Randy Rice
Rice Consulting Services, Inc.
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Randy Rice
Rice Consulting Services, Inc.
A leading author, speaker, and consultant with more than thirty years of experience in the field
of software testing and software quality, Randy Rice has worked with organizations worldwide to
improve the quality of their information systems and optimize their testing processes. He is
coauthor (with William E. Perry) of Surviving the Top Ten Challenges of Software Testing and
Testing Dirty Systems. Randy is an officer of the American Software Testing Qualifications
Board (ASTQB). Founder, principal consultant, and trainer at Rice Consulting Services, Randy
can be contacted at riceconsulting.com where he publishes articles, newsletters, and other
content about software testing and software quality. Visit Randy’s blog.

3. 4/29/2014
1
PRINCIPLES BEFORE
PRACTICES
HOW TO TRANSFORM YOUR TESTING
PRACTICES WITH BASIC AND
ADVANCED PRINCIPLES
RANDALL W. RICE, CTAL
RICE CONSULTING
SERVICES, INC.
© 2013, Rice Consulting Services, Inc.
2
A LITTLE BACKGROUND
• Many years ago, I heard a great
analogy about conveying and
applying knowledge.
• If I were to teach someone how to
wash dishes, I could:
• Teach all the variations needed for
every type of dish, cookware and
utensil, or
• I could teach principles and let the
person adapt the methods to
whatever they needed to wash.

4. 4/29/2014
2
3
THINGS LIKE…
• Rinse off the big foodstuff first.
• Save the really messy dishes until the
end so you don't get everything else in
the sink messy too.
• Use hot water, but not too hot or else
you will scald yourself.
• Be careful with sharp knives in sudsy
water
4
SIMILARLY, IN TESTING…
• Take some sample tests early and find
where the big problem areas seem to be.
• Don't test the really complex areas at first.
Get your bearings first.
• Have strong tests, but if you make every
test strong, you may not have time to
finish.
• Early testing is good, expect when the
thing you are testing isn't ready even for
early testing.
• Test automation is not automatic.

5. 4/29/2014
3
5
COMMON QUESTIONS
• Which tools do we use in
certain situations?
• Which techniques are “best” for
a project?
• How do we adjust tools and
techniques?
6
WHAT WE WANT
No Knowledge
Mastery

6. 4/29/2014
4
7
REALISTIC VIEW
No Knowledge
Mastery
“The Dip”
Years
8
TESTING IS VERY NUANCED
• There are small to large variables on any project that require
you to:
• Select certain techniques
• Apply a technique differently
• In essence, all testing is context-driven
• And, there are small things that really make a big difference in
how we may apply a technique.

7. 4/29/2014
5
9
FROM CLASSROOM TO PRACTICE
Learning Objectives
Retention
Instruction
Exercises
Visual Aids
Application
10
BUT, IT GETS WORSE
Learning Objectives
Instruction
Visual Aids
Exercises
Retention for multiple people
Application by
multiple people
Passage of time. Needs change, organizations change

8. 4/29/2014
6
11
WHY?
• Let’s look at the lowest four levels of the Bloom
Taxonomy:
K4 (Analyze) - Assimilate
K3 (Do) - Practices
K2 (Understand) –
Principles, concepts,
theory
K1 (Remember) - Facts
The great majority
of Foundation Level
learning objectives
4 Foundation
learning objectives
12
7 GENERAL TESTING PRINCIPLES
• Testing shows the presence of defects
• Exhaustive testing is impossible
• Early testing
• Defect clustering
• Pesticide paradox
• Testing is context dependent
• Absence-of-errors fallacy
Source: ISTQB Foundation Level Syllabus

9. 4/29/2014
7
13
THERE ARE MANY OTHER IMPLICIT
PRINCIPLES
• Not every failure is a defect.
• Not every test can or should be
automated.
• Test automation doesn’t replace the
need for manual testing
• It doesn’t matter how good your test
is, if you are testing the wrong
version.
14
CLIMBING K2
• People normally dislike this level of
learning.
• Conceptual
• Theoretical
• Hands-off
• Requires deeper understanding
• Takes longer to build the basis of
understanding

10. 4/29/2014
8
15
ANOTHER CHALLENGE
• “Just show us how to use the tool.”
• The students
• “We want a course with all practical
application and little or no theory.”
• Management
16
A LITTLE TIME GOES BY…
• “We don’t know really get how to
design tests for our applications. We
just learned on the case study
example.”
• Students
• “The students passed the exam but we
are still struggling in how to implement
the techniques.”
• Management

11. 4/29/2014
9
17
AN EXAMPLE FROM MUSIC
• It is possible to learn to play an
instrument from mimicking someone else.
• The Suzuki Method
• However, you are able to play only specific
songs.
• Professional musicians, even many rock
stars, understand music theory!
• Blues, heavy metal and other styles are all
based on different types of scales and
tunings.
18
WE SEE THE SAME
THING IN SPORTS
• Why do professional baseball
players, golfers, football players need
coaches?
• Objectivity
• Accountability
• Teaching from a broader background
of experience

12. 4/29/2014
10
19
WHAT SETS APART PROFESSIONAL
MUSICIANS AND ATHLETES?
• Practice!
• Professional golfers are on the driving
range every day and play most days.
• Football players practice in extreme
conditions.
• Musicians may practice hours for one
show.
20
HOW DO TESTERS
PRACTICE?
• Try new approaches
• Delve deeper into an approach or
technique you already know
• Beta test a new technology
• Play mental games

13. 4/29/2014
11
TWO EXAMPLES OF
TEST TECHNIQUES
THAT REQUIRE
DEEPER
UNDERSTANDING
22
EXAMPLE 1 – PAIRWISE TESTING
• Key Principles
• Used when there are distinct numbers of conditions that can be
tested together, but the testing of all combinations would be
infeasible.
• Instead of testing all combinations of conditions, all unique pairs
of conditions are tested.
• Finds defects when:
• The interaction between TWO conditions are incorrect
• The conditions make sense to relate to each other
• Primarily applied by using either orthogonal arrays or pairwise
tools.
• This can be a very interesting and effective way to deal with
testing high numbers of combinations.

14. 4/29/2014
12
23
Teacher, why do we need
to know this?
24
AN EXAMPLE
The IE Tools>Advanced
Options window
53 binary conditions
1 condition with 3 options
1 condition with 4 options
253
= 9,007,199,254,740,992
X 12
108,086,391,056,891,904
Possible combinations of
conditions!
Source: James Whittaker, How to Break Software

15. 4/29/2014
13
25
THIS WOULD REQUIRE…
At one second per test execution:
108,086,391,056,891,904
360
300,239,975,158,033.067= hours
12,509,998,964,918.04 days
34,273,969,766.9 years
To test all possible combinations!
26
WITH TEST CASE OPTIMIZATION
• The number of tests that
represent unique pairings = 7
• Obtained from using the AllPairs
tool

16. 4/29/2014
14
27
WHAT RESEARCH INDICATES
• There have been many studies done
where defects were researched after
the fact.
• Research has shown that the highest
likelihood of an integration defect
relates to interaction between pairs of
items or parameters.
• www.pairwise.org
28
EXAMPLE OF RESEARCH FINDINGS
• [Evaluating FDA recall class failures in
medical devices we established that] [...]
out of the 109 reports that [were] detailed
[enough], 98% showed that the problem
could have been detected by testing the
device with all pairs of parameter settings.
• D.R. Wallace and D.R. Kuhn, 2001
• http://csrc.ncsl.nist.gov/staff/kuhn/final-rqse.pdf

17. 4/29/2014
15
29
THE LIKELIHOOD OF TRIPLE-
MODE OR HIGHER DEFECTS
Condition B Condition DCondition CCondition A Condition E
Double-mode defects
are more likely
than Triple-mode
or higher defects.
30
EXAMPLE – AN OA.4.3.2.2 ARRAY
011Test case 4
101Test case 3
110Test case 2
000Test case 1
Condition 3Condition 2Condition 1

18. 4/29/2014
16
31
EXAMPLE – WITH CONDITIONS
PoolFarmRuralTest case 4
PondHighriseRuralTest case 3
PondFarmUrbanTest case 2
PoolHighriseUrbanTest case 1
Condition 3Condition 2Condition 1
Some paired conditions may make no sense, or may be infeasible
32
ONE MORE SPIN ON
THIS
• A client was using orthogonal arrays to optimize the testing of
stock trades.
• 15 broker types
• 3 trade types
• 2 customer types
• 7 amount levels (bands)
• 4 account types
• This would yield 15*3*2*7*4 = 2,520 tests for all combinations.
• We noticed that the 15 broker types could be safely divided into
5 equivalence classes.
• 5*3*2*7*4 = 840
• This resulted in a 67% reduction in test cases before any pairwise
analysis!

19. 4/29/2014
17
33
WHAT WE’VE SEEN
• Instead of going immediately to a tool or technique with a
set of categories and conditions, let’s look first to see:
• If the technique makes sense in the situation
• How do we deal with the downsides?
• In this example, can they be used as negative tests?
• What do the stakeholders think of the technique?
• I’ve seen some get very excited
• I’ve seen some get very nervous!
• Are we willing to accept the trade-offs of pairwise testing?
• It becomes very important which categories and
conditions we test together.
• We still need an oracle to know the correct outcome.
• Maybe there is a provision for a highrise building on a farm!
34
EXAMPLE #2 – RISK-BASED
TESTING
• Key Principles
• Used when you can’t test the basic layer of functionality and/or
when you need to prioritize testing to meet delivery schedules
• The higher the risk, the more intense the test should be
• Helps by:
• Focusing on areas that might experience high levels of loss
• Giving lesser attention to trivial or less important functions
• Tracing and mitigating risks identified during testing
• Can be used in just about any application as long as you are able
to intelligently identify and measure risk.
• It is possible to miss some risks and some defects.
• Risk is a possibility, not a certainty
• Our knowledge is limited
• Some things will likely not get tested

20. 4/29/2014
18
35
Teacher, isn’t all testing
risk-based?
36
Universe of Test Cases
Case 1
Case 2
Case 3
Case 4
Case 5
Case n
Case 6
Case 7
Case 11
Case 8
Case 12
Case 13
Case 10
Case 14
Case 9
CONSIDER THE FOLLOWING

21. 4/29/2014
19
37
Universe of Test Cases
Case 1
Case 2
Case 3
Case 4
Case 5
Case n
Case 6
Case 7
Case 11
Case 8
Case 12
Case 13
Case 10
Case 14
Case 9
High-risk cases
Low-risk cases
Moderate-risk cases
THE TEST CASE UNIVERSE
SEGMENTED BY RISK
38
Universe of Test Cases
Case 1
Case 2
Case 3
Case 4
Case 5
Case n
Case 6
Case 7
Case 11
Case 8
Case 12
Case 13
Case 10
Case 14
Case 9
High-risk cases
Low-risk cases
Moderate-risk cases
TEST CASES THAT SPAN RISK LEVELS
In testing this transaction, you
span all levels of risk. How do you
prioritize risk in this scenario?

22. 4/29/2014
20
39
HOW TO APPLY
• At various levels:
• Product risk
• Functional risk
• Attribute risk – Usability, performance, etc.
• Project risk
• But what about?:
• Residual risks
• Safety critical (when everything is a “high” risk)
• When risks are:
• Difficult to identify
• Change often
• Are an excuse for reduced testing
• Difficult to mitigate and track
This is the practice level!
40
A TIP TO HELP IDENTIFY YOUR
PRACTICES
Area/Principles Implications
Test Automation
• You can’t automate everything
• Automation doesn’t replace people
• The ROI of test automation is repetition
• Select tests carefully to automate
• Consider which skills will need to be built
• To get payback, start with simple tests
that are performed a lot.
Test Design
• Test conditions form the basis of test
cases
• Many things can serve as a basis for test
design, including experience
• Before writing test cases, have a smart
way to identify test conditions.
• If you don’t have a good written basis for
testing, you can create one.
Test Evaluation
• If you can’t observe the outcome of a
test, you can’t evaluate it
• Testing is measurement
• If we can’t define a way to see, report, or
measure the outcome of a test, it does
little good to plan it.
• Since measurement of something
(defects, performance, etc.) is the
deliverable of testing, we should do it
well.

23. 4/29/2014
21
HOW TO GAIN
UNDERSTANDING
GOING FROM PRINCIPLE TO PRACTICES
42
A FEW EXPECTATIONS
• This is not easy, automatic or instant
• This is a mental process more than
anything else
• This is also the result of practice and
experience
• This is not a classroom deliverable
• Mentoring and coaching can help you see
blind spots, but they can’t make you
change.

24. 4/29/2014
22
43
WHAT REALLY HAPPENS
Instructor says, “Do it like this ” You say ”OK”
But think, “I’m not so sure” or
“I like my way better.”
44
However, I know I need to
improve my game. (I think)

25. 4/29/2014
23
45
ACTUALLY, THAT’S FINE
• Each of us know internally
the way we work best.
• An external coach can
provide tips and advice, but
you know what works for you.
• A great resource for this is
the series, “The Inner Game”
by Timothy Gallwey.
46
WHAT WE ARE TALKING ABOUT IS
CHANGE
No Knowledge
Mastery
“The Dip”
Years
Change is hard!
We often fail

26. 4/29/2014
24
47
COMPETENCE LEVELS
You do things
badly but don’t
know it.
You do things
badly but do
know it.
You do things
well but have
to think
carefully about
what you are
doing.
You do things
well and really
don’t think
much about
the
mechanics.
48
Source: Will Taylor

27. 4/29/2014
25
49
IN TAKING EXAMS
• Don’t overthink the questions.
• Read it carefully and think it through, but
don’t go back and forth too much on one
question.
• Don’t rely on practice exams.
• They can be HUGELY misleading.
• Instead, ask “Can I demonstrate the
learning objectives?”
• Read the book “Choke”.
• It explains why we get all flustered and
don’t do well on exams.
50
IMPLEMENTING PRACTICES
• Pilot projects are great.
• Learn in the small before going to
the larger scale
• Use the “planned organic” approach
• Have senior management support
• Let others copy your success!

28. 4/29/2014
26
51
FINAL THOUGHT
• “the 10,000hr rule is a definite key in
success”
― Malcolm Gladwell, Outliers: The Story of
Success
52

29. 4/29/2014
27
53
BIO - RANDALL W. RICE
• Over 35 years experience in building and
testing information systems in a variety of
industries and technical environments
• ASTQB Certified Tester – Foundation level,
Advanced level (Full)
• Director, American Software Testing
Qualification Board (ASTQB)
• Chairperson, 1995 - 2000 QAI’’’’s annual
software testing conference
• Co-author with William E. Perry, Surviving
the Top Ten Challenges of Software
Testing and Testing Dirty Systems
54
CONTACT INFORMATION
Randall W. Rice, CTAL
Rice Consulting Services, Inc.
P.O. Box 892003
Oklahoma City, OK 73170
Ph: 405-691-8075
Fax: 405-691-1441
Web site: www.riceconsulting.com
e-mail: rrice@riceconsulting.com