Taming the Testing Beast

Taming the Testing Beast
Marc J. Balcer
Chief Architect
Model Compilers LLC

Test
Case
TAME – Test Authoring Made Easy
Test
Worksheet
TAME
Test
Case
Test
Case
Test
Case
Test
Case
Excel Open Office

Test Case
Employee
Add an expenseto an
expensereport

Employee
Add an expenseto an
expensereport
Test Case
Expense Type
Expense Date
Amount
Notes
Inputs
Expense Added
No Date Error
Bad Date Error
Amount Missing
Error
Amount Too
Large Warning
Expected Results
“Stateless”

Test Case
Expense Type
Expense Date
Amount
Notes
Inputs
Expense Added
No Date Error
Bad Date Error
Amount Missing
Error
Amount Too
Large Warning
No Project Error
Employee Error
Expected Results
Project Exists
Employee
Current
Other Items Exist
Preconditions
“Stateful”

Test
Case
TAME – Test Authoring Made Easy
Test
Worksheet
TAME
Test
Case
Test
Case
Test
Case
Test
Case
Inputs
Preconditions
Expected Results
Test Suite

How do you get
a good set of
test cases?

Generic Login Command
Input User ID Field
Empty
Too long
Has invalid characters
Valid Syntax
Input Password Field
Empty
Too long
Valid Syntax
Inputs
Characteristic
Values
(“choices”)

Kinds of Input Values
Numeric Symbolic Enumerated Composite
A number
Specified as
• Range
• Precision
(number of
decimal places)
May include a unit
of measure
A character string
Specified as
• Size
• Pattern
(regular
expression or
equivalent)
A list of values
Specified as the
set of values
May be ordered or
unordered
May have a
corresponding
numeric or
symbolic value
A single value
composed of
other values
Examples:
• Mailing Address
• Grid Coordinate

Numeric Type Examples
Type Name Description Examples
Year A four-digit integer between
1900 and one plus the current
year
2012 is good
1832 is not good
342.34 is not good
“huh” is not good
Money A US Dollar value between 0
and 1000.00 to a precision of 2
decimal places.
345.23 is good
-60 is not good
34.219 is not good
Count A nonnegative number used to
indicate a count of items
between 0 and 100.
46 is good
245 is not good
-10 is not good
Date A calendar date to the precision
of one day
15-Jan-2010 is good
4/3/2010 is good*
June 6 is not good
* How this is interpreted depends upon whether you are in the US or Europe…

Symbolic Type Examples
Type Name Description Examples
ISBN Book
Number
10 or 13 decimal digits
The last character may be an X
instead of a zero.
0-201-74804-5 is good
4-87197-308-5 is good
9781492217947 is good
9321G4E3 is not good
ISO Country
Code
2 capital letters US is good
GB is good
EUR is not good
Order Number 4 digits plus a dash plus 6
digits:
[0-9]{4}-[0-9]{6}
(or d{4}-d{6})
2014-002432 is good
1986-231321 is good
503-3324-324 is not
good
Title String 1 to 100 characters, including
non-ASCII (foreign language)
characters
Pretty much anything as
long as it’s not more that
100 characters long.

Common Input Values
• Errors
• Missing (empty, null, nothing, unchecked, …)
• Too short
• Too long
• Wrong characters (bad syntax)
• Out of range
• Acceptable Values
• Good syntax
• Special characters
(e.g. quotes, < > characters)
• Non-ASCII and non-Latin characters

Results
Errormessage
"EmptyUserID"is
displayed
UserIDTooLong
UserIDhasinvalid
characters
EmptyPassword
PasswordTooLong
UserIDorpassword
incorrect
UserIDorpassword
incorrect
Userloggedin
Input User ID Field
Empty x
Too long x
Has invalid characters x
Valid Syntax x x x
Empty x
Too long x
Valid Syntax x x x

Preconditions
Input User ID Field
Empty
Too long
Valid Syntax
Empty
Too long
Valid Syntax
Condition User
no user with the user ID
user with the user ID
Condition User Password
does not match the user ID
matches the user ID
Facts about
the environment
(usually set up
in advance)

Forming Combinations
Input User ID Field
Empty
Too long
Valid Syntax
Empty
Too long
Valid Syntax
Condition User
matches the user ID
4 choices
3 choices
2 choices
2 choices
4 × 3 × 2 × 2
= 48 test cases

All Combinations = Many Useless Test
Cases
Redundant (duplicate)
User ID empty
Password too long
User: none with the user ID
Password: does not match the user
ID
 Result: Empty User ID error
User ID: empty
Password: good syntax
User: none with the user ID
ID
 Result: Empty User ID error
Make No Sense
User ID empty
Password too long
User: user with the user ID
ID
User ID bad syntax
Password too long
User: user with the user ID
ID

Define Results
EmptyUserIDerror
UserIDTooLong
error
UserIDhasinvalid
characterserror
EmptyPassword
error
PasswordTooLong
error
UserIDorpassword
incorrecterror
UserIDorpassword
incorrecterror
Userloggedin
Input User ID Field
Empty
Too long
Valid Syntax
Empty
Too long
Valid Syntax
Condition User
matches the user ID
Expected Results

Simple Result Marking
EmptyUserIDerror
UserIDTooLong
error
UserIDhasinvalid
characterserror
EmptyPassword
error
PasswordTooLong
error
UserIDorpassword
incorrecterror
UserIDorpassword
incorrecterror
Userloggedin
Input User ID Field
Empty x
Too long x
Valid Syntax
Empty x
Too long x
Valid Syntax
Condition User
matches the user ID

Combination Results
EmptyUserIDerror
UserIDTooLong
error
UserIDhasinvalid
characterserror
EmptyPassword
error
PasswordTooLong
error
UserIDorpassword
incorrecterror
UserIDorpassword
incorrecterror
Userloggedin
Input User ID Field
Empty
Too long
Valid Syntax x
Empty
Too long
Valid Syntax x
Condition User
user with the user ID x
matches the user ID x

Multiple Paths to the Same Result
EmptyUserIDerror
UserIDTooLong
error
UserIDhasinvalid
characterserror
EmptyPassword
error
PasswordTooLong
error
UserIDorpassword
incorrecterror
UserIDorpassword
incorrecterror
Userloggedin
Input User ID Field
Empty
Too long
Valid Syntax x x
Empty
Too long
Valid Syntax x x
Condition User
no user with the user ID x
does not match the user ID x
matches the user ID
same

Fully Marked Worksheet
EmptyUserIDerror
UserIDTooLong
error
UserIDhasinvalid
characterserror
EmptyPassword
error
PasswordTooLong
error
UserIDorpassword
incorrecterror
UserIDorpassword
incorrecterror
Userloggedin
Input User ID Field
Empty x
Too long x
Valid Syntax x x x
Empty x
Too long x
Valid Syntax x x x
Condition User
user with the user ID x x

Forming Test Cases
EmptyUserIDerror
UserIDTooLong
error
UserIDhasinvalid
characterserror
EmptyPassword
error
PasswordTooLong
error
Input User ID Field
Empty x
Too long x
Valid Syntax
Empty x
Too long x
Valid Syntax
Condition User
matches the user ID
Test Case 1
Inputs:
User ID Field : Empty
Expected Results:
Empty User ID error
------------------------------------
Test Case 2
Inputs:
User ID Field : Too long
Expected Results:
User ID Too Long error
------------------------------------
Test Case 3
Inputs:
User ID Field :
Expected Results:
User ID has invalid
characters error

No Mark – Don’t Care
EmptyUserIDerror
UserIDTooLong
error
UserIDhasinvalid
characterserror
EmptyPassword
error
PasswordTooLong
error
Input User ID Field
Empty x
Too long x
Valid Syntax
Empty x
Too long x
Valid Syntax
Condition User
matches the user ID
Test Case 1
Inputs:
Expected Results:
Empty User ID error
Any choice for password
will result in this same
error message.

Default Choice
Select a default if you want
to put that “don’t care”
category into the test case.

Combine All Choices
Select all choices if you want
to combine all choices into
test cases.
Alternatively check the
category name.

Multiple Paths to the Same Result
UserIDorpassword
incorrecterror
UserIDorpassword
incorrecterror
Input User ID Field
Empty
Too long
Valid Syntax x x
Empty
Too long
Valid Syntax x x
Condition User
matches the user ID
Test Case 6
Preconditions:
User : no user with the user ID
Inputs:
User ID Field : Valid Syntax
Password Field : Valid Syntax
Expected Results:
User ID or password incorrect
--------------------------------------------
---
Test Case 7
Preconditions:
User : user with the user ID
User Password : does not
match
the user ID
Inputs:
Expected Results:

Combination Results
Userloggedin
Input User ID Field
Empty
Too long
Valid Syntax x
Empty
Too long
Valid Syntax x
Condition User
Test Case 8
Preconditions:
User : user with the user ID
User Password : matches the
user ID
Inputs:
Expected Results:
User logged in

Multiple Combinations
Generic Login Command (With Special Inputs)
Userloggedin
Input User ID Field
Empty
Too long
Valid Syntax x
Valid with non-ASCII chars x
Empty
Too long
Valid Syntax x
Valid with quotes x
Valid with HTML chars x
Condition User
Form all combinations
of these inputs.
2 × 4
= 8 test cases

Limits
Generic Login Command (With Special
Inputs)
LIMITS
Userloggedin
Input User ID Field
Empty
Too long
Valid Syntax x
Valid with non-ASCII chars 1 x
Empty
Too long
Valid Syntax x
Valid with quotes 1 x
Valid with HTML chars 1 x
Condition User
Each choice will only be
used in one test case.

Generating Test Cases
Test
CaseTest
Worksheet
TAME
Test
Case
Test
Case
Test
Case
Test
Case
Excel Open Office
Test Case List
Test Summary
Table
Test Protocol
Document

Test Case List
Test Cases for Generic Login Command (With Special
Inputs)
----------------------------------------------------
Test Case 1
Preconditions:
Inputs:
Expected Results:
Empty User ID error
----------------------------------------------------
Test Case 2
Preconditions:
Inputs:
User ID Field : Too long
Expected Results:
User ID Too Long error
----------------------------------------------------
Test Case 3
Preconditions:
Inputs:
User ID Field : Has invalid characters
Expected Results:
User ID has invalid characters error

Test Summary Table
User ID Field Password Field User User Password Results...
1 Empty Empty User ID error
2 Too long User ID Too Long error
3
Has invalid
characters
User ID has invalid characters
error
4 Empty Empty Password error
5 Too long Password Too Long error
6 Valid Syntax Valid Syntax no user with the user ID
error
7Valid Syntax Valid with quotes no user with the user ID
error
8 Valid Syntax Valid with HTML chars no user with the user ID
error
9 Valid Syntax Valid Syntax user with the user ID does not match the user ID
error
10 Valid Syntax Valid with quotes user with the user ID does not match the user ID
error
11 Valid Syntax Valid with HTML chars user with the user ID does not match the user ID
error
12 Valid Syntax Valid Syntax user with the user ID matches the user ID User logged in
13 Valid Syntax Valid with quotes user with the user ID matches the user ID User logged in
14 Valid Syntax Valid with HTML chars user with the user ID matches the user ID User logged in

Creating a Custom Protocol
How to log in:
• Click a link to go to the login page
• Enter the user ID
• Go to the password field
• Enter the password
• Click the login button
• Observe whether
• an error message is displayed, or
• the application indicates that the user is logged
in

Creating a Custom Protocol
How to log in:
• Click a link to go to the login page
• Enter the user ID
• Go to the password field
• Enter the password
• Click the login button
• Observe whether
• an error message is displayed, or
• the application indicates that the user is logged
in
We want real values
for each of these parameters.

Defining Values
LIMITS
EmptyUserIDerror
UserIDTooLong
error
UserIDhasinvalid
characterserror
EmptyPassword
error
PasswordTooLong
error
UserIDorpassword
incorrecterror
UserIDorpassword
incorrecterror
Userloggedin
UserID:
UserID:
abcdefghijklm
UserID:56%#@!*(a
UserID:gary*
UserID:charlie
UserID:garcía
Password:
Password:
56%#@!*(a
Password:Good*
Password:Bad
Password:G"oo"d
Password:Go<o>d&
Password:¿sueÑo
Input User ID Field
Empty x x
Too long x x
Has invalid characters x x
Valid Syntax x x x x x
Valid with non-ASCII chars 1 x x
Empty x x
Too long x x
Valid Syntax x x x x x
Valid with quotes 1 x x x x
Valid with HTML chars 1 x x x x
Valid with non-ASCII chars 1 x x
Condition User
no user with the user ID x x
user with the user ID x x x x x x x
does not match the user ID x x
matches the user ID x x x

Defining Values
UserID:
UserID:
abcdefghijklm
UserID:56%#@!*(a
UserID:gary*
UserID:charlie
UserID:garcía
Password:
Password:
56%#@!*(a
Password:Good*
Password:Bad
Password:G"oo"d
Password:Go<o>d&
Password:¿sueÑo
Parameter Name : value

Custom Protocol Template
Test Case «TestCaseNumber»
Preconditions
«TableStart:Precondition»«Cate
goryName»
«ChoiceName»«TableEnd:Precondition»
Inputs
«TableStart:Input»«CategoryNa
me»
«ChoiceName»«TableEnd:Input»
Results
«TableStart:Result»«ResultName»«TableEnd:Result»
Procedure:
(Warning: If you customize this section do not use numbered lists. Word MailMerge doesn’t have the
sense to reset the numbering.)
 Open the website
 Click the Login button
 Enter «User ID» into the User ID field.
 Hit tab
 Enter «Password» into the password field
Now verify that:
«TableStart:Result»
 «ResultName»
«TableEnd:Result»
Word
Mailmerge
Fields
User ID
and Password
values filled in
here

Simple Values
UserID:
UserID:
abcdefghijklm
UserID:56%#@!*(a
UserID:gary*
UserID:charlie
UserID:garcía
Password:
Password:
56%#@!*(a
Password:Good*
Password:Bad
Password:G"oo"d
Password:Go<o>d&
Password:¿sueÑo
Input User ID Field
Empty x
Too long x
Valid Syntax x x
Empty x
Too long x
Valid Syntax x x
Valid with quotes x
Condition User
user with the user ID x x x x x
matches the user ID x x

Complex Values
UserID:
UserID:
abcdefghijklm
UserID:56%#@!*(a
UserID:gary*
UserID:charlie
UserID:garcía
Password:
Password:
56%#@!*(a
Password:Good*
Password:Bad
Password:G"oo"d
Password:Go<o>d&
Password:¿sueÑo
Input User ID Field
Empty x
Too long x
Valid Syntax x x
Empty x
Too long x
Valid Syntax x x
Valid with quotes x
Condition User

Default Values
LIMITS
EmptyUserIDerror
UserIDTooLong
error
UserID:
abcdefghijklm
UserID:56%#@!*(a
UserID:gary*
UserID:charlie
UserID:garcía
Password:
Password:
56%#@!*(a
Password:Good*
Password:Bad
Password:G"oo"d
Password:Go<o>d&
Password:¿sueÑo
Input User ID Field
Empty x
Too long x x
Valid Syntax x x
Empty x
Too long x
Valid Syntax x x
Valid with quotes 1 x
Valid with HTML chars 1 x
Condition User
* indicates the
default value

Values Applied to the Protocol
Test Case 7
Preconditions
User no user with the user ID
Inputs
User ID Field Valid Syntax
Password Field Valid with quotes
Results
User ID or password incorrect error
Procedure:
 Enter charlie into the User ID field.
 Hit tab
 Enter G"oo"d into the password field
Now verify that:
 User ID or password incorrect error

Going Beyond the Basics
• Create variability in procedures
• Action Choices
• Using TAME when bugs are found
• Root Cause Analysis
• Modify the Tests
• Scaling Up
• Higher Level Tests
• Results become Preconditions

Additional variability
Test Case 7
Preconditions
Inputs
Results
Procedure:
 Hit tab
Now verify that:
- Enter a URL directly
- Click a link on the company
homepage
- Click a link on an unsecured page
- Tab to the next field
- Click into the password field
- Click the Login button
- Tab to the Login button and
press Enter

Action Choices
Test Case 7
Preconditions
Inputs
Results
Procedure:
 Hit tab
Now verify that:
• Enter a URL directly
• Click a link on the company
homepage
• Click a link on an unsecured page
• Tab to the next field
• Click into the password field
• Click the Login button
• Tab to the Login button and
press Enter
For each of these
• Create “action” choices
• Use the action choices
in the protocol

Unforseen Circumstances
A user ID that includes
spaces causes a crash
Three failed logins
lock the user’s account
• Add “contains spaces” as a
choice
• Add a value definition that
includes spaces
• Determine how to handle a user
ID that includes spaces: is it
allowed or is it an error?
• Add “Preceding failed logins” as a
new preconditions with several
choices
• Determine whether a locked
account presents a different error
message
• Add result selectors

Scaling Up
Employeehas
expensesto
report
Log In
CreateNew
ExpenseReport
Add Expense
Item
Submit to
Manager
Expense
Report
Submitted
Open
ExpenseApp
already logged in
not logged in
Login is just one step in a larger process

Scaling Up
Employeehas
expensesto
report
Log In
CreateNew
ExpenseReport
Add Expense
Item
Submit to
Manager
Expense
Report
Submitted
Open
ExpenseApp
already logged in
not logged in
Create separate
test specifications
for each step
in the process

Scaling Up – Business Processes
EmployeeManagerController
Employeehas
expensesto
report
Record
Expenses
Approve
Expense
Report
Pay
Reimbursement
Reimbursement
Reject
Expense
Report
Resubmit
Expense
Report
Submitted
Expense
Report
Rejected
Expense
Report
Resubmitted
Expense
Report
Approved
Expense
Report
Review
New
Expense
Report
Review
Resubmitted
Expense
Report
Cancel
Expense
Report
ExpenseReport
hasbeen
canceled
Consider
whether
to
resubmit
Employeehas
expensesto
report
Log In
CreateNew
ExpenseReport
Add Expense
Item
Submit to
Manager
Expense
Report
Submitted
Open
ExpenseApp
already logged in
not logged in

Comparative Testing Tools
• Run Tests
• White-box, code-centric
NUnit, JUnit
• UI-based, capture-playback
Ranorex, Telerik
• Design by Contract
• Write a specification, construct a single test
case
Cucumber / Gherkin

Where to use TAME
• Unit testing
• specify and generate test cases that can run
against software modules (classes, functions).
• GUI testing
• whether a traditional installed application, a
website, or a mobile app, use TAME to specify
the contents of UI pages, then generate test
cases to validate user-facing behavior.
• Regression testing
• update the test specification and the test cases
whenever new features are deployed or bugs
are fixed.

Where to use TAME
• Project Planning
• create initial test specifications to estimate the
number of test cases to be run and project the
cost of the test effort.
• Retrospectives
• use the test specifications to probe features to
better understand the root causes of failures.

TAME for All
• TAME helps the analyst
to think through what needs to be tested.
• TAME gives the tester
a platform for creating many test cases quickly.
• TAME gives the project manager
a means for sizing and scoping the testing effort.

TAME software
Training Courses
“Writing Testable Requirements
and Effective Test Cases”
.
“Modeling Techniques
for Business Analysts”

Thank you
Marc J. Balcer
Chief Architect
Model Compilers LLC
marc@modelcompilers.com
http://www.modelcompilers.com

Taming the Testing Beast

Recommended

Recommended

More Related Content

Similar to Taming the Testing Beast

Similar to Taming the Testing Beast (20)

Recently uploaded

Recently uploaded (20)

Taming the Testing Beast