3. What makes a
functionality tester
• Functionality testing is to test the “new” code.
Input Output
Tester
Software Function Spec Test Plan/Test Cases
Defect
Marketing Requirement Spec
s
Scripts
Developer's code
4. Test Case Is Very
Important
• Development is science, Testing is an ART.
• Test case is the foundation of a QA department.
• Test case is the Crown Jewel of the testing.
• What is “Garbage in, garbage out”?
5. Exercise
• A program called “sum” is developed in Linux, it
accepts two 32-bit signed integers, add them and
display the result in the next line.
• The result is also 32-bit signed integer, if error is
detected, error message will be displayed to stderr
and error code will be returned.
6. Solutions
• What is the range of 32-bit signed integer?
• What are the boundaries conditions?
o -231 <= X + Y <= 231-1
o -231 <= X or Y <= 231-1
• What are the negative conditions?
o Out of range
o Invalid inputs
• External factors
o System resources are low
o Simultaneous execution
o Execution from another program (GUI as front-end), exit status code
• Other testing
o Performance (speed of the execution)
o Spell check on error messages (Localization?)
o Size of the program
o Versioning
o Help
7. • Input types:
Input • Arguments:
o +<#> (acceptable?) o 0, 1, 3, 10 arguments
o 600A o <#> <alphabet>
o 600.60 o “ “ <#> <#>
o <#> <100 space> <#> o 2 arguments with 200-
o <#> <tab> <#> character each
o 0x<hex>? • Error messages:
o “<#> <#>” o Ensure all error condition
o 100-digit number generates appropriate error
o 0<#>? messages in current locale
o 100 “0”? o Ensure all error the exit status
o 231 + 30 negative number? code is non-zero
o 100 “-” follow by valid o Ensure error messages are
number printed out in stderr
o %d o Ensure no mis-spell and
o &, $, #, ^G grammar errors
o Ensure consistency in error
messages (error format, error
number)
8. •
External• Factor
OS related factor: Others
o Run in different varieties of o Use “time” to measure the
Linux distro performance of sum
o Run program in low- program. The execution
memory condition (1MB of elapse time should not
memory) exceed 0.2 second; the CPU
o Develop shell scripts to usage should be less than
invoke 20 “sum” in 20% during execution
background “&” and ensure o Find out the size of the
the results are correct program, it should not be
o Run “sum” from cron more than 2MB of disk space
o Use “ldd” to find out the o Ensure the version is
shared library “sum” is using, displayed in help string
install different versions of o Ensure the help program is
shared library to ensure the available
program works o Develop a script to run
10,000 sum program
overnight
oinstallation
9. Boundary Testing
231-1 X
X X
X
X
X
X X
-231 231-1
X
X X
X
X X
X-231
10. 5 Basic Testing Case
Design Techniques
• Equivalent class
• Boundary values
• Decision tables
• State-transition
• All pair
11. Equivalence class ( 等价
类)
• 等价列划分设计方法是把所有可能的输入数据，即程序
的输入域划分成若干部分（子集），然后从每一个子集
中选取少量具有代表性的数据作为测试用例。
Equivalence
class Example
12. Equivalence class ( 等价
类)
• Case #1: domain of data is an interval
o – Class(es) of valid values
o – Class of invalid values (out of boundary, inf. and sup.)
o – Class of non member (for numerical values, characters are non
members)
• Case #2: Domain of data is a number of values
o –N classes for valid values
o –One class: absence of values
o –One class: too much values
13. Equivalence class ( 等价
类)
• Case #3: data is a set of values that are
processed in different manner
o – One class for each valid value
o – One class of all other invalid values
• Case #4: data is a constraint (form, syntax,
mean)
o –One class for valid constraint
o –One class for constraint violation
14. Boundary value testing
• Often combined with Equivalence class testing
• Boundary value testing ensures proper
functionality at the boundary (or edges) or
allowable data input. Boundary values include
maximum, minimum, just inside/outside
boundary, typical values, and error (malformed
values).
15. Boundary value testing
• Boundary value testing depends greatly on
experience
• Don’t come to Developer to ask for boundary
value advise often
• We have an practical example for boundary
testing
18. All Pair Testing
Closing Cost
Bank Emp
Intro Rate
Refinance
Residence
Property
Region
Credit
NAV
LTV
NIV
tier
NY L 1 fam A+ Pri 80% Yes Yes Yes Cust Yes Yes
NJ M 2 fam A Vac 90% No No No Ban No No
k
FL H 3 fam A- Inv 100
%
TX H+1 4 fam B
CA H+2 Coop <B
DC H+3 Cond
o
Othe
r
19. All Pair Testing
Twelve variables, with varying numbers of values, have
7 x 6 x 6 x 5 x 3 x 3 x 2 x 2 x 2 x 2 x 2 x 2 = 725,760
combinations of values.
“All Pairs does it in 50.” (Bernie Berger, STAREast2003)
20. All Pair Testing
• Reduce the number of tests Clean
• Multiple test cases will find same defects (all
combination is an overkill)
• Tests that cover all pairwise
o For any two parametersp1 and p2 and any valid
values v1 for p1 and v2 for p2, there is a test in which p1 has the
value v1 and p2 has the value v2.
21. In Our Example
• All pair test cases: C = 12!/(2! x (12 - 2)!) = 66
2
12
• Triple pair test case: C312 = 12!/(3! x (12 - 3)!)
• Worst case:
∑1
12 p 12
C12 = 2 − 1 = 4095
P=1
22. Can it Work?
• 97% of failure are caused by 2 feature interaction
(NIST Medical software failures 2000)
23. •
What about coverage
We measured the coverage of combinatorial design test sets for 10 Unix commands:
basename, cb, comm, crypt, sleep, sort, touch, tty, uniq, and wc. […] The pairwise
tests gave over 90 percent block coverage.[D. M. Cohen et al., 1996][…]
• a set of 29 pair-wise AETG tests gave 90% block coverage for the UNIX sort
command. We also compared pair-wise testing with random input testing and found
that pair-wise testing gave better coverage.[D. M. Cohen et al., 1997]
• The block coverage obtained for [pairwise] was comparable with that achieved by
exhaustively testing all factor combinations […][I. S. Dunietz et al., 1997]
• Our initial trial of this was on a subset Nortel's internal e-mail system where we able
cover 97% of branches with less than 100 valid and invalid testcases, as opposed to
27 trillion exhaustive testcases.[K. Burr and W. Young, 1998]
• Compared to a traditional company that would use the quasiexhaustive strategy, an
innovative company using the [Combinatorial] strategy would reduce its system
level test schedule by sixty-eight percent (68%) and save sixty-seven percent (67%)
in labor costs associated with the testing.[J. Huller, 2000]
• [Evaluating FDA recall class failures in medical devices we established that] [...] out
of the 109 reports that [were] detailed [enough], 98% showed that the problem could
have been detected by testing the device with all pairs of parameter settings.[D. R.
Wallace and D. R. Kuhn, 2001]
• More than 70% of bugs were detected with two or fewer conditions (75% for browser
and 70% for server) and approximately 90% of the bugs reported were detected with
three or fewer conditions (95% for browser and 89% for server). [...] It is interesting that
a small number of conditions (n<=6) are sufficient to detect all reported errors for the
browser and server software.[R. Kuhn and M. J. Reilly, 2002]
25. Test Case Quality
• Single Test Case Quality
o Clean
o Exact
o Repeatable
• Test Case Coverage Quality
o How do you know you are not missing critical test cases?
o After a test, if no bugs found, how confident are you to say the
software is in good quality.
26. quality
• Chances are: functionality tester might be junior.
• Use standard testing procedures whenever
possible
• Interact with programmer.
• Rome is not built in one day, so is your test plan /
test cases
o Review modify and Review modify and Review modify….
o Bring developers, technical marketing engineers, product managers
into the review meeting
27. Create What ACL?
need a detailed
Bad Test Cases
CLI Here
Match What
Group? What is
• Create extended acl the group
• Match group name, what is
the cli?
• Associate match group to E-ACL Next Interface?
Need to explain
• Create action group with next interface as tunnelthat.
• Create PBR policy
• Associate pbr policy to the action and match
group
• Apply pbr policy on the ingress interface
Ingress Interface?
Which one is
the ingress
interface?
28. Functionality Testing (2
input, 2 talk)
• Understand features based on
software function specification and
Marketing Requirement Documentation.
• Talk to the one who propose this new
functionality (Why we need this feature).
• Talk to the one who wrote the function
specification
Developer Checklist
29. •
6 “Understand”
Try to understand on “Functional Points”
• Try to understand more “Use Case/Scenarios”
• Try to understand more Why we need this feature
(revenue impact)
• Try to understand more the modules will be
affected by this feature.
• Try to understand more the performance impact.
• Try to understand more the way it is going to be
implemented by the developers
OSPF Hello Protocol
30. Test Case Attributes
• Test ID
• Description
• Pre-Setup
• Platform
• Topology
Sample Test Plan
• Priority
• Complexity
• Steps/Expected results
31. Functionality Test Case
Coverage
• Need to be sure all functionality points are
covered
• Need to be sure all related modules combine test
are covered. (IPv6, HA?)
• Need to be sure to have a platform coverage list.
• Enough use cases (experience and
communication matters) (Build your Customer
Scenario Library: various sample topologies,
typical device configurations, typical
applications)
32. Code Coverage
• A quantum way to measure testing coverage.
• A special compiler patch that when build a software
image, can place mark on the new code
• After you execute your black box test cases, you can
know which code line is being executed.
• Very mature in Java (just Google “java code
coverage)
• C has open source and commercialized tools (hard to
integrate into a build environment if code base is
huge) http://gcc.gnu.org/onlinedocs/gcc/Gcov-
Intro.html
33. CLI Standard Testing Procedure Example - 1
• enable CLI, verify all options and parameters are implemented as defined
in the function spec.
• Enable CLI, do “show running-config”, verify it is in the memory
Disable CLI, do “show running-config” , verify it is not in the memory.
Do a “show xxx” to verify the CLI is taking effect.
• Repeat 1, but write the configure into flash, verify it is written in the flash
memory
Reload the box; verify the configuration is still there.
• Enable then disable CLI for at least 20 times, see if we have any memory
leak, or other instability.
If it is a CLI that can be repeated, like firewall rules. Enable many rules,
then disable many rules.
• Verify there is no typos in the CLI
Especially when there are some errors prompt messages. Verity those
messages have no typos.
• Conflict CLI on different features will report error
• CLI response time
34. CLI Standard Testing Procedure Example - 2
• Boundary testing on maximum and minimum CLI parameters
values that it can take, check with function spec to see if it is as
designed.
• Negative testing on CLI parameters on
o Numbers:
• Out of boundary numbers. 负 数？ Maximum number + 1 if
possible.
• empty
• Put a string in to a number field.
o strings:
• Strange characters. %(#*$*@#&#&^&<>:”:{}.
• long characters ( as long as the system can take it, as long
as you can type on a terminal)
• empty
36. Review Process
Internal peer
Team Internal
review
Review
Peer review from other External peer
team that has domain review
expert knowledge
Tiger Testing Team
is a elite group of
Testing Tiger Team Review team lead and
managers that has
rich testing
experience
37. Testing Coverage Quality
•
Control
1st to see what developer wants to say
• Test area coverage Developer Checklist
o identify all functionality points (areas) on the software
function specification
o Can do code coverage if image building environment
support it
• Test type coverage (required for all new feature testing)
o CLI or GUI testing
o Functionality testing
o Negative testing/Boundary testing
o Use Case Testing
o Performance testing
38. Test case execution
quality control
• Define the platform coverage
• For both manual execution and auto execution
(running scripts)
• Detailed step by step execution logs may need
to be saved and reviews for junior engineers
• Regular rotation to keep engineer excited about
different testing areas better execution quality
• Cross testing to discover new bugs
• Stringent schedule control
39. What If you can’t get the
•
Spec?
Whatever specs exist
• Software change memos that come with each new
internal version of the program
• User manual draft (and previous version’s manual)
• Product literature
• Published style guide and UI standards
• Published standards (C language or RFC)
• 3rd party product compatibility testing suites
• Published regulations
• Internal memos (e.g. project mgr.to engineers, describing
the feature definitions)
40. What If you can’t get the
•
Spec?
Marketing presentations, selling the concept of the product to
management
• Bug reports (responses to them)
• Reverse engineer the program
• Interview people
o Developer lead
o Tech writer
o Customer service
o Subject matter experts
o Project manager
• Look at header files, source code, database table definitions
• Specs and bug lists for all 3rd party tools that you use
• Prototypes or lab notes on the prototypes.
41. What If you can’t get the
Spec?
• •Look at compatible products
• Look at customer call records
