Functionality testing involves developing test cases to test new code based on software function specifications, marketing requirements, and developer code. Test cases are the foundation of quality assurance and should cover equivalence classes, boundary values, decision tables, state transitions, and all pairs to ensure thorough coverage. Quality functionality testing requires understanding the purpose of new features, communicating with developers, thoroughly designing test cases, carefully executing tests, and reviewing results.

1. Functionality
Testing
Technique
Liang Gao

2. Agenda
• Test case development
• Test case execution

3. What makes a
functionality tester
• Functionality testing is to test the “new” code.
Input Output
Tester
Software Function Spec Test Plan/Test Cases
Defect
Marketing Requirement Spec
s
Scripts
Developer's code

4. Test Case Is Very
Important
• Development is science, Testing is an ART.
• Test case is the foundation of a QA department.
• Test case is the Crown Jewel of the testing.
• What is “Garbage in, garbage out”?

5. Exercise
• A program called “sum” is developed in Linux, it
accepts two 32-bit signed integers, add them and
display the result in the next line.
• The result is also 32-bit signed integer, if error is
detected, error message will be displayed to stderr
and error code will be returned.

6. Solutions
• What is the range of 32-bit signed integer?
• What are the boundaries conditions?
o -231 <= X + Y <= 231-1
o -231 <= X or Y <= 231-1
• What are the negative conditions?
o Out of range
o Invalid inputs
• External factors
o System resources are low
o Simultaneous execution
o Execution from another program (GUI as front-end), exit status code
• Other testing
o Performance (speed of the execution)
o Spell check on error messages (Localization?)
o Size of the program
o Versioning
o Help

7. • Input types:
Input • Arguments:
o +<#> (acceptable?) o 0, 1, 3, 10 arguments
o 600A o <#> <alphabet>
o 600.60 o “ “ <#> <#>
o <#> <100 space> <#> o 2 arguments with 200-
o <#> <tab> <#> character each
o 0x<hex>? • Error messages:
o “<#> <#>” o Ensure all error condition
o 100-digit number generates appropriate error
o 0<#>? messages in current locale
o 100 “0”? o Ensure all error the exit status
o 231 + 30  negative number? code is non-zero
o 100 “-” follow by valid o Ensure error messages are
number printed out in stderr
o %d o Ensure no mis-spell and
o &, $, #, ^G grammar errors
o Ensure consistency in error
messages (error format, error
number)

8. •
External• Factor
OS related factor: Others
o Run in different varieties of o Use “time” to measure the
Linux distro performance of sum
o Run program in low- program. The execution
memory condition (1MB of elapse time should not
memory) exceed 0.2 second; the CPU
o Develop shell scripts to usage should be less than
invoke 20 “sum” in 20% during execution
background “&” and ensure o Find out the size of the
the results are correct program, it should not be
o Run “sum” from cron more than 2MB of disk space
o Use “ldd” to find out the o Ensure the version is
shared library “sum” is using, displayed in help string
install different versions of o Ensure the help program is
shared library to ensure the available
program works o Develop a script to run
10,000 sum program
overnight
oinstallation

9. Boundary Testing
231-1 X
X X
X
X
X
X X
-231 231-1
X
X X
X
X X
X-231

10. 5 Basic Testing Case
Design Techniques
• Equivalent class
• Boundary values
• Decision tables
• State-transition
• All pair

11. Equivalence class ( 等价
类)
• 等价列划分设计方法是把所有可能的输入数据，即程序
的输入域划分成若干部分（子集），然后从每一个子集
中选取少量具有代表性的数据作为测试用例。
Equivalence
class Example

12. Equivalence class ( 等价
类)
• Case #1: domain of data is an interval
o – Class(es) of valid values
o – Class of invalid values (out of boundary, inf. and sup.)
o – Class of non member (for numerical values, characters are non
members)
• Case #2: Domain of data is a number of values
o –N classes for valid values
o –One class: absence of values
o –One class: too much values

13. Equivalence class ( 等价
类)
• Case #3: data is a set of values that are
processed in different manner
o – One class for each valid value
o – One class of all other invalid values
• Case #4: data is a constraint (form, syntax,
mean)
o –One class for valid constraint
o –One class for constraint violation

14. Boundary value testing
• Often combined with Equivalence class testing
• Boundary value testing ensures proper
functionality at the boundary (or edges) or
allowable data input. Boundary values include
maximum, minimum, just inside/outside
boundary, typical values, and error (malformed
values).

15. Boundary value testing
• Boundary value testing depends greatly on
experience
• Don’t come to Developer to ask for boundary
value advise often
• We have an practical example for boundary
testing

16. Decision tables
Decision
able Based Testin

17. State-transition diagrams
• Often used on protocol conformance test
State
ion Testing Ex

18. All Pair Testing
Closing Cost
Bank Emp
Intro Rate
Refinance
Residence
Property
Region
Credit
NAV
LTV
NIV
tier
NY L 1 fam A+ Pri 80% Yes Yes Yes Cust Yes Yes
NJ M 2 fam A Vac 90% No No No Ban No No
k
FL H 3 fam A- Inv 100
%
TX H+1 4 fam B
CA H+2 Coop <B
DC H+3 Cond
o
Othe
r

19. All Pair Testing
Twelve variables, with varying numbers of values, have
7 x 6 x 6 x 5 x 3 x 3 x 2 x 2 x 2 x 2 x 2 x 2 = 725,760
combinations of values.
“All Pairs does it in 50.” (Bernie Berger, STAREast2003)

20. All Pair Testing
• Reduce the number of tests Clean
• Multiple test cases will find same defects (all
combination is an overkill)
• Tests that cover all pairwise
o For any two parametersp1 and p2 and any valid
values v1 for p1 and v2 for p2, there is a test in which p1 has the
value v1 and p2 has the value v2.

21. In Our Example
• All pair test cases: C = 12!/(2! x (12 - 2)!) = 66
2
12
• Triple pair test case: C312 = 12!/(3! x (12 - 3)!)
• Worst case:
∑1
12 p 12
C12 = 2 − 1 = 4095
P=1

22. Can it Work?
• 97% of failure are caused by 2 feature interaction
(NIST Medical software failures 2000)

23. •
What about coverage
We measured the coverage of combinatorial design test sets for 10 Unix commands:
basename, cb, comm, crypt, sleep, sort, touch, tty, uniq, and wc. […] The pairwise
tests gave over 90 percent block coverage.[D. M. Cohen et al., 1996][…]
• a set of 29 pair-wise AETG tests gave 90% block coverage for the UNIX sort
command. We also compared pair-wise testing with random input testing and found
that pair-wise testing gave better coverage.[D. M. Cohen et al., 1997]
• The block coverage obtained for [pairwise] was comparable with that achieved by
exhaustively testing all factor combinations […][I. S. Dunietz et al., 1997]
• Our initial trial of this was on a subset Nortel's internal e-mail system where we able
cover 97% of branches with less than 100 valid and invalid testcases, as opposed to
27 trillion exhaustive testcases.[K. Burr and W. Young, 1998]
• Compared to a traditional company that would use the quasiexhaustive strategy, an
innovative company using the [Combinatorial] strategy would reduce its system
level test schedule by sixty-eight percent (68%) and save sixty-seven percent (67%)
in labor costs associated with the testing.[J. Huller, 2000]
• [Evaluating FDA recall class failures in medical devices we established that] [...] out
of the 109 reports that [were] detailed [enough], 98% showed that the problem could
have been detected by testing the device with all pairs of parameter settings.[D. R.
Wallace and D. R. Kuhn, 2001]
• More than 70% of bugs were detected with two or fewer conditions (75% for browser
and 70% for server) and approximately 90% of the bugs reported were detected with
three or fewer conditions (95% for browser and 89% for server). [...] It is interesting that
a small number of conditions (n<=6) are sufficient to detect all reported errors for the
browser and server software.[R. Kuhn and M. J. Reilly, 2002]

24. Even have a website to
promote

25. Test Case Quality
• Single Test Case Quality
o Clean
o Exact
o Repeatable
• Test Case Coverage Quality
o How do you know you are not missing critical test cases?
o After a test, if no bugs found, how confident are you to say the
software is in good quality.

26. quality
• Chances are: functionality tester might be junior.
• Use standard testing procedures whenever
possible
• Interact with programmer.
• Rome is not built in one day, so is your test plan /
test cases
o Review  modify and Review  modify and Review  modify….
o Bring developers, technical marketing engineers, product managers
into the review meeting

27. Create What ACL?
need a detailed
Bad Test Cases
CLI Here
Match What
Group? What is
• Create extended acl the group
• Match group name, what is
the cli?
• Associate match group to E-ACL Next Interface?
Need to explain
• Create action group with next interface as tunnelthat.
• Create PBR policy
• Associate pbr policy to the action and match
group
• Apply pbr policy on the ingress interface
Ingress Interface?
Which one is
the ingress
interface?

28. Functionality Testing (2
input, 2 talk)
• Understand features based on
software function specification and
Marketing Requirement Documentation.
• Talk to the one who propose this new
functionality (Why we need this feature).
• Talk to the one who wrote the function
specification
Developer Checklist

29. •
6 “Understand”
Try to understand on “Functional Points”
• Try to understand more “Use Case/Scenarios”
• Try to understand more Why we need this feature
(revenue impact)
• Try to understand more the modules will be
affected by this feature.
• Try to understand more the performance impact.
• Try to understand more the way it is going to be
implemented by the developers
OSPF Hello Protocol

30. Test Case Attributes
• Test ID
• Description
• Pre-Setup
• Platform
• Topology
Sample Test Plan
• Priority
• Complexity
• Steps/Expected results

31. Functionality Test Case
Coverage
• Need to be sure all functionality points are
covered
• Need to be sure all related modules combine test
are covered. (IPv6, HA?)
• Need to be sure to have a platform coverage list.
• Enough use cases (experience and
communication matters) (Build your Customer
Scenario Library: various sample topologies,
typical device configurations, typical
applications)

32. Code Coverage
• A quantum way to measure testing coverage.
• A special compiler patch that when build a software
image, can place mark on the new code
• After you execute your black box test cases, you can
know which code line is being executed.
• Very mature in Java (just Google “java code
coverage)
• C has open source and commercialized tools (hard to
integrate into a build environment if code base is
huge) http://gcc.gnu.org/onlinedocs/gcc/Gcov-
Intro.html

33. CLI Standard Testing Procedure Example - 1
• enable CLI, verify all options and parameters are implemented as defined
in the function spec.
• Enable CLI, do “show running-config”, verify it is in the memory
Disable CLI, do “show running-config” , verify it is not in the memory.
Do a “show xxx” to verify the CLI is taking effect.
• Repeat 1, but write the configure into flash, verify it is written in the flash
memory
Reload the box; verify the configuration is still there.
• Enable then disable CLI for at least 20 times, see if we have any memory
leak, or other instability.
If it is a CLI that can be repeated, like firewall rules. Enable many rules,
then disable many rules.
• Verify there is no typos in the CLI
Especially when there are some errors prompt messages. Verity those
messages have no typos.
• Conflict CLI on different features will report error
• CLI response time

34. CLI Standard Testing Procedure Example - 2
• Boundary testing on maximum and minimum CLI parameters
values that it can take, check with function spec to see if it is as
designed.
• Negative testing on CLI parameters on
o Numbers:
• Out of boundary numbers. 负 数？ Maximum number + 1 if
possible.
• empty
• Put a string in to a number field.
o strings:
• Strange characters. %(#*$*@#&#&^&<>:”:{}.
• long characters ( as long as the system can take it, as long
as you can type on a terminal)
• empty

35. GUI Testing Checklist
GUI Testing
Checklist

36. Review Process
Internal peer
Team Internal
review
Review
Peer review from other External peer
team that has domain review
expert knowledge
Tiger Testing Team
is a elite group of
Testing Tiger Team Review team lead and
managers that has
rich testing
experience

37. Testing Coverage Quality
•
Control
1st to see what developer wants to say
• Test area coverage Developer Checklist
o identify all functionality points (areas) on the software
function specification
o Can do code coverage if image building environment
support it
• Test type coverage (required for all new feature testing)
o CLI or GUI testing
o Functionality testing
o Negative testing/Boundary testing
o Use Case Testing
o Performance testing

38. Test case execution
quality control
• Define the platform coverage
• For both manual execution and auto execution
(running scripts)
• Detailed step by step execution logs may need
to be saved and reviews for junior engineers
• Regular rotation to keep engineer excited about
different testing areas  better execution quality
• Cross testing to discover new bugs
• Stringent schedule control

39. What If you can’t get the
•
Spec?
Whatever specs exist
• Software change memos that come with each new
internal version of the program
• User manual draft (and previous version’s manual)
• Product literature
• Published style guide and UI standards
• Published standards (C language or RFC)
• 3rd party product compatibility testing suites
• Published regulations
• Internal memos (e.g. project mgr.to engineers, describing
the feature definitions)

40. What If you can’t get the
•
Spec?
Marketing presentations, selling the concept of the product to
management
• Bug reports (responses to them)
• Reverse engineer the program
• Interview people
o Developer lead
o Tech writer
o Customer service
o Subject matter experts
o Project manager
• Look at header files, source code, database table definitions
• Specs and bug lists for all 3rd party tools that you use
• Prototypes or lab notes on the prototypes.

41. What If you can’t get the
Spec?
• •Look at compatible products
• Look at customer call records