Functionality testing techniqu


Published on

Published in: Technology, Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Functionality testing techniqu

  1. 1. Functionality Testing Technique Liang Gao
  2. 2. Agenda• Test case development• Test case execution
  3. 3. What makes a functionality tester • Functionality testing is to test the “new” code. Input Output TesterSoftware Function Spec Test Plan/Test Cases DefectMarketing Requirement Spec s Scripts Developers code
  4. 4. Test Case Is Very Important• Development is science, Testing is an ART.• Test case is the foundation of a QA department.• Test case is the Crown Jewel of the testing.• What is “Garbage in, garbage out”?
  5. 5. Exercise• A program called “sum” is developed in Linux, it accepts two 32-bit signed integers, add them and display the result in the next line.• The result is also 32-bit signed integer, if error is detected, error message will be displayed to stderr and error code will be returned.
  6. 6. Solutions• What is the range of 32-bit signed integer?• What are the boundaries conditions? o -231 <= X + Y <= 231-1 o -231 <= X or Y <= 231-1• What are the negative conditions? o Out of range o Invalid inputs• External factors o System resources are low o Simultaneous execution o Execution from another program (GUI as front-end), exit status code• Other testing o Performance (speed of the execution) o Spell check on error messages (Localization?) o Size of the program o Versioning o Help
  7. 7. • Input types: Input • Arguments: o +<#> (acceptable?) o 0, 1, 3, 10 arguments o 600A o <#> <alphabet> o 600.60 o “ “ <#> <#> o <#> <100 space> <#> o 2 arguments with 200- o <#> <tab> <#> character each o 0x<hex>? • Error messages: o “<#> <#>” o Ensure all error condition o 100-digit number generates appropriate error o 0<#>? messages in current locale o 100 “0”? o Ensure all error the exit status o 231 + 30  negative number? code is non-zero o 100 “-” follow by valid o Ensure error messages are number printed out in stderr o %d o Ensure no mis-spell and o &, $, #, ^G grammar errors o Ensure consistency in error messages (error format, error number)
  8. 8. • External• Factor OS related factor: Others o Run in different varieties of o Use “time” to measure the Linux distro performance of sum o Run program in low- program. The execution memory condition (1MB of elapse time should not memory) exceed 0.2 second; the CPU o Develop shell scripts to usage should be less than invoke 20 “sum” in 20% during execution background “&” and ensure o Find out the size of the the results are correct program, it should not be o Run “sum” from cron more than 2MB of disk space o Use “ldd” to find out the o Ensure the version is shared library “sum” is using, displayed in help string install different versions of o Ensure the help program is shared library to ensure the available program works o Develop a script to run 10,000 sum program overnight oinstallation
  9. 9. Boundary Testing 231-1 X X X X X X X X -231 231-1 X X X X X X X-231
  10. 10. 5 Basic Testing Case Design Techniques• Equivalent class• Boundary values• Decision tables• State-transition• All pair
  11. 11. Equivalence class ( 等价 类)• 等价列划分设计方法是把所有可能的输入数据,即程序 的输入域划分成若干部分(子集),然后从每一个子集 中选取少量具有代表性的数据作为测试用例。 Equivalence class Example
  12. 12. Equivalence class ( 等价 类)• Case #1: domain of data is an interval o – Class(es) of valid values o – Class of invalid values (out of boundary, inf. and sup.) o – Class of non member (for numerical values, characters are non members)• Case #2: Domain of data is a number of values o –N classes for valid values o –One class: absence of values o –One class: too much values
  13. 13. Equivalence class ( 等价 类)• Case #3: data is a set of values that are processed in different manner o – One class for each valid value o – One class of all other invalid values• Case #4: data is a constraint (form, syntax, mean) o –One class for valid constraint o –One class for constraint violation
  14. 14. Boundary value testing• Often combined with Equivalence class testing• Boundary value testing ensures proper functionality at the boundary (or edges) or allowable data input. Boundary values include maximum, minimum, just inside/outside boundary, typical values, and error (malformed values).
  15. 15. Boundary value testing• Boundary value testing depends greatly on experience• Don’t come to Developer to ask for boundary value advise often• We have an practical example for boundary testing
  16. 16. Decision tables Decisionable Based Testin
  17. 17. State-transition diagrams • Often used on protocol conformance test Stateion Testing Ex
  18. 18. All Pair Testing Closing Cost Bank Emp Intro Rate Refinance Residence PropertyRegion Credit NAV LTV NIV tierNY L 1 fam A+ Pri 80% Yes Yes Yes Cust Yes YesNJ M 2 fam A Vac 90% No No No Ban No No kFL H 3 fam A- Inv 100 %TX H+1 4 fam BCA H+2 Coop <BDC H+3 Cond oOthe r
  19. 19. All Pair TestingTwelve variables, with varying numbers of values, have7 x 6 x 6 x 5 x 3 x 3 x 2 x 2 x 2 x 2 x 2 x 2 = 725,760combinations of values.“All Pairs does it in 50.” (Bernie Berger, STAREast2003)
  20. 20. All Pair Testing• Reduce the number of tests  Clean• Multiple test cases will find same defects (all combination is an overkill)• Tests that cover all pairwise o For any two parametersp1 and p2 and any valid values v1 for p1 and v2 for p2, there is a test in which p1 has the value v1 and p2 has the value v2.
  21. 21. In Our Example• All pair test cases: C = 12!/(2! x (12 - 2)!) = 66 2 12• Triple pair test case: C312 = 12!/(3! x (12 - 3)!)• Worst case: ∑1 12 p 12 C12 = 2 − 1 = 4095 P=1
  22. 22. Can it Work?• 97% of failure are caused by 2 feature interaction (NIST Medical software failures 2000)
  23. 23. • What about coverage We measured the coverage of combinatorial design test sets for 10 Unix commands: basename, cb, comm, crypt, sleep, sort, touch, tty, uniq, and wc. […] The pairwise tests gave over 90 percent block coverage.[D. M. Cohen et al., 1996][…]• a set of 29 pair-wise AETG tests gave 90% block coverage for the UNIX sort command. We also compared pair-wise testing with random input testing and found that pair-wise testing gave better coverage.[D. M. Cohen et al., 1997]• The block coverage obtained for [pairwise] was comparable with that achieved by exhaustively testing all factor combinations […][I. S. Dunietz et al., 1997]• Our initial trial of this was on a subset Nortels internal e-mail system where we able cover 97% of branches with less than 100 valid and invalid testcases, as opposed to 27 trillion exhaustive testcases.[K. Burr and W. Young, 1998]• Compared to a traditional company that would use the quasiexhaustive strategy, an innovative company using the [Combinatorial] strategy would reduce its system level test schedule by sixty-eight percent (68%) and save sixty-seven percent (67%) in labor costs associated with the testing.[J. Huller, 2000]• [Evaluating FDA recall class failures in medical devices we established that] [...] out of the 109 reports that [were] detailed [enough], 98% showed that the problem could have been detected by testing the device with all pairs of parameter settings.[D. R. Wallace and D. R. Kuhn, 2001]• More than 70% of bugs were detected with two or fewer conditions (75% for browser and 70% for server) and approximately 90% of the bugs reported were detected with three or fewer conditions (95% for browser and 89% for server). [...] It is interesting that a small number of conditions (n<=6) are sufficient to detect all reported errors for the browser and server software.[R. Kuhn and M. J. Reilly, 2002]
  24. 24. Even have a website to promote
  25. 25. Test Case Quality• Single Test Case Quality o Clean o Exact o Repeatable• Test Case Coverage Quality o How do you know you are not missing critical test cases? o After a test, if no bugs found, how confident are you to say the software is in good quality.
  26. 26. quality• Chances are: functionality tester might be junior.• Use standard testing procedures whenever possible• Interact with programmer.• Rome is not built in one day, so is your test plan / test cases o Review  modify and Review  modify and Review  modify…. o Bring developers, technical marketing engineers, product managers into the review meeting
  27. 27. Create What ACL? need a detailed Bad Test Cases CLI Here Match What Group? What is• Create extended acl the group• Match group name, what is the cli?• Associate match group to E-ACL Next Interface? Need to explain• Create action group with next interface as tunnelthat.• Create PBR policy• Associate pbr policy to the action and match group• Apply pbr policy on the ingress interface Ingress Interface? Which one is the ingress interface?
  28. 28. Functionality Testing (2 input, 2 talk)• Understand features based on software function specification and Marketing Requirement Documentation.• Talk to the one who propose this new functionality (Why we need this feature).• Talk to the one who wrote the function specification Developer Checklist
  29. 29. • 6 “Understand” Try to understand on “Functional Points”• Try to understand more “Use Case/Scenarios”• Try to understand more Why we need this feature (revenue impact)• Try to understand more the modules will be affected by this feature.• Try to understand more the performance impact.• Try to understand more the way it is going to be implemented by the developers OSPF Hello Protocol
  30. 30. Test Case Attributes• Test ID• Description• Pre-Setup• Platform• Topology Sample Test Plan• Priority• Complexity• Steps/Expected results
  31. 31. Functionality Test Case Coverage• Need to be sure all functionality points are covered• Need to be sure all related modules combine test are covered. (IPv6, HA?)• Need to be sure to have a platform coverage list.• Enough use cases (experience and communication matters) (Build your Customer Scenario Library: various sample topologies, typical device configurations, typical applications)
  32. 32. Code Coverage• A quantum way to measure testing coverage.• A special compiler patch that when build a software image, can place mark on the new code• After you execute your black box test cases, you can know which code line is being executed.• Very mature in Java (just Google “java code coverage)• C has open source and commercialized tools (hard to integrate into a build environment if code base is huge) Intro.html
  33. 33. CLI Standard Testing Procedure Example - 1• enable CLI, verify all options and parameters are implemented as defined in the function spec.• Enable CLI, do “show running-config”, verify it is in the memory Disable CLI, do “show running-config” , verify it is not in the memory. Do a “show xxx” to verify the CLI is taking effect.• Repeat 1, but write the configure into flash, verify it is written in the flash memory Reload the box; verify the configuration is still there.• Enable then disable CLI for at least 20 times, see if we have any memory leak, or other instability. If it is a CLI that can be repeated, like firewall rules. Enable many rules, then disable many rules.• Verify there is no typos in the CLI Especially when there are some errors prompt messages. Verity those messages have no typos.• Conflict CLI on different features will report error• CLI response time
  34. 34. CLI Standard Testing Procedure Example - 2• Boundary testing on maximum and minimum CLI parameters values that it can take, check with function spec to see if it is as designed.• Negative testing on CLI parameters on o Numbers: • Out of boundary numbers. 负 数? Maximum number + 1 if possible. • empty • Put a string in to a number field. o strings: • Strange characters. %(#*$*@#&#&^&<>:”:{}. • long characters ( as long as the system can take it, as long as you can type on a terminal) • empty
  35. 35. GUI Testing Checklist GUI Testing Checklist
  36. 36. Review Process Internal peer Team Internal review Review Peer review from other External peer team that has domain review expert knowledge Tiger Testing Team is a elite group ofTesting Tiger Team Review team lead and managers that has rich testing experience
  37. 37. Testing Coverage Quality• Control 1st to see what developer wants to say• Test area coverage Developer Checklist o identify all functionality points (areas) on the software function specification o Can do code coverage if image building environment support it• Test type coverage (required for all new feature testing) o CLI or GUI testing o Functionality testing o Negative testing/Boundary testing o Use Case Testing o Performance testing
  38. 38. Test case execution quality control• Define the platform coverage• For both manual execution and auto execution (running scripts)• Detailed step by step execution logs may need to be saved and reviews for junior engineers• Regular rotation to keep engineer excited about different testing areas  better execution quality• Cross testing to discover new bugs• Stringent schedule control
  39. 39. What If you can’t get the• Spec? Whatever specs exist• Software change memos that come with each new internal version of the program• User manual draft (and previous version’s manual)• Product literature• Published style guide and UI standards• Published standards (C language or RFC)• 3rd party product compatibility testing suites• Published regulations• Internal memos (e.g. project engineers, describing the feature definitions)
  40. 40. What If you can’t get the• Spec? Marketing presentations, selling the concept of the product to management• Bug reports (responses to them)• Reverse engineer the program• Interview people o Developer lead o Tech writer o Customer service o Subject matter experts o Project manager• Look at header files, source code, database table definitions• Specs and bug lists for all 3rd party tools that you use• Prototypes or lab notes on the prototypes.
  41. 41. What If you can’t get the Spec?• •Look at compatible products• Look at customer call records