Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

SoftwareTesting-cs16..

324 views

Published on

  • Be the first to comment

  • Be the first to like this

SoftwareTesting-cs16..

  1. 1. <ul><li>Jason I. Hong </li></ul><ul><li>CS169 - Software Engineering, Fall 1998 </li></ul>Testing Strategies
  2. 2. Types of Testing
  3. 3. Types of Testing <ul><li>Scope </li></ul><ul><ul><li>Unit Testing tests a subroutine, module, or class </li></ul></ul><ul><ul><li>Integration Testing tests multiple units together </li></ul></ul><ul><ul><li>System Testing tests entire system as a whole </li></ul></ul><ul><li>Some Testing Goals </li></ul><ul><ul><li>Functional Testing tests functions in system </li></ul></ul><ul><ul><li>Performance Testing tests system performance </li></ul></ul><ul><ul><ul><li>primarily execution time, memory footprint </li></ul></ul></ul><ul><ul><li>Reliability Testing certifies system reliability </li></ul></ul><ul><ul><ul><li>predictable execution time, low number of failures, etc. </li></ul></ul></ul>
  4. 4. Black Box Testing <ul><li>Define test cases based only on specifications </li></ul><ul><li>Try all boundary conditions </li></ul><ul><ul><li>Go from empty to full, full to empty </li></ul></ul><ul><ul><li>Array bounds </li></ul></ul><ul><ul><li>Empty or null values </li></ul></ul><ul><li>Should be done by someone other than the module programmer </li></ul><ul><ul><li>Needs a shift in mindset away from programming </li></ul></ul><ul><ul><li>Programmers don’t want to find bugs, testers do </li></ul></ul>
  5. 5. White Box Testing <ul><li>Define test cases by looking at code </li></ul><ul><li>Not as useful as code review or code coverage </li></ul><ul><li>Useful for regression testing </li></ul>
  6. 6. Code Coverage <ul><li>Create test cases that execute every block of code at least once </li></ul><ul><li>Complete code coverage does not imply correct code </li></ul><ul><li>Ideally, we would like to test every path but this is exponential </li></ul><ul><ul><li>e.g. all possibilities in switch statements </li></ul></ul><ul><li>Useful for regression testing </li></ul>
  7. 7. Code Review <ul><li>Read through someone else’s code </li></ul><ul><ul><li>Extremely effective at finding defects </li></ul></ul><ul><li>Look for common errors </li></ul><ul><ul><li>Possible unhandled error conditions </li></ul></ul><ul><ul><li>Potential array out of bounds (e.g. gets) </li></ul></ul><ul><ul><li>Bad or dangerous type casting </li></ul></ul><ul><ul><li>Memory leaks (e.g. alloc w/o dealloc) </li></ul></ul><ul><ul><li>Too many global variables (high complexity) </li></ul></ul><ul><ul><li>Not cleaning up after self (e.g. goto, return) </li></ul></ul><ul><ul><li>... </li></ul></ul>
  8. 8. Regression Testing <ul><li>Maintain suite of tests </li></ul><ul><li>Build product periodically and run test suite </li></ul><ul><ul><li>Automatically verify that output is correct </li></ul></ul><ul><ul><li>Find reason for failure (test wrong or code wrong) </li></ul></ul><ul><li>Good metric for quality </li></ul><ul><ul><li>Can always tell if going forward or not </li></ul></ul><ul><ul><li>Requires significant process maturity from company </li></ul></ul><ul><li>Continually add new tests as bugs are fixed </li></ul><ul><ul><li>Ensures that old bugs do not reappear </li></ul></ul><ul><ul><li>Forces discipline on developers </li></ul></ul>
  9. 9. Stress Testing <ul><li>Place large load on product (typically a server) </li></ul><ul><li>Each client sends multiple requests </li></ul><ul><ul><li>Some valid requests, some junk requests </li></ul></ul><ul><li>Look for: </li></ul><ul><ul><li>Increasing memory footprint (memory leak?) </li></ul></ul><ul><ul><li>Running out of resources (sockets, file descriptor) </li></ul></ul><ul><ul><li>Increased response time (slows over time?) </li></ul></ul><ul><ul><li>Incorrect results (possible race conditions?) </li></ul></ul>
  10. 10. User Testing <ul><li>Perhaps the most important kind of testing </li></ul><ul><ul><li>Can customers use the product easily? </li></ul></ul><ul><ul><li>How many errors do users make? </li></ul></ul><ul><ul><li>How long does it take users to learn the system? </li></ul></ul><ul><ul><li>Do customers like the product? Enjoy using it? </li></ul></ul><ul><ul><li>Without customers, you have no product. </li></ul></ul><ul><li>Best strategy is iterative development </li></ul><ul><ul><li>Expensive to change product at end, so build cheap prototypes and evaluate with users at beginning </li></ul></ul><ul><ul><li>Hire real designers! Programmers != designers </li></ul></ul><ul><ul><li>Take CS160 for more information </li></ul></ul>
  11. 11. Alpha and Beta Testing <ul><li>Alpha testing </li></ul><ul><ul><li>Product is released to very small and focused group </li></ul></ul><ul><li>Beta testing </li></ul><ul><ul><li>Product is released to wider range of people </li></ul></ul><ul><li>Difficult to make significant changes once Alpha and Beta Testing is reached </li></ul><ul><ul><li>Use for polishing, finding compatibility problems, fixing simple errors, and finding workarounds </li></ul></ul><ul><ul><li>Do not use for evaluating usability or quality, a common mistake still made by many companies </li></ul></ul>
  12. 12. Some Statistics <ul><li>Unit testing finds 10-50% of defects </li></ul><ul><li>System testing finds 20-60% of defects </li></ul><ul><li>Combined rate < 60% of defects </li></ul><ul><li>Informal code review finds 30-70% of defects </li></ul><ul><li>Formal inspection finds 60-90% of defects, net savings of 10-30% </li></ul><ul><li>Reviews are more cost effective than testing </li></ul><ul><ul><li>Using reviews with testing helps software quality </li></ul></ul>See Rapid Development pp72-75
  13. 13. Development Notes <ul><li>Back-to-Back algorithms </li></ul><ul><ul><li>Use two implementations and compare results </li></ul></ul><ul><ul><li>Microsoft Excel does this for cell calculations </li></ul></ul><ul><ul><li>UWash Kimera Verifier found bugs in Netscape, Sun, and Microsoft implementations of Java </li></ul></ul><ul><li>Use Static Verification </li></ul><ul><ul><li>Use tools that go through code statically </li></ul></ul><ul><ul><li>Have no errors or warnings on Lint and gcc -Wall </li></ul></ul><ul><li>Self-Testing Code </li></ul><ul><ul><li>Have each class be able to test itself </li></ul></ul><ul><ul><li>Each Java class can have its own main() method </li></ul></ul>
  14. 14. Development Notes (cont.) <ul><li>Use good tools </li></ul><ul><ul><li>Purify, test harnesses, code coverage analysis, etc </li></ul></ul><ul><li>Iterative Development </li></ul><ul><ul><li>Plan a little, build a little, verify a little, and repeat </li></ul></ul><ul><ul><li>Fix defects as you find them , not “at the end” </li></ul></ul><ul><ul><li>If not, others will waste time on known problem </li></ul></ul><ul><li>80/20 rule applies for defects </li></ul><ul><ul><li>80% of bugs likely to be in 20% of code </li></ul></ul><ul><ul><li>Use defect tracking to find this code </li></ul></ul><ul><ul><li>If a unit seems to have lots of errors, it may be easier just to rewrite from scratch! </li></ul></ul>
  15. 15. Testing Process Notes <ul><li>Begin test planning from beginning </li></ul><ul><ul><li>Start test cases once requirements are known </li></ul></ul><ul><ul><li>Use User Manual and Specification Document </li></ul></ul><ul><ul><li>Microsoft uses one tester per developer </li></ul></ul><ul><ul><li>Space Shuttle people use ten testers per developer </li></ul></ul><ul><li>Daily Build </li></ul><ul><ul><li>Product is built every day and put through tests </li></ul></ul><ul><ul><li>Supports incremental development </li></ul></ul><ul><ul><li>Minimizes integration (prevents dis-integration ) </li></ul></ul><ul><ul><li>Provides project visibility </li></ul></ul>
  16. 16. Testing Process Notes (cont.) <ul><li>Find the reason for error and ask questions </li></ul><ul><ul><li>Is it a common occurrence? Mistake? Sloppiness? </li></ul></ul><ul><ul><li>Programmers fix own bugs at Microsoft, prevents programmer from introducing new bugs elsewhere </li></ul></ul><ul><ul><li>How much will it cost to fix? Is it worth it? </li></ul></ul><ul><ul><li>How can bugs like this can be prevented in future? </li></ul></ul><ul><li>Keep a database of bugs </li></ul><ul><ul><li>Track all defects to closure </li></ul></ul><ul><ul><li>Graph open bugs and closed bugs over time </li></ul></ul>
  17. 17. Testing Process Notes (cont.) <ul><li>Remember that testing only shows the presence of defects, not absence </li></ul><ul><ul><li>Testing assesses quality, does not assure quality </li></ul></ul><ul><li>Have an environment of quality </li></ul><ul><ul><li>Inktomi hangs a toy pig over person’s cubicle for a short while if he/she messes up </li></ul></ul><ul><ul><li>Microsoft head once threw chair out window on broken build </li></ul></ul><ul><ul><li>Lives depend on Space Shuttle software developers </li></ul></ul><ul><ul><li>Lives may depend on software you write! </li></ul></ul>
  18. 18. Summary <ul><li>Plan quality from the very beginning </li></ul><ul><li>Plan testing from the very beginning </li></ul><ul><ul><li>People and resources needed, test cases </li></ul></ul><ul><li>Preventing bugs is easier than finding bugs </li></ul><ul><ul><li>Use assertions, preconditions, invariants, reviews </li></ul></ul><ul><li>Use a variety of methods for reducing defects, assessing quality, and assuring quality </li></ul><ul><li>The earlier you find a defect, the cheaper and easier it is to correct </li></ul>
  19. 19. References <ul><li>Rapid Development by Steve McConnell </li></ul><ul><li>Code Complete by Steve McConnell </li></ul><ul><li>Software Engineering by Ian Sommerville </li></ul><ul><li>Software Project Dynamics by Jim McCarthy </li></ul><ul><li>Software Management by Donald Reifer </li></ul><ul><li>Construx Software </li></ul><ul><li>Eric Brewer (CS169 Spring 1998) </li></ul>

×