Download It


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Download It

  1. 1. Testing Computer Software Mike Stewart Microsoft
  2. 2. Who Am I and Why Should You Care? <ul><li>Software Test Lead for the Visual FoxPro team </li></ul><ul><li>Numerous Knowledge Base articles and white papers for Microsoft </li></ul><ul><li>Consultant developing custom applications using VFP, C++, Delphi and VB </li></ul><ul><li>Unix sysadmin and systems programmer using C++ </li></ul>
  3. 3. What this Session is Not <ul><li>A lot of code examples – no example can fit every situation </li></ul><ul><li>Testing the “Microsoft way” – this is all based on software QA industry methodologies </li></ul><ul><li>One size fits all – everyone’s testing needs are different </li></ul><ul><li>For people with aversions to blue Power Point slides </li></ul>
  4. 4. Why Test? <ul><li>Customer satisfaction – testers are customer advocates </li></ul><ul><li>Cost – the later a bug is found, the more it costs to fix </li></ul><ul><li>Quality – the earlier a bug is found, the more likely it is to be fixed </li></ul>
  5. 5. What is the goal of testing? <ul><li>Find bugs – more specifically, get bugs fixed </li></ul><ul><li>Verify that it works </li></ul><ul><li>Prevent errors before they get into the product </li></ul><ul><li>Provide information on risk – how solid is the product? </li></ul>
  6. 6. Important Things to Remember <ul><li>“ A test that reveals a problem is a success. A test that did not reveal a problem was a waste of time.” - Kaner </li></ul><ul><li>You will not find all of the bugs </li></ul><ul><ul><li>Myers’ 20 line program: 100 trillion paths </li></ul></ul><ul><ul><li>Program takes two inputs of four-digit number: almost 400 million possibilities </li></ul></ul><ul><ul><li>Test something 10,000 iterations: what if it fails at 10K + 1? </li></ul></ul>
  7. 7. Exhaustive Testing <ul><ul><li>“ The only exhaustive testing there is, is so much testing that the tester is exhausted!” — Hetzel </li></ul></ul>
  8. 8. “White Box” vs. “Black Box” Testing <ul><li>White box testing treats the software as a box that you can see into. </li></ul><ul><ul><li>You are looking into the box to see how it works </li></ul></ul><ul><ul><li>Assumes you have source code </li></ul></ul><ul><li>Black box testing treats the software as a box you cannot see into. </li></ul><ul><ul><li>Put in data (input) and see what comes out of the box (output). </li></ul></ul><ul><ul><li>No source code needed. </li></ul></ul><ul><li>Black box testing on finds about 55% of the bugs </li></ul>
  9. 9. Why White Box Testing isn’t Covered <ul><li>Doesn’t test the software in the manner in which it will be used </li></ul><ul><li>It doesn’t do spec testing – it has no way of knowing if it is functioning as designed or not </li></ul><ul><li>It does not find missing code – some estimate this accounts for 30% of bugs </li></ul><ul><li>Not enough time! It’s a discipline of its own, and we’d be here all day. </li></ul>
  10. 10. Steps in testing <ul><li>Test planning </li></ul><ul><ul><li>Write test plans </li></ul></ul><ul><ul><li>Spec and code reviews </li></ul></ul><ul><li>Build verification – is it even stable enough to test? </li></ul><ul><li>Function and system testing – run the full suite of test cases </li></ul><ul><li>Media verification </li></ul><ul><ul><li>No viruses </li></ul></ul><ul><ul><li>Bits on the media are the release bits </li></ul></ul><ul><ul><li>The product can be installed from the media </li></ul></ul>
  11. 11. Essential Testing Issues <ul><li>Establish a bug tracking system </li></ul><ul><ul><li>ATS Web – free at http:// /vstudio </li></ul></ul><ul><ul><li>No sticky notes </li></ul></ul><ul><li>Establish processes </li></ul><ul><ul><li>Use cases </li></ul></ul><ul><ul><li>Test plans </li></ul></ul><ul><ul><li>Test cases </li></ul></ul><ul><ul><li>Bug entry and resolution </li></ul></ul><ul><li>Automate, automate, automate! </li></ul>
  12. 12. Bug Tracking <ul><li>Accountability </li></ul><ul><ul><li>Assures that the bug does not fall through the cracks </li></ul></ul><ul><ul><li>When closed, we are assured that it is fixed. </li></ul></ul><ul><li>Metrics </li></ul><ul><ul><li>How great is the severity of bugs found? </li></ul></ul><ul><ul><li>How fast are we closing bugs? </li></ul></ul><ul><ul><li>Are we ready to ship? </li></ul></ul><ul><li>Liability </li></ul><ul><ul><li>CYA – cover yourself against legal liability (I am not a lawyer, however) </li></ul></ul>
  13. 13. Test Plans <ul><li>A test plan is a general “plan of attack”, not a detailed list of test cases. </li></ul><ul><li>List areas to test, and how they will be tested. </li></ul><ul><li>Notepad File/Open example </li></ul><ul><li>“Notepad’s File/Open functionality will be tested for various file types, file size limits, file name limits and regional encoding. Various media need to be considered, such as removable media and networks.” </li></ul>
  14. 14. Test Cases <ul><li>Each task can be compartmentalized into separate automation programs, or conglomerated into one big program. </li></ul><ul><li>Keep test cases atomized: test one particular feature at a time. </li></ul><ul><li>Create a matrix, and build test cases from that </li></ul><ul><ul><li>Could be something as simple as an Excel spreadsheet </li></ul></ul><ul><li>Example: the Notepad File/Open dialog </li></ul>
  15. 15. Test Cases for File/Open/Filename <ul><li>Long file names – boundary testing </li></ul><ul><li>Single letter file names – boundary testing </li></ul><ul><li>Long extensions – boundary testing </li></ul><ul><li>Edit box text length – boundary testing </li></ul><ul><li>Invalid characters in file names – functional </li></ul><ul><li>Valid file names – functional </li></ul><ul><li>Wild card characters – functional </li></ul><ul><li>Edit keys (Ctrl-C, etc.) – functional or spec </li></ul><ul><li>Paste in binary from clipboard - pathological </li></ul><ul><li>MRU dropdown – functional </li></ul><ul><li>Path completion – functional </li></ul><ul><li>Large fonts – functional </li></ul><ul><li>Gain and lose focus – functional </li></ul><ul><li>High ASCII chars - functional </li></ul>
  16. 16. Bug Entry and Resolution <ul><li>Minimum to track: </li></ul><ul><ul><li>Severity </li></ul></ul><ul><ul><li>Priority </li></ul></ul><ul><ul><li>The tester who found it </li></ul></ul><ul><ul><li>The dev who fixed it </li></ul></ul><ul><li>Enter good bugs: </li></ul><ul><ul><li>The easier to reproduce, the more likely it is to get fixed </li></ul></ul><ul><ul><li>Check on a second machine – could be a configuration/OS issue </li></ul></ul><ul><ul><li>If it only happens on one OS, say so </li></ul></ul>
  17. 17. Bug Entry and Resolution <ul><li>If you’re org is tracking metrics, close resolved bugs as quickly as possible </li></ul><ul><li>Set severity appropriately, and leave the priority to the project manager(s) </li></ul><ul><li>Your org will have to decide how to set severity and priority – everyone has different needs </li></ul>
  18. 18. Testing Methodologies <ul><li>Smoke testing – basic functionality </li></ul><ul><li>Exploratory testing </li></ul><ul><ul><li>Testing while learning the product </li></ul></ul><ul><ul><li>Misses large areas of the product </li></ul></ul><ul><li>Boundary conditions </li></ul><ul><ul><li>Cannot test every value, so test for the most important values </li></ul></ul><ul><ul><li>Hex boundaries and the boundary of field length </li></ul></ul><ul><ul><li>Boundary -1, boundary, and boundary +1 </li></ul></ul><ul><ul><li>Also test largest value allowed, if it is not a hex value </li></ul></ul>
  19. 19. Testing Methodologies <ul><li>Stress </li></ul><ul><ul><li>What happens when a large number of users hit it? </li></ul></ul><ul><ul><li>Low memory, low disk space, large amounts of data, repetitive </li></ul></ul><ul><ul><li>Bugs can often not be reproduced </li></ul></ul><ul><ul><li>Web Application Stress Tool is great for web apps. </li></ul></ul>
  20. 20. Testing Methodologies <ul><li>Functional </li></ul><ul><ul><li>Does it do what the spec says it should do? </li></ul></ul><ul><ul><li>How does it handle error conditions? </li></ul></ul><ul><li>Performance </li></ul><ul><ul><li>A discipline unto itself, worthy of an entire session </li></ul></ul><ul><li>Regression testing </li></ul><ul><ul><li>Make sure nothing got broken </li></ul></ul><ul><ul><li>Does not find a lot of bugs </li></ul></ul>
  21. 21. Testing Methodologies <ul><li>Code Coverage – for VFP, use the Coverage Profiler </li></ul><ul><li>Code coverage testing tells you what code has not been hit, and therefore needs to have test cases that hit this code. </li></ul><ul><li>You are unlikely to get 100% coverage </li></ul><ul><li>The VFP Coverage Profiler can also reveal bottlenecks. </li></ul>
  22. 22. Testing Methodologies <ul><li>Race conditions </li></ul><ul><ul><li>Two pieces of code are colliding </li></ul></ul><ul><ul><li>Especially important in multi-threaded and multi-user apps </li></ul></ul><ul><li>Error conditions - does it fail gracefully? </li></ul><ul><li>UI </li></ul><ul><ul><li>Tab order is easy to miss if you are a mouse-addicted developer </li></ul></ul><ul><ul><li>When testing as a dev, do not use the mouse, because you know the mouse will work. </li></ul></ul>
  23. 23. Testing Methodologies <ul><li>Ad Hoc testing – “guerrilla testing” </li></ul><ul><ul><li>Great for checking basic functionality </li></ul></ul><ul><ul><li>Can give broader testing than more formal means </li></ul></ul><ul><ul><li>Hard to verify coverage </li></ul></ul>
  24. 24. Automation <ul><li>The Active Accessibility test harness included with VFP 7.0 </li></ul><ul><li>Other automation products: </li></ul><ul><ul><li>Rational – Visual Test </li></ul></ul><ul><ul><li>Mercury Interactive – Testing suite with various products for web and thick client applications </li></ul></ul>
  25. 25. Automation Tips <ul><li>The test should be self-documenting </li></ul><ul><ul><li>Do not use “pass” or “fail” as the test result </li></ul></ul><ul><ul><li>Test should say what it expected and what it received, and whether it passed or failed. </li></ul></ul><ul><li>Write “drivers” or hooks into your software if needed – for example, VFP’s Active Accessibility or Project Manager hooks </li></ul><ul><li>Tests should be self-sustaining – create your own data, and clean up. Expect nothing. </li></ul>
  26. 26. When Do You Ship? <ul><li>You will ship with bugs if your testing is adequate. </li></ul><ul><li>Define show-stoppers </li></ul><ul><ul><li>Will customers find the bug? </li></ul></ul><ul><ul><li>Does it crash your product? </li></ul></ul><ul><ul><li>Will only pathologic testers find it? </li></ul></ul><ul><li>Define risk </li></ul><ul><ul><li>Fixing a bug has a 60% risk of introducing new bugs </li></ul></ul><ul><li>When the show-stoppers are fixed, ship </li></ul>
  27. 27. Selling Testing to Client <ul><li>Bulk contracts – sell the job, not the hours to develop it, and the hours to test it. </li></ul><ul><li>Once the software is in the customer’s hands, bugs are much more expensive to fix </li></ul><ul><li>How much does it cost to have 30 users find a bug vs. a qualified, dedicated test effort at $x/hour, that assures quality? </li></ul><ul><li>Other ideas? </li></ul>
  28. 28. “Things They Should Teach in the Testing Stuff Class, but Don’t” <ul><li>Run a high-color scheme like Marine High-Color </li></ul><ul><li>Use large fonts (Display/Settings/Advanced) </li></ul><ul><li>Throw away your mouse </li></ul>
  29. 29. More information <ul><li>Testing Computer Software – Kaner </li></ul><ul><li>The Art of Software Testing - Myer </li></ul><ul><li>Software Quality Testing Engineering magazine: </li></ul><ul><li>STAR Conference – the DevCon for professional software testers ( </li></ul>
  30. 30. A DVISOR D EV C ON Web Update Page <ul><li> </li></ul><ul><li>This session WILL have updates. </li></ul>
  31. 31. <ul><li>Thank you! </li></ul><ul><li>Please remember to fill out your evaluation. </li></ul>