But Did You Test It


Published on

Moodle is a very flexible application with a large number of variables and roles. Testing upgrades and changes can be a challenge. This presentation should help attendees focus testing at their own workplace.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Be warned I talk very fast – feel free to ask questions during the presentation or ask me to clarify something
  • But Did You Test It

    1. 1. But-- Did You Test It? Ruth Blakely Quality Assurance Coordinator Athabasca University Canada
    2. 2. Today we will talk testing! <ul><li>Discussion of what ‘Quality Assurance’ is versus software testing </li></ul><ul><li>Talk about planning </li></ul><ul><li>Suggest how to build a test matrix </li></ul><ul><li>Suggest some resources </li></ul><ul><li>Answer questions </li></ul>
    3. 3. What is Quality Assurance <ul><li>Testing is part of Quality Assurance </li></ul><ul><li>Wikipedia describes Quality Assurance (QA) as covering all activities from design..., development, production, installation, servicing and documentation </li></ul><ul><li>QA in every part of PDCA (Plan-Do-Check-Act) </li></ul>
    4. 4. QA versus Testing <ul><li>Software testing is the process used to help identify the correctness , completeness , security , and quality of developed computer software . </li></ul><ul><li>The word testing is connoted to mean the dynamic analysis of the product—putting the product through its paces </li></ul>
    5. 5. ‘Testing is for wimps’ <ul><li>Some developers will assume their code works every time (many developers are not trained in test design or techniques) </li></ul><ul><li>NOT testing is risky behaviour </li></ul><ul><li>Confidential student and university information is put at risk without adequate testing </li></ul><ul><li>Need to justify testing? Reductions in the cost of software by using quality range from a whopping seventy percent of the total production cost to a minimum of twenty to thirty percent* </li></ul><ul><li>*R. Black, Managing the Testing Process, Second Edition . Wiley, NY. </li></ul>
    6. 6. Building a Test Plan <ul><li>Whether your organization does full fledged QA or just provides software testing, a plan is good practice </li></ul><ul><li>A QA plan is more than just matrix containing your test scenarios </li></ul><ul><li>It evaluates need and assesses risk </li></ul>
    7. 7. Test Plan Prep <ul><li>Determine the objectives for testing </li></ul><ul><li>Determine the types of tests that will be executed. </li></ul><ul><li>Determine the level of automation. What, if anything, will be automated? </li></ul><ul><li>Determine the test definition list </li></ul>
    8. 8. Build a Matrix <ul><li>A test matrix (at AU) is a collection of use cases (test cases, user actions) that will be executed to verify correctness. </li></ul><ul><li>In the case of Moodle the wide variety of roles can multiply the required tests into an unreasonably large matrix. </li></ul><ul><li>We use simple software tools to help focus testing and minimize the number of tests </li></ul><ul><li>Take the output and assign priority (with the user community). </li></ul>
    9. 9. Using Tools to Focus Testing <ul><li>Using combinatorial testing tools such as PICT or Allpairs, you can limit the required number of tests </li></ul><ul><li>Combinatorial tools use orthogonal arrays (aka pair-wise) to identify the ‘must tests’ </li></ul><ul><li>With 10 roles you would have to execute each of the 11 actions illustrated in the example 10 different times – do the math </li></ul>
    10. 10. And still more prep… <ul><li>Determine the priority of each item in the test definition list </li></ul><ul><li>Determine the tools and resources needed for testing. </li></ul><ul><li>Determine the schedule for testing. </li></ul>
    11. 11. How to Set Priority <ul><li>User Impact (low to high) </li></ul><ul><li>1 – user can work without this function </li></ul><ul><li>2 - user has a workaround but it would be noticeable </li></ul><ul><li>3 - user would be severely impacted by loss of this function </li></ul><ul><li>Frequency of use (low to high) </li></ul><ul><li>1 – executed relatively infrequently </li></ul><ul><li>2 - executed moderately frequently </li></ul><ul><li>3 - executed nearly all of the time </li></ul><ul><li>Development risk (low to high) </li></ul><ul><li>1 - known to be relatively stable </li></ul><ul><li>2 - average stability from development perspective </li></ul><ul><li>3 - suspected or known to be unstable- error prone in past - based on changing or new technology – logic is complicated </li></ul>
    12. 12. Regression <ul><li>The purpose of a regression test is to verify that: </li></ul><ul><li>Failure fixes have been made since last build </li></ul><ul><li>Related functionality that could be affected by fixes or modifications still exists </li></ul><ul><li>Critical functionality of software is covered </li></ul><ul><li>This may be the most appropriate place for automation (i.e. jMeter, Selenium) </li></ul>
    13. 13. Tools, Resources and Set Up <ul><li>What tools are required? </li></ul><ul><ul><li>Servers, pc’s, operating systems, special software </li></ul></ul><ul><li>What resources? </li></ul><ul><ul><li>Developers, primary tester, user acceptance team, documentation specialist </li></ul></ul>
    14. 14. Bug Tracking <ul><li>Use your test matrix or a bug tracking tool to follow reports, developer updates and retest results </li></ul><ul><li>AU uses Bugzilla as our test tracking tool </li></ul><ul><li>Determine bug flow before the start of the project </li></ul>
    15. 15. Back to The Plan <ul><li>Clearly state the objectives of your testing </li></ul><ul><ul><li>Each project will have a different set of requirements as to what is an acceptable level of testing. </li></ul></ul><ul><li>Articulate the scope of testing </li></ul><ul><ul><li>types of tests </li></ul></ul><ul><ul><li>level of automation if applicable </li></ul></ul><ul><li>List the process you will follow for testing </li></ul>
    16. 16. Sometimes Forgotten <ul><li>Determine ahead of time who will record failures </li></ul><ul><li>Set guidelines for reporting failures </li></ul><ul><ul><li>Train your testers to write good bug reports or set up a system so they can fill in the blanks </li></ul></ul><ul><li>Determine the procedures once failures are logged. (i.e. will it be fixed in this release, the next release, not at all) </li></ul><ul><li>Plan to retest </li></ul>
    17. 17. What Programmers say to testers <ul><li>&quot;That's weird...&quot; </li></ul><ul><li>&quot;It worked yesterday.&quot; </li></ul><ul><li>&quot;It must be a hardware problem.&quot; </li></ul><ul><li>&quot;Somebody must have changed my code.&quot; </li></ul><ul><li>&quot;It works on my machine&quot; </li></ul>
    18. 18. Testing is just PART of QA <ul><li>Remember from earlier--PDCA </li></ul><ul><li>P lan a pilot to test the improvement. </li></ul><ul><li>D o the improvement. </li></ul><ul><li>C heck that the process actually improved. </li></ul><ul><li>A ct to adopt, adjust or abandon the change. </li></ul>
    19. 19. Keys to Success <ul><li>Good Planning </li></ul><ul><li>Adequate Resources </li></ul><ul><li>Realistic Goals </li></ul><ul><li>Follow your processes </li></ul><ul><li>Foster teamwork </li></ul>
    20. 20. Places to go for help… <ul><li>www.sqatesters.com has white papers, online forums, strategy documents </li></ul><ul><li>http:// www.sqe.com (Software Quality Engineering) provides QA certification training </li></ul><ul><li>http:// www.softwareqatest.com / has excellent FAQ </li></ul><ul><li>All Pairs (which is open source Perl) is available at www.satisfice.com </li></ul><ul><li>PICT is freeware http://msdn.microsoft.com/en-us/testing/bb980925.aspx </li></ul><ul><li>www.bugzilla.com if you are interested in the tracking software that we use </li></ul>
    21. 21. Come for a virtual visit… <ul><li>www.athabascau.ca </li></ul><ul><ul><ul><li>Ruth Blakely ruthb@athabascau.ca </li></ul></ul></ul>