Tools for Managing Quality (Nov 2010 BenchmarkQA Software Quality Forum)

2,082 views
2,025 views

Published on

The right quality lifecycle management tools can cut defects and time out of the software development lifecycle. The wrong tools – or lack of resources to support the tools—can cost you precious time and waste your money.

Get Dr. Steven G. Anderson's take on how Microsoft's new tool set compares to Quality Center.

Published in: Technology
1 Comment
1 Like
Statistics
Notes
No Downloads
Views
Total views
2,082
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
64
Comments
1
Likes
1
Embeds 0
No embeds

No notes for slide

Tools for Managing Quality (Nov 2010 BenchmarkQA Software Quality Forum)

  1. 1. Tools for Managing Quality How Microsoft s Emerging Tool Set Stacks Up Against Quality Center Software Quality Forum Thursday, November 11, 2010 5:00 7:00 pm
  2. 2. BenchmarkQA helps project teams deliver higher quality software through: Quality Assurance Consulting Contract & Permanent Staffing QA/Test Training & Seminars Local Outsourcing Test Lab © 2010 BenchmarkQA
  3. 3. Agenda Deciphering the MS Marketing Comparisons 1. Project/requirements management 2. Test case management 3. Test execution 4. Defect Management 5. Reporting 6. Administration 7. Automated Testing Conclusions 11/15/2010 © 2010 BenchmarkQA Slide 3
  4. 4. Overview of MS Tools 4 primary components This is primarily a developer view of the world 11/15/2010 © 2010 BenchmarkQA Slide 4 Reporting Services
  5. 5. Deciphering the MS Marketing TFS Database (think Access application) Visual Studio Developer tool (unit testing) Automated testing (Coded UI) MTM Microsoft Test Manager Reporting Reporting Services 11/15/2010 © 2010 BenchmarkQA Slide 5
  6. 6. Team Foundation Server (TFS) Database repository Views from Visual Studio, Reporting Services, IE and MTM Query tools 11/15/2010 © 2010 BenchmarkQA Slide 6
  7. 7. Visual Studio Developer tool Unit testing Coded UI testing Source code management 11/15/2010 © 2010 BenchmarkQA Slide 7
  8. 8. Microsoft Test Manager (MTM) Test case management Test execution Integrated defect management 11/15/2010 © 2010 BenchmarkQA Slide 8
  9. 9. Reporting Reporting Services Primarily warehouse based Excel 11/15/2010 © 2010 BenchmarkQA Slide 9
  10. 10. © 2010 BenchmarkQA Comparison 1 PROJECT/ REQUIREMENTS MANAGEMENT
  11. 11. TFS Project/Requirements Management User Stories (Validation (UAT) tests linked here) Tasks (Verification tests linked here) 11/15/2010 © 2010 BenchmarkQA Slide 11
  12. 12. TFS Requirements User Stories High level More detailed user stories linked to higher levels States Active/Resolved/Closed Area Path Modules or groupings of application functionality Can be used to sort of create a tree structure of user stories in a report Iteration Can be used to manage releases 11/15/2010 © 2010 BenchmarkQA Slide 12
  13. 13. Task Form 11/15/2010 © 2010 BenchmarkQA Slide 13
  14. 14. Tasks Tasks (linked to user stories) Assigned to person Estimate of effort Actual hours Activity Design (Verification tests linked here) Development Testing Task States Active Closed 11/15/2010 © 2010 BenchmarkQA Slide 14
  15. 15. Requirement-Design Links Use task activities to deal with design elements 11/15/2010 © 2010 BenchmarkQA Slide 15
  16. 16. Scoring - Project Management Scheduling TFS Resource management TFS QC Can be added to requirements section Workflow would be challenging Additional reporting needed (burn down charts, etc.) 11/15/2010 © 2010 BenchmarkQA Slide 16
  17. 17. Scoring Requirement Management Enter requirements Wash Organize requirements QC Link requirements to test cases Wash Link requirements to defects Wash 11/15/2010 © 2010 BenchmarkQA Slide 17
  18. 18. Comparison 2 TEST CASE MANAGEMENT © 2010 BenchmarkQA
  19. 19. TFS Manual Test Cases Similar to QC Action/step Expected results Parameterized test fields Defined at test level rather than run-time Can import from Excel Test case importer (CodePlex) Needs customization Organization similar to Test Suite in QC 11/15/2010 © 2010 BenchmarkQA Slide 19
  20. 20. MTM Test Case Management 11/15/2010 © 2010 BenchmarkQA Slide 20
  21. 21. Test Case Entry Test case entry very similar to QC Action Result Parameters Shared steps (call another test in QC) 11/15/2010 Slide 21© 2010 BenchmarkQA
  22. 22. Scoring - Test Case Management Enter test cases QC Organize test cases QC Import from Excel QC Link test cases to requirements Wash Link test cases to defects Wash Integration of automated tests QC Unit tests - VS 11/15/2010 © 2010 BenchmarkQA Slide 22
  23. 23. Comparison 3 TEST EXECUTION © 2010 BenchmarkQA
  24. 24. MTM Test Execution Management Note configurations Progress bar 11/15/2010 © 2010 BenchmarkQA Slide 24
  25. 25. TFS Test Execution Very similar to QC Filter by environment Cannot stop and restart test Recorded as fail. Rearranges your desktop History not linked to test case List of tests to run 11/15/2010 © 2010 BenchmarkQA Slide 25
  26. 26. Scoring - Test Execution (Manual) Organize test runs Wash Execution speed Wash Restart test QC Environment support MTM Execution history - QC 11/15/2010 © 2010 BenchmarkQA Slide 26
  27. 27. Comparison 4 DEFECT MANAGEMENT © 2010 BenchmarkQA
  28. 28. TFS Defect Management Remember this is a developer view of the world Active/Resolved/Closed Changing workflow is not as easy as QC Option to verify defect Major issue-Defect verification is recorded in test suite defect was found rather than test suite it was fixed in. Only reruns if defect created from test run 11/15/2010 © 2010 BenchmarkQA Slide 28
  29. 29. Scoring - Defect Management Defect entry Wash Creating defect while running test Wash Managing defects QC Verifying defects - Wash 11/15/2010 © 2010 BenchmarkQA Slide 29
  30. 30. Comparison 5 REPORTING © 2010 BenchmarkQA
  31. 31. TFS Reporting Uses a warehouse so data is old Reporting services Much higher skill level and permissions required to create reports Good for roll-up/summary reporting Difficult to nearly impossible to generate accurate requirements coverage report 11/15/2010 © 2010 BenchmarkQA Slide 31
  32. 32. Scoring - Reporting No contest - QC 11/15/2010 © 2010 BenchmarkQA Slide 32
  33. 33. Comparison 6 ADMINISTRATION © 2010 BenchmarkQA
  34. 34. Scoring - Administration Project customization QC User administration Wash Defect workflow QC Other workflow Not easy in either 11/15/2010 © 2010 BenchmarkQA Slide 34
  35. 35. Comparison 7 AUTOMATED TESTING © 2010 BenchmarkQA
  36. 36. Unit Testing Integrated part of VS Easy for developer to create tests for methods that return values MOQ for visual unit test Code coverage 11/15/2010 © 2010 BenchmarkQA Slide 36
  37. 37. Coded UI Testing Capture/replay Manually create object repository Higher skill level than QTP to customize tests Much more flexible than QTP No waiting for support 11/15/2010 © 2010 BenchmarkQA Slide 37
  38. 38. Case Study 1 QTP 9 Crashed QTP on login screen QTP 10 QTP hung on login screen QTP 11 QTP recorded, could not replay login screen (some customization would get past this issue) VS 2010 Records and plays back for simple actions. Customization for dealing with grid actions 11/15/2010 © 2010 BenchmarkQA Slide 38
  39. 39. Case Study 2 QTP 10 Cannot find any objects on screen, not even the menus QTP 11 Can find most standard controls Cannot see any 3D objects VS2010 Sees all objects View object properties (center, radius, etc.) 11/15/2010 © 2010 BenchmarkQA Slide 39
  40. 40. Scoring - Automated Testing No contest Visual Studio 11/15/2010 © 2010 BenchmarkQA Slide 40
  41. 41. Summary Project management TFS Requirements QC (by a hair) Manual test case management QC Manual test execution QC (by a hair) Defects Toss up Reporting QC Administration QC Automated testing VS2010 11/15/2010 © 2010 BenchmarkQA Slide 41
  42. 42. Considerations Cost Initial (Project, QC, VS, QTP) Maintenance Conversion Customization Productivity improvements Skill level of testing staff Best of Breed vs. One Solution If starting from scratch (i.e. want to get off Excel) 11/15/2010 © 2010 BenchmarkQA Slide 42
  43. 43. And the winner is? Microsoft s Team Foundation Server/Visual Studio 2010 11/15/2010 © 2010 BenchmarkQA Slide 43
  44. 44. How can we help? Project engagements Tool selection and implementation As needed support Staffing Senior automation resources Quality Path Tools and Technology Broad and deep assessment Process improvement program 11/15/2010 Slide 44© 2010 BenchmarkQA
  45. 45. Thank You! For additional information on how BenchmarkQA can be of assistance, please contact: Warren McLeod 952.392.2400 warren.mcleod@benchmarkqa.com © 2010 BenchmarkQA

×