Are You Making These 7 'Testing Metric' Mistakes? Webinar - Mark Bentsen, Philip Lew
•Download as PPTX, PDF•
0 likes•536 views
Find out if you're making these common testing metric mistakes and what to do if you are. In this hour-long XBOSoft webinar, ARGO Data's Quality Assurance Manager will teach you how to improve your metrics and build better software testing and QA teams.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Are You Making These 7 'Testing Metric' Mistakes? Webinar - Mark Bentsen, Philip Lew
Similar to Are You Making These 7 'Testing Metric' Mistakes? Webinar - Mark Bentsen, Philip Lew (20)
More from XBOSoft
More from XBOSoft (20)
Recently uploaded
Recently uploaded (20)
Are You Making These 7 'Testing Metric' Mistakes? Webinar - Mark Bentsen, Philip Lew
- 1. XBOSoft, Inc. All Rights Reserved. 1 Are You Making These 7 Metric Mistakes? And What to Do If You Are with Mark Bentsen, ARGO Data #XBOSoftWebinar #TestingMetricMistakes
- 2. XBOSoft Dedicated to software quality improvement Founded in 2006 We speed products to market with our expert: • Software QA consulting • Software testing Global team with offices in San Francisco, Beijing and Amsterdam “Thorough, accurate and fast” XBOSoft, Inc. All Rights Reserved. 2
- 3. House Rules Participants other than the speakers are muted Ask questions in the GoToWebinar control on the R-side of your screen or through Twitter @XBOSoft Questions may be asked throughout the webinar - we’ll try to answer them at the end You’ll receive info on recording after the webinar XBOSoft, Inc. All Rights Reserved. 3 Event hashtags: #XBOSoftWebinar #TestingMetricMistakes
- 4. Meet Our Speakers Philip Lew CEO and Founder, XBOSoft • Relevant specialties and passions o Software quality process, evaluation, measurement and improvement o Software quality in use / UX design o Mobile User Experience and usability o Cycling and travel XBOSoft, Inc. All Rights Reserved. 4 Rachel Geibig Marketing Manager, XBOSoft • Emails you about our industry hot topic webinars • Invites you to download our latest trends in software testing white papers XBOSoft Team
- 5. Guest Presenter Mark Bentsen XBOSoft, Inc. All Rights Reserved. 5 • Quality Assurance Manager, ARGO Data CTAL-Full, CSTE, PMP, ASQ CMQ/OE • Leading cross-enterprise collaboration in the DFW area among Software Quality practitioners for the past seven years. Gathering corporately on a quarterly basis and on field trips to one another’s work places, the QA Trailblazers are pushing each other to expand the capabilities of the modern software testing organization. • He is the QA Manager of ARGO Data, a software development company providing mission-critical and analytical solutions for financial services and healthcare. • Mark & his wife Melissa are the President Couple of Better Marriages Texas and have been active in Marriage Enrichment since they said “I do” in 2001. Prior to working in technology, he worked in YWAM & Mercy Ships in Switzerland and Namibia. • Contact: Mark.Bentsen@argodata.com
- 6. Agenda XBOSoft, Inc. All Rights Reserved. 6 1. Why We’re Here 2. 7 Metrics Mistakes 3. What you can do about it!
- 7. Is There Ever TMI In Testing? XBOSoft, Inc. All Rights Reserved. 7 • No • But There Is TMD… Too Much Data • Reporting too much data and not enough information
- 8. What do we do? XBOSoft, Inc. All Rights Reserved. 8 • Reduce Risk & Eliminate Waste • Effective software testing teams: – Build confidence – Reduce “Risk & Surprises” – Detect defects early – Provide valuable information
- 9. What do we do? XBOSoft, Inc. All Rights Reserved. 9 • Reduce Risk & Eliminate Waste • Effective Quality Assurance teams: – Identify risks – Prevent defects – Focus on continuous improvement of SDLC quality – Guard the company brand
- 10. STEP ZERO Communication is what the LISTENER does XBOSoft, Inc. All Rights Reserved. 10
- 11. Communication Heuristics XBOSoft, Inc. All Rights Reserved. 11 • Misconception is that status and metrics only go out in email. • If you depend on email, you have one “where” and one “how” in your communication tool belt. There are a lot of other tools available to the wise test manager.
- 12. METRIC MISTAKES 7 XBOSoft, Inc. All Rights Reserved. 12
- 13. 1-Flash In The Pan XBOSoft, Inc. All Rights Reserved. 13 Too ambitious in the beginning • Symptom: Having trouble keeping metrics current and reported when they are needed. • Principle: Slow is fast. • Solution: Make a sustainable decision up front. • Then: Steadily report meaningful information.
- 14. 2-Heavy Lifting XBOSoft, Inc. All Rights Reserved. 14 Choosing to work hard instead of smart • Symptom: Overly manual processes required to produce. • Principle: Maintainability is a key quality attribute and a success criteria for process success. • Solution: Build in telemetry. • Then: Automate, or automation assist.
- 15. 3-Message In A Bottle XBOSoft, Inc. All Rights Reserved. 15 Communicating in a vacuum, throwing the bottle into the ether and expecting the best • Symptom: No one responds to status reports from testing. • Principle: Communication is what the listener (recipient) does.
- 16. 3-Message In A Bottle XBOSoft, Inc. All Rights Reserved. 16 Communicating in a vacuum, throwing the bottle into the ether and expecting the best • Solution: Know your audience. Who they are and what they care about. • Then: Answer the questions they have, and address their concerns.
- 17. 4-House on Fire XBOSoft, Inc. All Rights Reserved. 17 Ongoing state of emergency • Symptom: Poor reporting and insight into QA processes through reporting. • Principle: Reporting should be part of every testing process.
- 18. 4-House on Fire XBOSoft, Inc. All Rights Reserved. 18 Ongoing state of emergency • Solution: Validate you have meaningful metrics for each stage of your testing processes leading into a milestone. • Then: Remove the mystery from what is taking place.
- 19. 5-Mine Goes To Eleven XBOSoft, Inc. All Rights Reserved. 19 Reporting is too complicated & cumbersome • Symptom: No one understands the metrics or the implications, including the Test Analyst sending it out. • Principle: Keep it simple.
- 20. 5-Mine Goes To Eleven XBOSoft, Inc. All Rights Reserved. 20 Reporting is too complicated & cumbersome • Solution: Choose to be pragmatic over trying to be impressive. • Then: Be consistent & crystal clear in your narratives.
- 21. 6-Fish Out Of Water XBOSoft, Inc. All Rights Reserved. 21 Answering the wrong questions • Symptom: Suffering from “This is the way we always do it” mentality. • Principle: Know your audience.
- 22. 6-Fish Out Of Water XBOSoft, Inc. All Rights Reserved. 22 Answering the wrong questions • Solution: Answer questions that should be asked and avoid the status quo. • Then: Follow-up in person with key stakeholders and adjust.
- 23. 7-Rearview Mirror Driver XBOSoft, Inc. All Rights Reserved. 23 Haphazardly moving forward without knowing what to expect in the days/weeks ahead. • Symptom: You are spending all your time trying to find defects instead of preventing them. • Principle: An ounce of prevention is worth a pound of cure.
- 24. 7-Rearview Mirror Driver XBOSoft, Inc. All Rights Reserved. 24 Haphazardly moving forward without knowing what to expect in the days/weeks ahead. • Solution: Predict what your metrics will be in the next two weeks. Document it. • Then: Compare reality with predictions. Adjust until forecasting accurately.
- 25. AND NOW What to do about it… XBOSoft, Inc. All Rights Reserved. 25
- 26. Test Management Trifecta XBOSoft, Inc. All Rights Reserved. 26 • What have you completed? • What did you learn? • What remains?
- 27. Tactical versus Strategic XBOSoft, Inc. All Rights Reserved. 27 • Tactical – Standardize metrics across teams • Strategic – Align to KPIs – YoY performance – Enterprise Quality Roadmap
- 28. BLUF XBOSoft, Inc. All Rights Reserved. 28 • Bottom Line Up Front • Follow with a “Headline” • Impact to the triple constraint?
- 29. Less is More XBOSoft, Inc. All Rights Reserved. 29 "I would have written you a shorter letter if I had more time." - Blaise Pascal
- 30. Post your webinar questions on Twitter @XBOSoft Registrants will receive an email with information on where to view the recording and slides from today’s webinar. Join us to keep updated on all our webinars, reports and white papers: facebook.com/xbosoft +xbosoft linkedin.com/company/xbosoft Check out our blog: http://xbosoft.com/software-quality-blog/ Download our free white papers: http://xbosoft.com/knowledge-center/ Contact us with ideas for future webinars or questions regarding our services! services@xbosoft.com Thank you! Q+A www.xbosoft.com
Editor's Notes
- Reporting upward Goal is to build confidence that their involvement is not needed unless requested Content / details varies for audience (Team, Managers, Directors, Officers) Concise, accurate, confidence-building Start with less, but make sure you have asked the right questions and have answers Flows always help- high level, block flows with associated test cases
- Communication Heuristics – Does not always happen in email Always needs a headline. Would you read a newspaper or magazine article without one? What else do they use in their format? After meetings – big game hunting In an elevator Take the elevator challenge Ground to top floor/to lowest floor Kung Fu Reporting Take an opportunity (they call you) and you turn it When you “do” send email, make sure it is relevant. When are the meetings where people need your info. When do PM’s begin preparing their reports? Your report needs to be in their outlook before they start that process
- Flash In The Pan. Symptom: Too ambitious in the beginning. Principle: Slow is fast. Solution: Make a good decision up front: Then: Steadily report meaningful information Double is trouble Metrics on the same thing in multiple ways Be sure to be consistent
- Heavy Lifting – Choosing to work hard instead of smart. Symptom: Overly manual processes required to produce. Principle: Maintainability is a key quality attribute and a success criteria for process success. Solution – Build in telemetry Then: Automate, or automation assist.
- Message In A Bottle – communicating in a vacuum, throwing the bottle into the ether and expecting the best Symptom: No one responds to status reports from testing. Principle: Communication is what the listener (recipient) does. Solution – Know your audience. Who they are and what they care about. Sample informal review of internal customers to your testing organization What do you need & want from QA? How well are we meeting your needs? What is your perception of the QA team? (tell me the truth…) Then: Answer the questions they have and address their concerns.
- Message In A Bottle – communicating in a vacuum, throwing the bottle into the ether and expecting the best Symptom: No one responds to status reports from testing. Principle: Communication is what the listener (recipient) does. Solution – Know your audience. Who they are and what they care about. Sample informal review of internal customers to your testing organization What do you need & want from QA? How well are we meeting your needs? What is your perception of the QA team? (tell me the truth…) Then: Answer the questions they have and address their concerns.
- House on Fire. – The house is always on fire Symptom: Ongoing state of emergency is a symptom of poor reporting and insight into QA processes through reporting Principle: Reporting should be part of every testing process Solution: Validate you have meaningful metrics for each stage of your testing processes leading into a milestone. Are you on Track? What to adjust? Then: Remove the mystery from what is taking place
- House on Fire. – The house is always on fire Symptom: Ongoing state of emergency is a symptom of poor reporting and insight into QA processes through reporting Principle: Reporting should be part of every testing process Solution: Validate you have meaningful metrics for each stage of your testing processes leading into a milestone. Are you on Track? What to adjust? Then: Remove the mystery from what is taking place
- Mine Goes To Eleven – The opposite of simple Symptom: Exaggerated, Hollow, Pretentious. No one understands the metrics or the implications; this includes the person sending it out. Principle – Keep it simple. Gate guard example Solution: Choose to be pragmatic over trying to be impressive. Then: Be consistent. Be crystal clear in your narratives. Bonus – Measure surprises Good Bad Don’t let your reports surprise any of the recipients – prewire Forgetting who they are for – The “Order” of understanding You Then others, “who” can only understand the metrics if you do first Can you interpret what this means to your plan? Assumption: You have a plan.
- Mine Goes To Eleven – The opposite of simple Symptom: Exaggerated, Hollow, Pretentious. No one understands the metrics or the implications; this includes the person sending it out. Principle – Keep it simple. Gate guard example Solution: Choose to be pragmatic over trying to be impressive. Then: Be consistent. Be crystal clear in your narratives. Bonus – Measure surprises Good Bad Don’t let your reports surprise any of the recipients – prewire Forgetting who they are for – The “Order” of understanding You Then others, “who” can only understand the metrics if you do first Can you interpret what this means to your plan? Assumption: You have a plan.
- Are you providing numbers, but not answering relevant key questions for your senior stakeholders? The traditional questions Executives ask, “Are they outdated?” How many test cases did you run? How many are automated? How many bugs did you find? Defect metrics / Trends? You learned that QA probably can’t cover everything… What is important for you? Performance? Core functions? What didn’t QA test? What are the limitations? What are you afraid of? (don’t need data give me a gut feel) Would more time help or not? Appropriate Trends/Metrics Simple Data points followed up by complex appendix… When reporting: Start with a simple Yes or NO (on a release vote) Gut check, what am I worried about… Go a bit deeper with feature level data (Check, Check + etc...) Schedule compression number QA typically never gets the X days requested to test Publish a compression number to indicate risk We only had 10/15 days so 0.67 compression (risk is high) If you had calculated you need a full tank of gas to make it to Houston, would you go with 2/3? Go a bit deeper with regression coverage maps NO Test cases run, or defect metrics etc… Example: Yes , Go Release Areas for concern XYZ Because we got it late and configuration nightmare to test Schedule Compression = 0.67 (we got crunched badly) Test Case point total = 892/1000 (we got close but not all the way) New feature Coverage Map 2 areas yellow rest 6 Green Appendix: Test Cases run, defects trends, LOKI, Intermittent defect list etc., etc.… Mitigations for all of the above when possible Looks like a 1 page go/no go/ with a large appendix Think ESPN Box score, but with full stats later down in the article Also: – Wrong context of report for the testing Example – Functional API testing vs. Functional testing executed via Quality Center
- Are you providing numbers, but not answering relevant key questions for your senior stakeholders? “Are they outdated?” The traditional questions Executives ask: How many test cases did you run? How many are automated? How many bugs did you find? Defect metrics / Trends? You learned that QA can’t cover everything… What is important for you? Performance? Features and Functionality (F&F), especially critical functions? What didn’t QA test? What are the limitations? What are you afraid of? (don’t need data give me your thoughts based on intuition and expertise) Would more time help or not? Appropriate Trends/Metrics Simple Data points followed up by comprehensive appendix… When reporting: Start with a simple Yes or NO (on a release vote) Reality check, what am I worried about… Go a bit deeper with feature level data (Check, Check + etc...) Schedule compression number QA typically never gets the X days requested to test Publish a compression number to indicate risk We only had 10/15 days so 0.67 compression (risk is high) If you had calculated you need a full tank of gas to make it to Houston, would you go with 2/3? Go a bit deeper with regression coverage maps NO Test cases run, or defect metrics etc… Example: Yes , Go Release Areas for concern XYZ Because we got it late and configuration nightmare to test Schedule Compression = 0.67 (we got crunched badly) Test Case point total = 892/1000 (we got close but not all the way) New feature Coverage Map (2) areas yellow rest (6) Green Appendix: Test Cases run, defects trends, Intermittent defect list etc., etc.… Mitigations for all of the above when possible Looks like a 1 page go/no go/ with a large appendix Think Horse race grid, but with full stats provided below or in this case, later in the status Also: – Wrong context of report for the testing Example – Functional API testing vs. Functional testing executed via Quality Center
- Rearview Mirror Driver – Constant focus on the past without forecasting the future. Symptom: You are spending all your time trying to find defects instead of time allocated to activities that would prevent them. Principle: An ounce of prevention is worth a pound of cure. Solution: Predict what your metrics will be in the next two weeks. Document it. Then: Compare reality with your predictions. Adjust as appropriate until you can accurately forecast. SDLC Quality & Root Causes Are you capturing root causes or excuses? ARGO thirteen Definitions Accountability Process Make it actionable Predict outcomes and measure results Wash, rinse, and repeat
- Rearview Mirror Driver – Constant focus on the past without forecasting the future. Symptom: You are spending all your time trying to find defects instead of time allocated to activities that would prevent them. Principle: An ounce of prevention is worth a pound of cure. Solution: Predict what your metrics will be in the next two weeks. Document it. Then: Compare reality with your predictions. Adjust as appropriate until you can accurately forecast. SDLC Quality & Root Causes Are you capturing root causes or excuses? ARGO thirteen Definitions Accountability Process Make it actionable Predict outcomes and measure results Wash, rinse, and repeat
- Guidance Regarding What To Do About It
- The Test Management Trifecta What have you completed? What did you learn? Defects Risks What testing remains?
- Tactical vs. Strategic Tactical – Standardize across teams Strategic – Align to KPIs, YoY performance, Enterprise Quality Roadmap 97/97/250 Find 97% of the defects Fix 97% of those prior to release Drive the cost of a function point of development to $250 System health assessment. Need to know the definition of “healthy” in your context. GA Readiness Version Release (VR) Exit Criteria
- And Then (and then, and then) BLUF – know your bottom line and lead with that. Think in terms of triple constraint. Schedule – We need an additional three days to complete planned testing. We lost five days to configuration management issues, but are working more efficiently as a team, so I’m confident this amount of time will be sufficient. Scope – We will now need to define additional test scenarios based on the change request getting approved Resources – We need two people from the product team to assist us in testing the new logic for the business rules Quality – We would like an additional three days to test functionality that had higher than anticipated defects found in regression. Doug’s advice – Don’t have the cost of the project on the final slide of the deck. Start with it. This is going to cost a million dollars, and here’s why…
- Less is more. "I would have written you a shorter letter if I had more time." - Blaise Pascal There are three habits that keep most people from being succinct: The habit of over explaining The habit of underpreparing The habit to completely miss the point