InfoStretch - Business Driven Quality Metrics: Beyond Counting Defects
 

InfoStretch - Business Driven Quality Metrics: Beyond Counting Defects

on

  • 600 views

Inculcate metrics based optimization into your business process without losing sight of business needs and client focus. ...

Inculcate metrics based optimization into your business process without losing sight of business needs and client focus.


Defining quality metrics is one of the fundamental and most important tasks for QA managers. However, business goals and customer experience often get neglected while identifying QA metrics because of the silos QA teams work in. Quality assurance is not just about testing and finding bugs, it is in fact, crucial to the success of the business itself. But again, too much focus on alignment with business goals while defining QA metrics can put excess stress on the QA team and adversely affect their productivity.

If for example, the prime business goal is customer satisfaction, then finding customer defects is more important than the cost of finding those defects. While ‘customer defect metrics’ is important in this scenario, time taken to find these defects can’t be ignored. So, productivity related metrics still need to be considered. How do we strike the right balance between defining QA metrics that are in line with business goals without overburdening the QA team?

In this webinar, Infostretch CEO Rutesh Shah addresses the above question and provide a comprehensive approach along with tools that will help you inculcate metrics based optimization into your business process without losing sight of business needs and client focus. In this webinar, Rutesh focuses on following :

Defining business driven metrics – Going beyond SDLC - Industry focused metrics, Accounting for competition;
Benchmarking success metrics - Capturing business goals like costs, product leadership, operational efficiency into QA metrics
End-user/customer focused metrics - Metrics driven actions
Watch this webinar if you want to learn about how your QA team can become a significant contributor to the business baseline and goals.

Statistics

Views

Total Views
600
Views on SlideShare
600
Embed Views
0

Actions

Likes
0
Downloads
14
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Business Driven Quality Metrics - Beyond Counting DefectsWednesday, March 23, 2011 10:30 AM - 11:30 AM PDT- Show in my Time Zone Webinar RegistrationDefining quality metrics is one of the fundamental and most important tasks for QA managers.  However, business goals and customer experience often get neglected while identifying QA metrics because of the silos QA teams work in.  Quality assurance is not just about testing and finding bugs, it is in fact, crucial to the success of the business itself. But again, too much focus on alignment with business goals while defining QA metrics can put excess stress on the QA team and adversely affect their productivity. If for example, the prime business goal is customer satisfaction, then finding customer defects is more important than the cost of finding those defects. While ‘customer defect metrics’ is important in this scenario, time taken to find these defects can’t be ignored. So, productivity related metrics still need to be considered.    How do we strike the right balance between defining QA metrics that are in line with business goals without overburdening the QA team? In this webinar, Infostretch CEO Rutesh Shah will try to address the above question and provide a comprehensive approach along with tools that will help you inculcate metrics based optimization into your business process without losing sight of business needs and client focus.  In this webinar, Rutesh will focus on following : - Defining business driven metrics – Going beyond SDLC - Industry focused metrics, Accounting for competition; Benchmarking success metrics - Capturing business goals like costs, product leadership, operational efficiency into QA metrics - End-user/customer focused metrics - Metrics driven actions Attend this webinar if you want to learn about how your QA team can become a significant contributor to the business baseline and goals.

InfoStretch - Business Driven Quality Metrics: Beyond Counting Defects InfoStretch - Business Driven Quality Metrics: Beyond Counting Defects Presentation Transcript

  • Business Driven Quality MetricsBeyond Counting DefectsMarch, 2011phone: (408) 727-1100 | website: www.infostretch.com | email: info@infostretch.comAll trademarks are the property of their respective owners.©2004-2011 InfoStretch Corporation. All rights rights reserved.©2004-2010 InfoStretch Corporation. All reserved.
  • Agenda 1. Introduction 2. Types QA Metrics and Examples 3. Business Driven Metrics – Going Beyond SDLC  Industry Focused, Competitive - Success Metrics  Cost, Timelines, Effectiveness – Optimization Metrics  End-User / Customer focused Metrics 4. Metrics Driven Actions 5. Q&AINFOSTRETCH CORPORATE|phone: (408) 727-1100 website: www.infostretch.com | email: info@infostretch.com(p) (408) 727-1100 Corporation. All rights reserved.©2004-2011 InfoStretch | (w) ww.infostretch.com | (e) info@infostretch.com©2004-2010 InfoStretch Corporation. All rights reserved.
  • InfoStretch Overview Elevating Confidence(p) (408) 727-1100phone: (408) 727-1100| |(w) ww.infostretch.com website: www.infostretch.com info@infostretch.com | (e) | email: info@infostretch.com | Presenter: Rutesh Shah©2004-2011 InfoStretch Corporation. All rights reserved.
  • Overview Organization People Infrastructure  HQ in Silicon Valley  Average 6 years experience  Extensive Platform Compatibility  Offices in New York, Seattle  300+ Engineers/Consultants and Performance Test Labs 105+ Clients and 160+ Projects  Industry Expertise in Finance,  2000+ Mobile Handset Test Lab  Strong contributor to QA Retail, Mobile and Healthcare  ISO 9000-2001 Certified Community  Strong Project Mgmt. team  High Availability and 24 hours secure offshore centers with  ISO 27001 Audited PartnersINFOSTRETCH CORPORATE|phone: (408) 727-1100 website: www.infostretch.com | email: info@infostretch.com(p) (408) 727-1100 Corporation. All rights reserved.©2004-2011 InfoStretch | (w) ww.infostretch.com | (e) info@infostretch.com | Presenter: Rutesh Shah©2004-2011 InfoStretch Corporation. All rights reserved.
  • Recognition Unique Perspective – Leadership Team with Entrepreneurial, Startup, Fortune 500, Top Technology & Big 4 Consulting Firms Experience. Long Term Relationships – Focus on building long term relationships and competencies around client 100% reference-able. Clients including T-Mobile, AT&T, Amazon, Intel, TNS, Harmony & Wells Fargo Strong Growth – Strong Year over Year Growth and Profitable Operation Since 2004. Significant Investments in Platform Compatibility and Device Labs 19th among GESIA Rising Gartner Top 20 QA Awarded Inc Fast growing Part of 10 Most Star Award Service Provider 500/5000 privately Promising 2010 – Best Exceptional QA company of owned Testing Services SoftwareServices and Solutions the year 2010 companies in Company Product Bay Areaphone: (408) 727-1100 | website: www.infostretch.com | email: info@infostretch.com(p) (408) 727-1100 | (w) ww.infostretch.com | (e) info@infostretch.com©2004-2011 InfoStretch Corporation. All rights reserved.
  • QA Metrics We best Manage, What we can Measure(p) (408) 727-1100phone: (408) 727-1100| |(w) ww.infostretch.com website: www.infostretch.com info@infostretch.com | (e) | email: info@infostretch.com | Presenter: Rutesh Shah©2004-2011 InfoStretch Corporation. All rights reserved.
  • Metrics – The Map “Standards of measurement by which efficiency, performance, progress, or quality of a plan, process, or product can be assessed."phone: (408) 727-1100 | website: www.infostretch.com | email: info@infostretch.com©2004-2011 InfoStretch Corporation. All rights reserved.
  • If only Metrics ruled the World Note: Cartoon by Andy Gloverphone: (408) 727-1100 | website: www.infostretch.com | email: info@infostretch.com©2004-2011 InfoStretch Corporation. All rights reserved.
  • Quality Metrics – The Guide “Measuring whether or not a given process will produce products and services that meet quality standards.”phone: (408) 727-1100 | website: www.infostretch.com | email: info@infostretch.com©2004-2011 InfoStretch Corporation. All rights reserved.
  • Business Metrics – The Compass “Measuring the outcome of business objectives with- in the defined constraints – Predictability of process Resource utilization product/service quality Timeliness Market Impact Customer Satisfaction Financial Sanityphone: (408) 727-1100 | website: www.infostretch.com | email: info@infostretch.com©2004-2011 InfoStretch Corporation. All rights reserved.
  • Lost without a Guide or Quality Metricsphone: (408) 727-1100 | website: www.infostretch.com | email: info@infostretch.com©2004-2011 InfoStretch Corporation. All rights reserved.
  • What is a successful outcome? Customer Competition Satisfaction Impact Quality Happy Measurement Customer Emp & Business Happine ss Timeline Sales Financial & Budget < Cust. Support Success Impact Outcome Criteriaphone: (408) 727-1100 | website: www.infostretch.com | email: info@infostretch.com©2004-2011 InfoStretch Corporation. All rights reserved.
  • Influence Metrics Influence Metrics from Quantity to Qualityphone: (408) 727-1100 | website: www.infostretch.com | email: info@infostretch.com©2004-2011 InfoStretch Corporation. All rights reserved.
  • Why Quality Metrics? Companies that consistently produce high quality products and services will have high customer loyalty and a strong corporate brand. Is this a good business practice? Is this guaranteed? What about Cost, Timelines, Competitive Threat?phone: (408) 727-1100 | website: www.infostretch.com | email: info@infostretch.com©2004-2011 InfoStretch Corporation. All rights reserved.
  • Business Metrics Let us to do what is important to Businessphone: (408) 727-1100 | website: www.infostretch.com | email: info@infostretch.com©2004-2010 InfoStretch Corporation. All rights reserved.
  • Types of Business Metrics – QA Centricphone: (408) 727-1100 | website: www.infostretch.com | email: info@infostretch.com©2004-2011 InfoStretch Corporation. All rights reserved.
  • Quality Metrics Defect Reports Defect Density by Component Discover Submit 5% 6% Core Lookup 12% 33% Lists 3% Playlist Updates 30% 3% Defect trends Cache 8% 120 30 100 25 80 20 Total Defects 60 15 Total Fixed Defects Total Open defects 40 10 20 5 0 0phone: (408) 727-1100 | website: www.infostretch.com | email: info@infostretch.com©2004-2011 InfoStretch Corporation. All rights reserved.
  • Quality Metrics Defect Reports Defect Severity Report Severity Severity 4 (Minor) 1 (Catastrophic) 10% 9% Severity 3 (Moderate) 31% Open Defects (Aging Report) Severity 2 (Critical) 50% 100 90 80 70 60 50 Open Defects 40 30 20 10 0 <4 weeks 4-10 weeks >10 weeksphone: (408) 727-1100 | website: www.infostretch.com | email: info@infostretch.com©2004-2011 InfoStretch Corporation. All rights reserved.
  • Quality Metrics Defect Reports Defects By Release 20 18 16 14 Total Defects Defects 12 10 Defects Fixed 8 Defects Deferred 6 4 2 0 Release Pre and Post Product Defects Pre Production Defects Post Production Defects 18 13 12 11 11 8 8 6 4 4 4 1 1 1 1 1 0 0 0 0 0 0 0 0phone: (408) 727-1100 | website: www.infostretch.com | email: info@infostretch.com©2004-2011 InfoStretch Corporation. All rights reserved.
  • Quality Metrics Defect Reports Security Errors Logical Errors 8% DataBase 6% GUI 1% 2% Cosmetic GUI Performance/ Optimization 12% DataBase 1% Cosmetic Server Response Business Logic 21% Error Handling Compatibility Business Logic 0% Compatibility 47% Defect Removal Efficiency Error Handling 2% 120.00 100.00 80.00 60.00 40.00 20.00 0.00 Series1phone: (408) 727-1100 | website: www.infostretch.com | email: info@infostretch.com©2004-2011 InfoStretch Corporation. All rights reserved.
  • QA Metrics – Examples & Importance • These metrics give the status of quality of Defect Based Metrics products based on defects and its impact on quality Coverage Based • These metrics show the test as well as business requirement coverage as part of the testing Metrics effort Process • These metrics give the details of resource Effectiveness/ utilization and efficiency details which helps to optimize the QA/Testing operation. These metrics Resource Efficiency / give the details surrounding effectiveness of Optimization Metrics testing process to meet quality objectivesphone: (408) 727-1100 | website: www.infostretch.com | email: info@infostretch.com©2004-2011 InfoStretch Corporation. All rights reserved.
  • QA Metrics – Examples & Importance Quality Trend • These metrics give the overall quality picture for product over the life span which enables to Metrics understand what is working and what is not working Solution/Client •These are special purpose metrics created specifically Specific Metrics for business outcomes Automation ROI •It also includes the impact of Automation related ROI Metrics in testing organization Performance • These metrics are usually produced as part of Metrics performance testing engagements.phone: (408) 727-1100 | website: www.infostretch.com | email: info@infostretch.com©2004-2011 InfoStretch Corporation. All rights reserved.
  • Business Quality Metrics Efficiency and Optimization Testcase Maintenance For Release 400 350 300 250 200 Deleted Testcases 150 Modified Testcases 100 50 New Testcases 0 Resource Utilization and Deliverables 250 200 150 100 Testcases Developed 50 Defects Logged 0 Testcases Automatedphone: (408) 727-1100 | website: www.infostretch.com | email: info@infostretch.com©2004-2011 InfoStretch Corporation. All rights reserved.
  • Business Quality Metrics Efficiency and Optimization Product Test Optimization per Release Time For Testing 1400 180 160 1200 140 1000 120 Total Testcases 800 100 600 80 60 400 40 200 20 0 0 Total Testcases Time for Testingphone: (408) 727-1100 | website: www.infostretch.com | email: info@infostretch.com©2004-2011 InfoStretch Corporation. All rights reserved.
  • Business Quality Metrics Value Driven - Automation and ROI Automation ROI 1400 1200 1000 800 600 400 200 0 1 2 3 4 5 6 7 8 Manual Testcases Automated Testcases Manual Execution Time Automated Execution Timephone: (408) 727-1100 | website: www.infostretch.com | email: info@infostretch.com©2004-2011 InfoStretch Corporation. All rights reserved.
  • Business Quality Metrics End-Result Driven – User Satisfaction and Trend Usability Measure 20 15 10 5 0 Rel1 Rel2 Rel3 Rel4 Regression Efficiency 1600 New Features Features Used in 1st month 1400 Defects logged 1200 1000 Axis Title Regresion 800 Testcases 600 Regression Time 400 200 0 Rel1 Rel2 Rel3 Rel4phone: (408) 727-1100 | website: www.infostretch.com | email: info@infostretch.com©2004-2011 InfoStretch Corporation. All rights reserved.
  • Business Driven Quality Metrics Overall Indicator Stability Index Product - Stability Index 5.10 5.00reasonably overall quality 4.90 indicates variation from 4.80 product release to 4.70 quality release 4.60 4.50 Quality 4.40 indicators of 4.30 the testing Stability Index 4.20 effort using 4.10 complex formula Stability Index Index Description 1 Poor Quality 2 Becoming Stable 3 Needs Improvement 4 Desired Stability 5 Highly Stablephone: (408) 727-1100 | website: www.infostretch.com | email: info@infostretch.com©2004-2011 InfoStretch Corporation. All rights reserved.
  • Metrics Driven Action Make the change and measure to analyze the impactphone: (408) 727-1100 | website: www.infostretch.com | email: info@infostretch.com©2004-2010 InfoStretch Corporation. All rights reserved.
  • QA Metrics – Actions and Impacts Metrics Description Frequency Actions Effort & Effort spent versus Effort Monthly • Adjust Resources and/or Project Schedule planned & Actual End Date Timeline Variance Versus Planned End Date • Incorporate learning for next phase planning Time Number of releases that were Monthly • Calculate the cost/quality impact Variance successfully delivered to QA due to delay within the timelines • Development schedule checkphone: (408) 727-1100 | website: www.infostretch.com | email: info@infostretch.com©2004-2011 InfoStretch Corporation. All rights reserved.
  • QA Metrics – Actions and Impacts Metrics Description Frequency Actions Test # Defects detected in Testing / Monthly • Modify testcases and coverage asContainment (# Defects detected in Testing + necessary Efficiency Escaped defects detected by • Retire the testcases that are not Client or subsequent testing) aiding value or repetativeDefect Sanity Number of Valid Defects / Total Monthly • Review testcases and refersh tester Number of Invalid + Valid knowledge about the product defects) • Ensure completeness of defect descriptionphone: (408) 727-1100 | website: www.infostretch.com | email: info@infostretch.com©2004-2011 InfoStretch Corporation. All rights reserved.
  • QA Metrics – Actions and Impacts Metric Description Frequency Actions Test Case Number of Test Cases which are End of testing • Review and Enhance/Retire Efficiency finding defects Vs Total Number round testcases of Test Cases • Link testcases to requirement, Usecases and/or defects Cost of Ratio of Testing effort versus Total Monthly • Adjust tester focus based on the Testing Project effort. Cost effectiveness module/build/release importance for Onsite vs offshore testing cost. • Cross-train testers for different types of testingphone: (408) 727-1100 | website: www.infostretch.com | email: info@infostretch.com©2004-2011 InfoStretch Corporation. All rights reserved.
  • QA Metrics – Actions and Impacts Metric Description Frequency Actions ROI on Calculate the Return on Quarterly • Measure reduction in regression timeAutomation Investment on Automation (ROI and compare with faster time-to- Calculator) market • Framework/Keyword driven automation for reduced maintenanceUsability and Measure the Usability of Release • Try new features with different UIPerformance application with customer’s • Measure the usability and usage of features intuitively (How performance with customer data many tech support calls?) volume phone: (408) 727-1100 | website: www.infostretch.com | email: info@infostretch.com ©2004-2011 InfoStretch Corporation. All rights reserved.
  • Best Theoryphone: (408) 727-1100 | website: www.infostretch.com | email: info@infostretch.com©2004-2011 InfoStretch Corporation. All rights reserved.
  • InfoStretch Next Webinar Mobile App Design It’s an art, not a science Presenter: Morgan Russell, InfoStretch Corporation Mark your Calendar April 21, 2011 Mark your Calendar April 21, 2011All trademarks are the property of their respective owners.©2004-2011 InfoStretch Corporation. All rights reserved.
  • Website: www.infostretch.com Santa Clara New York Ahmedabad Mumbai Headquarters 17 Tanglewood Rd, 5th Floor, Regency Vashi InfoTech Park, 2880 Lakeside Drive, #200 Scarsdale, NY 10583 Plaza, Anand Nagar, B103, Tower # 1, Santa Clara, CA 95054 (703) 395-9791 Ahmedabad-15, Vashi Rail Station, (408) 727-1100 Gujarat, India New Mumbai, 91-79-40266266 Maharashtra, India 91-22-6947900 Partnersphone: (408) 727-1100 | website: www.infostretch.com | email: info@infostretch.com©2004-2011 InfoStretch Corporation. All rights reserved.