Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Testing as a Managed Service 
Using SLA’s and KPI’s 
Testing Conference, 
Chicago 
CONNECT WITH US: 
September 2014
How do you successfully manage outsourced testing? 
CONNECT WITH US: 2 
What is it? 
 Outsource testing is when a 
compan...
Measure to avoid the common mistakes 
CONNECT WITH US: 
Accelerate 
Problem 
Identification 
and contribute 
to the 
devel...
Key Performance Indicators vs Metrics 
CONNECT WITH US: 4 
KPIs 
Guidelines 
 KPIs highlight the things that are importan...
SLA’s are vital signs. 
You can tell where a 
testing organization is 
headed and also 
measure it’s maturity 
level. 
CON...
CONNECT WITH US: 6 
Understand your 
KPIs and determine 
what you need 
specific to your 
environment 
Remember to 
includ...
What are some of the Key go to items to have in your SLAs? 
 Product quality measurements 
 DRE (Defect Removal Efficien...
Snapshot of a SLA Metrics Dashboard 
DRE Process Compliance Business Process Knowledge 
CONNECT WITH US: 8 
General Incide...
How KPIs, SLAs, and Metrics work together 
Business Goals Main Project Drivers Individual Project Performance 
Metrics 
 ...
Our Governance Process at McKesson 
We have a three-tier model for leading the McKesson -Prolifics partnership 
and provid...
Automated SLA Reporting to Track Events in Real Time 
CONNECT WITH US: 
11
CONNECT WITH US: 
Sample SLA Report 
12
CONNECT WITH US: 
13 
Public | Copyright © 2014 Prolifics Q & A 
Thank You!
Stan.Adams@prolifics.com 
+1 510 689 8712 
CONNECT WITH US: 
14 
Public | Copyright © 2014 Prolifics Contact US 
www.proli...
Upcoming SlideShare
Loading in …5
×

Testing as a Managed Service using SLAs and KPIs

13,167 views

Published on

Stan Adams of Prolifics - Presentation for Test Expo in Chicago on 9/18/2014

Published in: Technology
  • Be the first to comment

Testing as a Managed Service using SLAs and KPIs

  1. 1. Testing as a Managed Service Using SLA’s and KPI’s Testing Conference, Chicago CONNECT WITH US: September 2014
  2. 2. How do you successfully manage outsourced testing? CONNECT WITH US: 2 What is it?  Outsource testing is when a company decides to do all or part of there testing from a 3rd party specialist either onsite, near shore or offshore.  Manage Services is when the 3rd party handles everything from Strategy through Sunset Why should I care?  Knowing the skillset increases your worth to the business  Not all Outsource Groups are created equal  The business will still look to you to ensure over all quality Your tools for success?  Proper Governance  Specific KPIs  Custom SLAs
  3. 3. Measure to avoid the common mistakes CONNECT WITH US: Accelerate Problem Identification and contribute to the development process Improve Quality of Software as well as Service Delivery The Aberdeen Group in 2010 estimated that nearly 50% of all outsource projects fail outright or fail to meet expectations 7
  4. 4. Key Performance Indicators vs Metrics CONNECT WITH US: 4 KPIs Guidelines  KPIs highlight the things that are important to your business and should tie directly to business goals  Financial Goals - QA costs as a % of total IT costs and as a % of revenue  Staffing Goals – QA FTE to total FTE Ratio  Operational Goals – Business Process Quality and Continuity as a % of Up Time  Guidelines for creating KPIS  The success measure must show clear, specific and measureable performance indicators  Costs to monitor can not exceed the value that will known from measuring  Linked to a strategic objective and easily understood within its context  Credible and under the control of the team attributed to
  5. 5. SLA’s are vital signs. You can tell where a testing organization is headed and also measure it’s maturity level. CONNECT WITH US: 5 Service Level Agreements
  6. 6. CONNECT WITH US: 6 Understand your KPIs and determine what you need specific to your environment Remember to include items you will typically delegate to internal staff to self manage Service Level Agreements (SLA’s) Before you begin…..
  7. 7. What are some of the Key go to items to have in your SLAs?  Product quality measurements  DRE (Defect Removal Efficiency)  Traceability, test case efficiency and code coverage  Defect identification quality (valid bugs identified % of the time)  Test plan quality and strategy will require no more than X reviews CONNECT WITH US: 7  Reporting and project management timelines  Defects will be reported within a specified time window by severity  Defects assigned for re-test are regressed within x hours  Testers will design X number of test cases per day per tester based on complexity  Manual testers will execute X number of test cases per day  Automation testers will automate X number of test cases a day  Knowledge Transfer and Core business understanding  Vendor staff are required to maintain X experts and Y intermediates in our core business process
  8. 8. Snapshot of a SLA Metrics Dashboard DRE Process Compliance Business Process Knowledge CONNECT WITH US: 8 General Incident Mgmt. Automated Regression Library - Execution Regression Library - Updates 100% 98% 100% 91% 100% 100%
  9. 9. How KPIs, SLAs, and Metrics work together Business Goals Main Project Drivers Individual Project Performance Metrics  Cost  On-Cost Delivery (Accuracy of cost estimation)  Estimated Effort Index (Actual vs. planned hours)  Time  On-Time Delivery (Accuracy of schedule estimation)  Schedule Variance  Productivity  Percentage Rework  Quality  Weighted Average Defect Density  Defect detection rate (based on function points or person hours)  Defect Management  Defects detected by phase and by type  Root cause distribution CONNECT WITH US: 9 SLAs  Delivery within budget  Defect Removal Effectiveness  Estimation and Sizing Turnaround  Test Design Efficiency  Process Compliance Score  Staff’s knowledge of business process  Accuracy of Metrics  General Incident Management  Automated Regression Test Library  Regression Library Maintenance KPIs  Cost of testing as a % of overall project cost  Test Case Effectiveness  Test Progress – Plan vs. Actual  Skill Set Matrix  Quality of Development  Quantifiable Regression Execution  Number of regression test cases updated by release/ project Illustrative list of typical performance measures
  10. 10. Our Governance Process at McKesson We have a three-tier model for leading the McKesson -Prolifics partnership and providing comprehensive program oversight. CONNECT WITH US: 10 Steering Committee Meeting  Set strategic direction  Review long term goals  Review areas of improvement  Optimize alignment across partners Leadership Team Meeting  Operational planning  Account management  SLA Reviews  Escalation management  New Initiatives Test Managers Meeting  Project progress review  Resource management  Prioritization  Knowledge management  Issue management McKesson  Head of testing practice  Account Executive  VP McKesson IT  Generics Director  SV&T Senior Director  SV& T Director Frequency Quarterly Meetings Monthly Meetings Weekly Meetings  Account Executive  Onsite Test Manager  Generics Director  SVCS Sr. Director  SV&T Senior Director  SV& T Director  Account Executive  Onsite Test Manager  Project /Program Manager Prolifics McKesson Prolifics McKesson Prolifics
  11. 11. Automated SLA Reporting to Track Events in Real Time CONNECT WITH US: 11
  12. 12. CONNECT WITH US: Sample SLA Report 12
  13. 13. CONNECT WITH US: 13 Public | Copyright © 2014 Prolifics Q & A Thank You!
  14. 14. Stan.Adams@prolifics.com +1 510 689 8712 CONNECT WITH US: 14 Public | Copyright © 2014 Prolifics Contact US www.prolifics.com

×