Stan Adams of Prolifics - Presentation for Test Expo in Chicago on 9/18/2014

1. Testing as a Managed Service
Using SLA’s and KPI’s
Testing Conference,
Chicago
CONNECT WITH US:
September 2014

2. How do you successfully manage outsourced testing?
CONNECT WITH US: 2
What is it?
 Outsource testing is when a
company decides to do all or part
of there testing from a 3rd party
specialist either onsite, near shore
or offshore.
 Manage Services is when the 3rd
party handles everything from
Strategy through Sunset
Why should I care?
 Knowing the skillset increases
your worth to the business
 Not all Outsource Groups are
created equal
 The business will still look to you
to ensure over all quality
Your tools for success?
 Proper Governance
 Specific KPIs
 Custom SLAs

3. Measure to avoid the common mistakes
CONNECT WITH US:
Accelerate
Problem
Identification
and contribute
to the
development
process
Improve Quality
of Software as
well as Service
Delivery
The
Aberdeen
Group in
2010
estimated
that nearly
50% of all
outsource
projects fail
outright or
fail to meet
expectations
7

4. Key Performance Indicators vs Metrics
CONNECT WITH US: 4
KPIs
Guidelines
 KPIs highlight the things that are important to your
business and should tie directly to business goals
 Financial Goals - QA costs as a % of total IT costs and
as a % of revenue
 Staffing Goals – QA FTE to total FTE Ratio
 Operational Goals – Business Process Quality and
Continuity as a % of Up Time
 Guidelines for creating KPIS
 The success measure must show clear, specific and
measureable performance indicators
 Costs to monitor can not exceed the value that will
known from measuring
 Linked to a strategic objective and easily understood
within its context
 Credible and under the control of the team attributed to

5. SLA’s are vital signs.
You can tell where a
testing organization is
headed and also
measure it’s maturity
level.
CONNECT WITH US: 5
Service Level Agreements

6. CONNECT WITH US: 6
Understand your
KPIs and determine
what you need
specific to your
environment
Remember to
include items you
will typically
delegate to internal
staff to self manage
Service Level Agreements (SLA’s)
Before you begin…..

7. What are some of the Key go to items to have in your SLAs?
 Product quality measurements
 DRE (Defect Removal Efficiency)
 Traceability, test case efficiency
and code coverage
 Defect identification quality (valid
bugs identified % of the time)
 Test plan quality and strategy will
require no more than X reviews
CONNECT WITH US: 7
 Reporting and project
management timelines
 Defects will be reported
within a specified time
window by severity
 Defects assigned for re-test
are regressed within x
hours
 Testers will design X
number of test cases per
day per tester based on
complexity
 Manual testers will
execute X number of test
cases per day
 Automation testers will
automate X number of test
cases a day
 Knowledge Transfer and Core
business understanding
 Vendor staff are required to
maintain X experts and Y
intermediates in our core
business process

8. Snapshot of a SLA Metrics Dashboard
DRE Process Compliance Business Process Knowledge
CONNECT WITH US: 8
General Incident Mgmt. Automated Regression
Library - Execution
Regression Library - Updates
100%
98%
100%
91%
100%
100%

9. How KPIs, SLAs, and Metrics work together
Business Goals Main Project Drivers Individual Project Performance
Metrics
 Cost
 On-Cost Delivery (Accuracy of
cost estimation)
 Estimated Effort Index (Actual
vs. planned hours)
 Time
 On-Time Delivery (Accuracy of
schedule estimation)
 Schedule Variance
 Productivity
 Percentage Rework
 Quality
 Weighted Average Defect
Density
 Defect detection rate (based on
function points or person hours)
 Defect Management
 Defects detected by phase and
by type
 Root cause distribution
CONNECT WITH US: 9
SLAs
 Delivery within budget
 Defect Removal Effectiveness
 Estimation and Sizing Turnaround
 Test Design Efficiency
 Process Compliance Score
 Staff’s knowledge of business
process
 Accuracy of Metrics
 General Incident Management
 Automated Regression Test
Library
 Regression Library Maintenance
KPIs
 Cost of testing as a % of
overall project cost
 Test Case Effectiveness
 Test Progress – Plan vs.
Actual
 Skill Set Matrix
 Quality of Development
 Quantifiable Regression
Execution
 Number of regression test
cases updated by release/
project
Illustrative list of typical performance measures

10. Our Governance Process at McKesson
We have a three-tier model for leading the McKesson -Prolifics partnership
and providing comprehensive program oversight.
CONNECT WITH US: 10
Steering Committee Meeting
 Set strategic direction
 Review long term goals
 Review areas of improvement
 Optimize alignment across
partners
Leadership Team Meeting
 Operational planning
 Account management
 SLA Reviews
 Escalation management
 New Initiatives
Test Managers Meeting
 Project progress review
 Resource management
 Prioritization
 Knowledge management
 Issue management
McKesson
 Head of testing
practice
 Account
Executive
 VP McKesson IT
 Generics Director
 SV&T Senior
Director
 SV& T Director
Frequency
Quarterly Meetings
Monthly Meetings
Weekly Meetings
 Account
Executive
 Onsite Test
Manager
 Generics Director
 SVCS Sr. Director
 SV&T Senior
Director
 SV& T Director
 Account
Executive
 Onsite Test
Manager
 Project /Program
Manager
Prolifics
McKesson Prolifics
McKesson Prolifics

11. Automated SLA Reporting to Track Events in Real Time
CONNECT WITH US:
11

12. CONNECT WITH US:
Sample SLA Report
12

13. CONNECT WITH US:
13
Public | Copyright © 2014 Prolifics Q & A
Thank You!

14. Stan.Adams@prolifics.com
+1 510 689 8712
CONNECT WITH US:
14
Public | Copyright © 2014 Prolifics Contact US
www.prolifics.com