TSP SYMPOSIUM Schneider, Henry.pptx

TSP Symposium
Mexico City
February 23 – 24, 2016

®2016 All Rights Reserved - PPQC 2 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
•  Core Concepts
•  Iden-fying Sta-s-cal Stability
•  Selec-ng Processes and Performance Measures
•  Ques-ons and Answers
•  Some Common Measures (reference only)

PPQC
PROCESS AND PRODUCT

PPQC
PROCESS AND PRODUCT
•  Performed Process
o  A process that accomplishes the needed work to produce work
products
•  Managed Process
o  A performed process that is planned and executed in accordance with
policy; employs skilled people having adequate resources to produce
controlled outputs; involves relevant stakeholders; is monitored,
controlled, and reviewed; and is evaluated for adherence to its
process descripLon
•  Deﬁned Process
o  A managed process that is tailored from the organizaLon’s set of
standard processes according to the organizaLon’s tailoring
guidelines; has a maintained process descripLon; and contributes
process related experiences to the organizaLonal process assets

PPQC
PROCESS AND PRODUCT
•  Special Cause of Varia-on
o  A cause of a defect that is speciﬁc to some transient circumstance and
is not an inherent part of a process
•  Common Cause of Varia-on
o  The variaLon of a process that exists because of normal and expected
interacLons among components of a process
•  Stable Process
o  The state in which all special causes of process variaLon have been
removed and prevented from recurring so that only common causes
of process variaLon of the process remain

PPQC
PROCESS AND PRODUCT
•  Quality and Process Performance Objec-ves (QPPOs)
o  QuanLtaLve objecLves and requirements for product quality, service
quality, and process performance
•  Capable Process
o  A process that can saLsfy its specified product quality, service quality,
and process-performance objecLves
•  Process Performance
o  A measure of actual results achieved by following a process. Process
performance is characterized by both process measures (e.g., effort,
cycle Lme, and defect removal efficiency) and product or service
measures (e.g., reliability, defect density, and response Lme)

PPQC
PROCESS AND PRODUCT
•  Process Performance Baseline (PPB)
o  A documented characterizaLon of process performance, which can
include central tendency and variaLon. A process performance
baseline can be used as a benchmark for comparing actual process
performance against expected process performance

PPQC
PROCESS AND PRODUCT
•  Process Performance Model (PPM)
o  A descripLon of the relaLonships among the measurable aZributes of
one or more processes or work products that is developed from
historical process performance data and is used to predict future
performance
o  One or more of the measureable aZributes represent controllable
inputs Led to a sub-process to enable performance of “what-if”
analyses for planning dynamic re-planning, and problem resoluLon
o  PPMs include staLsLcal, probabilisLc, and simulaLon based models
that predict interim or ﬁnal results by connecLng past performance
with future outcomes
o  PPMs model the variaLon of the factors and provide insight into the
expected range and variaLon of predicted results

PPQC
PROCESS AND PRODUCT
•  Compose the Deﬁned Process
o  Composing the Project’s Deﬁned Process (PDP) involves idenLfying
alternaLves to one or more processes or sub-processes, performing
quanLtaLve analysis of performance and selecLng the alternaLves
that are best able to help the project achieve its QPPOs
•  Sta-s-cal Process Control (SPC)
o  StaLsLcally based analysis of a process and measures of process
performance, which will idenLfy common and special causes of
variaLon in process performance and maintain process performance
within limits

PPQC
PROCESS AND PRODUCT
•  Sta-s-cal Techniques
o  Techniques adapted from the ﬁeld of mathemaLcal staLsLcs used for
acLviLes such as characterizing process performance, understanding
process variaLon, and predicLng outcomes.
o  Examples of staLsLcal techniques include sampling techniques,
analysis of variance (ANOVA), chi-squared tests, and process control
charts
•  Causal Analysis
o  The analysis of outcomes to determine their causes
o  Used to address selected issues and diagnose deﬁciencies in achieving
the project’s QPPOs

PPQC
PROCESS AND PRODUCT

PPQC
PROCESS AND PRODUCT
Before an organiza-on can perform High Maturity ac-vi-es, it
must have the following in place:
o  Capability to gather and use data at all organizaLonal levels, i.e.,
project members who
•  Gather data on their own work
•  Understand and use the data in planning and performing their work
o  Processes that specify how and when data are collected
o  Consistent execuLon of the deﬁned process, where tailoring is
handled in a controlled and disciplined fashion

PPQC
PROCESS AND PRODUCT
•  Everything we do in life, including product development and
service delivery, is a series of interconnected processes
•  All processes are variable
•  People performing the process is the largest source of
varia-on
•  Understanding varia-on is the basis for management by fact
and systema-c improvement:
o  Understand the past – quan-ta-vely
o  Control the present – quan-ta-vely
o  Predict the future – quan-ta-vely

PPQC
PROCESS AND PRODUCT
•  ScaOer Diagrams
•  Run Charts
•  Cause-and-Eﬀect Diagrams
•  Histograms
•  Bar Charts
•  Pareto Charts
•  Control Charts

PPQC
PROCESS AND PRODUCT
•  Real process behavior must be understood before drawing
reasonable conclusions about where performance needs to
be improved
•  Ask these ques-ons to ﬁnd out about real process behavior:
o  What is the normal or inherent process variaLon?
o  What diﬀerenLates inherent from anomalous variaLon?
o  What is causing the anomalous variaLon?
•  Sta-s-cal Process Control provides the methods and tools
needed to measure and analyze process behavior, draw
conclusions, and drive improvement

PPQC
PROCESS AND PRODUCT
•  A summary or characteriza-on of a distribu-on (i.e., a set of
numbers)
•  A characteriza-on of a central tendency (e.g., mean,
median, and mode)
•  A characteriza-on of
dispersion (e.g., variance,
standard devia-on,
interquar-le range, and
range)

PPQC
PROCESS AND PRODUCT
Statistics (specifically hypothesis testing) enable you to place a
confidence interval on the Central Tendency and Dispersion of the
population and on future samples
A data population consists of the
total possible observations with
which you are concerned but to
which you do not necessarily
have access (X1 thru X15).
X5
X15
X14 X13
X12
X11
X6
X10X8 X7
X9
X3
X2
X1
X4
A sample
is a set of observations selected
from a data population that you
can access.

PPQC
PROCESS AND PRODUCT
•  Hypothesis tes-ng uses sta-s-cal analysis as a formal way
of making a comparison and deciding whether or not the
difference is sta-s-cally significant
•  Hypothesis tes-ng consists of comparing a null and an
alterna-ve hypothesis:
o  The null hypothesis states that the members of the comparison are
staLsLcally equal; there is no difference (a concrete, default posiLon)
o  The alternaLve hypothesis states that there is a staLsLcally significant
difference; it is supported when the null hypothesis is rejected
•  The conclusion either rejects or fails to reject the null
hypothesis.

PPQC
PROCESS AND PRODUCT
When the p is low, the null must go.
When the p is high, the null must ﬂy.
Note: The p-value is the key output in staLsLcal analysis that students are taught to
idenLfy and use to draw a conclusion regarding the hypothesis test comparison or
regarding the signiﬁcance of a staLsLcal model

PPQC
PROCESS AND PRODUCT

PPQC
PROCESS AND PRODUCT
•  “While every process displays varia1on, some processes
display controlled varia1on, while others display
uncontrolled varia1on”
•  Rou-ne varia-on:
o  A stable paZern of random variaLon over Lme
o  Typical observaLon within a common cause system
•  A process is out of sta-s-cal control if it is opera-ng in the
presence of assignable causes
o  A factor that is not inherent in the process is aﬀecLng its performance
o  VariaLon in the process can be “assigned” to a speciﬁc cause
•  A process under sta-s-cal control is predictable

PPQC
PROCESS AND PRODUCT
•  Wheeler’s Empirical Rules:
o  60% ≤ ±1σ ≤ 75%
o  90% ≤ ±2σ ≤ 98%
o  99% ≤ ±3σ ≤ 100%
Wheeler’s Empirical Rules are robust with regard to a remarkable number of distribu-ons

PPQC
PROCESS AND PRODUCT
•  SPC is a technique that can be used to tell you how much
variability there is in a process
o  How much variability is due to rouLne variaLon
o  How much variability is due to some unique event or factor not
inherent in the process (assignable causes)
o  A control chart is a tool used in SPC to help do this
•  Control charts are generated from two data sets
o  PPB data – used to staLsLcally derive a process average, Upper
Control Limit (UCL), and Lower Control Limit (LCL) (±3σ)
o  Actual performance data – ploZed like a Run Chart and compared
against baseline data

PPQC
PROCESS AND PRODUCT
•  Which of these 20 projects should be acted on?
0
1
2
3
4
5
6
7
8
9
10
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Number of Non-compliances
Project ID

PPQC
PROCESS AND PRODUCT
•  Diﬀerent techniques can be used to understand process
varia-on
o  Simple graphs, such as frequency histograms and bar charts
o  StaLsLcal formulae, such as standard deviaLon and range
o  StaLsLcal process control tools, such as XmR charts and u-charts
o  Analysis of variance techniques, such as ANOVA and hypothesis test

PPQC
PROCESS AND PRODUCT
ExcepLonal
(unexpected)
variaLon
RouLne (expected) variaLon ExcepLonal
(unexpected)
variaLon

PPQC
PROCESS AND PRODUCT
0
1
2
3
4
5
6
7
8
9
10
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Number of Non-compliances
Project ID
Upper Control Limit = 11.7
Mean = 5
Lower Control Limit = 0

PPQC
PROCESS AND PRODUCT
Upper Control Limit
Lower Control Limit
Process Average
ExcepLonal process
variaLon
ExcepLonal
process variaLon
Actual Performance Data from Project XYZ

PPQC
PROCESS AND PRODUCT
1σ
2σ
3σ
3σ
2σ
1σ
ZONE A
ZONE A
ZONE B
ZONE B
ZONE C
ZONE C
UCL
LCL
Process
Average

PPQC
PROCESS AND PRODUCT
•  1 single point outside 3σ
•  2 of 3 successive values on same
side of centerline and outside of
2σ
•  8 successive values on same side
of centerline

PPQC
PROCESS AND PRODUCT
“… in so:ware development it is difficult to use control charts
in the formal SPC manner. It is a formidable task, if not
impossible, to aDempt to define the process capability of a
so:ware development process.”
- Stephen H. Kan (1995, p. 143)
•  Cri-cal assump-on for interpre-ng control charts – data are
affected by homogeneous sources of varia-on
o  Same populaLon
o  Same behavioral implementaLon of the process
o  Same common cause system

PPQC
PROCESS AND PRODUCT
“Despite these issues, control charts are useful for so:ware
quality improvement – when they are used in a relaxed manner
… control charts use in so:ware is not in terms of formal
sta1s1cal process control and process capability … used as a
tool for improving consistency and stability … more
appropriately called pseudo-control charts.”
- Stephen H. Kan (1995, p. 144)

PPQC
PROCESS AND PRODUCT
•  System development processes rou-nely operate within the
presence of mul-ple common cause systems – PPBs mix natural
varia-on with:
o  VariaLon in developer skills
o  VariaLon in aZributes of work products (e.g. complexity)
o  VariaLon in situaLons
•  The voice of the process is a cacophony of many voices from many
behavioral implementa-ons of the process:
o  Confounded by variaLon from many different sources
o  Control limits are excessively wide – occasionally 10X range
o  Out of control events are difficult to detect
o  ‘StaLsLcal stability’ may not offer control or predictability
o  A single PPB may provide poor characterizaLon

PPQC
PROCESS AND PRODUCT

Experienced Inexperienced
Method &
Support
Tools
No Method
& Support
Tools
Each cell
represents a
unique
common cause
system with
diﬀerent mean
results

PPQC
PROCESS AND PRODUCT
•  In the example, methods and experience levels are:
o  Sources of process variaLon that guide the idenLficaLon of common
cause systems
o  Factors whose variaLon must be controlled in establishing separate
common cause systems
o  Likely to appear as assignable causes of process variaLon if data from
several common cause systems are combined into a single PPB
•  Each cell is a different common cause system
o  Different PPBs
o  Different behavioral implementaLons of the process
•  The typical uses of control charts in system development
regularly mix data from mul-ple common cause systems

PPQC
PROCESS AND PRODUCT
•  Control charts are appropriate when:
o  The ploZed results characterize process performance within a single
common cause system
o  Data describe a single behavioral implementaLon of the process
o  Process fidelity is high (the process is well-defined and performed in
compliance with how it is defined)
•  When condi-ons are appropriate for control charts;
o  Process stability is a meaningful concept
o  AggregaLng PPBs provides a good basis for understanding
performance

PPQC
PROCESS AND PRODUCT
Individual
Aggregate
OR

PPQC
PROCESS AND PRODUCT
•  Typical ML 4 prac-ces result in plokng aggregated inspec-on results
o  Across different developers
o  Across different reviewers
o  Across components or work products with different aZributes

PPQC
PROCESS AND PRODUCT
•  Since people are the greatest source of varia-on in system development
a more Personal Solware Process (PSP)-like approach:
o  Treats individuals as separate common cause systems rather than as assignable
causes
o  Decreases the width of control limits
o  Makes unlikely or unexpected events easier to detect

PPQC
PROCESS AND PRODUCT
“A:er product size, people factors have the strongest inﬂuence
in determining the amount of eﬀort required to develop a
so:ware product.”
- Barry Boehm, et. al. (2000)

PPQC
PROCESS AND PRODUCT

PPQC
PROCESS AND PRODUCT
•  Establish and maintain the organization’s QPPOs,
which are traceable to business objectives
o  The QPPOs are derived from the organizaLon’s business objecLves
and will typically be speciﬁc to the organizaLon, group, or funcLon
o  The QPPOs take into account what is realisLcally achievable based
upon a quanLtaLve understanding (knowledge of variaLon) of the
organizaLon’s historic quality and process performance

PPQC
PROCESS AND PRODUCT
•  A simple task that many -mes does not specify enough
informa-on for use at Maturity Level 4 or 5
•  QPPOs need to be SMART
o Speciﬁc
o Measurable
o Achievable
o RealisLc
o Time-based

PPQC
PROCESS AND PRODUCT
•  Select measures and processes that provide insight into the
organiza-on’s ability to meet its QPPOs
o  Create/update clear unambiguous operaLonal deﬁniLons for the
selected measures
o  Revise and update the set of measures, analyses, and procedures as
warranted
o  In usage, be sensiLve to measurement error
o  The set of measures may provide coverage of the enLre lifecycle and
be controllable

PPQC
PROCESS AND PRODUCT

PPQC
PROCESS AND PRODUCT
•  Each project and organiza-on has unique informa-on needs
•  When iden-fying goals and objec-ves, consider sources such as
o  business and customer needs
o  constraints and assumpLons
o  project or organizaLon characterisLcs
•  domain, technology
•  people and skills
•  product features and acceptance criteria
•  cost, schedule, and milestones
o  risks
o  past experience of the team

PPQC
PROCESS AND PRODUCT
Enterprise
How do we perform,
compared to others?
Which
improvements
should we fund?
In what training or
tools must we
invest?
Are our customers
sa-sfied?
Process
How consistent is
our performance?
Which processes
require
improvement?
What is the ROI of
improvement
efforts?
How are
improvement efforts
progressing?
Project
How big is this
project?
Is the project plan
feasible?
How is the project
progressing?
When will we be
ready to deliver?
Product
How big is the
applica-on?
Which modules are
have the most
defects?
What is the cost to
maintain?
What are the quality
and growth trends?
Goals
Data

PPQC
PROCESS AND PRODUCT
Term Deﬁni-on
Data A collecLon of actual values assigned to a measure
(example: data for a source lines of code base measure
might be 100, 115, 121).
Measurement A set of operaLons having the object of determining a
value of a measure.
Measurement
process
The process for establishing, planning, performing, and
evaluaLng measurement within a project or organizaLon.
Measure (noun) The rule for assigning a quanLtaLve or categorical value to
one or more enLty/ object/aZribute (e.g. process, project,
resource).
Informa-on
Need
Reason for which data or measures are needed; generally
phrased as “<who> needs to know <what>, in order to <do
what>”

PPQC
PROCESS AND PRODUCT
• Process Compliance
•  Process Efficiency
• Process Effec-veness
Process
Perform-
ance
• Milestone CompleLon
• CriLcal Path Performance
• Work Unit Progress
• Incremental Capability
Schedule
and
Progress
• Physical Size and Stability
• FuncLonal Size and Stability
Product
Size and
Stability
Source: Prac=cal So>ware and Systems Measurement
• FuncLonal Correctness
• Supportability – Maintainability
• Efficiency
• Portability
• Usability
• Dependability - Reliability
Product
Quality
• Personnel Effort
• Financial Performance
• Environment and Support
Resources
Resources
and Cost
• Technology Suitability
• Technology VolaLlity
Technology
Effec-ve-
ness
• Customer Feedback
• Customer Support
Customer
Sa-sfac-on

PPQC
PROCESS AND PRODUCT
•  Balanced Scorecards provide a mechanism to balance
o  internal goals and external goals
o  past performance and drivers for future performance
o  short term and long term performance
o  People, Process, and Technology investment
•  Categories generally used

Financial Outcomes
Customer
Sa-sfac-on
Internal Business
Processes
Learning and
Growth
Balanced
Scorecard

PPQC
PROCESS AND PRODUCT
•  First, establish and communicate a strategy
o  set financial objecLves
o  idenLfy key needs of customers and market segments; link those to financial goals
o  focus on internal business processes and efficiencies in meeLng customer needs
o  invest in learning and growth to excel with the processes
•  Within the scorecard
o  idenLfy how to see financial goals are met
o  set targets for meeLng customer needs
o  idenLfy process changes or innovaLons needed
o  match learning and improvement objecLves to these

PPQC
PROCESS AND PRODUCT
Financial
Increase
proﬁtability
Increase
revenue
Increase
margins
Customer
Serve more
levels of
business
Use social
networking
services
Internal
Processes
Increase range
of services
Provide social
networking
services
Maintain social
networking
site
Learning
and Growth
Build
capability
Increase social
networking
skills
Business Goals QPPOs

PPQC
PROCESS AND PRODUCT
Area Measure Target
Financial Proﬁt
Revenue Growth
Margin
25% before tax
+30% over last year
+5% over last year
Customer Mean # business levels served
% using social networking
services
+10% over last year
15% this year
Internal
Processes (and
eﬃciency)
increase range of services
Minimize social networking
downLme
new PMO sales by July
web site up 99.9%
Learning and
Growth
# of PMPs
# of months of total social
networking experience
+3 over last year
+24 over last year

PPQC
PROCESS AND PRODUCT
•  In general, process improvement directly aﬀects the lower half of
the Balanced Scorecard
o  process development and/or maintenance
o  process performance
o  learning and growth of process users
o  a way of idenLfying and tracing QPPOs
•  Changes in these areas can link to
o  increased customer saLsfacLon
o  increase in customer count or transacLon value
o  reducLon in cost to meet customer needs
•  These, in turn, translate to ﬁnancial results

PPQC
PROCESS AND PRODUCT
Financial
Develop new
business
Reduce costs
Customer
Increase
Customer
Contact
Increase
customer
value
Internal
Processes
Increase
process
compliance
Improve
process quality
Improve
product quality
Learning
and Growth
Increase skills
Retain more
employees
Raise
employee
saLsfacLon
Overall Goal: Controlled Growth and Sa-sﬁed Return
Business Goals QPPOs

PPQC
PROCESS AND PRODUCT
Areas Objec-ves Measures
Financial •  Develop New Business
•  Reduce Cost
Total revenue
Revenue in new business segment
ProducLvity (LOC/project hr.)
Customer •  Increase Customer Contact
•  Increase Customer Value
% customers with annual interview
% employees visiLng customers
customer saLsfacLon
% with maintenance contracts
Internal
Processes
•  Increase Process Compliance
•  Improve Process Quality
•  Improve Product Quality
1-5 on key pracLces
% rework
% milestones on schedule
# warranty hours needed
Learning
and Growth
•  Increase Skills
•  Retain More Employees
•  Raise Employee SaLsfacLon
# cerLﬁed in key areas
% retenLon
employee saLsfacLon score

PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N GStrategic Goal
supporLng goal
supporLng goal
supporLng goal
supporLng goal
supporLng goal measure
measure
measure
measure
measure performance
performance measure
measure
measure
measure
Corporate Business Goals and
Objec-ves
Quality and Process
Performance Objec-ves

PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N GImproved Proﬁtability
OpLmize ROI
Improve customer
saLsfacLon
Improve posiLoning
in current market
Minimize cost of
business
Improve operaLons
eﬃciency
Overhead
rates
Product-
ivity
Customer
SaLsfacLon
Market
Share
Revenue
Corporate Business Goals and
Objec-ves
Minimize
development Lme
Improve review
process to remove
defects early
COQ
# CRs
# defects
Cycle Lme
Quality and Process
Performance Objec-ves

PPQC
PROCESS AND PRODUCT
Improvement Goals QPPO
For small features requested by
customers, provide a response
next year in half the Lme we do
today by using an incremental
release model
•  Reduce cycle Lme for a release
by 50%
Reduce negaLve impact of change
management on customer
saLsfacLon
•  Cut response Lme on defect
handling by 25%
•  Cut response Lme on change
requests by 10% each quarter
Use peer reviews for key parts of
requirements work products on all
projects
•  Reduce cost of rework from
requirements defects by 50%
next year

PPQC
PROCESS AND PRODUCT

PPQC
PROCESS AND PRODUCT
•  Core Base and Derived Measures
o  Requirements Measures
o  So{ware Size Measures
o  Cost and Eﬀort Measures
o  Schedule and Progress Measures
o  Defect Measures
•  Other Key Derived Measures
o  Earned Value
o  ProducLvity/Unit Costs
•  Summary Measures for a Program or Organiza-on

PPQC
PROCESS AND PRODUCT
•  How used:
o  requirements baselined and tracked throughout project
o  provides insight into growth and stability; also used to status progress
•  Useful measures:
o  number of requirements (growth over Lme)
o  number of new requirements, changed or deleted requirements (stability/
volaLlity)
o  requirements status (approved, tested, implemented)
•  Typical coun-ng issues:
o  how to count a “requirement” consistently, since size and diﬃculty vary
o  stated vs. derived requirements
o  traceability/lack of requirements tool use
o  in maintenance, more likely to see change requests

PPQC
PROCESS AND PRODUCT
0
100
200
300
400
500
600
700
800
Number of Requirements
Project Status Date
Requirements Stability
Plan
Actual
Project: OLDMS

PPQC
PROCESS AND PRODUCT
0
10
20
30
40
50
60
Feb Mar Apr May Jun Jul Aug
Person Hours of Work or Rework
Add Requirements
Change
Requirements

PPQC
PROCESS AND PRODUCT
70% 73% 73% 76% 79% 80% 81% 82% 82% 83%
8%
10% 10%
13%
13% 15% 14% 13% 13% 12%
4%
6% 6%
6%
6%
5% 4% 4% 4% 2%
18%
11% 11%
5% 2% 0% 1% 1% 1% 2%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
6/3/2009 6/10/2009 6/17/2009 6/24/2009 7/1/2009 7/8/2009 7/15/2009 7/22/2009 7/29/2009 8/5/2009
Passed Failed Blocked Unveriﬁed

PPQC
PROCESS AND PRODUCT
•  Size is a cri-cal component of measurement and management
o  basis for esLmaLng eﬀort, and quanLfying and managing scope
o  basis for producLvity measurement (eﬀort to build a unit/size)
o  allows normalizaLon and comparison across systems and projects
(defects/size)
•  Size Measures
o  # of modules, screens/reports
o  # of calls
o  # of boards
o  # of requirements

PPQC
PROCESS AND PRODUCT
0
50
100
150
200
250
Plan 1 Plan 2 Plan 3 Actual
Number of Components
Reused
New/Modiﬁed

PPQC
PROCESS AND PRODUCT
0.1
1
10
100
1000
10000
0.1 1 10 100 1000 10000
Staﬀ Months
Source Lines of Code Thousands
Size – Eﬀort
Es-ma-ng Rela-onship
Project Under
ConsideraLon
Average
-1 Standard
DeviaLon
+1 Standard
DeviaLon

PPQC
PROCESS AND PRODUCT
•  Effort is es-mated, staffing planned, and actual effort tracked to plan
o  overall project level
o  by Lme period, by work unit/task, build, phase
•  Variance monitored for consistent over or under-run
o  underesLmated by size or effort
o  unexpectedly complex or defect-prone
•  Cost usually derived from effort (plus other costs)
o  #of labor hrs/FTEs (by event, Lme period)
•  Typical coun-ng issues:
o  no Lme reporLng tool; tool not used or enforced
o  contractors vs. employees; mgt, admin, others not captured
o  lack of granularity of plan and actual effort data
0
5
10
15
20
25
30
Q1 Q2 Q3 Q4
Financial Summary
Plan
Actual

PPQC
PROCESS AND PRODUCT
0
2
4
6
8
10
12
14
1 2 3 4 5 6 7 8 9 10 11 12
Staﬀ Hours (thousands)
Repor-ng Periods
Plan
Actual

PPQC
PROCESS AND PRODUCT
0
1
2
3
4
5
6
7
8
9
10
1-Aug 1-Sep 1-Oct 1-Nov 1-Dec 1-Jan 1-Feb 1-Mar 1-Apr 1-May 1-Jun 1-Jul 1-Aug 1-Sep 1-Oct 1-Nov 1-Dec 1-Jan 1-Feb 1-Mar 1-Apr 1-May
Eﬀort Alloca-on
Subcontractor Actual Internal Actual

PPQC
PROCESS AND PRODUCT
Ac-vity % Plan Hours
Project Management 12% 9654
ConﬁguraLon Management 8% 6436
Quality Assurance 3% 2414
Training 4% 3218
Requirements 9% 7241
Design 12% 9654
Code and Unit Test 20% 16090
IntegraLon Test 16% 12872
User Acceptance Test 9% 7241
Maintenance 5% 4023
Customer Support 2% 1609
100% 80450

PPQC
PROCESS AND PRODUCT
0
10
20
30
40
50
60
70
80
90
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
Time to Close (days)
Ticket #

PPQC
PROCESS AND PRODUCT
•  How used:
o  examine planned schedule and milestones and actual aZainment of each
o  monitor task compleLon at a level of detail to be able to recover from problems
o  milestone dates (schedule)
o  days late
o  work unit progress (requirements, design, code, test, etc.)
•  # of units planned to be complete, # completed
o  schedule accuracy/performance
•  Typical Coun-ng Issues:
o  plan data not baselined; no CM control
o  work not broken down into manageable work units
o  work units of diﬀerent size, complexity, and importance

PPQC
PROCESS AND PRODUCT
Milestone Original Revised Actual
Proposal Complete 14 January 17 January 20 January
Approval to Proceed 1 03 February 03 February 03 February
Reqmts Signed Oﬀ 08 April 15 April 15 April
Approval to Proceed 2 02 May 02 May 05 May
Design Signed Oﬀ 10 June 14 July 15 July
System Test Complete 17 July 10 August
Beta Test Complete 03 October 07 November
Approval to Proceed 3 14 October 10 November
Release to Customer 10 November 16 December

PPQC
PROCESS AND PRODUCT

PPQC
PROCESS AND PRODUCT
0
20
40
60
80
100
120
140
# of Units Comple-ng Unit Test
Project Status Date
Unit Test Status
Plan
Actual

PPQC
PROCESS AND PRODUCT
•  How used:
o  idenLfies the number of defects (problem reports) in the so{ware product
o  provides insight into test progress and both pre- and post-producLon quality
o  provides insight into rework remaining
o  defect trends (open, closed, severity, age of open, types)
o  defect densiLes (by system, subsystem, module)
o  defect origins, detecLon effecLveness, and leakage
•  Typical Coun-ng Issues:
o  defects versus effort; rework not always 1:1
o  definiLon of a defect/classificaLon inconsistencies
o  cultural/behavioral issues associated with collecLng

PPQC
PROCESS AND PRODUCT
5 5
14
4
8
18
10 10
4
18
8
4
9 9 10
72
66
74
67
60 60
57 57
48
52
45
40
38 39
37
11
6
14 15
18
13
5
12 11 10 10 11
6
8
0
10
20
30
40
50
60
70
80
Number of Defects
Status of Severity 1 Defects
New Open (Including New) Worked oﬀ During the Week
Open Last Week =
Closed This Week =
Downgraded =

New This Week =
Upgraded =

Open This Week =

39
8
4
------
27
10
0
------
37

PPQC
PROCESS AND PRODUCT
1017 1028 1034 1039 1048 1063 1077 1078 1096 1097 1111 1118 1129 1143 1147 1165 1179 1191
71 72
108 124 125
379
454 454 484 509 510 522
605
653
726
776 751 782
0
200
400
600
800
1000
1200
1400
Number of Defects
Severity 2 Defects
Severity 2 DRs Total Discovered Severity 2 DRs Total Closed

PPQC
PROCESS AND PRODUCT
0
10
20
30
40
50
60
70
80
90
<1 1 - 2 3 - 4 5 - 8 9 - 18 >18
Number of Problem Reports
Weeks Open
Problem Report Aging
Open Problem Reports
Data as of 15 June 2009
Average = 5.7 weeks
Target < 8 weeks

PPQC
PROCESS AND PRODUCT
4
9
8
10
6
0
2
4
6
8
10
12
# of Severity 1 Defects
Status of 37 Open Severity 1 Defects
Pending
Valida-on
Defect Under
Analysis
Fix Started –
Work in Progress
Fix Made –
Pending Delivery
Fix Delivered –
Pending Test

PPQC
PROCESS AND PRODUCT
0
5
10
15
20
25
30
35
40
A B C D E F
# of Problem Reports
Module ID
Problem Report Status
Open Severity 1 and 2 by Module

PPQC
PROCESS AND PRODUCT
Module
Size
(KSLOC)
Defects
Defect Density
(Defects/KSLOC)
A 44 48 1.1
B 32 60 1.9
C 36 36 1.0
D 28 33 1.2
E 34 42 1.2
F 15 46 3.1
G 9 30 3.3
Total 198 295 1.5

PPQC
PROCESS AND PRODUCT
0
20
40
60
80
100
120
140
160
180
Reqmts Design Code Unit Test Integ Test Sys Test Del + 90
days
Defects/KSLOC
Life Cycle Phase
Defect Detec-on Proﬁle
All Projects
New Process

PPQC
PROCESS AND PRODUCT
Phase
Discovered
Phase Inserted
Reqmts Analysis Design Code Integ. Test Sys Test
Accept.
Test
Ops
Requirements 50
Analysis 65 65
Design 70 80 65
Code & Unit Test 74 83 85 70
Integra-on Test 76 89 94 88 55
System Test 85 93 95 93 70 50
Acceptance Test 94 95 96 94 75 70 70
1 year Opera-ons 98 97 98 96 88 90 90 76
Columns present cumula=ve % of defects removed

PPQC
PROCESS AND PRODUCT
o  Earned Value
•  Summary Measures for a Program or Organiza-on

PPQC
PROCESS AND PRODUCT
0
20
40
60
80
100
120
140
160
3 6 9 12 15 18 21
Cost ($K)
Months
ACWP
BCWP
Now
Cost Variance
Actual Cost of
Work
Performed
(ACWP)
Budgeted Cost
of Work
Performed
(BCWP)
Cost Variance (CV) is the diﬀerence between actual and budgeted costs (BCWP - ACWP)
CPI = .63
(<1 over budget)

PPQC
PROCESS AND PRODUCT
0
20
40
60
80
100
120
140
160
3 6 9 12 15 18 21
Cost ($K)
Months
BCWS
BCWP
Now
Schedule Varianc
Budgeted Cost
of Work
Scheduled
(BCWS)
Budgeted Cost
of Work
Performed
(BCWP)
Schedule Variance (SV) is the diﬀerence between budgeted cost for work performed and
work scheduled (BCWP- BCWS)
SPI = .52
(<1 behind
schedule)

PPQC
PROCESS AND PRODUCT
0
20
40
60
80
100
120
140
160
180
Proposal Actuals to Date Re-Plan 1 (2 builds) Re-Plan 2 (3 builds)
SLOCs/Person Month
Build Produc-vity
Build 1
Build 2
Build 3
15 June 2009

PPQC
PROCESS AND PRODUCT
o  Earned Value
•  Summary Measures for a Program or an Organiza-on

PPQC
PROCESS AND PRODUCT
Performance to Schedule
milestone baseline forecast actual
xxx 1/03 1/03 1/02
yyy 6/16 6/30
zzz 7/04 8/05
Risks
•  Descrip-ons
•  Ac-on Plans
Overall Status (Red/Yellow/Green): Project Alpha as of April
0
10
20
30
40
1st Qtr 2nd Qtr 3rd Qtr 4th Qtr
Financial Summary
Plan
Actual
0 200 400 600
1st Qtr
2nd Qtr
3rd Qtr
4th Qtr
Func-onality Delivered
Actual
Plan

PPQC
PROCESS AND PRODUCT
0
5
10
15
20
25
30
35
40
45
Monthly Eﬀort (FTE Hours)
Actual
Projected
Budget
0.72 0.72
0.17
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
Cumula-ve Project Costs ($100K)
Projected
Budget
Actual
Cost Situa-on
Current Forecast
Month Total Month Total
DeviaLon more than 30%
DeviaLon more than 15%
MeeLng budget
Milestone
Envision Start
Vision/Scope
Planning Ends
Devel Complete
V&V Compete
Beta Start
Beta Live
GA
Project Closed
Delay more than 60 days
On Target or Early
0
5
10
15
20
25
30
35
40
45
Monthly Eﬀort (FTE Hours)
Actual
Projected
Budget
0.72 0.72
0.17
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
Cumula-ve Project Costs ($100K)
Projected
Budget
Actual
Schedule Situa-on
Milestone Curr Last
Envision Start
Vision/Scope
Planning Ends
Devel Complete
V&V Compete
Beta Start
Beta Live
GA
Project Closed
On Target or Early

PPQC
PROCESS AND PRODUCT
Size
Cost and
Effort
Schedule Quality
Process
Perfor-
mance
Customer
Sa-sfac-
-on
Risk
Quick
Overview
Suppor-ng
Comments
<Project Name> Monthly Status Report
Technology Effec-veness
Size
Cost and Effort
Schedule
Quality and Process Performance
Risk and Customer Sa-sfac-on
Tech.
Effec-ve-
ness

PPQC
PROCESS AND PRODUCT
Suite of indicators for
each area of reporLng
for review by
project managers,
team members,
and business managers
Total Physical Size
-10
0
10
20
30
40
50
60
08-jun
22-jun
06-jul
20-jul
03-aug
17-aug
31-aug
KLOC
EsLmated Actual Milestone Trend Module Size
0 5 10 15 20
Module A
Module B
Module C
KLOC
Current esLmate Actual
Milestone Tracking
19-may 29-may 08-jun 18-jun 28-jun 08-jul
MST1 P
MST1 A
MST2 P
MST2 A
Milestones actual/planned
Milestone Status
Completed
on Lme
20%
Completed Late
10%
Not yet completed
70%
Cost and Schedule Performance
-40 -35
-30 -25 -20 -15
-10 -5 0
5 10
Apr-08
Jul-08
Oct-08
Jan-09
Apr-09
Jul-09
Oct-09
Jan-10
Apr-10
Jul-10
Oct-10
Percent variance Schedule Performance Variance
Cost Performance Variance
Cri-cal Ra-o
0,40
0,60
0,80
1,00
1,20
1,40
Apr-08
Jul-08
Oct-08
Jan-09
Apr-09
Jul-09
Oct-09
Jan-10
Apr-10
Jul-10
Oct-10
CriLcal RaLo
Threshold (red)
Threshold (yellow)

PPQC
PROCESS AND PRODUCT
•  Issues in solware projects generally require use of common
core measures to handle
o  requirements growth and stability
o  monitoring of product size
o  management of cost and eﬀort to budget
o  schedule and other progress indicators
o  monitoring of defect levels, to see trends
•  Derived measures provide insight with combina-ons of base
measures

PPQC
PROCESS AND PRODUCT
Address: Henry Schneider
President/Senior Principal Consultant
Process and Product Quality Consul-ng, LLC
2009 Simond Avenue, Suite B
Aus-n, TX 78723

Phone: 832-628-2486

E-mail: henry@ppqc.net

Web Site: www.ppqc.net

TSP SYMPOSIUM Schneider, Henry.pptx

More Related Content

Viewers also liked

Similar to TSP SYMPOSIUM Schneider, Henry.pptx

TSP SYMPOSIUM Schneider, Henry.pptx