QuantiMetrics provides benchmarking services to help clients improve software development performance. Their approach involves collecting project data from clients and comparing key metrics like costs, productivity and quality to industry benchmarks. This identifies areas for improvement. Case studies showed benchmarking helped a telecom company reduce cycle times with minimal productivity loss, and an insurance company lower costs by 54% while improving quality. Critical success factors include executive sponsorship, process maturity, collecting comparable data, ensuring confidentiality and taking a holistic view of interrelated factors.

1. Measure. Optimize. Deliver.
Phone +1.610.644.2856
Measuring Success
Benchmarking as a Tool for Optimising
Software Development Performance
Bram Meyerson
QuantiMetrics, a DCG Partner

2. ©2013 David Consulting Group and QuantiMetrics
QuantiMetrics
• A two decade track record as an independent trusted advisor
• Established in 1992 and remained independent
• Acquired Research Services Benchmarking (ex: Butler Cox) from CSC
in 2000
• Owns the largest validated and calibrated benchmark database
covering project and application data that spans thousands of projects
from hundreds of organisations
• Retained by world-class organisations and consultancies to help them
evaluate suppliers and to improve their in-house efficiencies
• Provides client-centric service … underpinned by trust, confidentiality
and teamwork
• Offices in London, Wiesbaden, Johannesburg
2

3. ©2013 David Consulting Group and QuantiMetrics
Proven Capabilities and a Focus
on Ongoing Innovation
• Fact-based IT executive advisory and benchmark provider
• Optimisation of software development and support processes
• Due diligence and professional review of portfolio, programs and
project budgets and plans
• Software application sizing (in-house development, outsourced
applications and vendor-supplied packages)
• Evaluation of 3rd party software development services
• Commercial software delivery and support dispute advisory
3

4. ©2013 David Consulting Group and QuantiMetrics
Agenda
• Why benchmark?
• Our unique approach to benchmarking
• Examples of how benchmarking has underpinned performance
improvement
• Critical success factors for benchmarking
4

5. ©2013 David Consulting Group and QuantiMetrics
Why benchmark?
• Provides an objective 3rd party scorecard
• Provides the cross-industry breadth perspective. Most IT executives have
a depth of MIS about their project environment
• Acts as a catalyst for change by setting realistic improvement targets
• Provides an accurate mechanism to calibrate project estimation models.
• Focuses attention on measurement and improvement
5

6. ©2013 David Consulting Group and QuantiMetrics
6
Our Unique Approach to Benchmarking:
Process
Applications
Development
Project portfolio
Proposed or
disputed Project
Standard
project
data
(current)
QPeP
Benchmark
Report
Calibrated
client
baseline
QuantiMetrics
Validated Project
Database
QEsT
Risk
Report
High Moderate Acceptable
<-------------- Estimation Scenarios --------------à
Efficiency
Staffing IndexCost per FTE
Unit cost
Elapsed months
All testing errors
Operational
errors
Schedule conformance
Budget conformance
Client Minimum Maximum
Median Top quartile Top decile
IT Development – Summary Performance
Efficiency
Staffing IndexCost per FTE
Unit cost
Elapsed months
All testing errors
Operational
errors
Schedule conformance
Budget conformance
Client Minimum Maximum
Median Top quartile Top decile
IT Development – Summary Performance
Identifies good practices
and successes – but
also highlights
inefficiencies and
problems in
methodology, staffing,
planning, productivity,
cost or capability across
sizes, types &
technologies – and
proposes improvement
actions
Identifies probability of
successful delivery against
time, budget & quality
targets – and proposes
alternative scenarios with
higher likelihood
Project
characteristics

7. ©2013 David Consulting Group and QuantiMetrics
• By project, programme, group of projects and organisation
• The impact of size on project performance (finding the “sweet spot”)
• Project type
• Development technologies
• The impact of time pressure on project performance (finding the ideal
time-box)
• Industry sector
7
Our Unique Approach to Benchmarking:
Key Focus on Normalization

8. ©2013 David Consulting Group and QuantiMetrics 8
Base Data
• Classification data
• Size data
• Resource data
• Testing errors
• Operational errors
• Qualitative data
Outcomes
• Productivity
• Duration
• Unit Cost
• Test Errors
• Reliability
• Time Slip
• Effort Overrun
Drivers
• Efficiency
• Staffing Style
• Technical Quality
• Planning Style
Context
• Technology
• Cost
• Project Size
Raw Metrics Derived Metrics
Influence
Assist in
interpreting
Our Unique Approach to Benchmarking:
Appropriate Level of Metrics

9. ©2013 David Consulting Group and QuantiMetrics
Good performance for Reliability & Effort Overrun; poorest performance in
Productivity, Unit Cost, & Test Errors.
The more the blue-
lined (Customer)
footprint
approaches the
outer edge of the
polygon, the better
the performance;
the closer to the
origin (centre), the
worse the
performance
Note: Project
sensitive cost
benchmarks are UK-
based and blended
where necessary to
take account of use
of external and off-
shore resources
Productivity
Duration
Unit cost
Test errorsReliability
Time slip
Effort overrun
Minimum Maximum Median
Top quartile PEP member Bottom quartile
OUTCOMES
Our Unique Approach to Benchmarking:
Graphical Reporting

10. ©2013 David Consulting Group and QuantiMetrics
Excellent budget conformance, weak underlying efficiency and typical staffing
levels.
Efficiency
Staffing
QualityPlan conformance
Budget
conformance
Minimum Maximum Median
Top quartile PEP member Bottom quartile
DRIVERS
The more the blue-
lined (Customer)
footprint approaches
the outer edge of the
polygon the better
the performance, the
closer to the origin
(centre) the worse
the performance
Our Unique Approach to Benchmarking:
Graphical Reporting

11. ©2013 David Consulting Group and QuantiMetrics
Technology gearing is typical, but projects are much smaller than typical, and staff
costs close to the top 25%.
Gearing examines the
power of the toolset /
languages (i.e. function
points per 1,000 lines of
code)
Gearing
FTE costSize
Minimum Maximum Median
Top quartile PEP member Bottom quartile
CONTEXT
These costs are fully
loaded. Benchmark rates
quoted here are global,
dominantly based on
internal staff plus
increasing contribution of
off-shore resourcing
The more the blue-
lined (Customer)
footprint
approaches the
outer edge of the
polygon, the better
the performance;
the closer to the
origin (centre), the
worse the
performance
Our Unique Approach to Benchmarking:
Graphical Reporting

12. ©2013 David Consulting Group and QuantiMetrics
Cost-driven approach – high use of low-cost, off-shore suppliers whose productivity
is low; time-driven characteristic largely accounts for this, but also contributes to
high efficiency … productivity and efficiency are both better than previous
Staffing Style
Efficiency
Current projects
Previous projects
Current Agile
Previous Agile
Current not-Agile
Lean Time
driven
UnproductiveStretched
H
H
L
LStaff cost
Productivity
Current projects
Previous projects
Current Agile
Previous Agile
Current not-Agile
Economic Skill driven
Cost driven Uneconomic
H
H
L
L
Our Unique Approach to Benchmarking:
Trade-offs

13. ©2013 David Consulting Group and QuantiMetrics
Agenda
• Why benchmark?
• Our unique approach to benchmarking
• Examples of how benchmarking has underpinned performance
improvement
• Critical success factors for benchmarking
13

14. ©2013 David Consulting Group and QuantiMetrics
Case Study 1: Large Telco
• Leading communications company, serving the needs of customers in
almost 200 countries
• Mandated to become more cost effective and embarked on an
aggressive performance transformation initiative. The company
participated in benchmarking to underpin their transformation journey
• Summary of achievements:
– Benchmarking reports have been used as a catalyst to speed up,
sharpen costs and improve reliability of software delivery
– Benchmarking deliverables identified the impact of the adoption of
various Agile practices and assisted in identifying which were/were
not making a difference
– Dramatically shortened cycle times, with small effort tradeoffs
– Quality of software products also improved
14

15. ©2013 David Consulting Group and QuantiMetrics
Case Study 1: Large Telco
Relative to equivalent benchmarks, cycle time reduced dramatically, offset
by small productivity loss
15
Achieved by:
• Increasing adoption
of time boxing (and
other elements of
Agile practice)
• Increasing proportion
of outsourcing to a
small set of suppliers
• The decision to go for speed was more of a concern than productivity, as simultaneous use of off-
shore cheaper resources would more than compensate.
• The client could not continue to do things the same old way; to remain competitive in an era of
deregulation required being smart
Relativetobenchmark
Time

16. ©2013 David Consulting Group and QuantiMetrics
• Leading Life, Pensions and Investment company
• Non measurable achievements: benchmarking reports have been used
as a catalyst to speed up, sharpen costs and improve reliability of
software delivery.
• Summary of achievements:
– Improved quality of software products
– 54% reduction in project costs ($/fp)
– More disciplined project management
– More accurate schedule and cost estimation
– Better technical quality (fewer errors)
Case Study 2: Large Insurance Company

17. ©2013 David Consulting Group and QuantiMetrics
Case Study 2: Large Insurance Company
Significant productivity and quality improvements
17
0%
2%
4%
6%
8%
10%
12%
14%
16%
18%
Proportionofmembers
Function points perstaff-month
IQR
Year 3Year 1
Inter-quartile
range
All testing errors
Earlyoperationerrors
Current proje
Previous proj
Current Agile
Previous Agil
Current not-A
Ineffective testing Poor quality
approach
Good quality
approach
Effective testing
H
H
L
L

18. ©2013 David Consulting Group and QuantiMetrics
Technical Quality
Efficiency
Previous projects
Current projects
Current Agile
Previous Agile
Current not-Agile
Quality deferred Quality led
Quality laggedQuality ignorant
H
H
L
L
Focus on Process Leads to Getting it Right the
First Time
18

19. ©2013 David Consulting Group and QuantiMetrics
Critical Success Factors
• Sponsorship
• Maturity
• Comparability
• Confidentiality
• Taking a systemic view
19

20. ©2013 David Consulting Group and QuantiMetrics
High
Productivity
(fp/sm)
Low resource
cost ($/FTE)
Low Unit cost
(R/fp)
Effective Testing
(reducing
Fault rework)
High technology
gearing (Kloc/fp)
Process
Efficiency
Quality
Process (approach)
High
requirements
churn
Right Staffing
(approach to
Staffing levels)
Planning
Efficiency
Effective
requirements
management
Higher skill
levels
Right project
size (fp)
Excessive business
(time) demands
Late business scope creep
High speed
Lean
Processes
Reuse
“Efficeincy
over
Utelisation”
A Systems-thinking View on
Project Performance

21. ©2013 David Consulting Group and QuantiMetrics
Contact Us
Alan Cameron, DCG-SMS, Europe
Email: a.cameron@dcg-sms.com
Phone: +44 843 2895174
Mobile: +44 7825 271344
http://www.dcg-sms.com
21
@DavidConsultGrp
/DavidConsultGrp
/company/David-Consulting-Group
Mike Harris, DCG, United States
Email: m.harris@davidconsultinggroup.com
Phone: +1 (610) 644-2856
Mobile: +1 (484) 753-3935
http://www.davidconsultinggroup.com
Measure. Optimize. Deliver.
Phone +1.610.644.2856