This presentation shows why it is important to benchmark the performance of software projects and organizations. Measurement of performance and comparing this to relevant peer groups provides the knowledge and understanding for managemenr to make informed decisions on where the organization stands and where it should go. This presentation was given at the Italian GUFPI-ISMA conference (December 2013) and addressed also the way the Italian industry is performing according to the ISBSG Country Analysis report.

1. Harold van Heeringen,
ISBSG president

2. Overview
Benchmarking
Software Project Industry
Functional Size Measurement
Software Metrics
Historical Data: ISBSG
Project Benchmark Example
Organization Benchmark
Other uses for Benchmarking
Country report – Italy
Data submissions Italy

3. Benchmarking (wikipedia)
Benchmarking is the process of comparing one's business processes
and performance metrics to industry bests or best practices from
other industries.
Benchmarking is used to measure performance using a
specific indicator (cost per unit of measure, productivity per unit
of measure, cycle time of x per unit of measure or defects per
unit of measure) resulting in a metric of performance that is then
compared to others
This then allows organizations to develop plans on how to make
improvements or adapt specific best practices, usually with the
aim of increasing some aspect of performance. Benchmarking
may be a one-off event, but is often treated as a continuous
process in which organizations continually seek to improve their
practices.

4. Where are we now?
“Even the most detailed navigation map of an
area is useless if you don’t know where you are”
?
?
?
?

5. Informed decisions
Senior Management of IT departments/organizations
need to make decisions need to make decisions
based on ‘where they are’ and ‘where they want to
go’.
Benchmarking is about determining ‘where you are’
compared to relevant peers, in order to make
informed decisions.
But how to measure and determine where you are?

6. Software project industry
Low ‘performance metrics’ maturity
Few Performance Measurement Process implemented
Few Benchmarking processes implemented
Most organizations don’t know how good or how bad
they are in delivering or maintaining software.
These organizations are not able to assess their
competitive position, nor able to make informed
strategic decisions to improve their competitive
position.

7. But…
Best in Class organizations deliver software up to 30
times more productively than Worst in Class
organizations
High Productivity, High Quality
More functionality for the users against lower costs – value
Shorter Time to Market – competitive advantage!
Worst in Class organizations will find themselves in
trouble in an increasingly competitive market
Outperformed by competition
Internal IT departments get outsourced
Commercial software houses fail to win new contracts
Important to know where you stand!
Benchmark is essential!

8. Difficulty – low industry maturity
How to measure metrics like productivity, quality,
time-to-market in such a way that a meaningful
comparison is possible?
Comparing apples to apples

9. Software is not easy to compare

10. Functional Size Measurement
Function Point Analysis (NESMA, IFPUG or COSMIC)
Measure the functional user requirements – size in function points;
ISO standards – objective, independent, verifiable, repeatable;
Strong relation between functional size and project effort needed;
What to do with the results?
Project effort/duration/cost estimation
Benchmarking/performance measurement
Use in Request for Proposal management (answer price/FP questions)
What about historical data?
Company data (preferably for estimation)
Industry data (necessary for external benchmarking)

11. Unit of Measure (UoM)
Why are Function Points the best UoM to use in Benchmarking?
Functionality is of value for the client/business. More functionality means
more value. More Lines of code (technical size) is not necessarily of value.
Function Points are measured independent from technical requirements
500 FP of functionality implemented in Java SOA architecture
=
500 FP of functionality implemented in Cobol mainframe
Function Points are measured independent from implementation method
500 FP delivered in an agile development project
=
500 FP delivered in a COTS package implementation

12. Software metrics – some examples
Productivity
Productivity Rate: #Function points per staff month
PDR: #Effort hours per function Point
Quality
Defect Density: #Defects delivered per 1000 function
points
Time to Market
Speed: #Function points delivered per calendar month

13. Performance measurement
Measure the size of completed projects
Project size in Function Points
Product size in Function Points
Collect and analyze the data
Effort hours, duration, defects
Normalize the data when necessary
Store the data in the corporate database
Benchmark the project, internally and externally
Report metrics and trends
Different reports for different stakeholders
Depending on goals of the stakeholder

14. External benchmark
How to benchmark your performance externally?
Gartner / McKinsey/other ??
Very expensive!
No insight into data used !!!
Do it yourself Benchmarking, use:
Historical data of completed projects!!
Low cost, more feeling for the data, decide yourself which
peer groups are most relevant, decide yourself which
data is relevant!

15. Historical data: ISBSG repositories
International Software Benchmarking Standards Group
Independent and not-for-profit
Full Members are non-profit organizations, like DASMA, IFPUG,
FiSMA, QESP and NESMA. GUFPI-ISMA is now associate member
Grows and exploits two repositories of software data:
New development projects and enhancements (> 6000 projects)
Maintenance and support (> 1000 applications)
Everybody can submit project data
DCQ on the site
Completely anonymous
Free benchmark report in return

16. ISBSG
Mission: “To improve the management of IT resources by both
business and government, through the provision and
exploitation of public repositories of software engineering
knowledge that are standardized, verified, recent and
representative of current technologies”.
All ISBSG data is
validated and rated in accordance with its quality guidelines
current
representative of the industry
independent and trusted
captured from a range of organization sizes and industries

17. Website and portal

18. Example project benchmark
Project X was completed, the following data was collected:
Primary programming language: Java
Effort hours spent: 5730
Duration: 11 months
Defects found after delivery: 23
The functional size of the project was measured: 411 FP
Software metrics:
Project Delivery Rate: 5730/411 = 13,9 h/FP
Project Speed: 411/11 = 37,4 FP per calendar month
Defect Density: (23/411) *1000 = 56,0 defects/1000 FP

19. Example benchmark
ISBSG ‘New Developments & Enhancements’
Select the right ‘peer group’
Data Quality A or B
Count approach: IFPUG 4.x or NESMA
Primary Programming Language = ‘Java’
300 FP < Project Size < 500 FP

20. Results project benchmark
PDR
Speed
N
488
N
428
Minimum
0,1
Minimum
9,4
Percentile 10
2,5
Percentile 10
23,1
Percentile25
4,7
Percentile 25
32,5
Median
9,8
Median
53,8
Percentile 75
18,4
Percentile 75
95,4
Percentile
Percentile 90
130,2
90
28,9
Maximum
476,0
Maximum
621,3
Average
70,9
Average
15,2
Project Delivery Rate: 5730/411 = 13,9 h/FP
Project Speed: 411/11 = 37,4 FP per calendar month
Defect Density: (23/411) *1000 = 56,0 defects/1000 FP
Defect
Density
N
Minimum
Percentile 10
Percentile 25
Median
Percentile 75
Percentile 90
Maximum
Average
This project was carried out less productive and slower
than market average, and the quality is worse than average.
154
0,0
0,0
0,0
3,7
17,9
40,1
366,5
18,6

21. Organization benchmark
Organization Y - Speed index
Organization Y - Productivity lndex
2,0
1,8
Speed Index
Productivity Index
2,2
2,2
2,0
1,8
1,6
1,4
1,2
1,0
0,8
0,6
0,4
1,6
1,4
1,2
1,0
0,8
0,6
<2009
2009
Organization Y PI
2010
2011
2012
0,4
<2009
Target (baseline +50%)
2009
2010
2011
2012
Organization Y Speed index
Industry Productivity
Industry Speed
Lower bound (baseline -40%)
Quality Index
2,0
1,8
1,6
1,4
1,2
1,0
0,8
0,6
0,4
0,2
2009
2010
2011
Lower bound (baseline -40%)
Analysis:
Until 2010, the organization was improving
After 2010/2011, the trends go the wrong way
Recommendation: find the cause and draw up
improvement plan
Organization Y - Quality lndex
<2009
Target (baseline +50%)
2012
organization Y - Quality index
Target (baseline +50%)
Industry Quality level
Lower bound (baseline -50%)

22. Other uses for ISBSG data
Vendor selection, based on productivity, speed or
quality metrics, compared to the industry.
Definition of SLA agreements (or other KPI’s) based on
industry average performance.
Establish a baseline from which to measure future
improvement.
Explain to the client/business that a project was
carried out in a ‘better-than-average’ way, while the
client may perceive otherwise.

23. Analysis of the data
Analyze the difference in productivity or quality
between two (or more) types of projects:
Traditional vs. Agile
Outsourced vs. In-house
Government vs. Non-government
One site, multi site
Reuse vs. no reuse
Etcetera.
ISBSG Special Analysis reports

24. Traditional vs. Agile
Agile productivity: 10 – 20% gain after 1 year of
adoption
Agile cost: 20-40% lower after 1 year of adoption
Agile time-to-market: 10-60% less
Agile quality (post production defects): 2-8% higher
10 Take Aways from Reifer Agile Report
(www.isbsg.com)

25. Special reports
Impact of Software Size on Productivity
Government and Non-Government Software Project
Performance
ISBSG Software Industry Performance report
ISBSG The Performance of Business Application, Real-Time and
Component Software Projects
Estimates – How accurate are they?
Planning Projects – Role Percentages
Team size impact on productivity
Manage your M&S environment – what to expect?
Many more

27. Country report

28. Italy (IFPUG)
Latest project: 2005

29. Italy (COSMIC)
Latest project: 2010

30. Government vs. Non-government

31. Role percentages

32. We need data!
Everybody wants to use data
But nobody wants to submit data… Why not?
Is it hard?
Is there a risk?
Is the reward not big enough?
Why not try it? You’ll get a free Benchmark report and
100 portal credits in return!!
Are there any factors preventing you?
WWW.ISBSG.ORG

33. H.S.2
GUFPI-ISMA Event offer
GUFPI-ISMA members
always get a 10% discount,
using the code provided
earlier this year.

34. Dia 33
H.S.2
van Heeringen; 15-11-2013

35. Thank you!

36. NEtherlands Software Metrics users Association

37. To celebrate our anniversary NESMA
will organise the IWSM Mensura
October 6-8, 2014
2014.iwsm-mensura.org

38. Historical landmarks

39. Modern landmarks

40. European Cultural Capital 2001

41. The city of modern architecture

42. Largest soccerstadium in the Netherlands

43. Welcome to
Rotterdam

44. Busiest port in Europe

45. I hope to see you next year
October 6-8, 2014
2014.iwsm-mensura.org