Abstract—This article provides insight into a ‘best practice’ used for the selection of software suppliers at the largest Dutch telecom operator, KPN[1]. It explains the metrics rationale applied by KPN when selecting only one preferred supplier (system integrator) per domain instead of the various suppliers that were previously active in each domain. Presently (Q2 2012) the selection and contracting process is entering its final phase. In this paper, the model that was built and used to assess the productivity of the various suppliers and the results of the supplier selection process are discussed. In addition, a number of lessons learned and recommendations are shared.
Metrics Based Software Supplier Selectionhans_kuijpers
Abstract-This presentation provides insight into a 'best practice' used for the selection of software suppliers at the largest Dutch telecom operator, KPN.
It is related to the paper with the same title, also available on slideshare.
Rfid roi-sme pilots presentation and results ueapme august 2012UEAPME
The main goal of RFID-ROI-SME is to boost the adoption of RFID technology by wide SME communities, while at the same time creating business opportunities for innovative RFID solution providers within the EU. The aim is to demonstrate tangible benefits of RFID technology across many different sectors and business cases. These benefits will then be disseminated to wider SME communities in the form of case studies, best practices and blueprints. To this end, the project has integrated, deployed, organized and coordinated eight RFID pilots in six European countries (including one international pilot) covering different sectors, starting in March 2010 until May 2012.
This is a presentation about the RFID-ROI-SME project pilots, their business evaluations, key performance indicators, major issues and lessons learnt.
Metrics Based Software Supplier Selectionhans_kuijpers
Abstract-This presentation provides insight into a 'best practice' used for the selection of software suppliers at the largest Dutch telecom operator, KPN.
It is related to the paper with the same title, also available on slideshare.
Rfid roi-sme pilots presentation and results ueapme august 2012UEAPME
The main goal of RFID-ROI-SME is to boost the adoption of RFID technology by wide SME communities, while at the same time creating business opportunities for innovative RFID solution providers within the EU. The aim is to demonstrate tangible benefits of RFID technology across many different sectors and business cases. These benefits will then be disseminated to wider SME communities in the form of case studies, best practices and blueprints. To this end, the project has integrated, deployed, organized and coordinated eight RFID pilots in six European countries (including one international pilot) covering different sectors, starting in March 2010 until May 2012.
This is a presentation about the RFID-ROI-SME project pilots, their business evaluations, key performance indicators, major issues and lessons learnt.
TIM SA is the biggest Polish distributor of tech-‐ electro materials.
The mul(channel character of sale is crucial for us – says Krzysztof Folta, CEO in TIM SA.
Sustainability In Government - Bim Webinar4 All of Us
CROSS-WHITEHALL SUSTAINABILITY PRACTITIONERS FORUM
ADDRESSING THE BENEFITS OF BIM - SPONSORED BY FORGE TRACK LTD
We kick-started the series with a webinar for the Cross-Whitehall Sustainability Practitioners Forum. These webinars will provide the practitioners with the latest information, knowledge and policy announcements on a range of sustainability issues and was decided to launch this series by addressing BIM. BIM is a new requirement for government which Sustainability Practitioners will need to be aware of and one that will help Departments in the achievement of the mainstreaming commitments outlined below.
The Green Government Commitments require transparency and, in addition to the above requirements, must address the following areas: climate change adaptation; biodiversity and natural environment; procurement of food and catering services; sustainable construction; and staff wellbeing and quality of life. The Government Construction Strategy published May 2011, set out the framework for a range of workstreams, all of which have the ultimate aim of reducing the cost of government construction projects by 15-20 per cent by the end of the current Parliament.
The BIM task group are supporting and helping deliver the objectives of this strategy and the requirement to strengthen the public sector's capability in BIM implementation with the aim that all central government departments will be adopting, as a minimum, collaborative level 2 BIM 2016.
This webinar examined how BIM can support central government departments in their achievement of collaborative level 2 Bim by 2016 and highlight the great work already taking place within this field.
Using the CMMI-SVC to Transform an Organization into a High-Functioning, Cust...Henry Schneider
As a company grows and matures from a startup entrepreneurial venture to a sustainable corporation, the departments and company services that begin as good ideas expand and evolve to support the company’s growing business. Many times these services simply develop without any strategic vision resulting in institutionalized behaviors that are incompatible with the company’s business goals and objectives. Consequently, the transition to a larger corporation becomes a challenge. A notable example is a company’s Engineering Services Department.
When people think of Engineering Services, the Customer Support or Help Desk team is what first comes to mind. However, other services such as Product Training, Field Services (product installation and troubleshooting), and Engineering Sales Support may be provided as well.
As a product development company begins selling product, the Customer Support function becomes one of its first service offerings whether or not it recognizes it as such. In addition, it is natural for the focus of the Customer Support function to be on pleasing their customer base, as many sales are contingent upon repeat business and word of mouth until the company and its product line become established in the marketplace. Nevertheless, without a clear idea of its charter and strategic direction to support business growth and identify new markets and service offerings, the Customer Support Specialists focus instead on supporting their customer base on non-company and non-product issues and questions that consume internal resources without any tangible benefit to the company. Once a company starts banging its head on the “glass ceiling” as it attempts to grow, the leadership may recognize that its current Engineering Services approach does not support its strategic business goals and objectives.
In these circumstances, the company is not necessarily interested in implementing the CMMI for Services (CMMI-SVC) and becoming appraised to either Maturity Level 2 or Maturity Level 3. However, by using the Continuous Representation, the CMMI-SVC can provide the needed guidance to help a company restructure and reorganize its Engineering Services approach in order to become a profit center or revenue generating function.
In this presentation, we will present a case study for OMNI Flow Computers, Inc., a company that specializes in the design, development, and manufacture of panel-mount multi-run, multi-tasking liquid and gas flow computers, and field-mount, hazardous area controllers/RTUs for liquid and gas custody transfer metering systems. The challenge facing OMNI was to develop its Engineering Services Department into a high-functioning, customer-driven profit center. OMNI’s Engineering Services Department consists of three groups: Customer Support, Training, and Engineering Field Services. Customer Support handles customer questions, concerns, and issues. The Training group provides training on the OMNI product line to its customers and users. Engineering Field Services provides on-site troubleshooting services on an as-needed basis.
As the Training and Engineering Field Services groups were recent additional capabilities, Customer Support presented the biggest obstacle to overcome. Noted management consultant Peter Drucker declared several years ago that Quality in a service or product is not what you put into it. It is what the client or customer gets out of it. Moreover, an obstacle to achieving this objective was one of the core challenges faced by the department: developing an appropriate customer focus and developing new service offerings. A major reason for these challenges is the nature of OMNI products. OMNI's customers integrate their products into custody transfer systems that involve a wide variety of large-scale hardware and electronic equipment from other manufacturers. OMNI’s customers usually develop and commission these systems for their clients and end users. Therefore,
Making Architecture Business Value Driven - Dave Guevara
The Denver IASA June 2008 meeting looked at the Business of Architecture. This discussion will look at one part of the broad topic, the business and strategic alignment processes that align IT to be business value driven.
The alignment that we will focus on is between the strategic intent of a company, and the capabilities that are needed to implement the business processes and technologies that will execute that strategic intent. A case study example will be used to illustrate some of the “how to” methods for doing this in your organization. This session will be interactive.
In addition to discussion these new concepts relating strategic intent to capabilities (Business, Organizational, Technical, Integration) we will also take a brief look at getting from capabilities down to User Story inventories and ultimately into services development. Recent feedback from the Agile conference was that these concepts and their use were spot on to where the thought leadership is moving to in the Agile field.
Metrics based software supplier selection - Best practice used in the largest...Hans Kuijpers
This article provides insight into a ‘best practice’ used for the selection of software suppliers at the largest Dutch telecom operator, KPN[1]. It explains the metrics rationale applied by KPN when selecting only one preferred supplier (system integrator) per domain instead of the various suppliers that were previously active in each domain. Presently (Q2 2012) the selection and contracting process is entering its final phase. In this paper, the model that was built and used to assess the productivity of the various suppliers and the results of the supplier selection process are discussed. In addition, a number of lessons learned and recommendations are shared.
Application Portfolio Management, the Basics - How much Software do I haveFrank Vogelezang
After two external benchmarks, the Software Application Support division decided that the only way to quantitatively manage application support was by establishing a sufficient estimate of the size of the application portfolio. Based on criteria from Gartner’s Application Benchmark a selection was made regarding which applications made up the application portfolio. Five different methods were used to size approximately three hundred applications, each with its own precision and cost efficiency: Gartner Fast FPA estimation, Backfiring from LoC for some older PL/1 and Assembler applications, detailed counts with NESMA FPA and COSMIC for newly developed systems and backtracking from budget for less important smaller applications. The size estimations were made by regular support personnel with little training in functional size measurement. A sample selection was reviewed by an experienced consultant, in order to detect possible pitfalls and ambiguities. The results from the review led to a re-evaluation of most of the FPA-estimations, with a higher precision and greater consistency. The results of these size estimates can now be used to compare parts of the portfolio and to quantitatively manage the (support of the) portfolio.
Faster responses to changing business needs and being more cost effective at the same time is a challenge for many organizations.
To help our customers focus more on innovation, Fujitsu will work to reduce the operation and maintenance costs of their existing ICT assets and build a platform for growth through the modernization of these assets. It enables customers to divert their investments from operations and maintenance to innovation and growth.
The Application Value Assessment (AVA) is a strategic consulting assignment, and it's the first step towards your Application/Legacy Modernization project.
ISMA 9 - van Heeringen - Using IFPUG and ISBSG to improve organization successHarold van Heeringen
Introduction to the International Software Benchmarking Standards Group and 3 cases in which function points together with ISBSG data really resulted in business value:
- Reality check of an estimate made by experts
- Assessing the competitive position of a department
- Selecting a single software supplier
TIM SA is the biggest Polish distributor of tech-‐ electro materials.
The mul(channel character of sale is crucial for us – says Krzysztof Folta, CEO in TIM SA.
Sustainability In Government - Bim Webinar4 All of Us
CROSS-WHITEHALL SUSTAINABILITY PRACTITIONERS FORUM
ADDRESSING THE BENEFITS OF BIM - SPONSORED BY FORGE TRACK LTD
We kick-started the series with a webinar for the Cross-Whitehall Sustainability Practitioners Forum. These webinars will provide the practitioners with the latest information, knowledge and policy announcements on a range of sustainability issues and was decided to launch this series by addressing BIM. BIM is a new requirement for government which Sustainability Practitioners will need to be aware of and one that will help Departments in the achievement of the mainstreaming commitments outlined below.
The Green Government Commitments require transparency and, in addition to the above requirements, must address the following areas: climate change adaptation; biodiversity and natural environment; procurement of food and catering services; sustainable construction; and staff wellbeing and quality of life. The Government Construction Strategy published May 2011, set out the framework for a range of workstreams, all of which have the ultimate aim of reducing the cost of government construction projects by 15-20 per cent by the end of the current Parliament.
The BIM task group are supporting and helping deliver the objectives of this strategy and the requirement to strengthen the public sector's capability in BIM implementation with the aim that all central government departments will be adopting, as a minimum, collaborative level 2 BIM 2016.
This webinar examined how BIM can support central government departments in their achievement of collaborative level 2 Bim by 2016 and highlight the great work already taking place within this field.
Using the CMMI-SVC to Transform an Organization into a High-Functioning, Cust...Henry Schneider
As a company grows and matures from a startup entrepreneurial venture to a sustainable corporation, the departments and company services that begin as good ideas expand and evolve to support the company’s growing business. Many times these services simply develop without any strategic vision resulting in institutionalized behaviors that are incompatible with the company’s business goals and objectives. Consequently, the transition to a larger corporation becomes a challenge. A notable example is a company’s Engineering Services Department.
When people think of Engineering Services, the Customer Support or Help Desk team is what first comes to mind. However, other services such as Product Training, Field Services (product installation and troubleshooting), and Engineering Sales Support may be provided as well.
As a product development company begins selling product, the Customer Support function becomes one of its first service offerings whether or not it recognizes it as such. In addition, it is natural for the focus of the Customer Support function to be on pleasing their customer base, as many sales are contingent upon repeat business and word of mouth until the company and its product line become established in the marketplace. Nevertheless, without a clear idea of its charter and strategic direction to support business growth and identify new markets and service offerings, the Customer Support Specialists focus instead on supporting their customer base on non-company and non-product issues and questions that consume internal resources without any tangible benefit to the company. Once a company starts banging its head on the “glass ceiling” as it attempts to grow, the leadership may recognize that its current Engineering Services approach does not support its strategic business goals and objectives.
In these circumstances, the company is not necessarily interested in implementing the CMMI for Services (CMMI-SVC) and becoming appraised to either Maturity Level 2 or Maturity Level 3. However, by using the Continuous Representation, the CMMI-SVC can provide the needed guidance to help a company restructure and reorganize its Engineering Services approach in order to become a profit center or revenue generating function.
In this presentation, we will present a case study for OMNI Flow Computers, Inc., a company that specializes in the design, development, and manufacture of panel-mount multi-run, multi-tasking liquid and gas flow computers, and field-mount, hazardous area controllers/RTUs for liquid and gas custody transfer metering systems. The challenge facing OMNI was to develop its Engineering Services Department into a high-functioning, customer-driven profit center. OMNI’s Engineering Services Department consists of three groups: Customer Support, Training, and Engineering Field Services. Customer Support handles customer questions, concerns, and issues. The Training group provides training on the OMNI product line to its customers and users. Engineering Field Services provides on-site troubleshooting services on an as-needed basis.
As the Training and Engineering Field Services groups were recent additional capabilities, Customer Support presented the biggest obstacle to overcome. Noted management consultant Peter Drucker declared several years ago that Quality in a service or product is not what you put into it. It is what the client or customer gets out of it. Moreover, an obstacle to achieving this objective was one of the core challenges faced by the department: developing an appropriate customer focus and developing new service offerings. A major reason for these challenges is the nature of OMNI products. OMNI's customers integrate their products into custody transfer systems that involve a wide variety of large-scale hardware and electronic equipment from other manufacturers. OMNI’s customers usually develop and commission these systems for their clients and end users. Therefore,
Making Architecture Business Value Driven - Dave Guevara
The Denver IASA June 2008 meeting looked at the Business of Architecture. This discussion will look at one part of the broad topic, the business and strategic alignment processes that align IT to be business value driven.
The alignment that we will focus on is between the strategic intent of a company, and the capabilities that are needed to implement the business processes and technologies that will execute that strategic intent. A case study example will be used to illustrate some of the “how to” methods for doing this in your organization. This session will be interactive.
In addition to discussion these new concepts relating strategic intent to capabilities (Business, Organizational, Technical, Integration) we will also take a brief look at getting from capabilities down to User Story inventories and ultimately into services development. Recent feedback from the Agile conference was that these concepts and their use were spot on to where the thought leadership is moving to in the Agile field.
Metrics based software supplier selection - Best practice used in the largest...Hans Kuijpers
This article provides insight into a ‘best practice’ used for the selection of software suppliers at the largest Dutch telecom operator, KPN[1]. It explains the metrics rationale applied by KPN when selecting only one preferred supplier (system integrator) per domain instead of the various suppliers that were previously active in each domain. Presently (Q2 2012) the selection and contracting process is entering its final phase. In this paper, the model that was built and used to assess the productivity of the various suppliers and the results of the supplier selection process are discussed. In addition, a number of lessons learned and recommendations are shared.
Application Portfolio Management, the Basics - How much Software do I haveFrank Vogelezang
After two external benchmarks, the Software Application Support division decided that the only way to quantitatively manage application support was by establishing a sufficient estimate of the size of the application portfolio. Based on criteria from Gartner’s Application Benchmark a selection was made regarding which applications made up the application portfolio. Five different methods were used to size approximately three hundred applications, each with its own precision and cost efficiency: Gartner Fast FPA estimation, Backfiring from LoC for some older PL/1 and Assembler applications, detailed counts with NESMA FPA and COSMIC for newly developed systems and backtracking from budget for less important smaller applications. The size estimations were made by regular support personnel with little training in functional size measurement. A sample selection was reviewed by an experienced consultant, in order to detect possible pitfalls and ambiguities. The results from the review led to a re-evaluation of most of the FPA-estimations, with a higher precision and greater consistency. The results of these size estimates can now be used to compare parts of the portfolio and to quantitatively manage the (support of the) portfolio.
Faster responses to changing business needs and being more cost effective at the same time is a challenge for many organizations.
To help our customers focus more on innovation, Fujitsu will work to reduce the operation and maintenance costs of their existing ICT assets and build a platform for growth through the modernization of these assets. It enables customers to divert their investments from operations and maintenance to innovation and growth.
The Application Value Assessment (AVA) is a strategic consulting assignment, and it's the first step towards your Application/Legacy Modernization project.
ISMA 9 - van Heeringen - Using IFPUG and ISBSG to improve organization successHarold van Heeringen
Introduction to the International Software Benchmarking Standards Group and 3 cases in which function points together with ISBSG data really resulted in business value:
- Reality check of an estimate made by experts
- Assessing the competitive position of a department
- Selecting a single software supplier
Controlling Project Performance by Using a Defect Model - SEPG NA 2008 - Ben ...Ben Linders
Wouldn’t it be nice if you had more insight in product quality, during development? To estimate defects inserted in the product, and check how effective test is in capturing defects? We developed a model to measure defects from requirements to customer feedback after release. It is used to control projects (putting quality next to planning/budget), evaluate risks, and decide on release and support. Baseline data is used to measure and improve process performance.
Making the Business Case for Remote Service CapabilitiesPTC
Manufacturers are looking to make their service more efficient and more profitable. Adopting remote service capabilities allows OEMs to move from vendor to partner, building customer trust along the way.
Improve Estimation maturity using Functional Size Measurement and Historical ...Harold van Heeringen
Many software projects still fail in recent years, also agile projects. Improving estimation maturity in order to start with a realistic estimate instead of an optimistic one can really save billions of dollars in most local software industries. The Chinese government may now be moving towards an active software estimation maturity improvement strategy in its new 5-year plan. Functional size measurement and relevant historical data as well as parametric estimation tools are key to such a strategy. This presentation was the key-note speech at the China System and Software Process Improvement Association conference on software estimation, Beijing China, May 27 2016.
Nowadays, as the software industry is slowly becoming more mature, software measurement and performance measurement are becoming increasingly important. Organizations need to know their productivity and competitiveness in software development projects for various reasons. In many software development contracts, targets are set for the suppliers to reach. These targets are based on software metrics like productivity, speed of delivery and software quality. In order to check if the targets are reached, it is necessary to measure the functional size of the software product that is delivered and also the functional size of the software development project that is carried out, as there is usually a difference between these two sizes. To be able to use functional size in contracts, it must be measured in an objective, repeatable, verifiable and therefore defensible way. That being the case, the industry’s best practice is to use an ISO/IEC standard for functional size measurement, e.g. Nesma, COSMIC or IFPUG function points. However, these methods only measure the functional user requirements from the total software requirements to be delivered. In activities like project estimation and productivity measurement, the influence of the non-functional requirements is expressed in the Project Delivery Rate (PDR) which is expressed in effort hours per function point. If more than the average amount of non-functional requirements need to be realized in a project (or more severe non-functional requirements), the PDR used should also be higher. In the industry it is customary to set productivity targets based on an average (or calibrated) influence of non-functional requirements and this works quite fine in traditional software projects. In software development projects that are executed in an agile way, this is not always the case. When working agile, there are forces that influence the traditional way of performance measurement significantly, resulting in a number of serious issues. In this paper these issues are explained and a method to overcome these issues is proposed.
Nowadays, as the software industry is slowly becoming more mature, software measurement and performance measurement are becoming increasingly important. Organizations need to know their productivity and competitiveness in software development projects for various reasons. In many software development contracts, targets are set for the suppliers to reach. These targets are based on software metrics like productivity, speed of delivery and software quality. In order to check if the targets are reached, it is necessary to measure the functional size of the software product that is delivered and also the functional size of the software development project that is carried out, as there is usually a difference between these two sizes. To be able to use functional size in contracts, it must be measured in an objective, repeatable, verifiable and therefore defensible way. That being the case, the industry’s best practice is to use an ISO/IEC standard for functional size measurement, e.g. Nesma, COSMIC or IFPUG function points. However, these methods only measure the functional user requirements from the total software requirements to be delivered. In activities like project estimation and productivity measurement, the influence of the non-functional requirements is expressed in the Project Delivery Rate (PDR) which is expressed in effort hours per function point. If more than the average amount of non-functional requirements need to be realized in a project (or more severe non-functional requirements), the PDR used should also be higher. In the industry it is customary to set productivity targets based on an average (or calibrated) influence of non-functional requirements and this works quite fine in traditional software projects. In software development projects that are executed in an agile way, this is not always the case. When working agile, there are forces that influence the traditional way of performance measurement significantly, resulting in a number of serious issues. In this paper these issues are explained and a method to overcome these issues is proposed.
The importance of benchmarking software projects - Van Heeringen and OgilvieHarold van Heeringen
Benchmarking is a crucial management activity that enables organizations to understand how competitive they are. Using functional size measurement methods and historical data enables organizations to improve their processes and become more succesful.
Avoid software project horror stories - check the reality value of the estima...Harold van Heeringen
Many large software projects turn into software horror stories, resulting in newspaper headlines and even political issues. Often, the project costs and schedule were estimated unrealistically optimistic, using immature estimation techniques. A relatively simple way to avoid many problems is to perform a reality check on the estimate. This presentation was given on the conference of the International Cost Estimating and Analysis Association (ICEAA2014), June 2014 (Denver, USA)
This presentation shows why it is important to benchmark the performance of software projects and organizations. Measurement of performance and comparing this to relevant peer groups provides the knowledge and understanding for managemenr to make informed decisions on where the organization stands and where it should go. This presentation was given at the Italian GUFPI-ISMA conference (December 2013) and addressed also the way the Italian industry is performing according to the ISBSG Country Analysis report.
Using the ISBSG data to improve your organization success - van Heeringen (Me...Harold van Heeringen
This presentation gives an introduction into the ISBSG organization and shows how the ISBSG products and data can help the software industry to become more mature. Three cases from personal experience are showed: a reality check of an expert estimate (that turned out to be optimistic), the analysis of the outgoing bids of a software supplier and the measurement of the performance of a single supplier, providing target productivity levels they should reach.
DevOps and metrics presentation, co-presentation by Dave van Herpen and Harold van Heeringen (both Sogeti Nederland B.V.). The key message of the presentation is the fact that metrics are important in DevOps environments and that it is important to do a thorough analysis of which metrics are important to collect and for which reasons.
In many organizations, bottom up estimation of software development projects is still the way to go. Management feels that top-down (parametric) estimation models require too much effort and cost. In this presentation, a random selection of 10 bids are analyzed and the bottom-up estimates and top-down estimates are compared with regard to accuracy and effort spend.
In many organizations, bottom up estimation of software development projects is still the way to go. Management feels that top-down (parametric) estimation models require too much effort and cost. In this paper, a random selection of 10 bids are analyzed and the bottom-up estimates and top-down estimates are compared with regard to accuracy and effort spend.
When clients outsource their software development projects, they need to make sure that these suppliers don't overprice the projects. As it is often not longer possible to select the best offer, there should be another mechanism to measure the value that they are getting in comparison to the money they are paying. Supplier Performance Measurement enables clients to keep in control of theis project costs in outsourcing situations and to negotiate performance improvements with the suppliers.
Project Control using functional size - which method to use?Harold van Heeringen
Project Control is one of the most difficult tasks of a project manager. To assess whether a project is still on track is very difficult because the team members often report overconfident status reports. Using functional size and specialized tooling, like QSM Control or SEER PPMC, helps the project manager to get a real good understanding of the status of the project. These tools produce accurate forecasts and offer the possibility to do what-if analysis in order to select the best way to get a project back on track. In this presentation, it is examined which method would produce the best results, NESMA or IFPUG FPA or COSMIC FPA.
Project Control using functional size - which method to use?
Metrics based software supplier selection - Best practice used in the largest Dutch telecom company
1.
2. Metrics Based Software Supplier Selection
Best practice used in the largest Dutch telecom company
Hans Kuijpers
Harold van Heeringen
Assisi, October 2012
3. Introduction
Harold van Heeringen
Hans Kuijpers
Sizing, Estimating & Control
Program Assurance & Methods
Harold.van.heeringen@sogeti.nl
hans.tm.kuijpers@kpn,.com
@haroldveendam
@_hanskuijpers
@Sogeti_SEC
Sogeti Nederland BV: KPN Nederland:
Senior Consultant Software Metrics Manager Metrics Desk
ISBSG: President Certified Scope Manager
NESMA: Board Member QSM: Special Interest Group Agile
NESMA: Working Group Chair COSMIC
NESMA: Working Group Chair Benchmarking
COSMIC: IAC Member
Metrics Based Software Supplier Selection Pagina 2
4. Agenda
• Introduction and Context
• Phases and Timeline
• The Model
• Results
• Conclusions & Recommendations
Metrics Based Software Supplier Selection -3-
5. Introduction and Context
Revenues: €13.000m
.Why supplier selection? KPN Board
EBITDA: € 5.100m
Employees: 31.000
• Consolidation # suppliers
• Cost reduction
Consumer Business Corporate KPN
• Supplier acts as SI & MSP Market Market Market NetCo E-Plus
Belgium
• 5 Year investment
• SPM is a best practice at KPN Customer
Fixed Mobile Wholesale IT Operations
Experience
Generic &
OSS Domains BSS
Traditional
Problem: no more competition between suppliers
instrument needed to avoid excessive cost Unit of Work pricing
Metrics Based Software Supplier Selection -4-
6. Agenda
• Introduction and Context
• Phases and Timeline
• The Model
• Results
• Conclusions & Recommendations
Metrics Based Software Supplier Selection -5-
7. Phases and Timeline
Why Productivity metric added?
• Objective selection criteria
• Supplier willingness to show their transparency
• Basis for productivity baseline
• Insight in quality level
• Negotiations for year on year cost reduction
• Relation to continous improvement steps
Metrics Based Software Supplier Selection -6-
8. Requested project information
• Data of 6 historical projects, max 3 KPN projects
• In scope of current technology domain
• Range 300 – 1000 FP
• Sizing method NESMA 2.1 or IFPUG 4.x
• DCF must be completely filled out
• No other template is allowed
In BAFO phase suppliers should show evidence of the size and productivity
figures by releasing FPA-reports, Data Collection Forms and/or insight in their
administrative systems.
Metrics Based Software Supplier Selection -7-
10. Historical Project Data form (2)
Per data field
requirements are
mentioned in the
template.
Metrics Based Software Supplier Selection -9-
11. Agenda
• Introduction and Context
• Phases and Timeline
• The Model
• Results
• Conclusions & Recommendations
Metrics Based Software Supplier Selection - 10 -
12. The Model
Characteristics:
• Degree of openess and compliancy
• Completeness and cohesion of submitted data
• Productivity benchmark against each other and industry
• Delivered Quality
• During the RFP phase the data will be considered as correct, but will be
checked on reality
The 3 test criteria:
A. Compliancy value (10%)
B. Reality value (30%)
C. Productivity - Quality value (60%)
Metrics Based Software Supplier Selection - 11 -
13. Used metrics and benchmarks
Project Delivery Rate (PDR) = spent project effort related to function point (h/FP)
Productivity Index (PI) = metric from QSM, derived from size, duration and effort
Quality: delivered defects per FP
Benchmarks:
• PI against the QSM Business trend line
• PDR against ISBSG Repository
• Adjusted = normalised to Construction+Test activities
Metrics Based Software Supplier Selection - 12 -
14. Compliancy value (10%)
Suppliers start with 10 points
The compliancy value is substracted with 2 points for each violation of rule:
a)Range 300 – 1000 Function Points
b)Method NESMA 2.1 or IFPUG 4.x
c) Each field of “Historical Project Data”-form must be filled out
Maximum value = 10, minimum value = 0
Metrics Based Software Supplier Selection - 13 -
15. Reality value (30%)
PI vs. Functional size (FP)
35
Unrealistic projects are discarded Unrealistic
from further analysis: 30
• PI > +2 sigma (95%) 25
• PDR < P25 ISBSG (best in class 20
projects)
PI
15
The reality value is substracted with 10
Supplier A
Supplier B
2 points for each unrealistic project 5
Supplier C
Supplier D
Supplier E
QSM Business
Maximum value = 10 0
Avg. Line Style
2 Sigma Line Style
50 150 250 350 450 550 650 750 850 950
Minimum value = 0 Effective FP
Functional Size (FP)
Metrics Based Software Supplier Selection - 14 -
16. Productivity - Quality value (60%)
Productivity - Quality value =
(Points PI score * 0,5) + (Points PDR score * 0,3) + (Points Quality score * 0,2)
ID PDR (h/FP) PDR ISBSG median PDR score ID Defects/FP Quality score
7 5,9 8,6 -2,7 15 41,7
8 6,0 8,6 -2,6 18 13,9
9 6,9 8,6 -1,7 21 66,7
11 6,2 8,6 -2,4 22 4,0
12 7,3 8,6 -1,3 23 10,0
Average: -2,1
Median 13,9
The lowest average most points The lowest median most points
For PI and PDR the average of the distance to the benchmark value is determined
For the quality the median is dertermined
The highest average most points
Metrics Based Software Supplier Selection - 15 -
17. Agenda
• Introduction and Context
• Phases and Timeline
• The Model
• Results
• Conclusions & Recommendations
Metrics Based Software Supplier Selection - 16 -
18. Results of Compliancy (1)
Projects discarded:
• Projects on going (4)
• Project sized in COSMIC (1) Result: 1 supplier has 3 violations,
the others 5 or more
Blank crucial fields
• Defect data Supplier Compliancy Value
• Effort data Supplier A 0
• Dates Supplier B 0
Supplier C 4
Other violations Supplier D 0
• Primary Language (example English) Supplier E 0
Metrics Based Software Supplier Selection - 17 -
20. Results of Reality
Projects unrealistic:
• 3 according to PI
• 1 according to PDR
Unrealistic Unrealistic
Discarded for further analysis projects PI projects PDR
Supplier criterion criterion Reality Value
Supplier A 1 1 6
Supplier B 1 0 8
Supplier C 0 0 10
Supplier D 0 0 10
Supplier E 1 0 8
Metrics Based Software Supplier Selection - 19 -
21. Results of Productivity / Quality
Rank Points PDR Rank Points
Supplier PI score PI score PI score Supplier score PDR score PDR score
Supplier A 3,9 2 8 Supplier A -3,2 1 10
Supplier B 5,0 1 10
+
Supplier B -2,1 2 8 +
Supplier C 3,4 3 6 Supplier C 16,6 4 4
Supplier D 3,0 5 2 Supplier D 6,2 3 6
Supplier E 3,2 4 4 Supplier E 18,3 5 2
Quality Rank Points Points Points Points Productivity/
Supplier Score Quality score Quality score Supplier PI score PDR score Quality score Quality value
Supplier A 3,1 1 10 Supplier A 8 10 10 9,0
Supplier B 13,9 2 8 Supplier B 10 8 8 9,0
Supplier C 52,6 3 6 = Supplier C 6 4 6 5,4
Supplier D 1000,0 5 2 Supplier D 2 6 2 3,2
Supplier E 94,6 4 4 Supplier E 4 2 4 3,4
weight 50% 30% 20%
Metrics Based Software Supplier Selection - 20 -
22. Results of Total Assessment
Recommendation from Metrics Desk: Supplier B and A score best in the model
Compliancy Reality Productivity/ Total
Supplier value value Quality value Points Rank
Supplier A 0 6 9,0 7,2 2
Supplier B 0 8 9,0 7,8 1
Supplier C 4 10 5,4 6,6 3
Supplier D 0 10 3,2 4,9 4
Supplier E 0 8 3,4 4,4 5
weight 10% 30% 60%
However suppliers B and C were selected for the next BAFO phase
Metrics Based Software Supplier Selection - 21 -
23. Findings BAFO phase
Metrics Desk investigated the provided project data of the selected suppliers B + C:
• Size
• Dates
• Hours
• Defects
Supplier B:
• Resistance: confidentiality clause with clients
• Client site visit
Supplier C:
• Size measurement by junior not certified measurers
Metrics Based Software Supplier Selection - 22 -
24. Agenda
• Introduction and Context
• Phases and Timeline
• The Model
• Results
• Conclusions & Recommendations
Metrics Based Software Supplier Selection - 23 -
25. Conclusions and recommendations
• The productivity assessment influenced the total outcome significantly
• The assessment and discussions afterwards gave insight in:
Conclusions:
• Transparency and CMMI level
• The results are used in the negotiations phase to maximize the baseline value
• Make sure the parties understand:
• the purpose of the assessment
• the use of the “Historical Project Data” form
• that the disclosed data will be validated and should not be confidental
• the consequences of violating the governance rules (e.g. penalty points)
Recommendations: • Because of many violations of the compliancy rules, consider 1 penalty point
per violation
• Construct model beforehand, but don’t communicate the model with suppliers
• Bring site visits when offered. This gives extra information next to the
productivity validation
Metrics Based Software Supplier Selection - 24 -
26. Productivity: don’t trust it, check it
Harold van Heeringen Hans Kuijpers
Sizing, Estimating & Control Software Metrics Consultant
Harold.van.heeringen@sogeti.nl hans.tm.kuijpers@kpn,.com
@haroldveendam @_hanskuijpers
@Sogeti_SEC
Metrics Based Software Supplier Selection - 25 -