Your SlideShare is downloading. ×
Metrics based software supplier selection - Best practice used in the largest Dutch telecom company
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Metrics based software supplier selection - Best practice used in the largest Dutch telecom company

235

Published on

Abstract—This article provides insight into a ‘best practice’ used for the selection of software suppliers at the largest Dutch telecom operator, KPN[1]. It explains the metrics rationale applied by …

Abstract—This article provides insight into a ‘best practice’ used for the selection of software suppliers at the largest Dutch telecom operator, KPN[1]. It explains the metrics rationale applied by KPN when selecting only one preferred supplier (system integrator) per domain instead of the various suppliers that were previously active in each domain. Presently (Q2 2012) the selection and contracting process is entering its final phase. In this paper, the model that was built and used to assess the productivity of the various suppliers and the results of the supplier selection process are discussed. In addition, a number of lessons learned and recommendations are shared.

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
235
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Metrics Based Software Supplier SelectionBest practice used in the largest Dutch telecom companyHans KuijpersHarold van HeeringenAssisi, October 2012
  • 2. Introduction Harold van Heeringen Hans Kuijpers Sizing, Estimating & Control Program Assurance & Methods Harold.van.heeringen@sogeti.nl hans.tm.kuijpers@kpn,.com @haroldveendam @_hanskuijpers @Sogeti_SECSogeti Nederland BV: KPN Nederland: Senior Consultant Software Metrics Manager Metrics DeskISBSG: President Certified Scope ManagerNESMA: Board Member QSM: Special Interest Group AgileNESMA: Working Group Chair COSMICNESMA: Working Group Chair BenchmarkingCOSMIC: IAC MemberMetrics Based Software Supplier Selection Pagina 2
  • 3. Agenda• Introduction and Context• Phases and Timeline• The Model• Results• Conclusions & RecommendationsMetrics Based Software Supplier Selection -3-
  • 4. Introduction and Context Revenues: €13.000m.Why supplier selection? KPN Board EBITDA: € 5.100m Employees: 31.000• Consolidation # suppliers• Cost reduction Consumer Business Corporate KPN• Supplier acts as SI & MSP Market Market Market NetCo E-Plus Belgium• 5 Year investment• SPM is a best practice at KPN Customer Fixed Mobile Wholesale IT Operations Experience Generic & OSS  Domains  BSS TraditionalProblem: no more competition between suppliers instrument needed to avoid excessive cost  Unit of Work pricing Metrics Based Software Supplier Selection -4-
  • 5. Agenda• Introduction and Context• Phases and Timeline• The Model• Results• Conclusions & RecommendationsMetrics Based Software Supplier Selection -5-
  • 6. Phases and TimelineWhy Productivity metric added?• Objective selection criteria• Supplier willingness to show their transparency• Basis for productivity baseline• Insight in quality level• Negotiations for year on year cost reduction• Relation to continous improvement stepsMetrics Based Software Supplier Selection -6-
  • 7. Requested project information• Data of 6 historical projects, max 3 KPN projects• In scope of current technology domain• Range 300 – 1000 FP• Sizing method NESMA 2.1 or IFPUG 4.x• DCF must be completely filled out• No other template is allowedIn BAFO phase suppliers should show evidence of the size and productivityfigures by releasing FPA-reports, Data Collection Forms and/or insight in theiradministrative systems.Metrics Based Software Supplier Selection -7-
  • 8. Historical Project Data form (1)Metrics Based Software Supplier Selection -8-
  • 9. Historical Project Data form (2) Per data field requirements are mentioned in the template.Metrics Based Software Supplier Selection -9-
  • 10. Agenda• Introduction and Context• Phases and Timeline• The Model• Results• Conclusions & RecommendationsMetrics Based Software Supplier Selection - 10 -
  • 11. The ModelCharacteristics:• Degree of openess and compliancy• Completeness and cohesion of submitted data• Productivity benchmark against each other and industry• Delivered Quality• During the RFP phase the data will be considered as correct, but will be checked on realityThe 3 test criteria:A. Compliancy value (10%)B. Reality value (30%)C. Productivity - Quality value (60%)Metrics Based Software Supplier Selection - 11 -
  • 12. Used metrics and benchmarks Project Delivery Rate (PDR) = spent project effort related to function point (h/FP) Productivity Index (PI) = metric from QSM, derived from size, duration and effort Quality: delivered defects per FPBenchmarks:• PI against the QSM Business trend line• PDR against ISBSG Repository• Adjusted = normalised to Construction+Test activitiesMetrics Based Software Supplier Selection - 12 -
  • 13. Compliancy value (10%)Suppliers start with 10 pointsThe compliancy value is substracted with 2 points for each violation of rule:a)Range 300 – 1000 Function Pointsb)Method NESMA 2.1 or IFPUG 4.xc) Each field of “Historical Project Data”-form must be filled outMaximum value = 10, minimum value = 0Metrics Based Software Supplier Selection - 13 -
  • 14. Reality value (30%) PI vs. Functional size (FP) 35Unrealistic projects are discarded Unrealisticfrom further analysis: 30• PI > +2 sigma (95%) 25• PDR < P25 ISBSG (best in class 20 projects) PI 15The reality value is substracted with 10 Supplier A Supplier B2 points for each unrealistic project 5 Supplier C Supplier D Supplier E QSM Business Maximum value = 10 0 Avg. Line Style 2 Sigma Line Style 50 150 250 350 450 550 650 750 850 950 Minimum value = 0 Effective FP Functional Size (FP)Metrics Based Software Supplier Selection - 14 -
  • 15. Productivity - Quality value (60%) Productivity - Quality value = (Points PI score * 0,5) + (Points PDR score * 0,3) + (Points Quality score * 0,2) ID PDR (h/FP) PDR ISBSG median PDR score ID Defects/FP Quality score 7 5,9 8,6 -2,7 15 41,7 8 6,0 8,6 -2,6 18 13,9 9 6,9 8,6 -1,7 21 66,7 11 6,2 8,6 -2,4 22 4,0 12 7,3 8,6 -1,3 23 10,0 Average: -2,1 Median 13,9 The lowest average most points The lowest median most points For PI and PDR the average of the distance to the benchmark value is determined For the quality the median is derterminedThe highest average most points Metrics Based Software Supplier Selection - 15 -
  • 16. Agenda• Introduction and Context• Phases and Timeline• The Model• Results• Conclusions & RecommendationsMetrics Based Software Supplier Selection - 16 -
  • 17. Results of Compliancy (1)Projects discarded:• Projects on going (4)• Project sized in COSMIC (1)  Result: 1 supplier has 3 violations, the others 5 or moreBlank crucial fields• Defect data Supplier Compliancy Value• Effort data Supplier A 0• Dates Supplier B 0 Supplier C 4Other violations Supplier D 0• Primary Language (example English) Supplier E 0Metrics Based Software Supplier Selection - 17 -
  • 18. Results of Compliancy (2)Metrics Based Software Supplier Selection - 18 -
  • 19. Results of RealityProjects unrealistic:• 3 according to PI• 1 according to PDR Unrealistic Unrealistic Discarded for further analysis projects PI projects PDR Supplier criterion criterion Reality Value Supplier A 1 1 6 Supplier B 1 0 8 Supplier C 0 0 10 Supplier D 0 0 10 Supplier E 1 0 8Metrics Based Software Supplier Selection - 19 -
  • 20. Results of Productivity / Quality Rank Points PDR Rank Points Supplier PI score PI score PI score Supplier score PDR score PDR scoreSupplier A 3,9 2 8 Supplier A -3,2 1 10Supplier B 5,0 1 10 + Supplier B -2,1 2 8 +Supplier C 3,4 3 6 Supplier C 16,6 4 4Supplier D 3,0 5 2 Supplier D 6,2 3 6 Supplier E 3,2 4 4 Supplier E 18,3 5 2 Quality Rank Points Points Points Points Productivity/Supplier Score Quality score Quality score Supplier PI score PDR score Quality score Quality valueSupplier A 3,1 1 10 Supplier A 8 10 10 9,0Supplier B 13,9 2 8 Supplier B 10 8 8 9,0Supplier C 52,6 3 6 = Supplier C 6 4 6 5,4Supplier D 1000,0 5 2 Supplier D 2 6 2 3,2Supplier E 94,6 4 4 Supplier E 4 2 4 3,4 weight 50% 30% 20% Metrics Based Software Supplier Selection - 20 -
  • 21. Results of Total AssessmentRecommendation from Metrics Desk: Supplier B and A score best in the model Compliancy Reality Productivity/ Total Supplier value value Quality value Points Rank Supplier A 0 6 9,0 7,2 2 Supplier B 0 8 9,0 7,8 1 Supplier C 4 10 5,4 6,6 3 Supplier D 0 10 3,2 4,9 4 Supplier E 0 8 3,4 4,4 5 weight 10% 30% 60% However suppliers B and C were selected for the next BAFO phaseMetrics Based Software Supplier Selection - 21 -
  • 22. Findings BAFO phaseMetrics Desk investigated the provided project data of the selected suppliers B + C: • Size • Dates • Hours • DefectsSupplier B:• Resistance: confidentiality clause with clients• Client site visitSupplier C:• Size measurement by junior not certified measurersMetrics Based Software Supplier Selection - 22 -
  • 23. Agenda• Introduction and Context• Phases and Timeline• The Model• Results• Conclusions & RecommendationsMetrics Based Software Supplier Selection - 23 -
  • 24. Conclusions and recommendations • The productivity assessment influenced the total outcome significantly • The assessment and discussions afterwards gave insight in:Conclusions: • Transparency and CMMI level • The results are used in the negotiations phase to maximize the baseline value • Make sure the parties understand: • the purpose of the assessment • the use of the “Historical Project Data” form • that the disclosed data will be validated and should not be confidental • the consequences of violating the governance rules (e.g. penalty points)Recommendations: • Because of many violations of the compliancy rules, consider 1 penalty point per violation • Construct model beforehand, but don’t communicate the model with suppliers • Bring site visits when offered. This gives extra information next to the productivity validation Metrics Based Software Supplier Selection - 24 -
  • 25. Productivity: don’t trust it, check it Harold van Heeringen Hans Kuijpers Sizing, Estimating & Control Software Metrics Consultant Harold.van.heeringen@sogeti.nl hans.tm.kuijpers@kpn,.com @haroldveendam @_hanskuijpers @Sogeti_SECMetrics Based Software Supplier Selection - 25 -
  • 26. Back-up sheetsMetrics Based Software Supplier Selection - 26 -
  • 27. Productivity index (PI)500 FP Effort 17 h/FP Productivity Index=18 14 h/FP Productivity Index=16 10 h/FP 9 h/FP 7 h/FP Duration - 27 -

×