Jurnal an example of using key performance indicators for software development
An Example of Using Key Performance Indicators for Software DevelopmentProcess Efficiency EvaluationŽ. AntolićR&D CenterEricsson Nikola Tesla d.d.Complete Address: Krapinska 45, Zagreb, HR-10000, CroatiaPhone: +385 1 365 4584 Fax: +385 1 365 4082 E-mail: firstname.lastname@example.orgAbstract - This paper gives an overview of possible KeyPerformance Indicators (KPI) that can be used forsoftware process efficiency evaluation. The overview isbased on currently used KPIs in software developmentprojects on CPP platform. The most important KPIs areanalyzed, and their usage in the process efficiencyevaluation is discussed. The outcome of the measurement isused to initiate further process adjustments andimprovements. In addition, there is possibility to performbenchmarking between different development projects,and based on collected data easier search for best practicesin the projects that can be broadly implemented. Someproposals and future directions in the area of processmeasurement are given.I. INTRODUCTIONAll successful software organizations implementmeasurement as part of their day-to-day managementand technical activities. Measurement provides theobjective information they need to make informeddecisions that positively impact their business andengineering performance. In successful softwareorganizations, measurement-derived information istreated as an important resource and is made available todecision makers throughout all levels of management.The way measurement is actually implemented andused in a software organization determines how muchvalue is realized in terms of business and engineeringperformance. Measurement is most effective whenimplemented in support of an organization’s businessand technical objectives and when integrated with theexisting technical and management activities that definea software project. Measurement works best when itprovides objective information related to the risks andproblems that may impact a project’s defined objectives.In other words, measurement works best when it isconsidered a significant, integral part of projectmanagement .Top-performing organizations design their technicaland management processes to make use of objectivemeasurement data. Measurement data and associatedanalysis results support both short and long-termdecision making. A mature software developmentorganization typically uses measurement to help planand evaluate a proposed software project, to objectivelytrack actual performance against planned objectives, toguide software process improvement decisions andinvestments, and to help assess overall business andtechnical performance against market-drivenrequirements. A top-performing organization usesmeasurement across the entire life cycle of a softwareproject, from inception to retirement. Measurement isimplemented as a proactive discipline, and measurementderived information is considered to be a strategicresource.Measurement is most important at the project level.Software measurement helps the project manager do abetter job. It helps to define and implement morerealistic plans, to properly allocate scarce resources toput those plans into place, and to accurately monitorprogress and performance against those plans. Softwaremeasurement provides the information required to makekey project decisions and to take appropriate action.Measurement helps to relate and integrate theinformation derived from other project and technicalmanagement disciplines. In effect, it allows the softwareproject manager to make decisions using objectiveinformation.In this article, the overview of process measurementin software development projects on CPP platform willbe given, and some Key Performance Indicators (KPIs)will be discussed. Also, one example of projectbenchmarking will be presented. At the end, someimprovement proposals, and directions for further workwill be given.II. ISO/IEC 15939 Software Measurement ProcessThe International Standard ISO/IEC 15939 identifiesthe activities and tasks that are necessary to successfullyidentify, define, select, apply, and improve softwaremeasurement within an overall project or organizationalmeasurement structure. It also provides definitions formeasurement terms commonly used within the softwareindustry . The software measurement process itself isshown on Fig. 1.Fig. 1. ISO/IEC 15939 Software Measurement ProcessTechnical andManagementProcessesPlanMeasurementPerformMeasurementEvaluateMeasurementEstablishand SustainCommitmentMeasurementPlanNewIssuesAnalysisResultsUserFeedbackAnalysis Resultsand PerformanceMeasuresObjectives andIssuesImprovementActionsScope ofMeasurementProcessMeasurementRequirements
III. CMMI Process Area Measurement and AnalysisAccording to Capability Maturity Model Integration(CMMI), the purpose of Measurement and Analysis is todevelop and sustain a measurement capability that isused to support management information needs .The Measurement and Analysis process area involvesthe following:• Specifying the objectives of measurement andanalysis such that they are aligned withidentified information needs and objectives;• Specifying the measures, data collection andstorage mechanisms, analysis techniques, andreporting and feedback mechanisms;• Implementing the collection, storage, analysis,and reporting of the data;• Providing objective results that can be used inmaking informed decisions, and takingappropriate corrective actions.The integration of measurement and analysisactivities into the processes of the project supports thefollowing:• Objective planning and estimating;• Tracking actual performance againstestablished plans and objectives;• Identifying and resolving process-relatedissues;• Providing a basis for incorporatingmeasurement into additional processes in thefuture.The initial focus for measurement activities is at theproject level. However, a measurement capability mayprove useful for addressing organization wideinformation needs. Projects may choose to storeproject-specific data and results in a project-specificrepository. When data are shared more widely acrossprojects, the data may reside in the organization’smeasurement repository.The Measurement and Analysis contexts according toCMMI model is shown on Fig. 2.Fig. 2. Measurement and Analysis ContextIV. Data Collection ModelAll projects have specific objectives that are typicallydefined in terms of system capability, resource budgets,milestones, quality, and business or system performancetargets. Project success depends largely on how wellthese objectives are achieved. Project issues are areas ofconcern that may impact the achievement of a projectobjective: risks, problems, and lack of information, forexample.The most information needs in one project can begrouped into general areas, called informationcategories. We can identify seven informationcategories, which represent key areas of concern for theproject manager :• Schedule and Progress;• Resources and Cost;• Product Size and Stability;• Product Quality;• Process Performance;• Technology Effectiveness;• Customer Satisfaction.The information about project performance iscollected in cycles. The typical data collection cycle isfour weeks. Based on achieved results, product supplier(software development project) performs analysis anddefines operation excellence action plan within twoweeks time frame. When actions are established, themeasurement results and operational excellence actionplans are ready for presentation on Operating SteeringGroup (OSG) for the project. Results form the allprojects and OSG meetings are input for R&D CenterSteering Group meeting, organized each quarter .The data collection process is shown on Fig. 3.Fig. 3. Overview of data collection processThe measurement of project performance givesorganization increased opportunities to improve andshare good practices, and to increase the possibility toreach wanted operational efficiency. The measurementactivities responsibility is on the corporate R&D level.This responsibility covers forum, processes, tools,definitions of metrics, collections, analyzing andreporting on corporate level.Measurement IndicatorsCollectMeasurementDataCommunicateResultsStoreData &ResultsAnalyzeMeasurementDataProvide Measurement ResultsMeasurementRepositoryMeasurement ObjectivesProcedures,ToolsSpecifyMeasuresEstablishMeasurementObjectivesSpecifyAnalysisProceduresSpecifyDataCollectionand StorageProceduresAlign Measurement Analysis ActivitiesKPI Data CollectionCycleSupplier ScorecardsProducedKPI Data CollectionCycleSupplier ScorecardsProducedSupplier ScorecardsProducedR&DSupplierSG2 weeks4 weeksSupplier OSG’s4 weeks 2 weeksSupplierOE ActionPlanningOE ActionsEstablishedSupplier OSG’scompletedR&D SupplierSG meetingcompletedStart ofQuarterEnd ofQuarterR&DSupplierSG2 weeks2 weeks4 weeks4 weeksSupplier OSG’s4 weeks4 weeks 2 weeks2 weeksSupplierOE ActionPlanningOE ActionsEstablishedOE ActionsEstablishedSupplier OSG’scompletedSupplier OSG’scompletedR&D SupplierSG meetingcompletedR&D SupplierSG meetingcompletedStart ofQuarterEnd ofQuarterData Collection End DateData Collection End DateDefine / Refine BaselineCollect DataValidate Data Create Scorecards12 month data windowCut-Off DateCut-Off Date
V. KPI Definitions for CPP Development ProjectsIn order to follow performance of CPP softwaredevelopment projects, we have defined set of KPIs.Some of KPIs are applicable for early projectdevelopment phases, some of them for complete lifecycle, and some of them only for product maintenance.The set of KPIs, their behavior and applicability areshown on Fig. 4 .For each KPI we have defined:• Description;• Results format;• Formula;• Frequency.A. Schedule AdherenceDefinition:Measures timeliness and ‘quality’ of deliveriesrelative to baseline schedule and acceptance criteria.Based on percentage deviation between planned andactual lead times .Result format:Reported as a percentage, 100% is the highest result.Formula:[1 – ABS (ALT – PLT) / PLT] x 100PLT = Planned Start Date – Planned Finish DateALT = Actual Finish Date – Planned Start DateIf no planned start date is specified for intermediate orparallel deliverables, the earliest planned start date (e.g.TG2 or assignment start date) may be used.Planned Start/Finish Dates are replaced with reviseddates in case of Ericsson caused/mandated CRs.B. Assignment Content AdherenceDefinition:Measures supplier’s ability to deliver full assignmentscope by end of assignment. It is based on percentage ofcompleted functionality/requirements .Result format:Reported as a percentage, 100% is the highest result.Formula:(No. of Compl. Req. / No. of Commit. Req.) x 100Requirements are smallest measurable ‘packages’ offunctionality; e.g. features, documents, items inStatement of Compliance, Requirement Specification,Requirement Management Tool, or ImplementationProposal.Number of Completed Requirements counts packagesof functionality delivered during the entire assignment.Total Number of Committed Requirements counts thepackages of functionality originally planned for theassignment; may be revised based on Change Requestguidelines.Frequency:Measured and reported at the end of an assignment.KPI measurement has to be based on requirementsthat are the smallest objects of measurement and easilymeasurable. For example, content adherence for anassignment with 2 major deliveries should not be basedat the ‘delivery level’ but rather based at the corefunctionalities/requirements within each delivery.Assignments where scope is not frozen at TG2 (ProjectGO decision) need to handle scope additions through theCR handling guidelines.Pre-TG2 andRXIDelivery to CommitmentDelivery of Full ScopeOverall AssignmentContent AdherenceContentEfficiency in resolving quality issuesReduction in cost of poor qualityCost per TR (CTR)Cost of QualityService LevelsAssignment PhaseDesignFollow-up(MaintenancePRA to GA)Maintenance(Post GA)TR response to CommitmentQuality of Supplier Deliverables to EricssonDelivery of fault free deliverables to EricssonDelivery to CommitmentAccurate EstimatingAvoidance of BuffersDelivery to CommitmentAccurate EstimatingAvoid Delaying Scope Until Later AssignmentsDesired Behaviour Development(TG2-PRA)Fault Slip Through(FST)TR Closure RateQualityCost AdherenceCostScheduleAdherenceTimeKPIsPerformanceMetric Pre-TG2 andRXIDelivery to CommitmentDelivery of Full ScopeOverall AssignmentContent AdherenceContentEfficiency in resolving quality issuesReduction in cost of poor qualityCost per TR (CTR)Cost of QualityService LevelsAssignment PhaseDesignFollow-up(MaintenancePRA to GA)Maintenance(Post GA)TR response to CommitmentQuality of Supplier Deliverables to EricssonDelivery of fault free deliverables to EricssonDelivery to CommitmentAccurate EstimatingAvoidance of BuffersDelivery to CommitmentAccurate EstimatingAvoid Delaying Scope Until Later AssignmentsDesired Behaviour Development(TG2-PRA)Fault Slip Through(FST)TR Closure RateQualityCost AdherenceCostScheduleAdherenceTimeKPIsPerformanceMetricKey: KPI Applicable to Assignment KPI N/A to Assignment May be applicable (decided case by case)Key: KPI Applicable to Assignment KPI N/A to Assignment May be applicable (decided case by case)Fig. 4. KPI Definitions for CPP projects
C. Cost AdherenceDefinition:Measures supplier’s ability to deliver assignmentscope within the agreed/committed cost, including man-hour, lab and travel costs. Based on deviation betweencommitted (baseline) and expected (actual + forecast)costs at assignment/deliverable level .Result format:Reported as a percentage, 100% is the highest result.Formula:[1 – (ECost – CCost) / CCost] x 100%Committed cost is the baseline at assignment start.Contingency value (buffer) should be specifiedseparately, if known.Expected Cost to Complete is (actual + forecast) eachmonth:• Actual costs incurred so far;• Forecast of all remaining Costs to Complete;• Forecast of contingency sums (optional).Delivering an Assignment under the Committed Costswill have neutral impact on the KPI. Aim is todiscourage unnecessarily using budgeted hours;Frequency:Measured monthly at assignment level, or at end ofeach major deliverable.Costs have to be defined at assignment level(mandatory), and optionally (if possible) at deliverablelevel, to enable precise change control.D. Fault Slip ThroughDefinition:Measures supplier’s ability to capture faults beforemaking deliveries to I&V>• Assuming that supplier conducts FunctionTesting (FT);• Supplier or external organization mayconduct I&V (Integration and Verification)Testing.Based on Trouble Report (TR) slippage between FTand I&V test phases.• Assuming that TRs are analyzed to identify‘true’ slipped TRs;• If TRs are not analyzed, then 0% may not bethe expected best result due to the differentscope in FT and I&V testing .Result format:Reported as a percentage, 0% is the lowest result.Formula:[1 – FT Faults / All Faults] x 100%Faults are classified as FT or I&V based on testingphase, not who does the testing. All parties conductingthe testing need to capture the Function Test and I&VFaults, based on assignment TR Handlingguidelines/tools.Frequency:Monthly from start to end of I&V (cumulative datacollecting); or at each ‘drop’ on completion of therespective I&V.TRs that do not relate to ‘genuine’ faults, i.e.cancelled, postponed, duplicated, and rejected TRs, areto be excluded. All ‘minor’ faults, faults that do notaffect the main operation of the system are to beexcluded.E. Trouble Report Closure RateDefinition:Measures supplier’s ability to answer TRs within thespecified goals. It is based on deviation between theactual TR answering times and TR goals, set by theAssignment Owner .Result format:Reported as lost days, averaged across TR priority.The lowest result is 0, indicating that the TRs areanswered within the goals.Formula:NLD / (OTR+ NTR)NLD = number of lost days within the time incrementfor all open and new TRsOTR = number of open TRs at beginning of the timeincrementNTR = number of new TRs during time incrementThe TR handling time starts at the point at which theTR enters the supplier organization, and ends at thepoint at which the TR is answered.Time increment is typically 12 months in the pastfrom reporting date.Frequency:Measurement is done on a monthly basis.F. Cost per Trouble ReportDefinition:Measures supplier’s efficiency in fixing TRs (answerplus solution), i.e. maintenance costs relative to TRsresolved, in man-hours .Result format:Reported as man-hours.Formula:Cost of Maintenance / Number of TRs ResolvedCost of Maintenance activities is total hours spent onTR Handling activities.Number of TRs resolved are TRs that include afix/solution.The result is expressed as a rolling average, over past12 months from the current reporting date, across allproduct areas in maintenance.Frequency:Measurement is done on a monthly basis.
VI. Project BenchmarkingBenchmark office measures and analyzes thedevelopment unit performance in order to improve theirOperational Efficiency, for example by good practicesharing across organization. The measurements arefocused towards the project perspectives of thedevelopment unit performance, and possibilities to makeexternal or internal benchmarking possible. TheBenchmark office supports the development unitsteering in analyzing their operational and processefficiency and improvements. The Benchmark Office isresponsible for the process, definitions and tools, as wellas performing analysis on corporate level.The one example of CPP project benchmarking isshown on Fig. 5. It can be seen from the figure that mostof the KPIs are on the commitment or stretched level.That means project has fulfilled its goals.The project marked with D1 is the oldest measuredproject, and it has the lowest achieved results. Thesuccessor projects have performed much better. That isachieved by performing the root cause analysis of theKPIs. The analysis has resulted with corrective andpreventive actions in the next projects, and positiveresult is visible.The project marked with D2 had problems withbudget (visible from Cost Adherence KPI). Detailedanalysis shown that initial estimations were toooptimistic, and 3rdparty supplier part of the project hasspent much more than it was planned.The benchmarking itself has no intention to initiateonly the competition between projects and organization.The full benefit can be achieved if results are deeplyanalyzed, and preventive and corrective actions are setfor the ongoing and future projects (learning fromexperience).VII. Improvements and Future DirectionsThe set of KPIs described in this article is the basicset, established 18 months ago. We are today in theposition where we have enough measurement results toperform precise analysis. But, it is obvious that this isnot complete list of KPIs that can be measured in thesoftware development project. Many other interestingdata can be collected.Two product life cycle phases are the most importantfor new, more advanced KPIs and measurements in thefuture; verification phase and maintenance phase.In the verification phase of the project we have tomeasure how efficient our verification activities are. It isnot enough to measure number of executed test cases,and pass rate. These measurements are not telling usmuch about expected product quality. The idea is toestablish fault rate measurement. The fault rate measureshow many faults we discover in certain time interval(typically one week). If fault rate decrease with time,that means product quality is improving by performingtest activities. Additionally, we can set lower fault ratelimit, in order to plan how long we will go with ourtesting, and when we can stop with testing assuming thatproduct has reached expected quality level.In the maintenance phase of the life cycle we wouldlike to measure product maintenance cost compared withdevelopment effort. At the moment we know theaverage cost to remove the fault. According to thismeasurement it is difficult to compare quality level fortwo different products. The new KPI can measure totalproduct maintenance cost in the first year of operation,and compare it with total development cost. The resultcan be expressed as percentage of product developmentcost. With this measurement we will be able to comparequality level for different products in the maintenancephase of the life cycle.KPIsCost per TR (man-hours)Schedule Adherence (%)24,4%Supplier AverageFault Slip Through (%)TR Closure Rate(lost days)100,0%49,4Content Adherence(at Assignment end)Cost Adherence (%)98,9%98,0%3,3KPI Performance Range of AssignmentsSupplier Assignment KPI Results vs. Sponsor ExpectationsMaintenance Assignment KPIs3510 71599%98%95%90%85%99%98%95%90%85%15%30%45%60%80%99%98%95%90%85%Development Assignment KPIs405080 60100D4D3D3D2D2D1D1D2D1M3DFU1DFU2M3DFU2DFU1D2D1D3D4D4D5D6Fig. 5. CPP Project Benchmarking
VIII. ConclusionThe measurement method and KPIs described in thisarticle are coming from the CPP software developmentprojects at Ericsson. We have started this process asmeasurement program, and have implemented in alldevelopment projects from the end of year 2006. Thedata were collected and analyzed on monthly basis, andused as input for further improvement activities in thedevelopment projects.The measurement process should be an integral partof the way business is conducted. Data must be providedearly enough to allow management to take actions.Results must be communicated throughout theorganization in a timely manner. Decisions should notwait for perfect data, but should be based on accuratedata, supported by risk management and root causeanalysis.Both the measurement process and the specific KPIsshould be periodically evaluated and improved.Measurement is an iterative process; the KPIs arerefined as information needs change and theorganization implements improvement actions.In the future, we can expect more demands onsoftware product quality, reduced project lead-time, andreduced project budgets. The possible answer on thesedemands is to always have accurate data about projectand product performance, and fast improvementprograms, preventive and corrective actions based onanalysis of key performance indicators in the project.References J.McGarry, “Measurement Key Concepts andPractices,” Practical Software & System Measurements,USA, 2003. ***, “Systems and Software Engineering - MeasurementProcess”, International Organization for Standardization,Geneva, 2002. M.B.Chrissis, M.Konrad, S.Shrum, “Capability MaturityModel Integration – Guidelines for Process Integrationand Product Improvement”, Pearson Education Inc.,Boston, 2004. D.Ishigaki, C.Jones, “Practical Measurement in theRational Unified Process”, Rational Software, USA,2003. ***, “Data Collection Process 2007 – R&D ConsultancySupplier Benchmark”, Internal Ericsson documentation,Stockholm, Sweden, 2007. Z.Antolic, “CPP KPI Measurements 2008”, InternalEricsson Documentation, Zagreb, Croatia, 2004. C.Braf, “KPI Definitions and Guidelines”, InternalEricsson Documentation, Stockholm, Sweden, 2006.