Software Management By Numbers

Software Management by Numbers Towards an Engineering Discipline SE-CURE AG (www.se-cure.ch) Dr. Hans Sassenburg T +41 33 733 4682 M +41 79 231 6600 E hsassenburg@se-cure.ch © SE-CURE AG 1

Contents 1. Software Industry Today 2. Empirical Information 3. Key Performance Indicators 4. Software Benchmarking Organization 5. Finally ... © SE-CURE AG 2

Software Industry Today • The software industry still isn't doing a very good job at managing by the numbers ”Oftentimes, software projects are managed by just three metrics: schedule/effort and critical defects found during testing” • This is a flatland view for a multi-dimensional terrain problem: "flying a plane using only a watch and a fuel gauge“ • Other metrics must be in the equation as well, representing what will be built (product scope/quality) and how it is built (process) © SE-CURE AG 3

Problem Statement ”You can’t fix what you refuse to measure” • When quantitative information is missing: – Management is like an emperor without clothes: decisions are not based on facts – Improvement initiatives result in satisfying standards/models instead of tangibly improving measured capability © SE-CURE AG 4

What is Improvement? • It is NOT about – Satisfying a model/standard – Implementing the next hype • It is about becoming – More predictable (performance) – Faster (efficiency) – More effective (scope) – Better (quality) © SE-CURE AG 5

Questions ... • Did you ever go after the world„s next model/standard or hype? – If yes, how (much) did it help you? • Are you able today to tell what targeted, actual and historical values of your main KPIs are? – If not, would you be interested to know? © SE-CURE AG 6

Being Successful • Knowing the (performance) capability of your organization through the collection and analysis of historical data • Making credible commitments in terms of what will be delivered when against what cost • Managing development once it starts; this requires earned value management, activity distribution management and effective scope and quality management • Analyzing the impact of new initiatives by assessing how capability is affected in which areas; this prevents organizations from chasing hypes © SE-CURE AG 7

2. Empirical Information • Empirical Models – Putnam SLIM – COCOMO II • Empirical Laws – Minimal development time! • Benchmarking Data – Guru: Capers Jones • Observation – Availability hardly known, seldom used ... © SE-CURE AG 8

Empirical Models characteristics (product, platform, personnel, project) schedule Empirical Models features (Putnam SLIM, COCOMO II) size + Benchmarking Data effort capability © SE-CURE AG 9

Empirical Laws effort Legend: Ts, Es : fastest schedule Es Tn, En : nominal schedule/effort Te, Ee : lowest effort Impossible > 100% Region! En -50% Ee Impractical Region Ts Tn schedule Te -25% +100% © SE-CURE AG 10

Schedule Acceleration log (effort) size ↓ productivity ↑ Question: staffing rate ↑ What would even be a better change? Tn‘ Tn log (schedule) Optimal development time © SE-CURE AG 11

Acceleration Measures • Higher Staffing Rate – This means being able to allocate more people to the project under the condition that they can do effective work • Reduced Functionality – This means scaling down the product thereby reducing the volume of code created • Increased Reuse – Component-based development can significantly accelerate the project schedule, re-use of models and documentation as well • Increased Productivity – This means generating more output against less effort • Or something else ...? – A real paradigm shift? – But what could that possibly be ...? © SE-CURE AG 12

Rule of Thumb Schedule = 4 * Effort 1/3 (average project size, embedded, unit: months) © SE-CURE AG 13

Productivity Index Business Information Scientific System Process Control Engineering Telecommunications Command and Control Real Time Avionics Real Time Microcode 0 2 4 6 8 10 12 14 16 18 20 22 24 Productivity Index (PI) w/ ±1 Standard Deviation Source: QSM © SE-CURE AG 14

Benchmark (Jones 2008) © SE-CURE AG 15

Benchmark - Process Efficiency (Jones/Sassenburg) © SE-CURE AG 16

3. Key Performance Indicators ”Getting too little or too much data is easy, identifying and extracting the relevant data and converting it to meaningful information for everyone is the challenge” © SE-CURE AG 17

KPIs versus Metrics • KPIs (= meaningful information) are meant to gauge progress toward or benchmark against vital, strategic objectives usually defined by upper management • Metrics (= data) at a lower detail level are measured attributes of a process or a product • The goal of KPIs is to foster greater visibility and faster reaction to opportunities and threats, hereby enabling informed decision-making © SE-CURE AG 18

KPI Characteristics 1. A KPI echoes organization goals 2. A KPI is decided by management 3. A KPI provides context 4. A KPI creates meaning on all organizational levels 5. A KPI is based on legitimate data 6. A KPI is easy to understand 7. A KPI leads to action © SE-CURE AG 19

KPI Selection Criteria • Must support project management in analysing, planning and monitoring projects • Must inform management where a project stands and in what direction it is heading • Must support business units in measuring their capability improvement over time • Must support organisations in comparing/benchmarking business units © SE-CURE AG 20

Questions to Answer • Project Performance – What is the prediction of the performance of the project? • Process Efficiency – How efficient is the development process? • Product Scope – How large and stable is the scope of the planned effort in terms of features and size? • Product Quality – What is the expected quality of the resulting product (reliability and maintainability)? © SE-CURE AG 21

Derived KPI Categories • Project Performance (= how predictable?) – Cost, schedule, staffing rate, productivity • Process Efficiency (= how fast?) – Effort distribution (Cost of Quality model) • Product Scope (= how much?) – Features, size • Product quality (= how well?) – Complexity, test coverage, removal efficiency, defect density © SE-CURE AG 22

Schedule (PP) © SE-CURE AG 23

Effort (PP) © SE-CURE AG 24

Staffing Rate (PP) © SE-CURE AG 25

Productivity (PP) © SE-CURE AG 26

Cost Categories Appraisal, Core Support Prevention rework Value-Added Non-Value-Added Essential Non-Essential = Quality Related © SE-CURE AG 27

Non-Value-Added, Non-Essential Quality Related Costs Cost of Conformance Cost of Non- Conformance Prevention Appraisal Failure Costs Costs Costs Internal External © SE-CURE AG 28

Core (PE) © SE-CURE AG 29

Support (PE) © SE-CURE AG 30

Prevention (PE) © SE-CURE AG 31

Appraisal/Rework (PE) © SE-CURE AG 32

Feature Size (PS) © SE-CURE AG 33

Deferred (PS) © SE-CURE AG 34

Size (PS) © SE-CURE AG 35

Re-use (PS) © SE-CURE AG 36

What is Product Quality? Internal View External View How easy is the How well does product to the product understand, work? maintain and test? influences 2. Test coverage 1. Complexity 3. Defect density 4. Removal efficiency © SE-CURE AG 37

1. Complexity How difficult is it to understand, maintain and test the product? “Low coupling, high cohesion !” ISO9126 Understand Maintain Test Number of dependencies Number of places that Number of required tests on other systems require attention upon and required test time Cyclomatic complexity modification Cyclomatic complexity Fan-out Change propagation Code duplication Complexity © SE-CURE AG 38

Cyclomatic Complexity Number of linearly independent execution paths = # edges - # nodes + 2 or Number of decision points + 1 ........ sequence: if / then: while loop: do / until loop: ??? 1-2+2 = 1 3-3+2 = 2 3-3+2 = 2 3-3+2 = 2 © SE-CURE AG 39

Any Preference? Cyclomatic complexity? © SE-CURE AG 40

Empirical Values (SEI) Cyclomatic Risk Bad fix Complexity Evaluation probability Low risk 1-10 5% testable code 11-20 Moderate risk 10% 21-50 High risk 30% Very high risk > 50 > 40% untestable code © SE-CURE AG 41

Complexity (PQ) © SE-CURE AG 42

2. Test Coverage How much code could lead to failure? Levels of coverage: Remark Below 100% statement coverage - (e.g., function coverage, call coverage) Immature Product quality 100% statement coverage 100% decision coverage 100% basic path coverage Cyclomatic complexity 100% condition/decision coverage 100% modified condition/ decision coverage DO-178B to ensure that Level A (Catastrophic) software is tested adequately 100% multiple condition coverage Hard (impossible) to achieve 100% path coverage + void foo(bool c, int& a) { Limitations of statement coverage int *p = NULL; if (c) T1: (c = True, a = 1) → 100% statement coverage { T2: (c = False, a = 1) → ERROR ! p = &a; } *p = 1; } © SE-CURE AG 43

Testing Productivity 100% 100% High Coverage Failures productivity Low Low productivity productivity High productivity 0% 0% effort/time effort/time Coverage rate Failure discovery rate © SE-CURE AG 44

Possible Coverage Strategy • Objective: obtain high testing productivity • Condition: reject complex code • Steps – Invoke at least <> functions in <>% of the source files (or classes) – Invoke <>% of the functions – Attain <>% condition/decision coverage in each function © SE-CURE AG 45

General Message Low (cyclomatic) complexity + High test coverage = High quality code © SE-CURE AG 46

Test Coverage (PQ) © SE-CURE AG 47

3/4. Defect Model Defects removed Process Step: Incoming Outgoing - remove some old defect defects - inject new defects defects (Residual) - remove some new defects New defects (Injected) © SE-CURE AG 48

Example Low Level Unit Integration System Defects Req. Arch. Coding Design Testing Testing Testing Residual 0 10 10 10 15 10 7 Injected 20 10 10 20 5 4 3 Removed 10 10 10 15 10 7 5 Remaining 10 10 10 15 10 7 5 Removal 50.0% 50.0% 50.0% 50.0% 50.0% 50.0% 50.0% efficiency Total injected 20 30 40 60 65 69 72 Total removed 10 20 30 45 55 62 67 Cumulative 50.0% 66.7% 75.0% 75.0% 84.6% 89.9% 93.1% efficieny Defect density is the number of remaining defects per size unit © SE-CURE AG 49

Defect Density (PQ) © SE-CURE AG 50

Removal Efficiency (PQ) © SE-CURE AG 51

Measuring Product Quality • Complexity – Fact extractor systems (for fan-out, cyclomatic complexity) – Archive data mining tools (for change propagation) – Code clone detectors (for clone detection) • Test coverage – Use dedicated code coverage systems – Choose a coverage measure beyond statement coverage • Defect density and removal efficiency – Use the defect reporting/tracking system © SE-CURE AG 52

Identifying Weak Areas An aggregated number is good for monitoring, but to act you need to know where in the code measures are necessary Look at the measurement data Raw data tables can hardly give insight in the problem at hand Use insight boosting techniques Software visualization can efficiently implement the link back to the product © SE-CURE AG 53

Tree Maps phoenix newBusinessInstallation newBusiness recon onlineQuote acc reportServer employerPortal phoenix Cyclomatic complexity 0 20 Usage: - show distribution of metrics (cyclomatic complexity, fan-out) on source code © SE-CURE AG 54

Hierarchical Edge Bundles Usage: - show distribution of dependencies (fan-out) and clones - correlate dependencies with metrics (cyclomatic complexity) © SE-CURE AG 55

Typical Maturization Effects Project Performance Product Scope Schedule Feature size Budget Deferred Staffing rate Size Productivity Re-use Process Efficiency Product Quality Core Complexity Support Test coverage Prevention Defect density Appraisal/rework Removal efficiency © SE-CURE AG 58

Key Points to Remember 1. Assessing real software performance capability requires a multi-dimensional view 2. Compliance to process models and/or standards is no guarantee for high performance capability, although it may help (pragmatism versus bureacracy) 3. The set of Key Performance Indicators allows measuring real performance capability improvements over time and benchmarking © SE-CURE AG 59

5. Benchmarking • Software Benchmarking Organization – International consortium • Founding partners – SE-CURE AG (www.se-cure.ch) – SolidSource BV (www.solidsourceit.com) • URL – www.sw-benchmarking.org © SE-CURE AG 60

Announcement • Benchmarking Study – Reference: best practice KPI set • Markets – Embedded software industry – Finance (banks, insurance) – Automotive • Objectives – Assess performance of study participants – Benchmark results with industry values (average, best-in-class) © SE-CURE AG 61

Reference Dr. Ir. Ton Tinus, Senior Development Director (Philips Healthcare, BU CV): “The benchmarking study executed by SE-CURE AG in April 2009 was very useful and has brought many new insights. In addition, benchmarking the results with industry values was a useful exercise. The introduced best practice KPI set is considered very powerful and we have adopted the entire set to monitor projects and improvements." © SE-CURE AG 62

Benchmarking Approach • For each participant – Make available project data for 1-3 projects – SE-CURE AG and partners will analyse the data and benchmark the results • Includes on-line visit with report as outcome • All – Overall (anonymized) results will be presented in Q3.2010 – Optionally, results will be published at conferences and/or in refereed journals • Interested? – Contact me at sassenburg@sw-benchmarking.org – See also: www.sw-benchmarking.org © SE-CURE AG 63

5. Finally ... • Once management starts actively using such KPIs, projects are forced to bring and keep not only their measurement process in place, but many other processes as well … – Requirements Management – Project Planning – Project Monitoring and Control –… © SE-CURE AG 64

Another Capability Model ... Level 1 No or Ad Hoc Measurements Level 2 Basic Measurements at Project Level Level 3 Standardized Measurements Across Projects Level 4 Quantitative Portfolio Management Level 5 Quantitative Performance Optimization © SE-CURE AG 65

Project Reporting Cockpit The Project Reporting Cockpit will enable you to define and monitor a set of Key Performance Indicators (KPIs) for your project. Historical, target and actual values are displayed in overviews and charts, from which information can be consumed at a glance. © SE-CURE AG 67

Project Support Toolkit The Project Support Toolkit provides powerful instruments to project managers and measurements specialists to plan, monitor and control their projects on a more detailed level, using the same set of KPIs as in the Project Reporting Cockpit. © SE-CURE AG 68

CMMI® Appraisal Recorder The CMMI® Appraisal Recorder supports both SCAMPIsm and internal appraisal teams in recording their CMMI® (version 1.2) appraisal findings and (configurable) scores for all process areas considered. Results are summarized in tables and radar/bar charts. © SE-CURE AG 69

Business Case Analyser The Business Case Analyser will enable you to define a business case, comparing two alternatives. By specifying benefits and costs over a span of years (1-5) for both alternatives, four benchmarks (NPV, IRR, Payback Period, ROI) are calculated automatically and graphically displayed. SE-CURE AG © 70

Workshop Overview Software Software Software Estimation Management Planning & Monitoring See www.se-cure.ch/Courses.html © SE-CURE AG 71

Software Management By Numbers

0 comments

Notes on slide 1

2 Favorites