April 1999, Vol. 12, No. 4Software MetricsWhat’s Practical About Software Measurement?Joyce Statz 4What Do Our Metrics Mean?Richard E. Zultner 11First, Get the Front End RightLawrence H. Putnam, Michael Mah, and Ware Myers 20The Secrets of Highly Successful Measurement ProgramsCarol A. Dekkers 29Back to the Basics:Metrics that Work for Software ProjectsDwayne Phillips 36
If we can’t tell signalsfrom noise, how will weever know if changes tothe process areimprovements —or illusions?If the number of programmer-hoursrequired to produce a release decreas-es, thats good. If we cannot tell if theprocess center is shifting, then wedont know how the process is perform-ing, we dont know if its getting betteror worse, and we dont know if ouractions had an effect or how bigan effect.Many organizations attempt to assessthis with a percentage changecalculation on their metrics. Whatdo you use?2. The dispersion (or variation) of theprocess can change. It can increase(bad) or decrease (good). Processeswith less dispersion have less variability,are more predictable, and generallyhave lower costs and faster throughput.If we cannot tell if the process variationis changing, then we dont know howthe process is performing. We dontknow if it is getting better or worse,and we dont know if our actions hadan effect or what that effect was.Many organizations have no way ofassessing the variability of their processfrom their metrics data. This meansthey are blind to half the changes thatcan occur in their processes and halfthe improvements as well. How do youknow if the processes your metricsmeasure are increasing, or decreasing,in their variability?We must be able to detect both of thesechanges in our process data, or ourmetrics are not adequate for monitoringor improvement.What Makes It Hard to DetectThese Changes?No matter what process metric youmeasure, your data will vary. Variation isinherent in all real-world processes and allreal-world measurement systems. So howdo we interpret our process metrics? Howdo we know what is noise the naturalvariation that is an inherent part of theprocess and what is a signal (a sign theprocess has changed)? If we cant tellsignals from noise, how will we ever knowif changes to the process are improvements or illusions?In many organizations, managers haveinformal mental limits that they use to tellif the changes in the data are big enoughto warrant investigation. Often these limitsare ±10%-15%. How do you know, foryour own metrics data, what change issignificant?Figure 2 shows a set of data presented at arecent software development conference.The presenter offered this data as evidencethat the process had changed and drewattention to the improvement from thefirst data point to the last data point (repre-sented here by the dashed line) as a resultof changes made to the process. The onlyother comment offered was that points 6and 20 represented unusually small deliv-eries, so that explained why their majorerrors were so few.Dont these conclusions seem reasonable?In many organizations, process metrics arecollected, plotted, and eyeballed to assessthe process. Unfortunately, it is very easy tobe misled by noise and either miss a signalthat the process has changed or see a falsesignal when no change has actuallyoccurred. In this case, not only is there noimprovement, the process is out of control.But the noise makes it very difficult to seethis by eyeballing the chart. How can weavoid these all-too-common mistakes?WHERE CAN WE LOOK FOR A SOLUTION?If we cannot filter out the noise in ourprocess data, we will not be able to12 April 1999 Vol. 12, No. 4 Get the Cutter Edge free: www.cutter.com/consortium/
correctly tell when the location or disper-sion of our processes is changing.Fortunately, the Father of Quality, WalterShewhart, solved this problem in the 1920s.His solution became the foundation of thequality field statistical process control(SPC). Developed by Shewhart at BellLaboratories while working in a WesternElectric manufacturing plant, the controlchart filters noise from signal uncon-trolled from controlled variation and isthe heart of SPC [7, 8].A process that exhibits only natural, orcontrolled, changes is stable, and its perfor-mance can be predicted. A process without-of-control changes, in either location ordispersion, is unstable, and its performancecannot be predicted. Metrics from an out-of-control process predict nothing. The deter-mination of whether a process is out ofcontrol is not a matter of judgment oropinion. It is not assessed by answers on aquestionnaire. It is determined by the samesimple statistical analysis that Shewhartderived over 70 years ago (from both theo-retical and empirical studies) on how to tellif a process is in statistical control. So howdo the control charts of SPC tell us what ourprocesses are doing?SOLUTION: CONTROL CHARTSThere are only two changes that can occurin our data: the location and the dispersion.We will use two control charts then, one todetect each type of change.Shewhart developed several different kindsof control charts. The two that are mostappropriate for software metrics are theIndividual control chart (also called the xGet the Cutter Edge free: www.cutter.com/consortium/ Vol. 12, No. 4 April 1999 13Figure 2: EEvviiddeennccee ooff iimmpprroovveemmeenntt?? The data varies — as all real-world data does — but doesit show improvement or just noise? How can we tell? How do you know when “changes”in your metrics are big enough to be meaningful?Metrics from anout-of-control processpredict nothing.
chart) and the moving Range control chart(also called the mR chart). These partic-ular charts make no assumptions about thedata or its distribution, so they can be usedwith any software process metric.A change in process dispersion is the moredamaging of the two possible processchanges. So first we will check if the processdispersion has changed with the movingRange (mR) chart. If any data points areabove the upper control limit (UCL), that isa signal the process is not stable and haschanged.In Figure 3, the same data from Figure 2 ispresented on an mR chart. The mR chart isa plot of the successive differences betweenthe data points the moving Range, ameasure of variability. All the data points arebelow the UCL, so the process is stable.There are no signals, no indication that theprocess has changed (or improved) its vari-ability. Since the process is stable, we canpredict that the process will continue toproduce an average variation of 0.50 majorerrors per page and will approach a variationof 1.62 major errors per page occasionally unless the process is changed.Then we check for a change in processlocation with the x chart. Shewhart devel-oped four rules for the individual chart totest if the process is in control:The Four Western Electric Zone Rulesn Rule 1: One point more than threestandard deviations away from thecenter linen Rule 2: Two out of three successivepoints more than two standard devi-ations away from the center line onthe same siden Rule 3: Four out of five successivepoints more than one standard devi-ation away from the center line onthe same sideFigure 3: MMoovviinngg RRaannggee ccoonnttrrooll cchhaarrtt.. Here there is only “common cause” or natural variation— half the data is above average, half below average. All points are below the upper controllimit. The process is stable, predictable, and in statistical control.14 April 1999 Vol. 12, No. 4 Get the Cutter Edge free: www.cutter.com/consortium/
Get the Cutter Edge free: www.cutter.com/consortium/ Vol. 12, No. 4 April 1999 15n Rule 4: Eight consecutive points onthe same side of the center lineIn Figure 4, the same data from Figure 2 ispresented on an x chart. The x chart is aplot of the actual data points, a center line,the control limit (at three standard devia-tions), and the one- and two-standard devi-ation lines. There is no signal that delivery20 was different from the others. There isno evidence the process has improved.There are three signals that the process isout of control. Points 1 and 16 are above theupper natural process limit (a Rule 1 signal).Points 6-13 (eight consecutive points)appear on one side of the center line (aRule 4 signal). These points did not occurby chance. They tell us that the process isout of control subject to special causes and that the location of the process iswandering unpredictably. The process haschanged for the worse. Immediate actionshould be taken to determine the causeand stabilize the process. Because theprocess is out of control, we cannot reliablypredict what the process will produce in thefuture. The mean of 0.83 major errors perpage does not tell us what to expect onaverage, and the upper natural process limitdoes not tell us what the bounds are forwhat the process will do in the future,because the process is out of statisticalcontrol.Note that being in control (stable) or out ofcontrol (unstable) tells us nothing aboutwhether the performance of the process isacceptable or not. The process could bestable at an unacceptably high number ofmajor errors per page or unstable aroundan extremely low number of errors. Gettingthe process stable is only the first steptoward real improvement.Figure 4: IInnddiivviidduuaall ccoonnttrrooll cchhaarrtt.. Here there are multiple signals (circled) that “special causes”are at work — and the process is out of control and unpredictable. No improvement hasoccurred. Our metrics don’t tell us anything about what the process will do in the future.
MISCONCEPTIONS ABOUTCONTROL CHARTSSince control charts have been around for70 years, why arent people using them forsoftware? Well, some people are usingthem for software [1, 2, 4, 5, 14], but manymisconceptions remain. Here I will discusssome of the most common questions raisedabout software SPC and control charts.Arent control charts for manufacturing?We dont produce the same result overand over as they do If you use adefined process for software development(a methodology), then you produce yoursoftware products with a process, andcontrol charts will tell you how that processis performing. If you have (or aspire tohave) ISO 9000 certification for your soft-ware development process, then you mustdescribe your process and follow yourdescription. This is your defined process.(If you dont think you use a process todevelop your software, then what will yoube changing to build better software?How will you improve? And what are youmeasuring with your process metrics?)Please understand that it is not necessaryto assume anything about your process touse control charts. Whether you think it isdefined or not, the control chart will tellyou, based on your data, if your process isstable (in control) or unstable (out ofcontrol). Improving the definition of yourprocess is often helpful in getting an out-of-control process into control. Control chartsdont depend on the results being thesame but on the process being the same and they will tell you whether yourprocess is, or isnt, the same. (By the way,modern lean manufacturing, whichproduces to order, doesnt produce thesame result over and over either.)Even if you use multiple processes duringthe same project, control charts will tell youif you really do have multiple processes, andthey will allow you to monitor, control, andimprove each process independently orsimultaneously. For example, if you suspectthat your experienced programmers use adifferent process than your inexperiencedprogrammers, control charts can confirmor disprove your hypothesis. If you thinkcomplex modules are designed differentlythan simple modules, control charts can tellyou if that is the case or not. If you think thatthe size of your objects makes a differencein how long it takes to design them, thenumber of defects they contain, or anyother attribute of interest, control chartscan determine the magnitude of the differ-ence and whether it exists.In more than a decade of working withsoftware organizations to improve theirprocesses, I have seen many superstitionsabout software processes disproved (thingsthey knew made a difference butdidnt) and many significant factors revealed(things that were not even suspected ofmaking a difference but did). In orderto tell what is actually happening in yourprocesses, you must filter out the noise,and only then can you really see if theprocess changed. The control chart is theoldest, simplest, and most robust tool forproperly analyzing process metrics.Myths, or Why we cant use controlcharts here. Unfortunately there are manycommon myths concerning control charts,even in manufacturing organizations thathave used them for many years. One is thatthe data must be normally distributed. Asecond is that the control chart worksbecause of the central limit theorem. A thirdis that the observations must be indepen-dent. And a fourth is that the data must beThe control chart is theoldest, simplest, and mostrobust tool for properlyanalyzing process metrics.16 April 1999 Vol. 12, No. 4 Get the Cutter Edge free: www.cutter.com/consortium/
in control before plotting (including the useof two sigma limits). Other misunderstand-ings exist as well .Some people may tell you that controlcharts require lots of data maybe morethan you have with your current metrics.In fact, control charts can work with just4-6 data points. Now if you only have a fewdata points, you can only detect big processchanges. The more data you have, thebetter you can detect small processchanges. For up to 16 data points, eachadditional point is adding increasing resolu-tion to your control chart. Beyond 16 datapoints, each data point adds less and less toyour ability to detect small process changes.And after 32 data points, you have effec-tively reached the limit of your measure-ment system to detect small processchanges .We arent ready. Were not matureenough Another myth-understanding isthat to use control charts or SPC, you mustbe at some particular level of maturity withyour software process. Nothing could befurther from the truth. Maturity is a completelyseparate issue from whether a process is incontrol (or out of control), is producingacceptable results (or not), or is capable ofbeing improved with SPC methods.I have worked with a number of improve-ment teams to improve specific softwareprocesses (those that were most unaccept-able from the standpoint of their customers)with SPC and control charts. Most were atCMM Level 1 when they began, and someeven had stable, but terrible, processes.After making substantial (and, thanks to thecontrol chart, demonstrable) improvements,their customers noticed the difference but they were still at Level 1 according tothe CMM. Control charts and SPC workregardless of your maturity.Dont wait to use SPC and control charts.If you want to improve, or just want toknow how your process is doing, docontrol charts immediately with the datayou have right now.APPLYING CONTROL CHARTSTO YOUR DATAApplying control charts to your data is easyeven if you do it manually and it is veryeasy with any spreadsheet. (The hardestpart is learning which options to set to getyour chart nicely formatted, but once youhave the format, you can reuse it.) Here isan overview of the procedure.Moving Range (mR) ChartThe first control chart you do is the mRchart to check dispersion.1. Calculate the moving Ranges: theabsolute value of the successive differ-ences between each pair of data points.So if your first five data points are 2.25,0.85, 0.85, 1.20, and 0.95, then your mov-ing Ranges will be __, 1.4, 0.0, 0.4, and0.3. (The first blank is because you willhave one fewer moving Range than youhave data points.) Plot these movingRanges on your chart.2. Calculate the mean of the movingRanges. (Add them up and divide byhow many there are.) For example, inFigure 3, this came out to 0.50. Plot this(mR bar) as the center line on yourchart.3. Multiply the mean by 3.268. Plot this lineas the upper control limit. For example,in Figure 3, 0.50 × 3.268 = 1.634. (Withall intermediate calculations carried outby the spreadsheet with full precision,the answer is 1.62 as plotted.) This lineis three standard deviations above themean. (There will not be a lower con-trol limit on this chart, ever.)Get the Cutter Edge free: www.cutter.com/consortium/ Vol. 12, No. 4 April 1999 17
Are all the moving Ranges inside the UCL?Then the process dispersion is stable.Individual (x) ChartThe second, and last, control chart you do isthe x chart to check location.1. Plot the individual data points (2.25,0.85, 0.85, 1.20, 0.95, and so on) on yourchart.2. Calculate the mean of the individualdata points. For example, in Figure 4,this came out to 0.82. Plot this (x bar)as the center line on your chart.3. Multiply the mean of the moving Range(mR bar) by 2.660 and add that quantityto the mean of the individual values (xbar). This is the upper natural processlimit (UNPL). For example, in Figure 4,(0.50 × 2.660) + 0.82 = 2.15. (With theintermediate calculations carried out bythe spreadsheet with full precision, theanswer is 2.14, as plotted.) This line isthree standard deviations above themean. Plot it on your chart.4. Multiply the mean of the moving Range(mR bar) by 2.660 and subtract thatquantity from the mean of the individualvalues (x bar). This is the lower naturalprocess limit (LNPL). For example, inFigure 4, 0.82 − (0.50 × 2.660) = −0.51.This line is three standard deviationsbelow the mean. If this line is abovezero (or your data can go belowzero), plot it on your chart. (In Figure4 this line was not plotted, as it wasbelow zero.)5. The distance from the center line tothe UNPL is three standard deviations.Divide this interval by three to get onestandard deviation. For example, inFigure 4, (2.14 − 0.82) ÷ 3 = 0.44, whichis one standard deviation. Draw a lineone standard deviation above andbelow the center line. Draw anotherline the same distance beyond thoselines. Now you should have lines drawnat equal intervals of one, two, and threestandard deviations above and belowthe center line. (If any of the lines isbelow zero, and your data cannot gobelow zero, you dont need to drawthose lines.) Apply the four WesternElectric Zone Rules stated above. Ifnone of the points is a signal accordingto the four rules, then the process loca-tion is stable.On the mR chart, we checked if the disper-sion of the data is changing. On the x chart,we checked if the location of the data ischanging. We now know objectively andempirically if our process is in statisticalcontrol and is therefore predictable.CONTROL CHARTS WORK FOR SOFTWAREPROCESS METRICSStatistical process control is much morethan just doing control charts it is acomplete system for process improvement[6, 13]. But control charts are a vitalelement, and they provide a simple, effec-tive, and robust way to filter out the noisethat is inherent in every process and everymeasurement system. They give you clearsignals using your own metrics as towhether your process is stable or out ofcontrol. If your process is stable, the controlcharts allow you to predict what yourprocess can be expected to deliver in thefuture. Can you do these things now?If you do control charts on your key processmetrics, for your critical software processes,then you will have repeatable, quantitativeevidence that your development process isin statistical control. And once you achievesuch demonstrable stability of your criticalprocesses, you will have the solid founda-tion needed to improve your performancefurther to excellence.Once you achievedemonstrable stability ofyour critical processes, youwill have the solidfoundation needed toimprove your performancefurther to excellence.18 April 1999 Vol. 12, No. 4 Get the Cutter Edge free: www.cutter.com/consortium/
Proven over 70 years, in use today in everyindustry, the control chart is the simplesttool available to give us essential insightsinto the meaning of our own metrics. Whynot try one today?REFERENCESThere is a lot more to SPC, and controlcharts, than I covered here. I highly recom-mend the books by Donald Wheeler [9-12],starting with Understanding Variation .For more information, including details andan example of an Excel spreadsheet for soft-ware control charts, e-mail a request to:SoftwareSPC@Zultner.com.1. Affourtit, Barba. Statistical ProcessControl Applied to Software. In TotalQuality Management for Software, ed. G.Gordon Schulmeyer and James I. McManus.Van Nostrand Reinhold, 1992.2. Card, David N., and Robert L. Glass.Measuring Software Design Quality.Prentice Hall, 1990.3. Deming, W. Edwards. Out of the Crisis.MIT Center for Advanced Engineering, 1986.4. Grady, Robert B. Practical SoftwareMetrics for Project Management and ProcessImprovement. Prentice Hall, 1992.5. Grady, Robert B., and Deborah L. Caswell.Software Metrics: Establishing a Company-Wide Program. Prentice Hall, 1987.6. Kume, Hitoshi. Statistical Methods forQuality Improvement. Translated by JohnLoftus. The Association for OverseasTechnical Scholarship, 1985. (Available fromASQ Quality Press.)7. Shewhart, Walter. Economic Control ofQuality of Manufactured Product. 1931. Witha foreword by W. Edwards Deming. Reprint,ASQ Quality Press, 1980.8. Shewhart, Walter. Statistical Method fromthe Viewpoint of Quality Control. 1939. Witha foreword by W. Edwards Deming. Reprint,Dover Publications, 1986.9. Wheeler, Donald J. UnderstandingVariation: The Key to Managing Chaos. SPCPress, 1993.10. Wheeler, Donald J. Advanced Topics inStatistical Process Control. SPC Press, 1995.11. Wheeler, Donald J., and Richard W.Lyday. Evaluating the Measuring Process,2nd ed. SPC Press, 1989.12. Wheeler, Donald J., and David S.Chambers. Understanding Statistical ProcessControl, 2nd ed. With a foreword by W.Edwards Deming. SPC Press, 1992.13. Zultner, Richard E. TQM for TechnicalTeams. Communications of the ACM, Vol.36, No. 10 (October 1993), pp. 79-91.14. Zultner, Richard E. Software SPC. InProceedings of the 4th InternationalConference on Software Quality. ASQSoftware Division, 1994.Get the Cutter Edge free: www.cutter.com/consortium/ Vol. 12, No. 4 April 1999 19Richard Show Me the DataZultner is an international consul-tant, educator, author, andspeaker. His primary focus ison efficient software processimprovement the rapid applica-tion to software development ofdaily management methodssuch as statistical process control(SPC), cross-functional manage-ment techniques such as qualityfunction deployment (QFD), andstrategic improvement approachessuch as theory of constraints(TOC). He is an avid and long-time student of W. EdwardsDeming.Mr. Zultner is a founder anddirector of the QFD Institute, anonprofit research organizationdedicated to the advancement ofQFD. In 1998, for his work inapplying QFD to software, hereceived the Akao Prize one of12 people in the world so honoredto date.He holds a masters degree inmanagement from the J.L. KelloggGraduate School of Managementat Northwestern University, and hehas professional certifications inquality, software quality, projectmanagement, software engi-neering, and theory of constraints.Mr. Zultner can be reached atZULTNER & COMPANY, 12Wallingford Drive, Princeton, NJ08540-6428 USA. Tel: +1 609 4520216; Fax: +1 609 452 2643;E-mail: Richard@Zultner.com;Web site: www.zultner.com.