Download as KEY, PPTX
Uploaded byErin Dees
Your Own Metric System
The document discusses the importance of creating personalized metrics in coding to enhance productivity and engagement while preventing poor information and structural changes that may introduce bugs. It emphasizes that metrics should serve developers by helping answer specific questions about code quality and maintainability rather than merely serving external reporting needs. Additionally, it highlights various common and custom metrics that can help identify issues such as code complexity, test coverage, and potential risks within the codebase.
More Related Content
![Inside PHP [OSCON 2012]](https://cdn.slidesharecdn.com/ss_thumbnails/insidephppresentation-120724002136-phpapp02-thumbnail.jpg?width=640&height=640&fit=bounds)
![Inside Python [OSCON 2012]](https://cdn.slidesharecdn.com/ss_thumbnails/insidepythonpresentation-120724001527-phpapp02-thumbnail.jpg?width=640&height=640&fit=bounds)
![A(n abridged) tour of the Rust compiler [PDX-Rust March 2014]](https://cdn.slidesharecdn.com/ss_thumbnails/rust-march-2014-140329160039-phpapp02-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Your Own Metric System
![[Paul Holland] Bad Metrics and What You Can Do About It](https://cdn.slidesharecdn.com/ss_thumbnails/paulhollandbadmetricsandwhatyou-140717230402-phpapp02-thumbnail.jpg?width=640&height=640&fit=bounds)
Your Own Metric System
- 1. Your Own MetricSystem Ian Dees ·@undees OSCON 2012
- 3.
- 5.
- 6. The forces againstus ❥ Entropy drags our code down ❥ Apathy drags us down
- 7. Stay engaged andproductive
- 8. Knowing our code canhelp us do our jobs and have more fun
- 9. Risk #1 Missing orpoor information can waste our time or lead us to cause harm
- 10. Two steps forward 1.Ask questions about your code 2. Choose metrics that answer those questions
- 11.
- 12. purpose of metrics Helpyou answer a question
- 13. purpose of metrics Whatmess should I clean up next?
- 14. purpose of metrics Theproduct backlog isn’t a substitute for your brain
- 15. Risk #2 Making structuralchanges can introduce new bugs (or expose existing ones)
- 16. purpose of metrics Whereare the bugs (likely to be)?
- 17. purpose of metrics /** * REMOVE THIS CODE * BEFORE WE SHIP! */
- 18. purpose of metrics Havewe forgotten anything for this release?
- 19. purpose of metrics Thesequestions are for us
- 20. purpose of metrics Notfor someone else
- 21. purpose of metrics Questionsfrom others: (outside the scope of our metrics)
- 22. purpose of metrics Shouldwe hold the release?
- 23. purpose of metrics errors/KLOC→ time →
- 24. purpose of metrics Who’s got the best KLOC or error rate?
- 25. purpose of metrics Itwas time to fill out the management form for the first time. When he got to the lines of code part, he thought about it for a second, and then wrote in the number: -2000. After a couple more weeks, they stopped asking Bill to fill out the form, and he gladly complied. —folklore.org
- 26. purpose of metrics Havewe met our target complexity or coverage?
- 28. purpose of metrics Metrics serve you, not the other way around
- 29. purpose of metrics Keepthe job fun
- 30. purpose of metrics More fun than actually working?
- 31. Risk #3 There isa trap here for the distractible
- 32.
- 33. common metrics Languages ❥ C: a case study ❥ Perl: the beginner’s experience ❥ <your lang> just ask!
- 34. common metrics Repo forthis talk github.com/undees/oscon
- 35.
- 36. common metrics E –N + 2P
- 37. common metrics 1. Startwith a score of 1 2. Add 1 for each if, case, for, or boolean condition
- 38. Volume speaking_volume( bool correct_room, bool correct_time) { if (correct_room && correct_time) { return INTELLIGIBLE; } else { // rehearsing return INAUDIBLE; } } complexity: 1
- 39. Volume speaking_volume( bool correct_room, bool correct_time) { if (correct_room && correct_time) { return INTELLIGIBLE; } else { // rehearsing return INAUDIBLE; } } complexity: 2
- 40. Volume speaking_volume( bool correct_room, bool correct_time) { if (correct_room && correct_time) { return INTELLIGIBLE; } else { // rehearsing return INAUDIBLE; } } complexity: 3
- 41. common metrics pmccabe parisc-linux.org/~bame/pmccabe
- 42. $ pmccabe *.c| sort -nr | head -10 3 3 3 6 8!oscon.c(6): speaking_volume 1 1 2 16 5!oscon.c(16): main
- 43.
- 44. sub speaking_volume { my $correct_room = shift; my $correct_time = shift; if ($correct_room && $correct_time) { return 'intelligible'; } else { # rehearsing return 'inaudible'; } }
- 45. $ countperl lib ... Tab-delimitedlist of subroutines, with most complex at top ----------------------------------------------------------- complexity sub path size 4 speaking_volume lib/OSCON.pm 9 ...
- 47. $ wc -loscon.c
- 48. Risk #4 Blindly reducingone number can add complexity and bugs
- 49.
- 50. common metrics 1. Instrumentyour program 2. Watch your tests run 3. Report which lines get executed
- 51. common metrics Addresses “epic confidence” fail opensourcebridge.org/sessions/923
- 52. common metrics Testable codeis more... testable
- 53.
- 54. int main() { assert(speaking_volume(true, true) == INTELLIGIBLE); return 0; }
- 55. $ gcc -fprofile-arcs -ftest-coverage -c oscon.c $ gcc -fprofile-arcs oscon.o
- 56. $ gcov oscon.c $cat oscon.c.gcov
- 57. 1: 6:Volume speaking_volume(bool correct_room, bool correct_time) { 1: 7: if (correct_room && correct_time) { 1: 8: return INTELLIGIBLE; -: 9: } else { -: 10: // rehearsing ####: 11: return INAUDIBLE; -: 12: } -: 13:}
- 58.
- 59. $ cover -test $cat cover_db/coverage.html
- 61. Risk #5 High codecoverage can make you think your code is good
- 62.
- 63.
- 64. custom metrics Carlin’s 7Dirty Words 1. 5. 2. 6. 3. 7. 4.
- 65. custom metrics Our 7 Dirty Words 1.XXX 5.HACK 2.TODO 6.#if 0 3.FIXME 7.#ifndef TESTING 4.TBD
- 66. custom metrics Our 7 Dirty Words 1. 5. 2. 6. 3. 7. 4.
- 67. $ ack -cl'XXX|TODO|FIXME' oscon.c:1
- 68.
- 69. use Test::Fixme; run_tests(where =>'lib', match => qr/XXX|TODO|FIXME/);
- 70. $ make test ... t/test-fixme.t.. 1/1 # Failed test ''lib/OSCON.pm'' # at t/test-fixme.t line 2. # File: 'lib/OSCON.pm' # 34 # XXX:remove the temp limit before we deploy # Looks like you failed 1 test of 1. t/test-fixme.t .. Dubious, test returned 1 (wstat 256, 0x100) Failed 1/1 subtests
- 71.
- 72. custom metrics Recently changedcode may have new bugs
- 73. custom metrics Frequently-changed code may have problems
- 74. git log --pretty=oneline --since=2012-05-04 oscon.c | wc -l
- 75.
- 76. custom metrics Errors bytime of day
- 77. custom metrics Play byPlay: Zed Shaw peepcode.com/products/play-by-play-zed-shaw
- 78. What do weget from all this?
- 79. Found a real dependencyproblem with pmccabe
- 80. Found dead codewith gcov
- 81. Did a quickchurn check at manual test time
- 82. Found places wecan DRY up the code
- 83.
- 84.
- 85. ❥ Find theanswers you need ❥ Look like heroes ❥ Have fun
- 86.
- 87.
Editor's Notes
- #2 Hello, and welcome. I&#x2019;m Ian.\n
- #3 By day, I make oscilloscopes. By night, I play guitar irresponsibly.\n
- #4 I also write books, mostly about Ruby topics. A group of us&#x2014;me, plus two major contributors to the Cucumber test framework&#x2014;are working on a new book of specific testing techniques. Ruby and its various test frameworks were my gateway drug to code metrics, though for this talk we&#x2019;ll be concentrating on other languages.\n
- #5 Oscilloscopes have been available commercially since the 1940s. Their architecture changes slowly. Software needs to last, and it tends to last whether we wish for a rewrite or not. Our team&#x2019;s exploration of this mix of old and new code led to our interest in code metrics.\n
- #6 And even if you&#x2019;re not working on a large legacy code base, there are likely issues that we face in common.\n
- #7 There are a lot of forces that push on us and our teams. Today, I want to talk about two very different forces that have surprisingly similar effects: the entropy that drags our code down over time, and the apathy that drags us down personally over time.\n
- #8 How do we fight these forces? How do we keep our interest after our tenth straight hour wading into the weeds of an incomprehensible legacy routine? How do we prevent the code we write today from being someone&#x2019;s nightmare tomorrow?\n
- #9 We have many tools in our chest; one is a good set of metrics&#x2014;information about our code base. My hope is that you&#x2019;ll consider code metrics at least as an intriguing, low-cost possibility for making the day go by a little better.\n
- #10 The risk with doing this&#x2014;and there&#x2019;s always a risk&#x2014;is that we might waste our time making changes we don&#x2019;t need, or worse, end up trashing our code in the name of blindly satisfying some target number.\n
- #11 How do we address that risk? By letting our project needs dictate our metric choices, not the other way around. It sounds simple. But as we&#x2019;ll see, it&#x2019;s possible to misapply a metric and make a big mess.\n
- #12 Since getting the reasons right is so important, let&#x2019;s talk about why we&#x2019;re gathering this data.\n
- #13 The purpose of any metric should be to help you answer a question. Since we&#x2019;re developers who maybe also do a little testing, let&#x2019;s ask a few example questions now.\n
- #14 For example, if several files need some love, where should I concentrate my efforts?\n
- #15 Something else may be giving us guidance on what part of the code to work in&#x2014;like the product backlog. But you may be in a situation where you&#x2019;ve got a little more leeway, like an explicit charter to pay down technical debt.\n
- #16 That said, when you do wander off the map, you do risk creating a bug. With legacy code, you may also uncover an existing bug and get the blame nonetheless. One way to address this risk is to improve your test coverage, and make small changes at a time. Another is to choose the right metrics; fixing static analysis warnings has anecdotally been one of the lowest-risk change activities I&#x2019;ve ever seen.\n
- #17 Here&#x2019;s another question we might ask. Where are the bugs? Where are the old bugs we haven&#x2019;t found yet? Where are the new ones we might have created recently?\n
- #18 We can also turn to our code for ideas of what questions to ask. Has anyone seen something like this comment in production code? The number of these red flags in your code is a kind of code metric you can measure and reduce.\n
- #19 That quantitative measurement&#x2014;number of bad comments in the code&#x2014;is helping us make a qualitative determination.\n
- #20 The questions we&#x2019;ve heard so far are things we might ask,...\n
- #21 ...not things someone else might ask.\n
- #22 Not that other people&#x2019;s questions aren&#x2019;t legitimately interesting, or that they might not apply metrics of their own.\n
- #23 For example, the SQA team might be looking for red flags that could hold up the release.\n
- #24 So they might look at aggregate errors per thousand lines of code. Not something I necessarily use to make decisions as a developer, but it doesn&#x2019;t scare me if this metric is in use somewhere.\n
- #25 On a more sinister note, tracking rates of code production or error creation/resolution are outright destructive of teams.\n
- #26 There was apparently a brief, dark time at Apple when employees were tracked by lines of code produced, until Bill Atkinson showed that you can improve and shorten the code at the same time.\n
- #27 Another, more subtle trap is setting absolute thresholds for various metrics.\n
- #28 Doing so is like blindly obeying a GPS device: sooner or later, you&#x2019;ll drive off a cliff.\n
- #29 Metrics are supposed to be here for our benefit.\n
- #30 And indeed, in addition to answering specific questions about our projects, they can make coding seem a little bit like a game where the side effect is to produce better code...\n
- #31 ...as long as we still get around to writing the code eventually.\n
- #32 We have to be careful not to spend all day writing fancier shell scripts and slapping our stats onto elaborate dashboards (though there are quick-and-cheap dashboards I like; see the Tranquil project).\n
- #33 Now that we have a few questions in mind about our code base, let&#x2019;s look at some metrics commonly used by many projects. (Later, we&#x2019;ll look at writing our own.) The nice thing about prefab metrics is that we can find open source implementations and supporting research.\n
- #34 Rather than present you with a laundry list, I&#x2019;m going to stick to a few targeted examples in C and Perl. But similar tools likely exist for your language; catch me in the hall afterwards if you&#x2019;d like to explore that together.\n
- #35 The code samples you&#x2019;re about to see are on GitHub; feel free to send a pull request if you&#x2019;d like your favorite language to be included.\n
- #36 The granddaddy of modern code metrics is McCabe Cyclomatic Complexity. It&#x2019;s meant to be a loose measure of how many different paths there are through a piece of code.\n
- #37 The fancy explanation is that you draw a graph of control flow through your function, then calculate a score from the number of edges, nodes, and return points.\n
- #38 The simpler explanation is that we walk through the code and add a point for each decision the code has to make.\n
- #39 So we&#x2019;d start with a value of 1 for this code sample...\n
- #40 ...add 1 point for the if statement...\n
- #41 ...and add 1 final point for the boolean operator. Depending on the implementation, we might add a point for the multiple returns.\n
- #42 One easy-to-use implementation of this metric for C code is pmccabe.\n
- #43 When we run it, it prints the complexity, size, and location of each function in our project.\n
- #44 CPAN has several metrics modules for Perl; Perl::Metrics::Simple is an easy one to get started with.\n
- #45 Here&#x2019;s a Perl subroutine similar to the one we saw.\n
- #46 Similar to pmccabe, Perl::Metrics::Simple gives us the size and complexity of each method.\n
- #47 Speaking of size and complexity, this paper reexamined several previous studies and found that several popular code metrics were effectively just expensive ways...\n
- #48 ...of counting lines. The paper didn&#x2019;t consider cyclomatic complexity alone (and there were other issues dealt with in subsequent papers by other authors), but we should always be skeptical of our own metrics. Fortunately, most tools give us both a line count and a complexity metric; we can decide for ourselves.\n
- #49 Some teams set complexity targets. In the degenerate case, they turn their code into a bunch of tiny functions that do nothing&#x2014;making the overall code base more complex and prone to bugs.\n
- #50 Another widely used metric is the percentage of your code that gets executed by your tests.\n
- #51 Measuring this typically involves instrumenting your code, so that you can watch it as it runs your tests.\n
- #52 Knowing our test coverage helps address the &#x201C;epic confidence&#x201D; problem that Laura Thomson described in her Open Source Bridge talk, &#x201C;How Not To Release Software.&#x201D; Teams afflicted by this bug assert without evidence that their tests are great.\n
- #53 In addition to combating hubris, measuring coverage helps us make our code more testable. Testability is not an end in itself, but a property with beneficial side effects.\n
- #54 For C projects, it&#x2019;s easy to measure coverage. GCC comes with the gcov coverage tool.\n
- #55 Here&#x2019;s a test that exercises just one branch of our code from earlier.\n
- #56 First, we&#x2019;d compile and link our program with a couple of gcov&#x2019;s required flags.\n
- #57 Then, we&#x2019;d run our tests and point gcov at the logfiles.\n
- #58 The result is a list of what lines did and didn&#x2019;t get executed. In this case, we never ran the &#x201C;else&#x201D; clause.\n
- #59 Not to be outdone, Perl provides Devel::Cover.\n
- #60 You just point Devel::Cover at your tests, and it produces an HTML report for you.\n
- #61 Devel::Cover gives us more information than gcov did. We executed line 26 once, but didn&#x2019;t exercise both sides of the &#x201C;&&&#x201D;.\n
- #62 Which brings us to another thing to keep in mind. Hitting each line of code once isn&#x2019;t the same as hitting each combination of branches. Code coverage is meant to help you look for holes, not to lull you into false security.\n
- #63 The advantage of applying commonly used measurements is good support. The downside is lack of context; the creators of those metrics have nowhere near the knowledge of your project that you do. So you may want to supplement common metrics with a few of your own. I can&#x2019;t tell you what those metrics are, but I can tell you a couple of the ones I&#x2019;ve seen used.\n
- #64 First, let&#x2019;s look at what I&#x2019;ll call X-rated-ness.\n
- #65 Just as George Carlin gave us his famous list of words you can&#x2019;t say on television,...\n
- #66 ...software teams have their own lists of bad words.\n
- #67 (Sorry, I should have blurred those out. ;-)\n
- #68 This is dead simple to do with ack, the modern-day replacement for grep. Just count string occurrences across your files, and optionally do a little sorting.\n
- #69 Grepping works on nearly every language, of course. But Perl has its own specific implementation of this metric.\n
- #70 All you have to do is throw a couple of lines into a &#x201C;.t&#x201D; file...\n
- #71 ...and Perl won&#x2019;t even let your tests pass if you&#x2019;ve got a naughty word in your code.\n
- #72 Another metric that&#x2019;s not universally used, but can still come in handy, is code churn: how often does a given piece of code change?\n
- #73 Churn can tell us what parts have changed recently; those parts may have new bugs.\n
- #74 Churn can also tell us what parts change often; those parts can become trouble spots.\n
- #75 You can get as crazy as you want with churn: examining which lines have changed the most, which functions have had the most people working on them, and so on. Git can tell you a lot more than a simple metric can, but if you&#x2019;re on a centralized system you may want to just grab the data yourself and pick it apart with UNIX tools.\n
- #76 If you&#x2019;re writing code that&#x2019;s going to get used by developers outside of your team, you might use a metric like documentation coverage to identify the parts of the code that most badly need docs.\n
- #77 Most of the metrics we&#x2019;ve seen so far have been one-shot numbers. But it&#x2019;s also possible to track things over time, like occurrences of compiler errors or test failures.\n
- #78 Zed Shaw does a great demo of this in his Play by Play screencast with PeepCode.\n
- #79 We&#x2019;ve talked about the kinds of questions we want to ask about our code, and the metrics that can help us answer those questions. Now for the bigger question: what&#x2019;s the effect on our software? Well, here are some of the things that happened with my team.\n
- #80 One, I found a surprisingly high complexity number in what was supposed to be a simple math routine. Somebody had snuck in an unwanted dependency on an unrelated system.\n
- #81 While looking for untested code, we found some code that didn&#x2019;t need any tests&#x2014;because it was never called anyway!\n
- #82 I personally like to look at what features have changed when it&#x2019;s time to do manual testing.\n
- #83 Some designs come at a time when our understanding of the domain is imperfect. As our understanding improves, we refactor the code. Complexity metrics can be handy for prioritizing.\n
- #84 One of the common themes woven through much of this discussion is that absolute limits for code metrics are not as helpful as relative measures within a project.\n
- #85 My hope is that you come away from this session with a couple of ideas for metrics you&#x2019;d like to try, and with the well-founded belief that you can get started with very little time investment.\n
- #86 I hope you find the answers you need for your project, and that you have fun getting them.\n
- #87 Thank you, and have a fantastic OSCON.\n
- #88 The images in this presentation were used by permission under the terms of a Creative Commons license.\n