<Insert Picture Here>Code coverage.The pragmatic approach.Александр ИльинJava Quality architectOracle
What it is about?Should the testing be stopped at 100% coverage?Should 100% be the goal?How (else) to use code coverage in...
<Insert Picture Here>Preface
What is the code coverage data forMeasure to which extent source code is coveredduring testing.                           ...
CC – how to get • Create a templateTemplate is a collection of all the code there is to cover • “Instrument” the source/co...
CC – kinds of• Block / primitive block• Line• Condition/branch/predicate• Entry/exit• Method• Path/sequence               ...
CC – how to use                           for testbase improvement • 1: Measure (prev. slide)Performed repeatedly, so reso...
<Insert Picture Here>Mis-usages
CC – how not to use                            mis-usages • Must get to 100%May be not. • 100% means no more testingNo it ...
<Insert Picture Here>Mis-usagesTest generation
Test generation“We present a new symbolic execution tool, ####,capable of automatically generating tests thatachieve high ...
Test generation cont.if ( b != 3 ) {    double a = 1 / ( b – 2);} else {    …}
Test generation cont.if ( b != 3 ) {    double a = 1 / ( b – 3);} else {    …}Reminder: testing is ...A set of activities ...
Test generation - conclusionGenerated tests could not test that the code workas expected because they only know how the co...
Who watches the watchmen?• Test logic gotta be right• No way to verify the logic   • No metrics   • No approaches   • No t...
<Insert Picture Here>Mis-usagesWhat does 100% coverage mean?
100% block/line coverage                           1                           false
100% branch coverage                       1      -1                       true   false
100% domain coverage                       0   .1   -.1                       0   .1    Exception
100% sequence coverage                   (-1,-1)                             (1,1)                                     (0,...
100% sequence coverage                              (-1,-1)                                        (-1,1)                 ...
100% sequence coverage• Has conceptual problems   • Code semantics   • Loops• One of the two  • Assume libraries has no er...
100% coverage - conclusion100% block/line/branch/path coverage, even ifreachable, does not prove much.Hence …No need to tr...
<Insert Picture Here>Mis-usagesTarget value
CC target value - cost                                                                                                Test...
CC target value - effectiveness                                                              Defect Coverage by Code Block...
CC target value - ROI                                                                                                     ...
100% coverage - conclusion100% block/line/branch/path coverage, even ifreachable, does not prove much.Hence …No need to tr...
Target value - conclusionTrue target value for block/line/branch/path comesfrom ROI, which is really hard to calculate and...
<Insert Picture Here>Usages
CC – how to use • Test base improvement.Right. How to select which tests to develop first • Dead code.Barely an artifact •...
<Insert Picture Here>CC as a metric
What makes a good metricSimple to explainSo that you could explain your boss why is that importantto spend resources onSim...
Is CC a good metric?Simple to explain +Is a metric of quality of testing.Simple to work towards +(Relatively) easy to map ...
Public API*Is a set of program elements suggested for usage bypublic documentation.For example: all functions and variable...
Public API
True Public API (c)Is a set of program elements which could be accesseddirectly by a library userPublic API+all extensions...
True public API example   My code                          ArrayList.java
True Public API how to get• Get public API with interfaces• Filter template so that it only contains implementations  and ...
UI coverageIn a way, equivalent to public API but for a UI product• %% of UI elements shown – display coverage• %% user ac...
Action coverage – how to get• Collect CC• Extract all implementations ofjavax.swing.Action.actionPerformed(ActionEvent)orj...
“Controller” code coverageModelContains the domain logicViewImplements user interactionControllerMaps the two. Only contai...
“Important” code• Development/SQE marks class/method as important   • We use an annotation @CriticalForCoverage• List of m...
Examples of non-generic metrics• BPEL elements• JavaFX properties   • A property in JavaFX is something you could set, get...
CC as a metric - conclusionThere are multiple ways to filter CC data to a set ofcode which needed to be covered in full.Th...
<Insert Picture Here>Test prioritization
Test prioritization 100500 uncovered lines of code! “OMG! Where do I start?” Metric   • Develop tests to close the metric ...
Age of the codeNew code is better be tested before getting to customer.Improves bug escape rate, BTWOld code is more likel...
Whats a bug escape metric?Ratio of defects sneaked out unnoticed                # defects not found before releaseIn theor...
Number of changesMore times a piece of code was changed, more atomicimprovements/bugfixes were implemented in it.Hence …Hi...
Number of lines changedMore lines changed – more testing it needs.Better all – number of uncovered lines which werechanged...
Bug densityAssuming all the pieces were tested equally well …Many bugs means there are, probably, even more  • Hidden behi...
Code complexityAssuming the same engineering talent and the sametechnology …More complex the code is – more bugs likely to...
Putting it togetherA formula(1 – cc) * (a1*x1 + a2*x2 + a3*x3 + ...)Wherecc – code coverage (0 - 1)xi – a risk of bug disc...
Putting it together(1 – cc) * (a1*x1 + a2*x2 + a3*x3 + ...)The ones with higher value are first to cover• Fix the coeffici...
Test prioritization - conclusionCC alone may not give enough information.Need to accompany it with other characteristics o...
<Insert Picture Here>Test prioritizationExecution
Decrease test execution timeExclude tests which do not add coverage (*).But, be careful! Remember that CC is not all and e...
Deep analysisStudy the coverage report, see what test code exerciseswhich code. (*).Recommended for developers.(*) Also re...
Controlled code changesDo not allow commits unless all the new/changed code iscovered.Requires simultaneous commits of tes...
Code coverage - conclusion100% CC does not guarantee that the code is working right100% CC may not be neededIt is possible...
Coverage data is not free• Do just as much as you can consume *   • Requires infrastructure work   • Requires some develop...
Coverage data is not free• Do just as much as you can consume   • Requires infrastructure work   • Requires some developme...
<Insert Picture Here>Code coverage.The pragmatic approach.Александр ИльинJava Quality architectOracle
Upcoming SlideShare
Loading in …5
×

Метрики покрытия. Прагматичный подход

1,025 views
958 views

Published on

Доклад Александра Ильина на конференции SQA Days-12, 30 ноября-1 декабря, Минск

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,025
On SlideShare
0
From Embeds
0
Number of Embeds
351
Actions
Shares
0
Downloads
27
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Метрики покрытия. Прагматичный подход

  1. 1. <Insert Picture Here>Code coverage.The pragmatic approach.Александр ИльинJava Quality architectOracle
  2. 2. What it is about?Should the testing be stopped at 100% coverage?Should 100% be the goal?How (else) to use code coverage information? What it is not about?Tools 2
  3. 3. <Insert Picture Here>Preface
  4. 4. What is the code coverage data forMeasure to which extent source code is coveredduring testing. consequently … Code coverage isA measure of how much source code is coveredduring testing. finally … Testing isA set of activities aimed to prove that the system undertest behaves as expected. 4
  5. 5. CC – how to get • Create a templateTemplate is a collection of all the code there is to cover • “Instrument” the source/compiled code/bytecodeInsert instructions for dropping data into a file/network, etc. • Run testing, collect dataMay need to change environment • Generate reportHTML, DB, etc 5
  6. 6. CC – kinds of• Block / primitive block• Line• Condition/branch/predicate• Entry/exit• Method• Path/sequence 6
  7. 7. CC – how to use for testbase improvement • 1: Measure (prev. slide)Performed repeatedly, so resource-efficiency is really important • Perform analysisFind what code you need to cover.Find what tests you need to develop. • Develop more tests • Find dead code • GOTO 1 7
  8. 8. <Insert Picture Here>Mis-usages
  9. 9. CC – how not to use mis-usages • Must get to 100%May be not. • 100% means no more testingNo it does not. • CC does not mean a thingIt does mean a fair amount if it is used properly. • There is that tool which would generate tests for us and were doneNope. 9
  10. 10. <Insert Picture Here>Mis-usagesTest generation
  11. 11. Test generation“We present a new symbolic execution tool, ####,capable of automatically generating tests thatachieve high coverage on a diverse set of complexand environmentally-intensive programs.” #### tool documentation
  12. 12. Test generation cont.if ( b != 3 ) { double a = 1 / ( b – 2);} else { …}
  13. 13. Test generation cont.if ( b != 3 ) { double a = 1 / ( b – 3);} else { …}Reminder: testing is ...A set of activities aimed to prove that the system undertest behaves as expected.
  14. 14. Test generation - conclusionGenerated tests could not test that the code workas expected because they only know how the codeworks and not how it is expected to. Because theonly thing they possess is the code which mayalready be not working as expected. :)Hence …Generated tests code coverage should not bemixed with regular functional tests code coverage. 14
  15. 15. Who watches the watchmen?• Test logic gotta be right• No way to verify the logic • No metrics • No approaches • No techniques • Code review – the only way• Sole responsibility of test developer
  16. 16. <Insert Picture Here>Mis-usagesWhat does 100% coverage mean?
  17. 17. 100% block/line coverage 1 false
  18. 18. 100% branch coverage 1 -1 true false
  19. 19. 100% domain coverage 0 .1 -.1 0 .1 Exception
  20. 20. 100% sequence coverage (-1,-1) (1,1) (0,0) b 1 1 NaN
  21. 21. 100% sequence coverage (-1,-1) (-1,1) (1,-1) (1,1) (0,0) b 1 -1 -1 1 NaNBut … isPositive(float) has a defect!
  22. 22. 100% sequence coverage• Has conceptual problems • Code semantics • Loops• One of the two • Assume libraries has no errors • Done in depth – with the libraries• Very expensive • A lot of sequences: 2# branches, generally speaking • Very hard to analyze data
  23. 23. 100% coverage - conclusion100% block/line/branch/path coverage, even ifreachable, does not prove much.Hence …No need to try to get there unless ... 23
  24. 24. <Insert Picture Here>Mis-usagesTarget value
  25. 25. CC target value - cost Test Dev. Effort by Code Block Coverage.Industry dataindicates that effort 90.00increasesexponentially with 80.00coverage. Relative Test Dev. Effort (1 at 50% code block coverage) 70.00We scale to makeeffort relative to theeffort of getting 50% 60.00coverage. 50.00 f  x =k e r xIntuition: the effortneeded to get more k =e−50r ⇒ f 50=1 40.00coverage is dfproportional to the =r f  x  30.00 dxtotal effort needed toget current coverage. 20.00Model not reliablebelow 50% coverage, 10.00except maybe very bigprojects. 0.00 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 Code Block Coverage (%)
  26. 26. CC target value - effectiveness Defect Coverage by Code Block CoverageDefect coverage by 120.00code block coverage...def ned in terms of i 100.00effort per code coverageand defect coverage byeffort. H  x =h  f  x  80.00 f  x =k e r x Defect Coverage(%)Intuition: discovery rate − s y Bis proportional to the h  y = B1−e percentage of bugs 60.00remaining and the effort dH H  x  df =s 1−  xneeded to get current dx B dxcoverage. 40.00Model not reliable below50% coverage except 20.00maybe very big projects. 0.00 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 Code Block Coverage(%)
  27. 27. CC target value - ROI Benef t(c) = DC(c) DD COD, where i DC(c): Defect Coverage Cost-Benefit Analysis DD: Defect Density. Example: 50bug/kloc 1200.00 COD: Cost Of Defect. Example: $20k/bug 1000.00 800.00Benefit ($/size), Cost ($/size), ROI (%) ROI = Benef t(c)/ Cost(c) - 1 i 600.00 400.00 200.00 0.00 Cost(c) = F + V * RE(c), where RE(c): Relative Effort, RE(50%) = 1 F: Fixed cost of test. Example: -200.00 $50k/kloc 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 V: Variable cost of test. Example: $5k/kloc Code Block Coverage (%)
  28. 28. 100% coverage - conclusion100% block/line/branch/path coverage, even ifreachable, does not prove much.Hence …No need to try to get there unless … 100% is the target value.Which could happen if cost of a bug is really bigand/or the product is really small. 28
  29. 29. Target value - conclusionTrue target value for block/line/branch/path comesfrom ROI, which is really hard to calculate andjustify. 29
  30. 30. <Insert Picture Here>Usages
  31. 31. CC – how to use • Test base improvement.Right. How to select which tests to develop first • Dead code.Barely an artifact • MetricBetter have a good metric. • Control over code development • Deep analysis 31
  32. 32. <Insert Picture Here>CC as a metric
  33. 33. What makes a good metricSimple to explainSo that you could explain your boss why is that importantto spend resources onSimple to work towardsSo that you know what to do to improveHas a clear goalSo you could tell how far are you.
  34. 34. Is CC a good metric?Simple to explain +Is a metric of quality of testing.Simple to work towards +(Relatively) easy to map uncovered code to missed tests.Has a clear goal -Nope. ROI – too complicated.Need to filter the CC dataso only that is left which must be covered
  35. 35. Public API*Is a set of program elements suggested for usage bypublic documentation.For example: all functions and variables which aredescribed in documentation.For a Java library: all public and protected methods andfields mentioned in the library javadoc.For Java SDK: … of all public classes in java and javaxpackages. (*) Only applicable for a library or a SDK
  36. 36. Public API
  37. 37. True Public API (c)Is a set of program elements which could be accesseddirectly by a library userPublic API+all extensions of public API in non-public classes
  38. 38. True public API example My code ArrayList.java
  39. 39. True Public API how to get• Get public API with interfaces• Filter template so that it only contains implementations and extensions of the public API (*)• Filter the data by template(*) This assumes that you either• Use a tool which allows such kind of filteringor• Have the data in a parse-able format and develop the filtering on your own
  40. 40. UI coverageIn a way, equivalent to public API but for a UI product• %% of UI elements shown – display coverage• %% user actions performed – action coverageOnly “action coverage” could be obtained from CC data (*).(*) For UI toolkits which the presenter is familiar with.
  41. 41. Action coverage – how to get• Collect CC• Extract all implementations ofjavax.swing.Action.actionPerformed(ActionEvent)orjavafx.event.EventHandler.handle(Event)• Inspect all the implementationsorg.myorg.NodeAction.actionPerformed(ActionEvent)• Add to the filter:org.myorg.NodeAction.nodeActionPerformed(Node myNode)• Extract, repeat
  42. 42. “Controller” code coverageModelContains the domain logicViewImplements user interactionControllerMaps the two. Only contains code which is called as aresult of view actions and model feedbacks.Controller has very little boilerplate code. A goodcandidate for 100% block coverage.
  43. 43. “Important” code• Development/SQE marks class/method as important • We use an annotation @CriticalForCoverage• List of methods is obtained which are marked as important • We do that by an annotation processor right while main compilation• CC data is filtered by the method list• Goal is 100%
  44. 44. Examples of non-generic metrics• BPEL elements• JavaFX properties • A property in JavaFX is something you could set, get and bind• Insert your own.
  45. 45. CC as a metric - conclusionThere are multiple ways to filter CC data to a set ofcode which needed to be covered in full.There are generic metrics and there is a possibilityto introduce product specific metric.Such metrics are easy to use, although not alwaysso straightforward to obtain. 45
  46. 46. <Insert Picture Here>Test prioritization
  47. 47. Test prioritization 100500 uncovered lines of code! “OMG! Where do I start?” Metric • Develop tests to close the metric • Pick another metric“Metrics for managers. Me no manager! Me write code!” Consider mapping CC data to few other source code characteristics.
  48. 48. Age of the codeNew code is better be tested before getting to customer.Improves bug escape rate, BTWOld code is more likely to be tested by usersorNot used by users.
  49. 49. Whats a bug escape metric?Ratio of defects sneaked out unnoticed # defects not found before releaseIn theory: # defects in the product # defects found after releasePractical: # defects found after + # defects found before
  50. 50. Number of changesMore times a piece of code was changed, more atomicimprovements/bugfixes were implemented in it.Hence …Higher risk of introducing a regression.
  51. 51. Number of lines changedMore lines changed – more testing it needs.Better all – number of uncovered lines which werechanged in the last release.
  52. 52. Bug densityAssuming all the pieces were tested equally well …Many bugs means there are, probably, even more • Hidden behind the known ones • Fixing existing ones may introduce yet more as regressions
  53. 53. Code complexityAssuming the same engineering talent and the sametechnology …More complex the code is – more bugs likely to be there.Any complexity metric would work: from class size tocyclomatic complexity
  54. 54. Putting it togetherA formula(1 – cc) * (a1*x1 + a2*x2 + a3*x3 + ...)Wherecc – code coverage (0 - 1)xi – a risk of bug discovery in a piece of codeai – a coefficient
  55. 55. Putting it together(1 – cc) * (a1*x1 + a2*x2 + a3*x3 + ...)The ones with higher value are first to cover• Fix the coefficients• Develop tests• Collect statistics on bug escape• Fix the coefficient• Continue
  56. 56. Test prioritization - conclusionCC alone may not give enough information.Need to accompany it with other characteristics oftest code to make a decision.Could use a few of other characteristicssimultaneously. 56
  57. 57. <Insert Picture Here>Test prioritizationExecution
  58. 58. Decrease test execution timeExclude tests which do not add coverage (*).But, be careful! Remember that CC is not all and even100% coverage does not mean a lot.While excluding tests get some orthogonal measurementas well, such as specification coverage.(*) Requires “test scales”
  59. 59. Deep analysisStudy the coverage report, see what test code exerciseswhich code. (*).Recommended for developers.(*) Also requires “test scales”
  60. 60. Controlled code changesDo not allow commits unless all the new/changed code iscovered.Requires simultaneous commits of tests and thechanges.
  61. 61. Code coverage - conclusion100% CC does not guarantee that the code is working right100% CC may not be neededIt is possible to build good metrics with CCCC helps with prioritization of test developmentOther source code characteristics could be used with CC 61
  62. 62. Coverage data is not free• Do just as much as you can consume * • Requires infrastructure work • Requires some development • Requires some analysis(*) The rule of thumb
  63. 63. Coverage data is not free• Do just as much as you can consume • Requires infrastructure work • Requires some development • Requires some analysis• Do just a little bit more than you can consume * • Otherwise how do you know how much you can consume?(*) The rule of thumb
  64. 64. <Insert Picture Here>Code coverage.The pragmatic approach.Александр ИльинJava Quality architectOracle

×