Your SlideShare is downloading. ×
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Pragmatic Code Coverage
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Pragmatic Code Coverage

690

Published on

0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
690
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
0
Comments
0
Likes
2
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. And now for something completely different ...
  • 2. <Insert Picture Here>Code coverage.The pragmatic approach.Александр ИльинJava Quality architectalexandre.iline@oracle.com
  • 3. What it is about?Should the testing be stopped at 100% coverage?Should 100% be the goal?How (else) to use code coverage information?3What it is not about?Tools
  • 4. <Insert Picture Here>Preface
  • 5. What is the code coverage data forMeasure to which extent source code is coveredduring testing.5consequently …Code coverage isA measure of how much source code is coveredduring testing.Testing isA set of activities aimed to prove that the system undertest behaves as expected.finally …
  • 6. CC – how to get• Create a template6• “Instrument” the source/compiled code/bytecode• Run testing, collect data• Generate reportTemplate is a collection of all the code there is to coverInsert instructions for dropping data into a file/network, etc.May need to change environmentHTML, DB, etc
  • 7. • Block / primitive block• Line• Condition/branch/predicate• Entry/exit• Method• Path/sequence7CC – kinds of
  • 8. CC – how to use•8• Perform analysis• Develop more tests• GOTO 1Performed repeatedly, so resource-efficiency is really importantFind what tests you need to develop.• 1: Measure (prev. slide)for testbase improvementFind what code you need to cover.Find what code you need to cover.• Find dead codeMeasure (prev. slide)
  • 9. <Insert Picture Here>Mis-usages
  • 10. • Must get to 100%10• 100% means no more testing• CC does not mean a thing• There is that tool which would generate testsfor us and were doneCC – how not to usemis-usagesMay be not.No it does not.It does mean a fair amount if it is used properly.Nope.
  • 11. <Insert Picture Here>Mis-usagesTest generation
  • 12. Test generation“We present a new symbolic executiontool, ####, capable of automaticallygenerating tests that achieve highcoverage on a diverse set of complex andenvironmentally-intensive programs.”#### tool documentation
  • 13. Test generation cont.if ( b != 3 ) {double a = 1 / ( b – 2);} else {…}
  • 14. Test generation cont.if ( b != 3 ) {double a = 1 / ( b – 3);} else {…}Reminder: testing is ...A set of activities aimed to prove that the system undertest behaves as expected.
  • 15. Test generation - conclusionGenerated tests do not verify conformance of thecore to the requirements. *Hence …Generated tests code coverage should not bemixed with regular functional tests code coverage.15(*) Same is true for all static analysis techniques
  • 16. Who watches the watchmen?• Test logic gotta be right• No way to verify the logic• No metrics• No approaches• No techniques• Code review – the only way• Sole responsibility of test developer
  • 17. <Insert Picture Here>Mis-usagesWhat does 100% coverage mean?
  • 18. 100% block/line coverage1false
  • 19. 100% branch coverage1true-1false
  • 20. 100% domain coverage00.1√.1-.1Exception0 e
  • 21. 100% sequence coverage(-1,-1)1(1,1)(0,0)1 NaNb(-1,1)(1,-1)1 1
  • 22. 100% sequence coverage(-1,-1)1(1,1)(0,0)1 NaNb(-1,1)(1,-1)-1 -1But … isPositive(float) has a defect!
  • 23. 100% sequence coverage• Has conceptual problems• Code semantics• Loops• One of the two• Assume libraries has no errors• Done in depth – with the libraries• Very expensive• A lot of sequences: 2# branches, generally speaking• Very hard to analyze data
  • 24. 100% coverage - conclusion100% block/line/branch/path coverage, even ifreachable, does not prove much.Hence …No need to try to get there unless ...24
  • 25. 100% coverage - conclusionMost importantly ...A code coverage only measures coverage of acode which has been written.25
  • 26. <Insert Picture Here>Mis-usagesTarget value
  • 27. Block coverage target value
  • 28. 100% coverage - conclusion100% block/line/branch/path coverage, even ifreachable, does not prove much.Hence …No need to try to get there unless …100% is the target value.Which could happen if cost of a bug is really bigand/or the product is really small.31
  • 29. Target value - conclusionTrue target value for block/line/branch/path comesfrom ROI, which is really hard to calculate andjustify.32
  • 30. <Insert Picture Here>Usages
  • 31. • Test base improvement.34• Dead code.• Metric• Control over code developmentCC – how to useRight. How to select which tests to develop firstBarely an artifactBetter have a good metric.• Control flow analysis
  • 32. <Insert Picture Here>CC as a metric
  • 33. What makes a good metricSimple to explainSimple to work towardsHas a clear goalSo that you could explain your boss why is that importantto spend resources onSo that you know what to do to improveSo you could tell how far are you.
  • 34. Is CC a good metric?Simple to explainSimple to work towardsHas a clear goalIs a metric of quality of testing.(Relatively) easy to map uncovered code to missed tests.Nope. ROI – too complicated.++-
  • 35. Filter code coverage… to only leave code whichshould be covered completelyExamples• Public API coverage• UI coverage• Controller code coverage• “Important code” coverage
  • 36. Public API*Is a set of program elements suggested for usage bypublic documentation.For example: all functions and variables which aredescribed in documentation.For a Java library: all public and protected methods andfields mentioned in the library javadoc.For Java SDK: … of all public classes in java and javaxpackages.(*) Only applicable for a library or a SDK
  • 37. Public API
  • 38. True Public API (c)Is a set of program elements which could be accesseddirectly by a library userPublic API+all extensions of public API in non-public classes
  • 39. True public API exampleArrayList.javaMy code
  • 40. UI coverage
  • 41. UI coverageIn a way, equivalent to public API but for a UI product• %% of UI elements shown – display coverage• %% user actions performed – action coverageOnly “action coverage” could be obtained from CC data (*).(*) For UI toolkits which the presenter is familiar with.
  • 42. Action coveragejavax.swing.Action.actionPerformed(ActionEvent)javafx.event.EventHandler.handle(Event)org.myorg.NodeAction.actionPerformed(ActionEvent)org.myorg.NodeAction.nodeActionPerformed(Node myNode)
  • 43. “Controller” code coverageModelContains the domain logicViewImplements user interactionControllerMaps the two. Only contains code which is called as aresult of view actions and model feedbacks.Controller has very little boilerplate code. A goodcandidate for 100% block coverage.
  • 44. “Important” code• Development/SQE marks class/method as important• We use an annotation @CriticalForCoverage• List of methods is obtained which are marked asimportant• We do that by an annotation processor right while maincompilation• CC data is filtered by the method list• Goal is 100%
  • 45. Examples of non-generic metrics• BPEL elements• JavaFX properties• A property in JavaFX is something you could set, get and bind• Project type coverage in NetBeans• Insert your own
  • 46. CC as a metric - conclusionThere are multiple ways to filter CC data to a set ofcode which needed to be covered in full.There are generic metrics and there is a possibilityto introduce product specific metric.Such metrics are easy to use, although not alwaysso straightforward to obtain.51
  • 47. <Insert Picture Here>Test prioritization
  • 48. Test prioritization100500 uncovered lines of code!Metric• Pick a metric• Develop tests“Metrics for managers. Me no manager! Me write code!”Consider mapping CC data to few other source codecharacteristics.“OMG! Where do I start?”
  • 49. Age of the codeNew code is better be tested before getting to customer.(Improves bug escape rate, BTW)Old code is more likely to be tested by usersorNot used by users.
  • 50. Whats a bug escape metric?Ratio of defects sneaked out unnoticed# defects not found before release# defects in the productIn theory:# defects found after + # defects found beforePractical:# defects found after release
  • 51. Number of changesMore times a piece of code was changed, more atomicimprovements/bugfixes were implemented in it.Hence …Higher risk of introducing a regression.
  • 52. Number of lines changedMore lines changed – more testing it needs.Better all – number of uncovered lines which werechanged in the last release.
  • 53. Bug densityAssuming all the pieces were tested equally well …Many bugs means there are, probably, even more• Hidden behind the known ones• Fixing existing ones may introduce yet more as regressions
  • 54. Code complexityAssuming the same engineering talent and the sametechnology …More complex the code is – more bugs likely to be there.Any complexity metric would work: from class size tocyclomatic complexity
  • 55. Putting it togetherA formula(1 – cc) * (a1*x1+ a2*x2+ a3*x3+ ...)Wherecc – code coverage (0 - 1)xi– a risk of bug discovery in a piece of codeai– a coefficient
  • 56. Putting it together(1 – cc) * (a1*x1+ a2*x2+ a3*x3+ ...)The ones with higher value are first to cover• Fix the coefficients• Develop tests• Collect statistics on bug escape• Fix the coefficient• Continue
  • 57. Test prioritization - conclusionCC alone may not give enough information.Need to accompany it with other characteristics oftest code to make a decision.Could use a few of other characteristicssimultaneously.62
  • 58. <Insert Picture Here>Test prioritizationExecution
  • 59. Decrease test execution timeExclude tests which do not add coverage (*).But, be careful! Remember that CC is not everything andeven 100% coverage does not mean a lot.While excluding tests get some orthogonal measurementas well, such as specification coverage.(*) Requires “test scales”
  • 60. Control flow analysisStudy the coverage report, see what test code exerciseswhich code. (*).Recommended for developers.(*) Also requires “test scales”
  • 61. Controlled code changesDo not allow commits unless all the new/changed code iscovered.Requires simultaneous commits of tests and thechanges.
  • 62. Code coverage - conclusion100% CC does not guarantee that the code is working right100% CC may not be neededIt is possible to build good metrics with CCCC helps with prioritization of test developmentOther source code characteristics could be used with CC67
  • 63. Coverage data is not free• Do just as much as you can consume *• Requires infrastructure work• Requires some development• Requires some analysis(*) The rule of thumb
  • 64. Coverage data is not free• Do just as much as you can consume• Requires infrastructure work• Requires some development• Requires some analysis• Do just a little bit more than you can consume *• Otherwise how do you know how much you can consume?(*) The rule of thumb
  • 65. <Insert Picture Here>Code coverage.The pragmatic approach.Александр ИльинJava Quality architectalexandre.iline@oracle.com

×