Test What Matters Most

1,684 views

Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Test What Matters Most

  1. 1. Test What Matters MostJohnny Willemsen, Remedy ITKeri Sprinkle, CoverityJon Jarboe, Coverity
  2. 2. Agenda2 For Coverity and Partner use only. Copyright Coverity, Inc., 2013• Part 1: Remedy IT and ACE• Introduction• ACE and Coverity Scan• Testing challenges• Creating an ENFORCEABLE testing policy• 4 Step policy with a big impact• Part 2: Coverity• Overview of Coverity Test Advisor• Coverity suite of development testing solutions• Development Testing Maturity Model• Q & A
  3. 3. Remedy ITACE3For Coverity and Partner use only. Copyright Coverity, Inc., 2013
  4. 4. Remedy IT4 For Coverity and Partner use only. Copyright Coverity, Inc., 2013• Company of specialists focused on middleware andcomponent technologies• Dedicated to open standards andpreferable open source• Actively involved in the development ofACE/TAO/CIAO/DAnCE/OpenDDSopen source software suite
  5. 5. What is ACE?5 For Coverity and Partner use only. Copyright Coverity, Inc., 2013ADAPTIVE Communication Environment• Open-source object-oriented (OO) framework thatimplements many core patterns for concurrentcommunication software• Strong adoption in the telecom, medical, aerospace,defense and financial services industries
  6. 6. ACE and Coverity Scan6 For Coverity and Partner use only. Copyright Coverity, Inc., 2013• What is Coverity Scan?• Free static analysis for open source projects• Find and fix defects in C/C++ or Java• Tests every line of code and potential execution path• Explains root cause of each defect making it easy to fix bugs• ACE and Coverity Scan• Joined in April 2011• Fixed 3243 defects as of June 2013• 1827 defects outstanding• Defect density of 0.24
  7. 7. ACE testing challenges7 For Coverity and Partner use only. Copyright Coverity, Inc., 2013Goal: reduce risk through actionable, efficient testing• Large, decentralized project with distributed development teams• 7.8 million lines of C++ code• 20 primary developers, 100 developers committing code to the repository,about 2375 developers who have contributed additions/patches• Last release on May 28, 2013• Testing a priority, but not enforceable, so most code remains untested• Current overall code coverage is 27% (57% for critical components)• Over 275,000 functions not fully tested including examples and other codethat does not require testing
  8. 8. Creating an ENFORCEABLE test policy8 For Coverity and Partner use only. Copyright Coverity, Inc., 2013• Many concerns before the project began• Where do I start, and how do I prioritize, with so much untested code?• How much real risk is there that I need to address ASAP?• How do I hold developers accountable for testing their own code?• How do I ensure code is tested as new features are developed?• ACE developed a basic Coverity Test Advisor policy toaddress these concerns• Filtering rules to determine which code is important to test• Analyzing untested code against those rules to identify missing testsand prioritize them
  9. 9. 4 Step policy with a big impact9 For Coverity and Partner use only. Copyright Coverity, Inc., 2013Step RemainingViolationsTesting Policy0 275,313 Simple line coverageAll insufficiently‐ or un‐tested functions1 12,833 Focus on core componentsPremise: certain components are more important to test than others2 2,967 Only called code (ignore uncalled)Premise: we only write tests for important functions3 2,757 Exclude debug, logging codePremise: diagnostic messages are not important to test4 2,588Since 6.1:  605Exclude error cases, prioritize violations by releasePremise: testing error cases provides minimal value and it is often difficult to trigger this codeNote: These numbers are for the exact same code
  10. 10. Using Coverity10 For Coverity and Partner use only. Copyright Coverity, Inc., 2013• ACE performs a weekly build dedicated to Coverity scan• Coverity scan analyzes the code and publishes the Scanresults• New issues are emailed to all developers• The ACE build czar will enforce that any new issue getsresolved by the original developer• Supports our goal that quality increases weekly and neverdecreases
  11. 11. Addressing the issues11 For Coverity and Partner use only. Copyright Coverity, Inc., 2013• Test violations in the core code are most important• Try to resolve high priority violations with funding throughmaintenance and support• When resolving bugs, test violations are used to focus thenew unit tests on what is really needed• When adding a feature to ACE, Test Advisor is used todetermine that the feature is well tested• After a decrease in number of test violations, the basic TestAdvisor policy can be relaxed
  12. 12. Coverity12For Coverity and Partner use only. Copyright Coverity, Inc., 2013
  13. 13. Automated Testing Is Not EffectivelyMeasured13 For Coverity and Partner use only. Copyright Coverity, Inc., 2013Effort to Develop Tests%ofCodeTest1. Diminishing return forincreased test effort2. Not all code is testable:• Unreachable statements• Dead code3. Not all tested code addsequal value:• Non critical code• Debug code• Exception handling code100%
  14. 14. Coverity Test Advisor14 For Coverity and Partner use only. Copyright Coverity, Inc., 2013Improves unit testing effectiveness and efficiencyFocus testingtime whereit mattersAnddon’t wastetime writing testsyou don’t needHighRiskCode
  15. 15. Creating an enforceable processThe high priority code that mustbe thoroughly testedDefineAnalyzeRemediateGovernCode that has changedand been impacted by changesCode that has been insufficientlytestedManage progress to improve testcoverage15
  16. 16. What code needs to be tested based onyour high risk criteriaDefineAnalyzeRemediateGovernWith patent-pending techniques basedon code behavior and change impactSurface issues in your workflow andefficiently manage to closureCreate a testing stage gate and enforcedeveloper accountabilityGeneric example:• All code changes for next release …• And code impacted by those changes …• Must have 100% coverage …• Not counting exception handling and debugcodePolicy definition example16 For Coverity and Partner use only. Copyright Coverity, Inc., 2013ACE:• Focus on core components• Only called code (ignore uncalled)• Exclude debug, logging code• Exclude error cases, prioritize violations byrelease• Exclude error code• Prioritize violations by release
  17. 17. Remediate Testing ViolationsUnified viewof testviolations anddefectsUnified viewof testviolations anddefectsCleardescription ofthe violationCleardescription ofthe violationAutomaticallyassignviolations toownersAutomaticallyassignviolations toownersIdentify wherethe violationoccurs in thecodeIdentify wherethe violationoccurs in thecodeFor Coverity and Partner use only. Copyright Coverity, Inc., 201317
  18. 18. Coverity ConnectCoverity SAVE™Static Analysis Verification EngineTestAdvisorSDLCIntegrationsTestExecutionThird PartyMetricsCodeCoverageSCMCoverity Development Testing Platform18 For Coverity and Partner use only. Copyright Coverity, Inc., 2013SecurityAdvisorTestAdvisorAnalysis PacksCoverity SAVE™Static Analysis Verification EnginePolicy ManagerQualityAdvisorArchitectureAnalysisDynamicAnalysisAnalysisIntegrationFindBugs™ | FxCopAnalysisIntegrationToolkitCoverity ConnectTestExecutionThird PartyMetricsBuild/ContinuousIntegrationALMHP | IBMIDECodeCoverageDefectTrackingSCMProprietary Code | Open Source Code(Scan)
  19. 19. Development Testing Maturity Model19 For Coverity and Partner use only. Copyright Coverity, Inc., 2013Development Testing AdoptionIntegrationintoSDLCLevel 1Detection ofcritical quality andsecurity defects aspart of SW buildprocess.No new defectsintroduced.Level 3Integration intothe existing SDLCusing a commonworkflow for alldefects and testeffectivenessissues.Level 4Establish andenforce consistentsource codequality andsecurity policies.Establish sourcecode acceptancecriteria.Level 5All legacy defectseliminated, buildfails if new defectsare introduced.All critical codeand code impactedby change istested.HighHighLevel 2Identification ofareas of riskcaused byinsufficientautomated testing.Ensure criticalcode is prioritizedand tested.
  20. 20. Q & A20 For Coverity and Partner use only. Copyright Coverity, Inc., 2012• Remedy IT:www.remedy.nl• ACE:www.cs.wustl.edu/~schmidt/ACE.html• Coverity:coverity.com• Coverity Scan:scan.coverity.com

×