Test What Matters Most


Published on

Presentation that gives an overview of how ACE/TAO/CIAO/DAnCE are using Coverity scan

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Test What Matters Most

  1. 1. Test What Matters Most Develop Testing: Totaalvoetbal for Software Development 12th September, Amsterdam Johnny Willemsen, CTO Remedy IT jwillemsen@remedy.nl
  2. 2. Remedy IT Company of specialists focused on middleware and component technologies Dedicated to open standards and preferable open source Actively involved in the development of ACE/TAO/CIAO/DAnCE/OpenDDS open source software suite For more information take a look at our website www.remedy.nl 2 Copyright © Remedy IT
  3. 3. What is ACE? ADAPTIVE Communication Environment Open-source object-oriented (OO) framework that implements many core patterns for concurrent distributed communication software Strong adoption in the telecom, medical, aerospace, defense and financial services industries TAO, CIAO, DAnCE, and OpenDDS are products that are created on top of ACE and form together the ATCDO open source suite. 3 Copyright © Remedy IT
  4. 4. ACE and Coverity Scan What is Coverity Scan? • Free static analysis for open source projects • Find and fix defects in C/C++ or Java • Tests every line of code and potential execution path • Explains root cause of each defect making it easy to fix bugs ACE and Coverity Scan • Joined in April 2011 • Fixed 3263 defects as of September 2013 • 1795 defects outstanding • Defect density of 0.24 4 Copyright © Remedy IT
  5. 5. ACE testing challenges Goal: reduce risk through actionable, efficient testing Large, decentralized project with distributed development teams: • 7.8 million lines of C++ code • 20 primary developers, 100 developers committing code to the repository, 2375 developers contributed Testing a priority, but not enforceable, so most code remains untested • Current overall code coverage is 27% (57% for critical components) • Over 275,000 functions not fully tested including examples and other code that does not require testing 5 Copyright © Remedy IT
  6. 6. Creating an ENFORCEABLE test policy Many concerns before the project began • Where do I start, and how do I prioritize, with so much untested code? • How much real risk is there that I need to address ASAP? • How do I hold developers accountable for testing their own code? • How do I ensure code is tested as new features are developed? 6 Copyright © Remedy IT
  7. 7. Creating an ENFORCEABLE test policy ACE developed a basic Coverity Test Advisor policy to address these concerns • Filtering rules to determine which code is important to test • Analyzing untested code against those rules to identify missing tests and prioritize them 7 Copyright © Remedy IT
  8. 8. 4 Step policy with a big impact 8 Step Remaining Violations Testing Policy 0 275,313 Simple line coverage All insufficiently- or un-tested functions 1 12,833 Focus on core components Premise: certain components are more important to test than others 2 2,967 Only called code (ignore uncalled) Premise: we only write tests for important functions 3 2,757 Exclude debug, logging code Premise: diagnostic messages are not important to test 4 2,588 Since 6.1: 605 Exclude error cases, prioritize violations by release Premise: testing error cases provides minimal value and it is often difficult to trigger this code Note: These numbers are for the exact same code Copyright © Remedy IT
  9. 9. Using Coverity ACE performs a weekly build dedicated to Coverity scan • Takes 27 hours to compile all source code with gcc • Takes 19 hours to analyze using Coverity scan Coverity scan analyzes the code and publishes the Scan results New issues are emailed to all developers The ACE build czar will enforce that any new issue gets resolved by the original developer Supports our goal that quality increases weekly and never decreases 9 Copyright © Remedy IT
  10. 10. Addressing the issues Test violations in the core code are most important Try to resolve high priority violations with funding through maintenance and support When resolving bugs, test violations are used to focus the new unit tests on what is really needed When adding a feature to ACE, Test Advisor is used to determine that the feature is well tested After a decrease in number of test violations, the basic Test Advisor policy can be relaxed 10 Copyright © Remedy IT
  11. 11. Want to know more? Remedy IT Postbus 81 6930 AB Westervoort The Netherlands tel.: +31(0)88 053 0000 e-mail: sales@remedy.nl website: www.remedy.nl Twitter: @RemedyIT Slideshare: RemedyIT Subscribe to our mailing list Join Coverity Scan 11 Copyright © Remedy IT