Presentation from Coverity at the European Merge World Tour - Coverity Development Testing

1. Coverity Development Testing
Accelerating Risk Mitigation through Continuous
Integration and Development Testing

2. “Software is Eating the World”
Software
- Marc Andreessen
Health
Financial
Communications
SCM / Logistics
Enterprise
Mobile
81% of business leaders believe technology is a fundamental
element of their business model
Over 60 million tablets and 175 million smart phones will be in
the workplace by the end of 2012
By 2016, open source software will be included in mission-
critical applications within 99% of Global 2000 enterprises
Automotive

4. Development Testing …
… is transforming software development by:
Reducing operational costs
Accelerating development and time to market
Protecting brands from catastrophic failure

5. Why All the Risk?
Software Complexity and Speed have Outpaced Legacy Testing Methods
Development
Testing
Software Complexity
Time to Market
Testing MethodsSecurity Testing
Functional Testing
Performance Testing
Manual Testing

6. Fewer defects escape development
Design Development
Quality &
Security
Assurance
Product Release
& Management
Development Testing
Transform software testing, from reactive to proactive

7. Transformation Maturity Model
Level 1
Automatic Defect
Detection
No new defects
introduced.
Level 3
Developer
Workflow
Optimization
Feeding all
components into
the developer
workflow
Level 4
Code Governance
Establish source
code acceptance
criteria
Complete
Enterprise Code
Assurance
All critical code
and code impacted
by change is tested
IntegrationintoSDLCHigh
Development Testing Adoption High
Level 2
Identification of
Residual Risk
Ensure critical code
is prioritized &
tested

8. How Coverity Static Analysis Works
Mimicks the
behavior of
dozens of
compilers
Integrates with
existing build
systems
Statically tests
all execution
paths
Finds defects
and inconsistent
coding patterns
AnalyzeBuild
Explains the
location and root
cause of defects
Manage and
share triage of
defects across
teams
Present &
Manage

9. Meaningful, real results
Focus on finding real defects, not style violations or superficial issues.
Over 12 years of experience analyzing open source and commercial code.
Industry-leading low false positive/negative rate
False positive rates typically below 15%
False positives waste time, hinder adoption, and reduce trust in the results.
Broadest Checker Library + Deepest Algorithms
Optimal balance of breadth, depth, and scalability to large code bases.
High Quality Results

10. Sample Project: PostgreSQL
Defects Fixed in 2012 per Category
Category # Defects Impact
Memory
–
corrup,ons
20
High
Memory
-­‐
illegal
accesses
10
High
Resource
leaks
43
High
Unini,alized
variables
10
High
API
usage
errors
1
Medium
Control
flow
issues
4
Medium
Error
handling
issues
14
Medium
Incorrect
expression
3
Medium
Insecure
data
handling
24
Medium
Integer
handling
issues
8
Medium
Null
pointer
dereferences
43
Medium
Code
maintainability
issues
58
Low
Security
best
prac,ces
viola,ons
15
Low
Grand
Total
253
• ~20 Developers
• Weekly Build
• 680k LOC
• False Positive Rate: 11.1%
• Defect Density: 0,273

11. We Find Critical Defects
• Tomcat Webserver 5.5.17
• Among several hundred defects, we found a “reverse lock
bug” that can lead to deadlock of the entire server

12. Focus testing
time where
it matters
… don’t waste
time writing tests
you don’t need
Test Advisor
Improving Unit Testing Effectiveness and Efficiency
High
Risk
Code
High
Risk
Code

13. Risk Mitigation Architecture
Test Advice
Actionable work items to address risk
due to inadequate testing
Test Policy Evaluation
• Critical code analysis
• Change impact analysis
• Test execution analysis
Test Monitoring
Code Ownership
and Change History
Static Code
Analysis
Customized
Test Policy

14. Move Quality into the
Inner Loop of Development
Code
Build
Test
Nightly Build
Continuous
Integration
Finding and Fixing
Quality Defects

15. QA
Development Testing Workflow
Code Check In
Development Security Audit
Static Analysis
Results
Nightly/Continuous Build
Regression Test
• Built into development process
• Retesting minimized
• Immediately actionable by developers
• Reduces burden on auditing team
Developer QA Security

16. Issue Responsibility Is Critical

17. Ingredients for Success
Code
Build
Test
Nightly Build
Continuous
Integration
High-Fidelity
Code
Compilation
High-
Performance
Analysis
Low False
Positive Rate
Detecting
Critical
Defects
Easy Defect
Navigation and
Comprehension
Comprehensive
Triage and
Remediation
Management
Visibility and
Governance
Team
Collaboration

18. Governance with Metrics
Automated high-fidelity
analysis on daily basis
18
Fast and educated triage
of results to categorize
and prioritize issuesAccurate
Data
Precise actions based
on comprehensive
data analysis
Trusted
Data

19. Policy Definition and Monitoring
Definition of organizational-wide policies for code quality
Aggregated sanity view of code by component, team, supplier

20. Supplier SLA Enforcement
Supplier self-certification based upon policies

21. Transformation Maturity Model
Level 1
Automatic Defect
Detection
No new defects
introduced.
Level 3
Developer
Workflow
Optimization
Feeding all
components into
the developer
workflow
Level 4
Code Governance
Establish source
code acceptance
criteria
Complete
Enterprise Code
Assurance
All critical code
and code impacted
by change is tested
IntegrationintoSDLCHigh
Development Testing Adoption High
Level 2
Identification of
Residual Risk
Ensure critical code
is prioritized &
tested

22. Coverity Development Testing Platform
Security
Advisor
Test
Advisor
Analysis Packs
Coverity SAVE™
Static Analysis Verification Engine
SDLC
Integrations
Policy Manager
Quality
Advisor
Architecture
Analysis
Dynamic
Analysis
FindBugs™
Analysis
Analysis
Integration
Toolkit
Coverity Connect
Test
Execution
Third Party
Metrics
Build/
Continuous
Integration
HP ALM
IDE
Code
Coverage
Defect
Tracking
SCM

23. ü Proven significant operational cost reductions
ü Metric visibility of code estate onshore and offshore
ü Proven history of finding crash causing or
unexpected behavior causing defects
ü Process Improvement of the Application Lifecycle
Management
Coverity Summary

24. Questions ?!?!
!
!