Software reliability

Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Journ´ee des Technologies de l’Information et de la
Communication
Baptiste Wicht
EIA-FR
June 3, 2014
Baptiste Wicht EIA-FR Software Reliability

Software Validation
Conclusion
EIA-FR

Software Validation
Conclusion
Table of Contents
1 Software Reliability
2 Software Validation
3 Defensive Programming
4 Software Analysis Tools
5 Conclusion

Software Validation
Conclusion
Deﬁnitions
Examples
Facts
Solutions
Contents
Deﬁnitions
Examples
Facts
Solutions
5 Conclusion

Software Validation
Conclusion
Definitions
Examples
Facts
Solutions
Definitions
Definition
Probability that a software will work properly in a specified
environment for a given amount of time.
What makes a program reliable ?
No bug
Must work under any possible condition
Must comply to the specifications

Software Validation
Conclusion
Definitions
Examples
Facts
Solutions
Difficulties
properly, specified environment, given amount of time
No specification is perfect
Specifications can change
Software is complex

Software Validation
Conclusion
Deﬁnitions
Examples
Facts
Solutions
Measuring reliability
Diﬃcult problem
Hard to specify
Software is complicated to quantify
Models are available
More than 200 models exist
No single model completely represent reliability
Basic idea: Reliability is function of the number of defects

Software Validation
Conclusion
Deﬁnitions
Examples
Facts
Solutions
Examples
1991: Telephone outage in California
Three lines of code changed
several-million lines of code program
⇒ ≈ 200’000 phone calls lost

Software Validation
Conclusion
Deﬁnitions
Examples
Facts
Solutions
Examples
1991: Telephone outage in California
Three lines of code changed
several-million lines of code program
⇒ ≈ 200’000 phone calls lost
1991: Patriot Missile fails to intercept Scud Missile
Time stored in tenths of a second, multiplied by 1/10 to get
seconds
24 bit ﬁxed-point register, 1/10 has 0.000000095 error
After 100 hours, error of 0.34 seconds
=> 28 dead

Software Validation
Conclusion
Definitions
Examples
Facts
Solutions
Examples (cont.d)
1996: Ariane 5 explodes after 37 seconds of flight
16bit integer overflow
Overflow detected ⇒ useless corrections ⇒ other
malfunctions ⇒ self-destruction
System taken from Ariane 4, not used in Ariane 5
⇒ 500M $ lost

Software Validation
Conclusion
Definitions
Examples
Facts
Solutions
Examples (cont.d)
1996: Ariane 5 explodes after 37 seconds of flight
16bit integer overflow
Overflow detected ⇒ useless corrections ⇒ other
malfunctions ⇒ self-destruction
System taken from Ariane 4, not used in Ariane 5
⇒ 500M $ lost
2014: Heartbleed (OpenSSL)
Missing bound check
Allows more data to be read than should be allowed
Introduced in 2011
⇒ Millions of website vulnerable

Software Validation
Conclusion
Deﬁnitions
Examples
Facts
Solutions
Facts
Perfect software does not exist
The bigger the program ⇒ the more bugs
working today = working tomorrow
small change may have big impact

Software Validation
Conclusion
Deﬁnitions
Examples
Facts
Solutions
Solutions
Software Validation

Software Validation
Conclusion
Unit Testing
Integration Testing
Software Validation
Conclusion
Contents
Unit Testing
Integration Testing
Software Validation
Conclusion
5 Conclusion

Software Validation
Conclusion
Unit Testing
Integration Testing
Software Validation
Conclusion
Unit Testing
Deﬁnition
Set of tests to verify that a small unit of code is correct
Automated tests
Should be run after each change to the program and under
production-like environment
Monitor code coverage

Software Validation
Conclusion
Unit Testing
Integration Testing
Software Validation
Conclusion
Integration Testing
Deﬁnition
Set of tests to verify that a set of units forming a component is
correct
Can be automated or performed by human
If human-performed, can be very costly
Should be performed as much as possible

Software Validation
Conclusion
Unit Testing
Integration Testing
Software Validation
Conclusion
Software Validation
Deﬁnition
Tests to ensure that the complete system meets its speciﬁcations
Was the right software built ?
Was the software built right ?
Generally performed by Q/A specialists

Software Validation
Conclusion
Unit Testing
Integration Testing
Software Validation
Conclusion
Conclusion
No part of the code should be neglected
No change should be neglected
Small modules easily covered by a unit test should always be
unit tested
Unit/Integration tests should always be used to avoid
regression
Unit testing should be done as part of Continuous Integration
Automated tests are less costly that human-performed tests

Software Validation
Conclusion
Language Features
Contract Programming
Conclusion
Contents
Language Features
Conclusion
5 Conclusion

Software Validation
Conclusion
Language Features
Conclusion
Deﬁnition
Make a program more reliable by ensuring that code is used in the
correct way and it does what it is supposed to do.
Protection against failures and fault
Written as code
Accurate information to the source of the error
Veriﬁed at compile-time or runtime

Software Validation
Conclusion
Language Features
Conclusion
Language Features
Diﬀerent languages provide diﬀerent level of security
int array [100];
int b = array [100]; // What should happen ?

Software Validation
Conclusion
Language Features
Conclusion
Language Features
Different languages provide different level of security
int array [100];
int b = array [100]; // What should happen ?
C/C++: Undefined
Java/Python: Runtime error
Generally: lower level ⇒ harder to debug

Software Validation
Conclusion
Language Features
Conclusion
Assertions
Deﬁnition
Predicates (true-false statements) inserted into code indicating
what the developer thinks should be true at a certain point.
Indicates that a certain condition must be true
Typically aborts execution if condition not met
May be replaced by exceptions
Is generally disabled in production
Should not have any side eﬀect

Software Validation
Conclusion
Language Features
Conclusion
Assertions (cont.d)
double log(int x){
assert(x > 0);
//...
}
template <typename T>
class Z {
static_assert(sizeof(T) > 2, "Invalid␣Type");
//...
}

Software Validation
Conclusion
Language Features
Conclusion
Assertions (cont.d)
Pros
Available in almost every languages
Very simple
Generally useful to check input parameters

Software Validation
Conclusion
Language Features
Conclusion
Assertions (cont.d)
Pros
Available in almost every languages
Very simple
Generally useful to check input parameters
Cons
Only veriﬁed at runtime
Some languages supports compile-time assertions
Only local to a function
Can make the code more complex
Can be disabled

Software Validation
Conclusion
Language Features
Conclusion
Contract of the code speciﬁed into the program
Pre/postconditions, Invariants, ...
Can be implemented with assertions
Coupled with powerful static analysis, some errors can be
found without running the program

Software Validation
Conclusion
Language Features
Conclusion
Conclusion
Pros
Catch errors with nice error messages
Stored with the code
Cons
Make the code more complex
May introduce new errors
May incur a runtime cost
Dependent on the developer

Software Validation
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Contents
Compilers
Tools
Exercises
SonarQube
Conclusion
5 Conclusion

Software Validation
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Many available tools
Automatic search for bugs, performance improvements, style
problems
Two types:
Static Analysis: Source code level
Dynamic Analysis: Runtime level

Software Validation
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Compilers
Every modern compiler produces warnings
Only when asked
May produce too much warnings
Often forgotten
Recommendation
New project: As much as possible and -Werror
Existing project: Fix existing warnings and enable warnings one
by one
CLang/GCC: -Wall -Wextra

Software Validation
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
cppcheck
Static Analysis for C/C++ programs
Free software, GPL license
Cross-platform
Active project, latest stable release 10 May 2014
Available on every mainline distribution
Contains a large set of checkers
Integrated in several IDE (Eclipse, Visual Studio,
Code::Blocks, CodeLite...)
Generate text, XML and HTML reports

Software Validation
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
CLang Static Analyzer
Static Analysis for C/C++ and Objective-C programs
Free software, MIT license
Developed as part of the LLVM/CLang project
Slower than compilation
Still young project
Several checkers implemented
Can be extended with checker plugins
Generate text and HTML reports

Software Validation
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Valgrind
Framework for dynamic analysis tools
Free Software, GPL License
Work on Linux and Mac OS X
Most known for Memcheck tool
Runs the program into a virtual machine
Program runs slower (4-5 times slower with None tool)
Generate text reports

Software Validation
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Tools
Memcheck: Memory debugger
Default and most used tool
Very large performance lost (20 to 40 times slower)
Massif: Heap proﬁler
Helgrind: Detect race conditions
Cachegrind: Cache proﬁler
Callgrind: Call-graph analyzer
exp-sgcheck: Experimental tool for memory overruns

Software Validation
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Exercises
Some exercises
Machines ready with Ubuntu
Launch terminal
> cd examples

Software Validation
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Memory Leak
int main (){
int* a = new int (4);
// Something
a = new int (5);
// Something else
delete a;
}
> g++ leak.cpp
> valgrind ./a.out

Software Validation
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Memory Leak
==15386== HEAP SUMMARY:
==15386== in use at exit: 4 bytes in 1 blocks
==15386== total heap usage: 2 allocs, 1 frees, 8 bytes allocated
==15386==
==15386== LEAK SUMMARY:
==15386== definitely lost: 4 bytes in 1 blocks
==15386== indirectly lost: 0 bytes in 0 blocks
==15386== possibly lost: 0 bytes in 0 blocks
==15386== still reachable: 0 bytes in 0 blocks
==15386== suppressed: 0 bytes in 0 block
4 bytes have been lost
No information on location
> valgrind --leak -check=full ./a.out

Software Validation
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Memory Leak
==6434== 4 bytes in 1 blocks are definitely lost in loss record 1 of 1
==6434== at 0x4028812: operator new(unsigned long) (vg_replace_malloc.c:319)
==6434== by 0x400641: main (in /home/wichtounet/dev/analysis-examples/a.out)
The source of the allocation is now shown
Only binary location, no source location
> g++ -g leak.cpp

Software Validation
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Memory Leak
==9604== at 0x4028812: operator new(unsigned long) (vg_replace_malloc.c:319)
==9604== by 0x400641: main (leak.cpp:2)
Source location (leak.cpp:2) is now complete
Valgrind will only show the sources of allocated leaked memory
Sometimes it may still be complicated to ﬁnd why not freed

Software Validation
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Heap Overrun
int main (){
int* heap = new int [5];
heap [5] = 42;
delete [] heap;
return 0;
}
> g++ -g heap_overrun.cpp
> valgrind ./a.out

Software Validation
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Heap Overrun
==16922== Invalid write of size 4
==16922== at 0x40064E: main (heap_overrun.cpp:4)
==16922== Address 0x50c4054 is 0 bytes after a block of size 20 alloc’d
==16922== at 0x4028F40: operator new[](unsigned long) (vg_replace_malloc.c:384)
==16922== by 0x400641: main (heap_overrun.cpp:2)
Source location of the invalid write (heap overrun.cpp:4)
Source location of the allocation (heap overrun.cpp:2)

Software Validation
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Stack Overrun
int main (){
int i, a[10];
for (i = 0; i <= 10; i++)
a[i] = 42;
return 0;
}
> g++ -g stack_overrun.cpp
> valgrind ./a.out

Software Validation
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Stack Overrun
==20444== HEAP SUMMARY:
==20444== in use at exit: 0 bytes in 0 blocks
==20444== total heap usage: 0 allocs, 0 frees, 0 bytes allocated
==20444==
==20444== All heap blocks were freed -- no leaks are possible
Overrun is not detected
Size of memory block is detected by malloc implementation
> valgrind --tool=exp -sgcheck ./a.out

Software Validation
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Stack Overrun
==24703== Invalid write of size 4
==24703== at 0x400572: main (stack_overrun.cpp:5)
==24703== Address 0xfff0000c8 expected vs actual:
==24703== Expected: stack array "a" of size 40 in this frame
==24703== Actual: unknown
==24703== Actual: is 0 after Expected
Overrun is detected!

Software Validation
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Stack Overrun 2
int main (){
int stack [5];
stack [5] = 0;
return 0;
}
> g++ -g stack_overrun_2 .cpp

Software Validation
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Stack Overrun 2
==28007== exp-sgcheck, a stack and global array overrun detector
==28007== NOTE: This is an Experimental-Class Valgrind Tool
==28007== Copyright (C) 2003-2013, and GNU GPL’d, by OpenWorks Ltd et al.
==28007== Using Valgrind-3.9.0 and LibVEX; rerun with -h for copyright info
==28007== Command: ./a.out
==28007==
==28007==
==28007== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0
Overrun not detected!
Need a valid access before the invalid one
> cppcheck stack_overrun_2 .cpp

Software Validation
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Stack Overrun 2
Checking stack_overrun_2.cpp...
[stack_overrun_2.cpp:4]: (error) Array ’stack[5]’ accessed at index 5, which is out of bounds.
Overrun detected!
Tools put together are very powerful

Software Validation
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Complex Memory Leak
int main (){
int* array [42];
for(int i = 0; i < 42; ++i){
array[i] = new int;
}
// Something
for(int i = 0; i < 41; ++i){
delete array[i];
}
}
> cppcheck --enable=all leak_2.cpp

Software Validation
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Complex Memory Leak
Checking leak_2.cpp...
Checking usage of global functions..
cppcheck static analysis not powerful enough
> g++ -g leak_2.cpp
==12868== at 0x4C2C167: operator new(unsigned long) (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-li
==12868== by 0x400739: main (leak_2.cpp:7)

Software Validation
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Yet Another Leak
int main (){
unsigned int* mem =
(unsigned int*) malloc (4);
if(mem)
return 1;
*mem = 0xdeadbeaf;
free(mem);
return 0;
}

Software Validation
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Yet Another Leak
> clang --analyze leak_3.cpp
leak_3.cpp:7:16: warning: Potential leak of memory pointed to by ’mem’
return 1;
ˆ
leak_3.cpp:8:10: warning: Dereference of null pointer (loaded from variable ’mem’)
*mem = 0xdeadbeaf;
˜˜˜ ˆ
2 warnings generated
> c++ analyzer leak_3.cpp
return 1;
ˆ
*mem = 0xdeadbeaf;
˜˜˜ ˆ
2 warnings generated.

Software Validation
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Yet Another Leak
> scan -build clang ++ leak_3.cpp
scan-build: Using ’/usr/bin/clang’ for static analysis
return 1;
ˆ
*mem = 0xdeadbeaf;
˜˜˜ ˆ
2 warnings generated.
scan-build: 2 bugs found.
scan-build: Run ’scan-view /tmp/scan-build-2014-06-02-103126-14357-1’ to examine bug reports.
> scan -view /tmp/scan -build -...

Software Validation
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Yet Another Leak
> scan -build -enable -checker alpha make all
> scan -view /tmp/scan -build -...

Software Validation
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
SonarQube
Web platform for software quality management
Multi-project, Multi-language
Open Source and free
Some plugins are commercial (e.g. C++)
Centralized view for code source quality metrics
LOC, Cyclomatic Complexity
Warnings, style errors
Unit Test Results, Code Coverage
Documentation Coverage
Duplications
Views by components (package, folders, classes, ...)

Software Validation
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Demo
Online demo: http://nemo.sonarqube.org/

Software Validation
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Conclusion
Numerous other tools exist:
Python: Pylint
Java: FindBugs, PMD, Checkstyle
Javascript: JSLint, JSHint
Powerful commercial tools: PVS-Studio, Coverity, ...
...
Tools helps in ﬁnding existing very quickly
Very powerful when coupled with continuous integration

Software Validation
Conclusion
Contents
5 Conclusion

Software Validation
Conclusion
Conclusion
Build reliable software is hard
Software testing is very important
Numerous tools are available
No tool will ever ﬁnd all bugs
Some tools are overlapping

Software Validation
Conclusion
Recommendations
No change should be ignored when testing
Tests should be run in production-like environment
Continuous Integration and Testing
Make use of the existing analysis features at each level
No tool is perfect ⇒ Several tools can be used together

Software Validation
Conclusion
Questions
Questions ?

Software reliability

More Related Content

What's hot

Viewers also liked

Similar to Software reliability

Recently uploaded

Software reliability