4. Владимир Иванов
Разработчик HotSpot JVM
Александр Ильин
Архитектор тестирования Oracle JDK
Pолевыеигры
Хардкорный девелопер
Тролль из отдела тестирования
5. Program testing can be used to show the
presence of bugs, but never to show their
absence.
[“Structured programming”, Dahl O.J., Dijkstra E.W. and Hoare C.A.R.]
[When] you have given the proof of [a
program's] correctness, … [you] can
dispense with testing altogether.
[“Software engineering”, Naur P., Randell B.]
(1972)
(1969)
6. Testing
is
Running the tested software
– in different environment
– with different data
in an attempt to
– Certify conformance
– Prove program correctness
– Prove incorrectness
7. Just a few years after “Structured programming” ...
We prove … that properly structured tests are
capable of demonstrating the absence of
errors in a program.
[“Toward a Theory of Test Data Selection”, John B. Goodenough, Susan L.
Gerhart] (1975)
Fundamental Test Theorem
8. Fundamental Test Theorem
COMPLETE(T ,C)=(∀d ∈T OK (d )⇒∀d ∈DOK (d ))∨(∀d ∈T ¬OK (d )⇒∀d ∈D¬OK (d ))
SUCCESSFUL(T )=∀t∈T OK (t)
RELIABLE(C)=(∀T1 ,T2⊂D)COMLPETE (T1 ,C)∧COMPLETE (T2 ,C)⇒
(SUCCESSFUL(T1)≡SUCCESSFUL(T2))
VALID(C)=∀d ∈D¬OK (d)⇒(∃T ⊆D)(COMPLETE (T ,C)∧¬SUCCESSFUL(T ))
∃T⊆D ,∃C
(COMPLETE(T ,C)∧RELIABLE(C)∧VALID(C)∧SUCCESSFUL(T ))⇒
∀d ∈DOK (d)
Program F(d) for domain D
Requirements: OUT(d, F(d)) = OK(d)
Data selection criteria: C
9. But wait! It's not over yet!
I hope to have convinced you that by its very nature
responsible system design and development must
be an activity of an undeniably mathematical nature.
… programming became an industrial activity at a
moment that the American manager was extremely
fearful of relying on the education and the
intelligence of his company's employees. And
management tried to organize the industrial
programming task in such a way that each
individual programmer would need to think as little
as possible.
[“Why correctness must be a mathematical concern” E. W Dijkstra] (1979)
10. But wait! It's not over yet!
"Arrogance in computer science is measured in
nano-Dijkstras."
Alan Kay
11. But wait! It's not over yet!
"Arrogance in computer science is measured in
nano-Dijkstras."
Alan Kay
"… and micro-Kays".
Unknown source ;-)
12. Testing
is
Running the tested software
– in different environment
– with different data
in an attempt to
– Certify conformance
– Prove program correctness (requires formal proof)
– Prove program incorrectness (practically)
Dynamic
13. Static testing
is
Analysis of artifacts
– source code
– binaries
– data files
in an attempt to
– discover errors
– identify suspicious patterns
– verify conformance of the artifacts
14. Static testing
includes
● Using static analyzers
– Big number of false positives
● Code reviews
– Tedious manual work
– Many errors missed
– Non formalizable
15. What defects
could by found by dynamic testing
Any defect!
You just need to invent enough test :)
only ...
It may take an indefinite amount of tests
So, the testing is, effectively, endless
16. What defects
could by found by static testing
Any defect!
You just need to look on the whole source long
enough
only ...
You can not know which ones you are detecting
and
You never know how many are left
17. What defects
are hard to find by dynamic testing
● Intermittent problems
– You may just missed it
● Platform/environment specific problem
– You just may not have the environment
18. What defects
are hard to find by static analysis
● Bugs in deep and wide class inheritance
– Virtual methods are resolved in runtime
● Bugs in modular system
– Many modules implement the same features,
modules are registered somewhere, etc.
– Same reason – modules are registered as runtime
19. Formal verification
is
Formal verifi cation is the act of proving or
disproving the correctness of intended algorithms
underlying a system with respect to a certain
formal specifi cation or property, using
formal methods of mathematics.
20. Formal verification vs Testing
● Testing
– Upper bound for program quality
● Passed test says nothing about quality
● What matters is when test fails
● Formal verification
– Lower bound for program quality
● Passed test guarantees absence of some type of
failures in a program
23. Formal verification
applied
(a1
…ak
)10....0Let's take a binary representation of a:
m >= 0
a > 0 => binary presentation of a has a least one 1 bit
a-1 = (a1
…ak
)01....1 => a&(a-1) = (a1
…ak
)00....0
m m
a&(a-1) = 0 => a1
,...,ak
= 0 => a = 2m
a = 2n
=> m=n, a1
,...,ak
= 0 => a&(a-1) = 0
∀0<a∈N :a &(a−1)=0⇔∃n∈N :a=2n
24. Formal verification
is
Formal verifi cation is the act of proving or
disproving the correctness of intended algorithms
underlying a system with respect to a certain
formal specifi cation or property, using
formal methods of mathematics.
Another approach is deductive verifi cation. It
consists of generating from the system and its
specifi cations (and possibly other annotations) a
collection of mathematical proof obligations,
the truth of which imply conformance of the
system to its specifi cation.
25. Deductive Verification
Theorem proving
● Four color theorem (proved in 1976)
● Curry-Howard isomorphism
– (Theorem, Proof) <=> (Type, Program)
● Theorem provers
– Interactive environments for constructing proofs
– Coq, Agda, Isabelle, HOL
● Real-world example
– COMPCERT: C Verified Compiler
26. Using tools
how about ...
● We create a program
– Is capable of proving something about another
program
– Is itself proven (yeah, yeah, a recursion)
● Use the program to prove something about
another program
● Let's call it a “prover”
Is this still a formal verification?
Sure!
27. Formal verification
compiler is a prover on it's own
● Formal verification for Java is performed by
Java compiler
– Types
– Uninitialized variable
– Missing of return statement
– Uncaught exceptions
– etc.
– etc.
28. Annotations in Java
@Stateless @LocalBean
public class GalleryFacade {
@EJB
private GalleryEAO galleryEAO;
@TransactionAttribute(SUPPORTS)
public Gallery findById(Long id) { ... }
@TransactionAttribute(REQUIRED)
public void create(String name) { … }
29. Annotations in Java
● Introduced in Java 5
● Metadata
● May be reflective
– SOURCE, CLASS, RUNTIME
● Standard (e.g. @Override) & custom annotations
● Extensively used nowadays
– JavaEE 6, IoC containers, test harnesses, etc
30. Annotations: pre-Java 8
● Allowed on declarations only
– Class declaration
@A public class Test {
@B private int a = 0;
@C public void m(@D Object o) {
@E int a = 1;
...
}
}
31. Annotations: pre-Java 8
● Allowed on declarations only
– Field declaration
@A public class Test {
@B private int a = 0;
@C public void m(@D Object o) {
@E int a = 1;
...
}
}
32. Annotations: pre-Java 8
● Allowed on declarations only
– Method declaration
@A public class Test {
@B private int a = 0;
@C public void m(@D Object o) {
@E int a = 1;
...
}
}
33. Annotations: pre-Java 8
● Allowed on declarations only
– Method parameter declaration
@A public class Test {
@B private int a = 0;
@C public void m(@D Object o) {
@E int a = 1;
...
}
}
34. Annotations: pre-Java 8
● Allowed on declarations only
– Local variable declaration
@A public class Test {
@B private int a = 0;
@C public void m(@D Object o) {
@E int a = 1;
...
}
}
38. Limitations
● Consider @NonNull annotation
● How to declare a Map with non-null keys and
values?
@NonNull Map<K,V>?
NO!
● Map<@NonNull K, @NonNull V>
… but incorrect in Java 7 and earlier
Type annotations in Java 8 for the rescue!
39. Type annotations in Java 8
● Allowed anywhere you would write a type
… including generics and casts
… for array levels and receivers
40. Type annotations in Java 8:
Examples
● Class inheritance
class UnmodifiableList<T>
implements @ReadOnly List<T> { ... }
● Casts
myDate = (@ReadOnly Date) roDate;
● Type tests
myString instanceof @NonNull String;
● Arrays
String @NonNull [] messages;
41. Type annotations in Java 8:
Examples
● Generics
List<@Interned String> messages;
● Type parameter bounds
Collection<? super @Exists File>
● Generic type arguments in a generic method
o.<@NonNull String>toString("...");
42. Pluggable types
● User-defined (pluggable) type system
● Extend built-in type system
– express extra information about types via
type qualifiers
● Permit more expressive compile-time checking
and guarantee the absence of additional errors
43. Checker Framework
● Collection of compiler plugins (“checkers”)
● Relies on Pluggable types and Type
Annotations in Java 8
● Find different sorts of bugs or verify their
absence
– 14 checkers are already provided
● Supports custom compiler plugins (provides
API)
– 5 third-party checkers
47. Example: Tainting Checker
● Use case:
– Trusted vs untrusted data
– Verify before use
● Examples
– SQL injection attack
● validate SQL query before executing it
– information leakage
● secret data vs data displayed to a user
48. Example: Tainting Checker
● Annotations
– @Untainted
● A type that includes only untainted, trusted values
– @Tainted
● A type that includes only tainted, untrusted values
54. Credit card number
Conclusion
● A card number in an account is always validated
● That is guaranteed at compile time
● You do not need to test with invalid numbers
● You do need to test
– All @SuppressWarnings("credit.card")
– checkLuhn(String cardNum)
● Better all … prove it!
55. More real life examples
String getProperty(@PropertyKey String key);
HashMap <@Adult Person, @NonNull Address>
findSobutylnik(@NonNull Location);
void monitorTemperature()
throws @Critical TemperatureException;
56. Checkers Framework:
Advanced features
● Linear checker
– Implements linear types (based on linear logic)
– control aliasing and prevent re-use
– Single ownership abstraction
● Prevents absence of ownership and multiple owners
● Dependent types
– @Dependent annotation
– Changes the type depending on qualified type of
the receiver (this)
– Example
List[N] – list with it's length encoded into it's type
57. How to start using
● No need to wait Java 8 release
– modified compiler already available
● Incremental program annotation
– Partial program checking
– Warnings during compilation
– Easily convertible into compilation errors
● -Werror flag to javac
– Default annotations for types w/o annotations
● Ability to annotate external libraries
