Формальная верификация как средство тестирования (в Java)
Upcoming SlideShare
Loading in...5
×
 

Формальная верификация как средство тестирования (в Java)

on

  • 1,240 views

Доклад Владимира Иванова, Александра Ильина на конференции SQA Days-12, 30 ноября-1 декабря, Минск

Доклад Владимира Иванова, Александра Ильина на конференции SQA Days-12, 30 ноября-1 декабря, Минск

Statistics

Views

Total Views
1,240
Slideshare-icon Views on SlideShare
1,153
Embed Views
87

Actions

Likes
0
Downloads
22
Comments
0

6 Embeds 87

http://sqadays.com 54
http://www.sqadays.com 19
http://sqadays.byte-force.yar.ru 7
http://sqadays.localhost 5
https://twitter.com 1
http://webcache.googleusercontent.com 1

Accessibility

Categories

Upload Details

Uploaded via as OpenOffice

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • A static type system helps programmers to detect and prevent errors. However, a language’s built-in type system does not help to detect and prevent enough errors, because it cannot express certain important invariants. A user-defined, or pluggable, type system enriches the built-in type system by expressing extra information about types via type qualifiers. Pluggable types permit more expressive compile-time checking and guarantee the absence of additional errors.
  • @GuardedBy("MyClass.myLock") Object myMethod() { ... } // reassignments without holding the lock are OK. @GuardedBy("MyClass.myLock") Object x = myMethod(); @GuardedBy("MyClass.myLock") Object y = x; Object z = x; // ILLEGAL (assuming no lock inference), // because z can be freely accessed. x.toString() // ILLEGAL because the lock is not held synchronized(MyClass.myLock) { y.toString(); // OK: the lock is held } =================================================== void helper1(@GuardedBy("MyClass.myLock") Object a) { a.toString(); // ILLEGAL: the lock is not held synchronized(MyClass.myLock) { a.toString(); // OK: the lock is held } } @Holding("MyClass.myLock") void helper2(@GuardedBy("MyClass.myLock") Object b) { b.toString(); // OK: the lock is held } void helper3(Object c) { helper1(c); // OK: passing a subtype in place of a the @GuardedBy supertype c.toString(); // OK: no lock constraints } void helper4(@GuardedBy("MyClass.myLock") Object d) { d.toString(); // ILLEGAL: the lock is not held } void myMethod2(@GuardedBy("MyClass.myLock") Object e) { helper1(e); // OK to pass to another routine without holding the lock e.toString(); // ILLEGAL: the lock is not held synchronized (MyClass.myLock) { helper2(e); helper3(e); helper4(e); // OK, but helper4's body still does not type-check } }
  • @GuardedBy("MyClass.myLock") Object myMethod() { ... } // reassignments without holding the lock are OK. @GuardedBy("MyClass.myLock") Object x = myMethod(); @GuardedBy("MyClass.myLock") Object y = x; Object z = x; // ILLEGAL (assuming no lock inference), // because z can be freely accessed. x.toString() // ILLEGAL because the lock is not held synchronized(MyClass.myLock) { y.toString(); // OK: the lock is held } =================================================== void helper1(@GuardedBy("MyClass.myLock") Object a) { a.toString(); // ILLEGAL: the lock is not held synchronized(MyClass.myLock) { a.toString(); // OK: the lock is held } } @Holding("MyClass.myLock") void helper2(@GuardedBy("MyClass.myLock") Object b) { b.toString(); // OK: the lock is held } void helper3(Object c) { helper1(c); // OK: passing a subtype in place of a the @GuardedBy supertype c.toString(); // OK: no lock constraints } void helper4(@GuardedBy("MyClass.myLock") Object d) { d.toString(); // ILLEGAL: the lock is not held } void myMethod2(@GuardedBy("MyClass.myLock") Object e) { helper1(e); // OK to pass to another routine without holding the lock e.toString(); // ILLEGAL: the lock is not held synchronized (MyClass.myLock) { helper2(e); helper3(e); helper4(e); // OK, but helper4's body still does not type-check } }
  • @GuardedBy("MyClass.myLock") Object myMethod() { ... } // reassignments without holding the lock are OK. @GuardedBy("MyClass.myLock") Object x = myMethod(); @GuardedBy("MyClass.myLock") Object y = x; Object z = x; // ILLEGAL (assuming no lock inference), // because z can be freely accessed. x.toString() // ILLEGAL because the lock is not held synchronized(MyClass.myLock) { y.toString(); // OK: the lock is held } =================================================== void helper1(@GuardedBy("MyClass.myLock") Object a) { a.toString(); // ILLEGAL: the lock is not held synchronized(MyClass.myLock) { a.toString(); // OK: the lock is held } } @Holding("MyClass.myLock") void helper2(@GuardedBy("MyClass.myLock") Object b) { b.toString(); // OK: the lock is held } void helper3(Object c) { helper1(c); // OK: passing a subtype in place of a the @GuardedBy supertype c.toString(); // OK: no lock constraints } void helper4(@GuardedBy("MyClass.myLock") Object d) { d.toString(); // ILLEGAL: the lock is not held } void myMethod2(@GuardedBy("MyClass.myLock") Object e) { helper1(e); // OK to pass to another routine without holding the lock e.toString(); // ILLEGAL: the lock is not held synchronized (MyClass.myLock) { helper2(e); helper3(e); helper4(e); // OK, but helper4's body still does not type-check } }

Формальная верификация как средство тестирования (в Java) Формальная верификация как средство тестирования (в Java) Presentation Transcript

  • Методы формальной верификации
  • Методы формальной верификации … в Java 8
  • Методы формальной верификации … в Java 8 … да и вообще...
  • Pол евы игр е ы Владимир Иванов Разработчик HotSpot JVM Хардкорный девелопер Александр ИльинАрхитектор тестирования Oracle JDK Тролль из отдела тестирования
  • Program testing can be used to show thepresence of bugs, but never to show theirabsence.[“Structured programming”, Dahl O.J., Dijkstra E.W. and Hoare C.A.R.] (1972)[When] you have given the proof of [aprograms] correctness, … [you] candispense with testing altogether.[“Software engineering”, Naur P., Randell B.] (1969)
  • Testing isRunning the tested software – in different environment – with different datain an attempt to – Certify conformance – Prove program correctness – Prove incorrectness
  • Fundamental Test TheoremJust a few years after “Structured programming” ...We prove … that properly structured tests arecapable of demonstrating the absence oferrors in a program.[“Toward a Theory of Test Data Selection”, John B. Goodenough, Susan L.Gerhart] (1975)
  • Fundamental Test Theorem Program F(d) for domain D Requirements: OUT(d, F(d)) = OK(d) Data selection criteria: CCOMPLETE (T ,C )=(∀d ∈T OK (d )⇒∀d ∈ D OK (d ))∨(∀d ∈T ¬OK (d )⇒∀d ∈ D ¬OK (d )) SUCCESSFUL(T )=∀t ∈T OK (t) RELIABLE (C )=(∀T1 ,T2⊂ D)COMLPETE (T1 ,C )∧COMPLETE (T2 , C )⇒ (SUCCESSFUL(T1)≡SUCCESSFUL(T2)) VALID(C )=∀d ∈ D ¬OK (d )⇒(∃T ⊆ D)(COMPLETE (T ,C )∧¬SUCCESSFUL(T )) ∃T ⊆ D ,∃C (COMPLETE (T ,C )∧RELIABLE (C )∧VALID(C )∧SUCCESSFUL(T ))⇒ ∀d ∈ D OK (d )
  • But wait! Its not over yet!I hope to have convinced you that by its very natureresponsible system design and development mustbe an activity of an undeniably mathematical nature.… programming became an industrial activity at amoment that the American manager was extremelyfearful of relying on the education and theintelligence of his companys employees. Andmanagement tried to organize the industrialprogramming task in such a way that eachindividual programmer would need to think as littleas possible.[“Why correctness must be a mathematical concern” E. W Dijkstra] (1979)
  • But wait! Its not over yet!"Arrogance in computer science is measured in nano-Dijkstras." Alan Kay
  • But wait! Its not over yet!"Arrogance in computer science is measured in nano-Dijkstras." Alan Kay "… and micro-Kays". Unknown source ;-)
  • Dynamic Testing isRunning the tested software – in different environment – with different datain an attempt to – Certify conformance – Prove program correctness (requires formal proof) – Prove program incorrectness (practically)
  • Static testing isAnalysis of artifacts – source code – binaries – data filesin an attempt to – discover errors – identify suspicious patterns – verify conformance of the artifacts
  • Static testing includes● Using static analyzers – Big number of false positives● Code reviews – Tedious manual work – Many errors missed – Non formalizable
  • What defects could by found by dynamic testing Any defect! You just need to invent enough test :) only ...It may take an indefinite amount of tests So, the testing is, effectively, endless
  • What defects could by found by static testing Any defect!You just need to look on the whole source long enough only ...You can not know which ones you are detecting and You never know how many are left
  • What defects are hard to find by dynamic testing● Intermittent problems – You may just missed it● Platform/environment specific problem – You just may not have the environment
  • What defects are hard to find by static analysis● Bugs in deep and wide class inheritance – Virtual methods are resolved in runtime● Bugs in modular system – Many modules implement the same features, modules are registered somewhere, etc. – Same reason – modules are registered as runtime
  • Formal verification isFormal verification is the act of proving ordisproving the correctness of intended algorithmsunderlying a system with respect to a certainformal specification or property, usingformal methods of mathematics.
  • Formal verification vs Testing is also● Testing – Upper bound for program quality ● Passed test says nothing about quality ● What matters is when test fails● Formal verification – Lower bound for program quality ● Passed test guarantees absence of some type of failures in a program
  • Formal verification requires● Correctness of – Language – Compiler – “Core” of the program● The specification is self-consistent
  • Formal verification appliedboolean isPowerOfTwo(int a) { return (a&(a-1)) == 0;}
  • Formal verification applied n∀0< a∈ N : a &(a−1)=0⇔∃ n∈ N : a=2a > 0 => binary presentation of a has a least one 1 bit m >= 0 Lets take a binary representation of a: (a1…ak)10....0 m m a-1 = (a1…ak)01....1 => a&(a-1) = (a1…ak)00....0 a&(a-1) = 0 => a1,...,ak = 0 => a = 2m a = 2n => m=n, a1,...,ak = 0 => a&(a-1) = 0
  • Formal verification isFormal verification is the act of proving ordisproving the correctness of intended algorithmsunderlying a system with respect to a certainformal specification or property, usingformal methods of mathematics.Another approach is deductive verification. Itconsists of generating from the system and itsspecifications (and possibly other annotations) acollection of mathematical proof obligations,the truth of which imply conformance of thesystem to its specification.
  • Deductive Verification Theorem proving● Four color theorem (proved in 1976)● Curry-Howard isomorphism – (Theorem, Proof) <=> (Type, Program)● Theorem provers – Interactive environments for constructing proofs – Coq, Agda, Isabelle, HOL● Real-world example – COMPCERT: C Verified Compiler
  • Using tools how about ...● We create a program – Is capable of proving something about another program – Is itself proven (yeah, yeah, a recursion)● Use the program to prove something about another program● Lets call it a “prover” Is this still a formal verification? Sure!
  • Formal verification compiler is a prover on its own● Formal verification for Java is performed by Java compiler – Types – Uninitialized variable – Missing of return statement – Uncaught exceptions – etc. – etc.
  • Annotations in Java@Stateless @LocalBeanpublic class GalleryFacade { @EJB private GalleryEAO galleryEAO; @TransactionAttribute(SUPPORTS) public Gallery findById(Long id) { ... } @TransactionAttribute(REQUIRED) public void create(String name) { … }
  • Annotations in Java● Introduced in Java 5● Metadata● May be reflective – SOURCE, CLASS, RUNTIME● Standard (e.g. @Override) & custom annotations● Extensively used nowadays – JavaEE 6, IoC containers, test harnesses, etc
  • Annotations: pre-Java 8● Allowed on declarations only – Class declaration @A public class Test { @B private int a = 0; @C public void m(@D Object o) { @E int a = 1; ... } }
  • Annotations: pre-Java 8● Allowed on declarations only – Field declaration @A public class Test { @B private int a = 0; @C public void m(@D Object o) { @E int a = 1; ... } }
  • Annotations: pre-Java 8● Allowed on declarations only – Method declaration @A public class Test { @B private int a = 0; @C public void m(@D Object o) { @E int a = 1; ... } }
  • Annotations: pre-Java 8● Allowed on declarations only – Method parameter declaration @A public class Test { @B private int a = 0; @C public void m(@D Object o) { @E int a = 1; ... } }
  • Annotations: pre-Java 8● Allowed on declarations only – Local variable declaration @A public class Test { @B private int a = 0; @C public void m(@D Object o) { @E int a = 1; ... } }
  • Limitations● Consider @NonNull annotation● How to declare a Map with non-null keys and values?
  • Limitations● Consider @NonNull annotation● How to declare a Map with non-null keys and values? @NonNull Map<K,V>?
  • Limitations● Consider @NonNull annotation● How to declare a Map with non-null keys and values? @NonNull Map<K,V>? NO!
  • Limitations● Consider @NonNull annotation● How to declare a Map with non-null keys and values? @NonNull Map<K,V>? NO!● Map<@NonNull K, @NonNull V> … but incorrect in Java 7 and earlier Type annotations in Java 8 for the rescue!
  • Type annotations in Java 8● Allowed anywhere you would write a type … including generics and casts … for array levels and receivers
  • Type annotations in Java 8: Examples● Class inheritance class UnmodifiableList<T> implements @ReadOnly List<T> { ... }● Casts myDate = (@ReadOnly Date) roDate;● Type tests myString instanceof @NonNull String;● Arrays String @NonNull [] messages;
  • Type annotations in Java 8: Examples● Generics List<@Interned String> messages;● Type parameter bounds Collection<? super @Exists File>● Generic type arguments in a generic method o.<@NonNull String>toString("...");
  • Pluggable types● User-defined (pluggable) type system● Extend built-in type system – express extra information about types via type qualifiers● Permit more expressive compile-time checking and guarantee the absence of additional errors
  • Checker Framework● Collection of compiler plugins (“checkers”)● Relies on Pluggable types and Type Annotations in Java 8● Find different sorts of bugs or verify their absence – 14 checkers are already provided● Supports custom compiler plugins (provides API) – 5 third-party checkers
  • Example: Nullness Checker● Annotations – @NonNull – @Nullable● Partial type hierarchy
  • Example: Nullness Checker● Annotations – @NonNull – @Nullable● Example: @Nullable Object o1; // might be null @NonNull Object o2; // never null o1.toString(); // warning o2 = o1; // warning if (o2 == null) // warning: redundant test
  • Example: Nullness Checker● Annotations – @NonNull – @Nullable● Example: public <@NonNull T> T process(T);
  • Example: Tainting Checker● Use case: – Trusted vs untrusted data – Verify before use● Examples – SQL injection attack ● validate SQL query before executing it – information leakage ● secret data vs data displayed to a user
  • Example: Tainting Checker● Annotations – @Untainted ● A type that includes only untainted, trusted values – @Tainted ● A type that includes only tainted, untrusted values
  • Example: Tainting Checker● Annotations – @Untainted – @Tainted● Example void execute(@Untainted String sql) throws SQLException; @Untainted String validate(@Tainted String) throws SQLException;
  • Credit card number Annotation@Documented@Retention(RetentionPolicy.RUNTIME)@Target({ElementType.TYPE_USE, ElementType.TYPE_PARAMETER})@TypeQualifier@SubtypeOf(Unqualified.class)public @interface CreditCard {}
  • Credit card number Checker@TypeQualifiers(CreditCard.class)@SuppressWarningsKey("credit.card")public class CreditCardChecker extends BaseTypeChecker {…}
  • Credit card number Usagepublic class Account { private final @CreditCard String cardNumber; public Account(@CreditCard String number) { this.cardNumber = number; } public @CreditCard String getCardNumber() { return cardNumber; }
  • Credit card number Sources@SuppressWarnings("credit.card")@CreditCard String convert(String input) { if(checkLuhn(input)) return input; else throw IllegalArgumentException("...")}new Account("4111111111111111");new Account("4111111111111110");
  • Credit card number Conclusion● A card number in an account is always validated● That is guaranteed at compile time● You do not need to test with invalid numbers● You do need to test – All @SuppressWarnings("credit.card") – checkLuhn(String cardNum)● Better all … prove it!
  • More real life examplesString getProperty(@PropertyKey String key);HashMap <@Adult Person, @NonNull Address> findSobutylnik(@NonNull Location);void monitorTemperature() throws @Critical TemperatureException;
  • Checkers Framework: Advanced features● Linear checker – Implements linear types (based on linear logic) – control aliasing and prevent re-use – Single ownership abstraction ● Prevents absence of ownership and multiple owners● Dependent types – @Dependent annotation – Changes the type depending on qualified type of the receiver (this) – Example List[N] – list with its length encoded into its type
  • How to start using● No need to wait Java 8 release – modified compiler already available● Incremental program annotation – Partial program checking – Warnings during compilation – Easily convertible into compilation errors ● -Werror flag to javac – Default annotations for types w/o annotations● Ability to annotate external libraries
  • Links● Type Annotations Specification (JSR-308) http://types.cs.washington.edu/jsr308/specification/java-● Checker Framework http://types.cs.washington.edu/checker-framework/curre
  • Q&A
  • Владимир Ивановvladimir.x.ivanov@oracle.com Александр Ильин alexandre.iline@oracle.com
  • Алло, мы ищем таланты! Приходите к нам работать! alexandre.iline@oracle.co m