Finding Bugs Efficiently with a SAT Solver      Julian Dolby, Mandana Vaziri, Frank Tip      IBM Thomas J. Watson Research C...
Outline• Background• Miniatur Contributions• Example• Miniatur Evaluation• Related Work• Conclusions and Future Work
Background—Finding Bugs• Finding bugs must balance coverage and precision  – Tools must find bugs to be useful  – Tools mus...
Background—Finding Bugs• Testing approaches find real bugs  – Each test is an actual execution, so all bugs real  – Coverag...
Background—Finding Bugs• Conservative static analysis approaches find all bugs  – Conservative approximation of all executi...
Background—Systematic Underapproximation• Systematic underapproximation blends testing and analysis  – Explores all possib...
Background—Systematic Underapproximation• Key issue is choosing a “good” set of execution  – Set must be tractable  – Set ...
Background—Finding Bugs with SAT• Given program, specification, find concrete counter example• Use relational first-order log...
Miniatur Contributions• Supports much larger range of integers than previous work   – Integers represented by one atom per...
Example—HashMap Dataclass HashMap ... {  Entry[] table; ...         HashM ap ← {H1}  class Entry {              Array ← {A...
Example—HashMap Dataclass HashMap ... {  Entry[] table; ...         HashM ap ← {H1}  class Entry {              Array ← {A...
Example—HashMap.putpublic Object put(Object k, Object v) {  int h = k.hashCode();  int i = h % table.length;  for (Entry e...
Example—HashMap.putpublic Object put(Object k, Object v) {  int h = k.hashCode();  int i = h % table.length;  for (Entry e...
Example—HashMap.putpublic Object put(Object k, Object v) {  int h = k.hashCode();  int i = h % table.length;  Entry e = ta...
Example—HashMap.putpublic Object put(Object k, Object v) {  int h = k.hashCode();  int i = h % table.length;  Entry e = ta...
Example—HashMap.putpublic Object put(Object k, Object v) {  int h = k.hashCode();  int i = h % table.length;  Entry e = ta...
Expression for assert table[i]1.next == null             ¬ (Guard(A) → E[table[i].next == null])E[table[i].next == null]  ...
Expression for assert table[i]1.next == null             ¬ (Guard(A) → E[table[i].next == null])E[table[i].next == null]  ...
Expression for assert table[i]1.next == null             ¬ (Guard(A) → E[table[i].next == null])E[table[i].next == null]  ...
Expression for assert table[i]1.next == null             ¬ (Guard(A) → E[table[i].next == null])E[table[i].next == null]  ...
Expression for assert table[i]1.next == null             ¬ (Guard(A) → E[table[i].next == null])E[table[i].next == null]  ...
Expression for assert table[i]1.next == null             ¬ (Guard(A) → E[table[i].next == null])E[table[i].next == null]  ...
Expression for assert table[i]1.next == null             ¬ (Guard(A) → E[table[i].next == null])E[table[i].next == null]  ...
Expression for assert table[i]1.next == null             ¬ (Guard(A) → E[table[i].next == null])E[table[i].next == null]  ...
Guard for assert table[i]1.next == nullE[e2! = null] = F alse ∨ E[e1! = null] = F alse ∨ E[e! = null] = F alse
Evaluation• Miniatur tool embodies our techniques  – Uses WALA for control dependence and other analyses  – Uses Kodkod to...
Evaluation Results: Testing java.utilclass        formula                                     Time (s)   LinesLinkedList  ...
Evaluation Results: Testing Equals Contracts• Test each property of each equals method using harness  public static void e...
Related Work• Most-closely related work is checking Java with Alloy  – Vaziri et al, Taghdiri et al, Dennis et al  – Minia...
Conclusions and Future Work• Miniatur makes SAT-based checking for Java practical  – Extends prior work to better handle i...
Upcoming SlideShare
Loading in …5
×

Miniatur at FSE 2007

481 views
406 views

Published on

This is the presentation I gave about the Miniatur model checking for Java project at FSE 2007

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
481
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Miniatur at FSE 2007

  1. 1. Finding Bugs Efficiently with a SAT Solver Julian Dolby, Mandana Vaziri, Frank Tip IBM Thomas J. Watson Research Center FSE 2007—Dubrovnik, Croatia—September 6, 2007
  2. 2. Outline• Background• Miniatur Contributions• Example• Miniatur Evaluation• Related Work• Conclusions and Future Work
  3. 3. Background—Finding Bugs• Finding bugs must balance coverage and precision – Tools must find bugs to be useful – Tools must minimize false positives to be useful
  4. 4. Background—Finding Bugs• Testing approaches find real bugs – Each test is an actual execution, so all bugs real – Coverage depends upon test suite, so often spotty
  5. 5. Background—Finding Bugs• Conservative static analysis approaches find all bugs – Conservative approximation of all executions – Can find false positives due to approximation
  6. 6. Background—Systematic Underapproximation• Systematic underapproximation blends testing and analysis – Explores all possible concrete execution within a finite set – Real bugs and total coverage within for set of executions
  7. 7. Background—Systematic Underapproximation• Key issue is choosing a “good” set of execution – Set must be tractable – Set must cover interesting range of all executions• Use “small scope hypothesis” to bound set of executions – Hypothesis: most data structure bugs need only few objects – E.g. Collection bugs can be seen by inserting few elements – Explore executions with small heaps (also bound loops) (small scope) (other approximation)
  8. 8. Background—Finding Bugs with SAT• Given program, specification, find concrete counter example• Use relational first-order logic (FOL) (Alloy [Jackson et al]) – Universe of atoms, and bounded relations over atoms – Relational operators, first-order formulae over relations – Tool (Kodkod[Torlak et al]) translates to CNF for SAT solver• Java in Relational FOL [Vaziri et al] [Taghdiri et al] [Dennis et al] – Heap objects as atoms, types as unary relations – Fields as binary relations, code as formulae – Encode integer operations using bit sets – Solve FOL formula program ∧ ¬(spec)
  9. 9. Miniatur Contributions• Supports much larger range of integers than previous work – Integers represented by one atom per power of 2 – Enables e.g. use of hashCode in updating a collection• Novel sparse representation of array objects – Allows subset of array indices to have values – Enables Miniatur to handle array-based collections• Novel sliced translation based on control dependence• General theoretical condition for slicing relational formulae• Demonstration on real collections and a few real programs – Arrays, integers enable analyzing all Java collections – Integers allow checking Java equality contracts
  10. 10. Example—HashMap Dataclass HashMap ... { Entry[] table; ... HashM ap ← {H1} class Entry { Array ← {A1} Object key; Entry ← {E1} Object value; Object ← {A1, E1, H1, K1, V 1} int hash; table ← { H1, A1 } Entry next;...}} key ← { E1, K1 } value ← { E1, V 1 } hash ← { E1, #65 } I ← { A1, N 1, #5 } V ← { A1, N 1, E1 }
  11. 11. Example—HashMap Dataclass HashMap ... { Entry[] table; ... HashM ap ← {H1} class Entry { Array ← {A1} Object key; Entry ← {E1} Object value; Object ← {A1, E1, H1, K1, V 1} int hash; table ← { H1, A1 } Entry next;...}} key ← { E1, K1 } value ← { E1, V 1 } hash ← { E1, #1 , E1, #64 } I ← { A1, N 1, #1 , A1, N 1, #4 } V ← { A1, N 1, E1 }
  12. 12. Example—HashMap.putpublic Object put(Object k, Object v) { int h = k.hashCode(); int i = h % table.length; for (Entry e = table[i]; e != null; e = e.next) { ... } modCount++; table[i] = new Entry(h, k, v, table[i]); return null;
  13. 13. Example—HashMap.putpublic Object put(Object k, Object v) { int h = k.hashCode(); int i = h % table.length; for (Entry e = table[i]; e != null; e = e.next) { ... } modCount++; table[i] = new Entry(h, k, v, table[i]); return null;
  14. 14. Example—HashMap.putpublic Object put(Object k, Object v) { int h = k.hashCode(); int i = h % table.length; Entry e = table[i]; if (e != null) { ... e = e.next if (e != null) { ... e = e.next if (e != null) goto end; } } modCount++; table[i] = new Entry(h, k, v, table[i]); return null;
  15. 15. Example—HashMap.putpublic Object put(Object k, Object v) { int h = k.hashCode(); int i = h % table.length; Entry e = table[i]0; if (e != null) { ... e1 = e.next; if (e1 != null) { ... e2 = e1.next; if (e2 != null) goto end; } } modCount1 = modCount0 + 1; table[i]1 = new Entry(h, k, v, table[i]0); return null;
  16. 16. Example—HashMap.putpublic Object put(Object k, Object v) { int h = k.hashCode(); int i = h % table.length; Entry e = table[i]0; if (e != null) { ... e1 = e.next; if (e1 != null) { ... e2 = e1.next; if (e2 != null) goto end; } } modCount1 = modCount0 + 1; table[i]1 = new Entry(h, k, v, table[i]0); A: assert table[i]1.next == null; // perfect hashing return null;
  17. 17. Expression for assert table[i]1.next == null ¬ (Guard(A) → E[table[i].next == null])E[table[i].next == null] ← E[table[i].next] = {N ull}E[table[i].next] ← E[table[i]].E[next] E[table[i]] ← {E[table].x.E[V 1] |sum(E[table].x.E[I1]) = E[i] } ... sum(x) ← xi ∈x int(xi ) int(j) ← int(#1) ← 1, int(#2) ← 2, int(#4) ← 4, . . . table ← {A1} , next ← { E1, N ull } T 1 ← { A1, N 1 } , i ← 5 I1 ← { A1, N 1, #1 , A1, N 1, #4 } V 1 ← { A1, N 1, E1 }
  18. 18. Expression for assert table[i]1.next == null ¬ (Guard(A) → E[table[i].next == null])E[table[i].next == null] ← E[table[i].next] = {N ull}E[table[i].next] ← E[table[i]].E[next] E[table[i]] ← {E[table].x.E[V 1] |sum(E[table].x.E[I1]) = E[i] } ... sum(x) ← xi ∈x int(xi ) int(j) ← int(#1) ← 1, int(#2) ← 2, int(#4) ← 4, . . . table ← {A1}, next ← { E1, N ull } T 1 ← { A1, N 1 } , i ← 5 I1 ← { A1, N 1, #1 , A1, N 1, #4 } V 1 ← { A1, N 1, E1 }
  19. 19. Expression for assert table[i]1.next == null ¬ (Guard(A) → E[table[i].next == null])E[table[i].next == null] ← E[table[i].next] = {N ull}E[table[i].next] ← E[table[i]].E[next] E[table[i]] ← {{A1} .x.E[V 1] |sum({A1} .x.E[I1]) = 5 } ... sum(x) ← xi ∈x int(xi ) int(j) ← int(#1) ← 1, int(#2) ← 2, int(#4) ← 4, . . . table ← {A1} , next ← { E1, N ull } T 1 ← { A1, N 1 } , i ← 5 I1 ← { A1, N 1, #1 , A1, N 1, #4 } V 1 ← { A1, N 1, E1 }
  20. 20. Expression for assert table[i]1.next == null ¬ (Guard(A) → E[table[i].next == null])E[table[i].next == null] ← E[table[i].next] = {N ull}E[table[i].next] ← E[table[i]].E[next] E[table[i]] ← {{A1} . {N 1} .E[V 1] |sum({A1} . {N 1} .E[I1]) = 5 } ... sum(x) ← xi ∈x int(xi ) int(j) ← int(#1) ← 1, int(#2) ← 2, int(#4) ← 4, . . . table ← {A1} , next ← { E1, N ull } T 1 ← { A1, N 1 } , i ← 5 I1 ← { A1, N 1, #1 , A1, N 1, #4 } V 1 ← { A1, N 1, E1 }
  21. 21. Expression for assert table[i]1.next == null ¬ (Guard(A) → E[table[i].next == null])E[table[i].next == null] ← E[table[i].next] = {N ull}E[table[i].next] ← E[table[i]].E[next] E[table[i]] ← {{A1} . {N 1} .E[V 1] |sum({#1, #4}) = 5 } ... sum(x) ← xi ∈x int(xi ) int(j) ← int(#1) ← 1, int(#2) ← 2, int(#4) ← 4, . . . table ← {A1} , next ← { E1, N ull } T 1 ← { A1, N 1 } , i ← 5 I1 ← { A1, N 1, #1 , A1, N 1, #4 } V 1 ← { A1, N 1, E1 }
  22. 22. Expression for assert table[i]1.next == null ¬ (Guard(A) → E[table[i].next == null])E[table[i].next == null] ← E[table[i].next] = {N ull}E[table[i].next] ← E[table[i]].E[next] E[table[i]] ← {{A1} . {N 1} .E[V 1] |5 = 5 } ... sum(x) ← xi ∈x int(xi ) int(j) ← int(#1) ← 1, int(#2) ← 2, int(#4) ← 4, . . . table ← {A1} , next ← { E1, N ull } T 1 ← { A1, N 1 } , i ← 5 I1 ← { A1, N 1, #1 , A1, N 1, #4 } V 1 ← { A1, N 1, E1 }
  23. 23. Expression for assert table[i]1.next == null ¬ (Guard(A) → E[table[i].next == null])E[table[i].next == null] ← E[table[i].next] = {N ull}E[table[i].next] ← E[table[i]].E[next] E[table[i]] ← {E1} ... sum(x) ← xi ∈x int(xi ) int(j) ← int(#1) ← 1, int(#2) ← 2, int(#4) ← 4, . . . table ← {A1} , next ← { E1, N ull } T 1 ← { A1, N 1 } , i ← 5 I1 ← { A1, N 1, #1 , A1, N 1, #4 } V 1 ← { A1, N 1, E1 }
  24. 24. Expression for assert table[i]1.next == null ¬ (Guard(A) → E[table[i].next == null])E[table[i].next == null] ← E[table[i].next] = {N ull}E[table[i].next] ← {N ull} E[table[i]] ← {E1} ... sum(x) ← xi ∈x int(xi ) int(j) ← int(#1) ← 1, int(#2) ← 2, int(#4) ← 4, . . . table ← {A1} , next ← { E1, N ull } T 1 ← { A1, N 1 } , i ← 5 I1 ← { A1, N 1, #1 , A1, N 1, #4 } V 1 ← { A1, N 1, E1 }
  25. 25. Guard for assert table[i]1.next == nullE[e2! = null] = F alse ∨ E[e1! = null] = F alse ∨ E[e! = null] = F alse
  26. 26. Evaluation• Miniatur tool embodies our techniques – Uses WALA for control dependence and other analyses – Uses Kodkod to generate CNF – Uses Minisat as backend SAT solver• Evaluated structural assertions in java.util collections – Check that size fields accurately reflect data structure – Use driver that inserts arbitrary objects into collection• Evaluated java equality contracts on open-source codes – (1) reflexive, (2) symmetric, (3) transitive, (4) non-null, – (5) hashCode, (6-9) compareTo properies
  27. 27. Evaluation Results: Testing java.utilclass formula Time (s) LinesLinkedList this.size = |this.header.next∗ − null| 8.2 111TreeMap this.size = 67.4 12592 |count(this.root.(lef t + right)∗ − null|TreeSet this.m.size = 80.0 12667 |this.m.root.(lef t + right)∗ − null|HashMap this.size = |this.table[].next∗ − null| 30.0 1107HashSet this.map.size = 34.8 1129 |(this.map.table[].next∗ − null|• Encode correctness of size fields• Check expressive properties against complex collections• Heap size per type, Loop unrolling
  28. 28. Evaluation Results: Testing Equals Contracts• Test each property of each equals method using harness public static void equalsTester(Object a, Object b) { if (a.equals(b)) assert b.equals(a); }• Evaluated several common, open-source benchmarks – Antlr, BCEL, Hsqldb, java cup – Found 20 concrete violations (of 2, 3, 5, 6, 7) – Most tests ran within 2 minutes• Bugs illustrated by concrete counter-examples argument a argument b ArrayType3 ArrayType2 type=54304 type=9181 basic type=UninitializedObjectType3 basic type=anonType3 ... ...
  29. 29. Related Work• Most-closely related work is checking Java with Alloy – Vaziri et al, Taghdiri et al, Dennis et al – Miniatur handles more of Java, scales better• SATURN checks C using SAT – Uses manually-tailored summaries – Checks less-rich properties – Better scaling than Miniatur• SMT solvers – Mix SAT with theories for arithmetic, arrays, etc – Can prove properties within particular theories – Current SMT solvers less expressive
  30. 30. Conclusions and Future Work• Miniatur makes SAT-based checking for Java practical – Extends prior work to better handle integers, arrays – New, sound slicing mechanism for scalability → Handles real properties on real programs → Deep structural properties on real data structures• Future work – Refinement-based approach to unrolling loops – Model concurrency

×