19. Extending JPF - SearchListener public interface SearchListener { /* got the next state */ void stateAdvanced (Search search); /* state was backtracked one step */ void stateBacktracked (Search search); /* a previously generated state was restored (can be on a completely different path) */ void stateRestored (Search search); /* JPF encountered a property violation */ void propertyViolated (Search search); /* we get this after we enter the search loop, but BEFORE the first forward */ void searchStarted (Search search); /* there was some contraint hit in the search, we back out could have been turned into a property, but usually is an attribute of the search, not the application */ void searchConstraintHit (Search search); /* we're done, either with or without a preceeding error */ void searchFinished (Search search); }
20. Extending JPF - VMListener public interface VMListener { void instructionExecuted (JVM vm); // VM has executed next instruction void threadStarted (JVM vm); / / new Thread entered run() method void threadTerminated (JVM vm); // Thread exited run() method void classLoaded (JVM vm); // new class was loaded void objectCreated (JVM vm); // new object was created void objectReleased (JVM vm); // object was garbage collected void gcBegin (JVM vm); // garbage collection mark phase started void gcEnd (JVM vm); // garbage collection sweep phase terminated void exceptionThrown (JVM vm); // exception was thrown void nextChoice (JVM vm); // choice generator returned new value }
21. Extending JPF - Listener Example public class HeapTracker extends GenericProperty implements VMListener , SearchListener { class PathStat { .. int heapSize = 0; .. } // helper to store additional state info PathStat stat = new PathStat(); Stack pathStats = new Stack(); public boolean check (JVM vm, Object arg) { // GenericProperty return (stat.heapSize <= maxHeapSizeLimit); } public void stateAdvanced (Search search) { // SearchListener if (search.isNewState()) {.. pathStats.push(stat); stat = (PathStat)stat.clone(); .. } public void stateBacktracked (Search search) { // SearchListener .. if (!pathStats.isEmpty()) stat = (PathStat) pathStats.pop(); } public void objectCreated (JVM vm) {.. // VMListener ElementInfo ei = vm.getLastElementInfo(); ..stat.heapSize += ei.getHeapSize(); .. } public void objectReleased (JVM vm) { // VMListener ElementInfo ei = vm.getLastElementInfo(); ..stat.heapSize -= ei.getHeapSize(); .. } ... }
32. JPF Perspective State consists of 2 main components, the state of the JVM and the current and next choice Generator (i.e. the objects encapsulating the choice enumeration that produces new transitions) Transition is the sequence of instructions that leads from one state. There is no context within a transition, it's all in the same thread. There can be multiple transitions leading out of one state Choice is what starts a new transition. This can be a different thread, i.e. scheduling choice, or different “random” data value.
33. Role of Choices In other words, possible existence of Choices is what terminates the last Transition, and selection of a Choice value precedes the next Transition. The first condition corresponds to creating a new ChoiceGenerator, and letting the SystemState know about it. The second condition means to query the next choice value from this ChoiceGenerator (either internally within the JVM, or in an instruction or native method).
45. Concrete Execution Path (example) x = 1, y = 0 1 >? 0 x = 1 + 0 = 1 y = 1 – 0 = 1 x = 1 – 1 = 0 0 – 1 >? 0 int x, y; if (x > y) { x = x + y; y = x – y; x = x – y; if (x – y > 0) assert(false); }
46. Symbolic Execution Tree (example) x = X, y = Y int x, y; if (x > y) { x = x + y; y = x – y; x = x – y; if (x – y > 0) assert(false); } X >? Y [ X > Y ] y = X + Y – Y = X [ X > Y ] x = X + Y – X = Y [ X > Y ] Y - X >? 0 [ X <= Y ] END [ X > Y ] x = X + Y [ X > Y, Y – X <= 0 ] END [ X > Y, Y – X > 0 ] END
52. Algorithm (illustration) consider executing next = t.next; E0 next E1 next t null t E0 next E1 next ? next E0 next E1 t next E0 next E1 next t E0 next E1 next t Precondition: acyclic list E0 E1 next t null next t E0 E1 next ? next next
53. Implementation via Instrumentation program instrumentation counterexample(s)/test suite [heap+constraint+thread scheduling] model checking decision procedure instrumented program correctness specification continue/ backtrack state: original program path condition (data) heap configuration thread scheduling
54.
55. Red-Black Trees (1) The root is BLACK (2) Red nodes can only have black children (3) All paths from a node to its leaves contain the same number of black nodes. Self-balancing Binary Search Trees Java TreeMap Implementation (4) Acyclic (5) Consistent Parents repOk(): conditions (1)-(5)
56. repOk() Fragment boolean repOk(Entry e) { // root has no parent, root is black,… // RedHasOnlyBlackChildren workList = new LinkedList(); workList.add(e); while (!workList.isEmpty()) { Entry current=(Entry)workList.removeFirst(); Entry cl = current.left; Entry cr = current.right; if (current.color == RED) { if(cl != null && cl.color == RED) return false; if(cr != null && cr.color == RED) return false; } if (cl != null) workList.add(cl); if (cr != null) workList.add(cr); } // equal number of black nodes on left and right sub-tree… return true; }
57.
58. Symbolic Execution of repOk() Example public static boolean repOk() { if (root == null ) return true ; if (root.color == RED) return false; … Size 1
59.
60. repOk() x 2 abstract and concrete Symbolic Execution of Code During Lazy Initialization check Abstract repOk() When coverage is achieved, solve the symbolic constraints to create concrete inputs Concretize inputs by symbolic execution of Concrete repOk() over symbolic structures - as with Black-box TIG - Abstract repOk() : Symbolic Structure {true,false,don’t know} Concrete repOk() : Symbolic Structure Concrete Structure
63. Symbolic Execution for white-box TIG if (p.left != null && p.right != null ) { ... Symbolic structure before executing branch Concretize Concrete structure that will cover the code The symbolic structure is used as input to repOk() and lazily executed to obtain the concrete structure Symbolic structure(s) that cover the branch This structure “passes” the abstract repOk()
64. API Based Testing SUT ENV (m,n) m is the seq. length of API calls & n is the number of values used in the parameters of the calls API … put(v) del(v) Evaluate different techniques for selecting test-cases from ENV(m,n) to obtain maximum coverage
65. Framework SUT with minor instrumentation ENV TestListener Abstraction Mapping + State Storage Coverage Manager JPF
66. Environment Skeleton M : sequence length N : parameter values A : abstraction used for (int i = 0; i < M; i++) { int x = Verify.random(N - 1); switch (Verify.random(1)) { case 0: put(x); break; case 1: remove(x); break; } } Verify.ignoreIf(checkStateMatch());
67. Symbolic Environment Skeleton M : sequence length A : abstraction used for (int i = 0; i < M; i++) { SymbolicInteger x = new SymbolicInteger(“X“+i); switch (Verify.random(1)) { case 0: put(x); break; case 1: remove(x); break; } } Verify.ignoreIf(checkStateMatch());
68. Sample Output Test case number 77 for '15,L+R+P-REDroot ': put(0);put(4);put(5);put(1);put(2);put(3);remove(4); Unique ID for the test Branch Number Predicate Values Test-case to achieve above coverage Test case number 7 for '32,L-R-P+RED': X2 (0) == X1 (0) && X2 (0) < X0 (1) && X1 (0) < X0 (1) put( X0 );put( X1 );remove( X2 ); Test case number 7 for '32,L-R-P+RED': put(1);put(0);remove(0); Concrete Symbolic Path Condition with solutions Symbolic TC
81. State Matching in JPF VM State Matchable + Restorable Stored State (hashed) Matchable compression Collaborator Peter Dillinger Abstract State erase some parts of the state could be lossy Matchable
82.
83. New Architecture VM/Search DefaultBacktracker VM bool int[] Restorer Serializer Serialized StateSet int[] set VM
84. Old Scheme in the New Architecture VM/Search DefaultBacktracker FullStateSet int[] set bool int[] Collapsing (de)Serializer int[] VM
85. New Architecture VM/Search CollapsingRestorer FilteringSerializer DefaultBacktracker VM bool int[] objects This is the default setting for JPF at the moment: vm.backtracker.class = gov.nasa.jpf.jvm.DefaultBacktracker vm.restorer.class = gov.nasa.jpf.jvm.CollapsingRestorer vm.serializer.class = gov.nasa.jpf.filter.FilteringSerializer vm.storage.class = gov.nasa.jpf.jvm.JenkinsStateSet JenkinsStateSet int[] set VM
86.
87. New Architecture Revisited - Abstraction VM/Search CollapsingRestorer AbstractingSerializer DefaultBacktracker VM bool int[] objects This is the setting for the above configuration: vm.backtracker.class = gov.nasa.jpf.jvm.DefaultBacktracker vm.restorer.class = gov.nasa.jpf.jvm.CollapsingRestorer vm.serializer.class = gov.nasa.jpf.abstraction.abstractingSerializer vm.storage.class = gov.nasa.jpf.jvm.JenkinsStateSet JenkinsStateSet int[] set VM
88.
89.
90.
91.
Editor's Notes
To handle this case, we have to leave the ideal world of model checking (that considers all possible choices), and make use of what we know about the real world - we have to use heuristics to make the set of choices finite and manageable. However, heuristics are application and domain specific, and it would be a bad idea to hardcode them into the test drivers we give JPF to analyze. This leads to a number of requirements for the JPF choice mechanism: choice mechanisms have to be decoupled (i.e. thread choices should be indpendent of data choices, double choices from int choices etc.) choice sets and enumeration should be encapsulated in dedicated, type specific objects. The VM should only know about the most basic types, and otherwise use a generic interface to obtain choices selection of classes representing (domain specific) heuristics, and parametrization of ChoiceGenerator instances should be possible at runtime, i.e. via JPF's configuration mechanism (properties) The diagram shown above depicts this with an example that uses a &quot;randomly&quot; chosen velocity value of type double. As an example heuristic we use a threshold model, i.e. we want to know how the system reacts below, at, and above a certain application specific value (threshold). We reduce an infinite set of choices to only three &quot;interesting&quot; ones. Of course, &quot;interesting&quot; is quite subjective, and we probably want to play with the values (delta, threshold, or even used heuristic) efficiently, without having to rebuild the application each time we run JPF. The code example does not mention the used ChoiceGenerator class (DoubleThresholdGenerator) at all, it just specifies a symbolic name &quot;velocity&quot;, which JPF uses to look up an associated class name from its configuration data (initialized via property files or the command line - see Configuring JPF Runtime Options ). But it doesn't stop there. Most heuristics need further parameterization (e.g. threshold, delta), and we provide that by passing the JPF configuration data into the ChoiceGenerator constructors (e.g. the 'velocity.threshold' property). Each ChoiceGenerator instance knows its symbolic name (e.g. &quot;velocity&quot;), and can use this name to look up whatever parameters it needs.
Transition is the sequence of instructions that leads from one state to the next. There is no context switch within a transition, it's all in the same thread. There can be multiple transitions leading out of one state (but not