Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Abhik-Satish-dagstuhl

34,916 views

Published on

Talk given at Dagstuhl Seminar on Symbolic execution and Constraint Solving, October 27-30 2014

Published in: Education
  • Be the first to comment

Abhik-Satish-dagstuhl

  1. 1. Symbolic Techniques for Software Debugging Abhik Roychoudhury, National University of Singapore Satish Chandra, Samsung Research Dagstuhl, October 27-30 2014 1
  2. 2. Testing & Analysis – Abhik Roychoudhury •Interested in testing, debugging, repair, binary analysis •Using symbolic execution, dependency analysis, … Testing and Analysis – under Software Engineering umbrella Two past works on symbolic execution: DARWIN – FSE09 – Qi, Roychoudhury, Liang, Vaswani (joint work NUS + MSR-I) First work to use symbolic execution for debugging [regression errors] -> Use symbolic execution to glean intended behavior from past version Follow up works in FSE10, FSE11, ICSE13, FSE13, ISSTA13 – on symbolic analysis of changes. SEMFIX – ICSE13 – Nguyen, Qi, Roychoudhury, Chandra (joint work NUS + IBM) First work to use symbolic execution for program repair. -> Use symbolic execution to glean intended behavior from failed + passing tests. Dagstuhl, October 27-30 2014 2
  3. 3. Symbolic Analysis & Automated Debugging – Satish Chandra • Symbolic analysis with applications to bug finding and automated debugging PLDI 2009: A Powerful Approach to Weakest Preconditions – Chandra, Fink, Sridharan POPL 2010: Programming with Angelic Nondeterminism – Bodik et al. (joint work Berkeley) ICSE 2011 : Angelic Debugging – Chandra, Torlak, Barman and Bodik (joint work Berkeley) CAV 2012: Scalable Bug Detection using Alternate and Learn – Sinha, Singhania, Chandra ICSE 2013: SEMFIX – Nguyen, Qi, Roychoudhury, Chandra (joint work NUS) VSSTE 2014: What Gives? A Hybrid Algorithm for Error Explanation – Vijayaraghavan et al. Dagstuhl, October 27-30 2014 3
  4. 4. Path exploration based symbolic execution 4 input in; if (in >= 0) a = in; else a = -1; return a; input in; in >= 0 Keep both a = in; a = -1; return a in ==   ≥ 0  out ==  <0  out == -1 Yes No
  5. 5. Use of path conditions – (1) ESEC-FSE 2013 Tutorial, Russia 5 input x, y; a = 0; b = 0; if (x > y) a = x; else a = y; if (x + y > 10) b = a; return b; Passing inputs: Continue the search for failing inputs, those which do not go through the “same” path. Path condition of (x == 0, y == 0) x ≤ y  x + y ≤ 10 x == 0, y == 0  x > y a = x a = y x +y >10 b = a return b Cover more paths x ≤ y  x + y ≤ 10 x ≤ y   x + y ≤ 10  x ≤ y Directed D Automated A Random R Testing T
  6. 6. Use of path conditions – (2) in == 5 ESEC-FSE 2013 Tutorial, Russia 6 input in; z = 0; x = 0; if (in > 0){ z = in *2; x = in +2; x = x + 2; } else … if ( z > x){ return error; } in == 5 X Failing inputs: Which other inputs follow the “same” error path? Path condition of in == 5 in>0  (2*in > in +4)  in > 4 Program P Program P’  X Can help in Regression debugging DARWIN (later)
  7. 7. Outline of today’s briefing • Symbolic execution • Debugging • Sample symbolic debugging techniques – Regression errors [FSE09, 10, …] – Cause clue clauses [PLDI11] – Error invariants [FM12] – Angelic debugging [ICSE11] Dagstuhl, October 27-30 2014 7
  8. 8. A quote from many years ago “Even today, debugging remains very much of an art. Much of the computer science community has largely ignored the debugging problem….. over 50 percent of the problems resulted from the time and space chasm between symptom and root cause or inadequate debugging tools.” Hailpern & Santhanam, IBM Systems Journal, 41(1), 2002 Any progress in 2002 – 2014? How can symbolic execution help? Dagstuhl, October 27-30 2014 8
  9. 9. Debugging vs. Bug Hunting • Debugging – Have a problematic input i, or ``counter-example” trace. – Does not match expected output for i. – Not sure what desired ``property” is violated. – Amounts to implicitly alerting programmer about program’s intended specifications as well. • Bug Hunting via Model Checking / SE – Have a desired ``property” – Tries to find a counter-example trace, and hence an input which violates the property. 9
  10. 10. Debugging: comparing to the intended behavior Dagstuhl, October 27-30 2014 10 input = 0 P output = 0 Specification about observable output Test input What went wrong? What would have been right? Execution
  11. 11. Trace Comparison based Debugging Successful Run Pool Generate Input Failing Run Successful Run Compare Execution Choose Difference As Diagnostics Testing Change Failing Difference Metric Copyright 2011-12 by Abhik Roychoudhury 11
  12. 12. Example Input: a, index 1. base = a; 2. sentinel = base; 3. offset = index; 4. address = base + offset; 5. output address, sentinel Dagstuhl, October 27-30 2014 12 Test 1 <a, index==10> assert sentinel <= address assert address < a + 10 Test 2 <a, index==9> assert sentinel <= address assert address < a + 10
  13. 13. Fix determines fault Input: a, index 1. base = a; 2. sentinel = base; 3. offset = index; 4. address = base + offset; 5. output address, sentinel Off-by-one error Dagstuhl, October 27-30 2014 13 Input: a, index 1. base = a - 1; 2. sentinel = base; 3. offset = index; 4. address = base + offset; 5. output address, sentinel Input: a, index 1. base = a; 2. sentinel = base; 3. offset = index - 1; 4. address = base + offset; 5. output address, sentinel
  14. 14. What is the intended behavior? Only in the programmer’s mind? Assertions capturing programmer’s intent at each statement Too much overhead on programmer: almost as much work as a proof Source of Information Name of Symbolic Technique Internal inconsistency Cause Clue Clauses [PLDI 11] Error Invariants [FM 12] Passing Tests Angelic Debugging [ICSE 11] Previous version / Golden implementation Regression Debugging [FSE09, FSE10, FSE11] Dagstuhl, October 27-30 2014 14
  15. 15. Outline of today’s briefing • Introduction to debugging • Symbolic execution • Sample symbolic debugging techniques – Regression errors [FSE09, 10, …] – Cause clue clauses [PLDI11] – Error invariants [FM12] – Angelic debugging [ICSE11] Dagstuhl, October 27-30 2014 15
  16. 16. Regression debugging Test Input t Old Stable Program P New Buggy Program P’ Dagstuhl, October 27-30 2014 16
  17. 17. Trace Comparison? Root cause Compare failing test with a similar, successful test. Requirement: How do we find such an execution? Question : why ignore the evolution? Dagstuhl, October 27-30 2014 17
  18. 18. Adapting trace comparison Directly Compare σ and π Dagstuhl, October 27-30 2014 18 Old Stable Program P Test Input t New Buggy Program P’ Path σ for t Path π for t New Input t’
  19. 19. How to obtain the new test? • We have: – Two versions of the program. (P and P’). – A test t that fails on P’ but passes on P. • Key requirement: Similarity – Test t and t’ are similar if they induce • same control flow path in P but • different paths in P’. Old Pgm P New Pgm P’ t’ t t t’ Dagstuhl, October 27-30 2014 19
  20. 20. How to obtain the new test? The new test input Dagstuhl, October 27-30 2014 20 Old Pgm. P New Pgm. P’ Buggy input
  21. 21. DARWIN approach [FSE09] New Input t’ Path condition f Path condition f’ 1. Solve to get another input t’ 2. Compare π and π’ to get diagnostics Dagstuhl, October 27-30 2014 21 Old Stable Program P Test Input t New Buggy Program P’ Path σ for t Path π for t f f ' Path π’ for t’
  22. 22. Simple Example 0,-1,-2,.. 0,-1,-2,… 9 Dagstuhl, October 27-30 2014 22 int inp, outp; scanf("%d", &inp); if (inp >=1){ outp = g(inp); if (inp>9){ outp=g1(inp); } } else{ outp = h(inp); } printf("%d", outp); int inp, outp; scanf("%d", &inp); if (inp >= 1){ outp = g(inp); /* if (inp>9){ outp=g1(inp); } */ } else{ outp = h(inp); } printf("%d", outp); 1,2,..,9 10,11,… 1,2,…,9, 10,11,… Explain inp == 100 using ??
  23. 23. Path condition f (inp >= 1)&& (inp>9) Path condition f’ (inp >= 1) inp==100 f 'f  (inp 1)&&(inp 9) STP Solver inp==9 int inp, outp; scanf("%d", &inp); if (inp >=1){ outp = g(inp); if (inp>9){ outp=g1(inp); } } else{ outp = h(inp); } printf("%d", outp); int inp, outp; scanf("%d", &inp); if (inp >= 1){ outp = g(inp); /* if (inp>9){ outp=g1(inp); } */ } else{ outp = h(inp); } printf("%d", outp); f f ' (inp  9)&&(inp 1) STP Solver No soln. Dagstuhl, October 27-30 2014 23
  24. 24. Generating new input Solve f f ' Check for satisfiability of Dagstuhl, October 27-30 2014 24 b1 b3 b6 b2 b4 b5 1  1  2  3  4  5   2 3   ' ( ... ) 1 2 m f      1 f  1 2 f   1 2 3 f     f ' At most m alternate inputs !! Remove trace comparison altogether 
  25. 25. DARWIN approach [FSE 09] Dagstuhl, October 27-30 2014 25 Old Stable Program P f:Path condition of t in P Test Input t New Buggy Program P’ Alternative Input t’ Concrete and Symbolic Execution STP Solver and input validation Satisfiable sub-formulae from f / f’ f':Path condition of t in P’ f f ' Diagnostics (Assembly level) Diagnostics (Source level)
  26. 26. Specification discovery • Given a reference implementation – Symbolic execution extracts specification of intended behaviour for a failed test t. – Can generate • Another input / class of inputs whose behaviour captures the intended behaviour of t [FSE09, FSE11] • Or, a direct semantic differencing of the behavior of t in two program version [FSE10] – Scalability + Precise diagnostics (~5min, ~5 LOC) • Program versions e.g. libPNG • Embedded SW – Busybox vs. Coreutils • Protocol impl. – e.g. Webservers – miniweb vs. Apache. Dagstuhl, October 27-30 2014 26
  27. 27. Outline of today’s briefing • Introduction to debugging • Symbolic execution • Sample symbolic debugging techniques – Regression errors [FSE09, 10, …] – Cause clue clauses [PLDI11] – Error invariants [FM12] – Angelic debugging [ICSE11] Dagstuhl, October 27-30 2014 27
  28. 28. Example - Revisited Input: a, index 1. base = a; 2. sentinel = base; 3. offset = index; 4. address = base + offset; 5. output address, sentinel Dagstuhl, October 27-30 2014 28 Test 1 <a, index==10> assert sentinel <= address assert address < a + 10 Test 2 <a, index==9> assert sentinel <= address assert address < a + 10
  29. 29. CCC : General idea Input: a, index 1. base = a; 2. sentinel = base; 3. offset = index; 4. address = base + offset; 5. assert sentinel <= address  address < a + 10 == false Dagstuhl, October 27-30 2014 29 <a, index == 10>  Failing input  Program formula  Observed Output == false
  30. 30. CCC: General idea Input: a, index 1. base = a; 2. sentinel = base; 3. offset = index; 4. address = base + <a, index == 10>   offset; 5. assert sentinel <= address address < a + 10 == false index== 10  base == a  sentinel == base  offset == index  address == base + offset  sentinel <= address  address < a + 10 == false Dagstuhl, October 27-30 2014 30 Hard constraint Soft constraint Hard constraint
  31. 31. First iteration index== 10  base == a  sentinel == base  offset == index  address == base + offset  address < a + 10 == false Dagstuhl, October 27-30 2014 31 Hard constraint Soft constraint Hard constraint Running Partial MAXSAT, we get base == a as a soft constraint that can be removed. Corresponds to the fix: Input: a, index 1. base = a - 1; 2. sentinel = base; 3. offset = index; 4. address = base + offset; 5. output address, sentinel
  32. 32. Moving further index== 10  base == a  sentinel == base  offset == index  address == base + offset  address < a + 10 == false Dagstuhl, October 27-30 2014 32 Hard constraint Hard & Soft constraints Hard constraint We mark base == a as hard now, and run Partial MaxSAT again, to get offset == index. Corresponds to the fix: Input: a, index 1. base = a; 2. sentinel = base; 3. offset = index - 1; 4. address = base + offset; The clause sentinel==base 5. output address, sentinel does not help (or hurt)
  33. 33. Usage and limitations • CoMSS: Complement of maximal satisfiable set – Set of soft clauses of minimum cardinality whose removal would make Φ satisfiable • MSS is maximal => CoMSS is minimal • CoMSS constitutes the suspect portion – Cause Clue Clauses -> Map clauses to program lines – Construct diagnostic information • Limitations – Does not deal with missing code errors. – Scalability: Program formula PF may blow up. – Might point to too many possible causes. Dagstuhl, October 27-30 2014 33
  34. 34. Specification discovery? • Find statements that cause inconsistency in the failing execution – Removal of that inconsistency makes the error go away • Minimal inconsistency  cause – Starting point for repair – Simple specification discovery • Removing statement S causes error to disappear • Do not know what S should have been! Dagstuhl, October 27-30 2014 34
  35. 35. Outline of today’s briefing • Introduction to Debugging • Symbolic Execution • Sample symbolic debugging techniques – Regression debugging [FSE09, FSE 10] – Cause Clue Clauses [PLDI 11] – Error invariants [FM 12] – Angelic debugging [ICSE 11] Dagstuhl, October 27-30 2014 35
  36. 36. Craig Interpolation Given: an unsatisfiable conjunction of formulas A  B A Craig interpolant for A  B is a formula I such that • A implies I • I  B is unsatisfiable • all free variables in I are shared between A and B A I B Dagstuhl, October 27-30 2014 36
  37. 37. Use interpolants for debugging Trace formula (not program formula) … … Position p X  Y  false Dagstuhl, October 27-30 2014 37 input valuation expected output X  Y Find interpolant at position p X  Ip  Y Helpful in understanding error propagation
  38. 38. Interpolants - example <a, index == 10> Input: a, index 1. base = a; 2. sentinel = base; 3. offset = index; 4. address = base + offset; 5. assert sentinel <= address assert address < a + 10 base = a, index = 10 base = a base = a, index = 10, sentinel = base base = a base = a, index = 10, sentinel = base, offset = 10 base = a  offset = 10 base = a, index = 10, sentinel = base, offset = 10, address = a + 10 address = a + 10 address = base + offset Interpolants contain just enough information to understand the failure. Dagstuhl, October 27-30 2014 38
  39. 39. Interpolants - example <a, index == 10> Input: a, index 1. base = a; 2. sentinel = base; 3. offset = index; 4. address = base + offset; 5. assert sentinel <= address assert address < a + 10 base = a base = a Unchanged! base = a  offset = 10 address = a + 10 address = a + 10 Interpolants need not be unique: for program paths, any formula between WP and SP Unchanged! Dagstuhl, October 27-30 2014 39
  40. 40. Interpolants - example Input: a, index 1. base = a; Dagstuhl, October 27-30 2014 40 2. 3. offset = index; 4. address = base + offset; 5. <a, index == 10> assert address < a + 10 base = a base = a  offset = 10 address = a + 10 Interpolants need not be unique: for program paths, any formula between WP and SP The trick is in finding interpolants cleverly!
  41. 41. Discussion • How do error invariants compare to CCC ? – The other side of the coin: looks at the “unsatisfiable core” in an error trace where as CCC finds a maximal satisfiable subprogram – Formal connection between “min core” and interpolation based invariants [VSSTE 2014] • Are error invariants good error explanations? – Will the unsatisfiable core be small enough to be useful? – Is trace minimization necessarily the best form of error explanation … human issue! Dagstuhl, October 27-30 2014 41
  42. 42. Outline of today’s briefing • Introduction to Debugging • Symbolic Execution • Sample symbolic debugging techniques – Regression Debugging [FSE09,FSE10] – Cause Clue Clauses [PLDI 11] – Error invariants [FM 12] – Angelic debugging [ICSE 11] Dagstuhl, October 27-30 2014 42
  43. 43. “Almost working” programs … Dagstuhl, October 27-30 2014 43
  44. 44. Debugging method Candidate defect locations Dagstuhl, October 27-30 2014 44 Buggy Program P Failing test t Debugging Method P Define possible defect locations by identifying expressions which can fix the fault!
  45. 45. General idea – fix failing tests,… Input: a, index 1. base = a; 2. sentinel = base; 3. offset = index; 4. address = base + <a, index == 10>   offset; 5. assert sentinel <= address assert address < a + 10 == false index== 10  base == ??  sentinel == base  offset == index  address == base + offset  sentinel <= address  address < a + 10 base = a is a valid fix location. Note: Does not suggest the repaired statement base = a – 1. Dagstuhl, October 27-30 2014 45
  46. 46. Fix failing tests, … Input: a, index 1. base = a; 2. sentinel = base; 3. offset = index; 4. address = base + <a, index == 10>   offset; 5. assert sentinel <= address assert address < a + 10 == false index== 10  base == a  sentinel == base  offset == ??  address == base + offset  sentinel <= address  address < a + 10 offset = index is another valid fix location. Dagstuhl, October 27-30 2014 46
  47. 47. …, and do not break passing tests Input: a, index 1. base = a; 2. sentinel = base; 3. offset = index; 4. address = base + <a, index == 1>   offset; 5. output address, sentinel assert sentinel == a Dagstuhl, October 27-30 2014 47 Input: a, index 1. base = ??; 2. sentinel = base; 3. offset = index; 4. address = base + offset; 5. output address, sentinel Input: a, index 1. base = a; 2. sentinel = base; 3. offset = ??; 4. address = base + offset; 5. output address, sentinel OK !! NEW
  48. 48. Is e a valid fix candidate? Find a candidate value replacement for e that makes the failing test pass For each passing test, find an alternate value for e that lets the test pass Dagstuhl, October 27-30 2014 48 Exists α .Test(P[α/e], If) Forall i. Exists α . ( Test(P[α/e], Ip_i) ∧ α ≠ Eval(P, Ip_i, e) )
  49. 49. Limitations • Assumption of 1-fixable – Try out different expressions too! • Quality of filtering depends on the goodness of test suite – Resemblance to mutation testing • Subject to implementation of the symbolic analysis Dagstuhl, October 27-30 2014 49
  50. 50. Specification discovery? • Passing tests tell us which expressions are “inflexible” – The better your test suite is, the more you know! • Therefore, the bug must be in one of the flexible expressions Dagstuhl, October 27-30 2014 50
  51. 51. Retrospective Debugging – some milestones – Manual era: prints and breakpoints – Statistical fault localization [e.g. Tarantula ] – Dynamic slicing [e.g. JSlice] – Trace comparison and delta debugging • Look for workarounds – how to avoid the error? – Symbolic techniques • Replace repeated experimentation with constraint solving. • Discover and (partially) infer intended semantics by symbolic analysis – Future: repair (hints) Dagstuhl, October 27-30 2014 51
  52. 52. Acknowledgements • Funding – NRF, MoE, DRTech Singapore • Co-authors – NUS: Zhenkai Liang, Dawei Qi, Ansuman Banerjee,… – MSRI: Kapil Vaswani – UCB: Ras Bodik, Emina Torlak, Shaon Barman • Rupak Majumdar helped clarify the CCC method • Thomas Weis generously shared one slide on error invariants. Dagstuhl, October 27-30 2014 52

×