Enabling and Supporting the Debugging of Software Failures (PhD Defense)

291 views
230 views

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
291
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Enabling and Supporting the Debugging of Software Failures (PhD Defense)

  1. 1. ENABLING AND SUPPORTINGTHE DEBUGGINGOF SOFTWARE FAILURESThesis DefenseJames Clause
  2. 2. DEFINITIONS‣ mistake: a human action thatproduces an incorrect result‣ fault: an incorrect step,process, or data definition ina computer program‣ failure: the inability of asystem or component toperform its requiredfunctions within specifiedrequirementsDebugging
  3. 3. DEBUGGING IS EXPENSIVE
  4. 4. • “...departments tend to spend about half of their applications staff time onmaintenance” – Lientz and Swanson, 1981DEBUGGING IS EXPENSIVE
  5. 5. • “...departments tend to spend about half of their applications staff time onmaintenance” – Lientz and Swanson, 1981• “Boehm, Brooks, Myers, andYourdon and Constantine indicate that testingand debugging alone represent approximately half the cost of new systemdevelopment.” –Vessey, 1985DEBUGGING IS EXPENSIVE
  6. 6. • “...departments tend to spend about half of their applications staff time onmaintenance” – Lientz and Swanson, 1981• “Boehm, Brooks, Myers, andYourdon and Constantine indicate that testingand debugging alone represent approximately half the cost of new systemdevelopment.” –Vessey, 1985• “According to an informal industry poll, 85 to 90 percent of the IS[Information Services] budget goes to legacy system operation andmaintenance.” – Erlikh, 2000DEBUGGING IS EXPENSIVE
  7. 7. • “...departments tend to spend about half of their applications staff time onmaintenance” – Lientz and Swanson, 1981• “Boehm, Brooks, Myers, andYourdon and Constantine indicate that testingand debugging alone represent approximately half the cost of new systemdevelopment.” –Vessey, 1985• “According to an informal industry poll, 85 to 90 percent of the IS[Information Services] budget goes to legacy system operation andmaintenance.” – Erlikh, 2000• “...the national annual costs of an inadequate infrastructure for softwaretesting is estimated to range from $22.2 to $59.5 billion” – NIST, 2002DEBUGGING IS EXPENSIVE
  8. 8. • “...departments tend to spend about half of their applications staff time onmaintenance” – Lientz and Swanson, 1981• “Boehm, Brooks, Myers, andYourdon and Constantine indicate that testingand debugging alone represent approximately half the cost of new systemdevelopment.” –Vessey, 1985• “According to an informal industry poll, 85 to 90 percent of the IS[Information Services] budget goes to legacy system operation andmaintenance.” – Erlikh, 2000• “...the national annual costs of an inadequate infrastructure for softwaretesting is estimated to range from $22.2 to $59.5 billion” – NIST, 2002• “24,191 people … were involved in either opening, handling, commentingon, or resolving WindowsVista bugs. That is an order of magnitude greaterthan the ∼2,000 developers who wrote code forVista” – Guo, 2010DEBUGGING IS EXPENSIVE
  9. 9. THESIS STATEMENTProgram analysis techniques can enable andsupport the debugging of failures in widely-usedapplications by:1) capturing, replaying, minimizing, and, as muchas possible, anonymizing failing executions2) highlighting subsets of failure-inducing inputsthat are likely to be helpful for debuggingsuch failures
  10. 10. TECHNICAL CONTRIBUTIONS
  11. 11. TECHNICAL CONTRIBUTIONSRecording andreplaying executions
  12. 12. TECHNICAL CONTRIBUTIONSRecording andreplaying executionsInput minimization✘
  13. 13. TECHNICAL CONTRIBUTIONSRecording andreplaying executionsInput anonymizationInput minimization✘
  14. 14. TECHNICAL CONTRIBUTIONSRecording andreplaying executionsInput anonymizationInput minimization✘Highlighting failure-relevant inputs
  15. 15. TECHNICAL CONTRIBUTIONSRecording andreplaying executionsInput anonymizationInput minimization✘Highlighting failure-relevant inputsEnable
  16. 16. TECHNICAL CONTRIBUTIONSRecording andreplaying executionsInput anonymizationInput minimization✘Highlighting failure-relevant inputsSupportEnable
  17. 17. TECHNICAL CONTRIBUTIONSRecording andreplaying executionsInput anonymizationInput minimization✘Highlighting failure-relevant inputsSupportEnable
  18. 18. MOTIVATION
  19. 19. MOTIVATION
  20. 20. MOTIVATION
  21. 21. MOTIVATION
  22. 22. MOTIVATIONFailures can be difficultto reproduce.
  23. 23. ENVIRONMENT INTERACTIONS
  24. 24. ENVIRONMENT INTERACTIONSStreams
  25. 25. ENVIRONMENT INTERACTIONSStreamsFiles
  26. 26. LIMITATIONSNot applicable in every situation
  27. 27. LIMITATIONS• May not be enough space to store accessed data• databases• long running executionsNot applicable in every situation
  28. 28. LIMITATIONS• May not be enough space to store accessed data• databases• long running executions• May have unacceptable runtime overhead• webservers, real-time applicationsNot applicable in every situation
  29. 29. LIMITATIONS• May not be enough space to store accessed data• databases• long running executions• May have unacceptable runtime overhead• webservers, real-time applicationsNot applicable in every situationEvaluation demonstrates that it can be usefulfor some common application types.
  30. 30. EVALUATIONAcceptableruntime overheadFailures reproducedsuccessfully
  31. 31. EVALUATIONPrototype implementation:• maps libc function calls tointeraction eventsSubjects:• several cpu intensive applications(e.g., bzip, gcc)Results:• negligible overheads• data size is acceptable• all failures successfully replayed
  32. 32. TECHNICAL CONTRIBUTIONSRecording andreplaying executionsInput anonymizationInput minimization✘Highlighting failure-relevant inputsSupportEnable
  33. 33. PRACTICALITY ISSUES
  34. 34. 345PRACTICALITY ISSUESLarge in size
  35. 35. 345345PRACTICALITY ISSUESLarge in size Contain sensitiveinformation
  36. 36. 345345PRACTICALITY ISSUESLarge in size Contain sensitiveinformationMinimize✘Highlight
  37. 37. 345345PRACTICALITY ISSUESLarge in size Contain sensitiveinformationAnonymizeMinimize✘Highlight
  38. 38. TECHNICAL CONTRIBUTIONSRecording andreplaying executionsInput anonymizationInput minimization✘Highlighting failure-relevant inputsSupportEnable
  39. 39. MINIMIZATION✘24:15
  40. 40. MINIMIZATION✘Timeminimization 2:5524:15
  41. 41. MINIMIZATION✘✂Dataminimization 2:55Timeminimization 2:5524:15
  42. 42. MINIMIZATION✘✂Dataminimization 2:55Timeminimization 2:5524:15Oracle Oracle
  43. 43. TIME MINIMIZATIONEvent log:Environment data (streams):KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NETWORK: {3405}<html><body>... ❙ {202}...FILE foo.1POLL KEYBOARD NOKPOLL KEYBOARD OKPULL KEYBOARD 5POLL NETWORK OKPULL NETWORK 1024FILE bar.1POLL NETWORK NOKPOLL NETWORK OKFILE foo.2...PULL NETWORK 1024FILE foo.2POLL KEYBOARD NOK...
  44. 44. TIME MINIMIZATIONEvent log:Environment data (streams):KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NETWORK: {3405}<html><body>... ❙ {202}...FILE foo.1POLL KEYBOARD NOKPOLL KEYBOARD OKPULL KEYBOARD 5POLL NETWORK OKPULL NETWORK 1024FILE bar.1POLL NETWORK NOKPOLL NETWORK OKFILE foo.2...PULL NETWORK 1024FILE foo.2POLL KEYBOARD NOK...Remove idle time
  45. 45. TIME MINIMIZATIONEvent log:Environment data (streams):KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NETWORK: {3405}<html><body>... ❙ {202}...FILE foo.1POLL KEYBOARD NOKPOLL KEYBOARD OKPULL KEYBOARD 5POLL NETWORK OKPULL NETWORK 1024FILE bar.1POLL NETWORK NOKPOLL NETWORK OKFILE foo.2...PULL NETWORK 1024FILE foo.2POLL KEYBOARD NOK...Remove idle time
  46. 46. TIME MINIMIZATIONEvent log:Environment data (streams):KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NETWORK: {3405}<html><body>... ❙ {202}...FILE foo.1POLL KEYBOARD NOKPOLL KEYBOARD OKPULL KEYBOARD 5POLL NETWORK OKPULL NETWORK 1024FILE bar.1POLL NETWORK NOKPOLL NETWORK OKFILE foo.2...PULL NETWORK 1024FILE foo.2POLL KEYBOARD NOK...Remove idle timeRemove delays
  47. 47. TIME MINIMIZATIONEvent log:Environment data (streams):KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NETWORK: {3405}<html><body>... ❙ {202}...FILE foo.1POLL KEYBOARD NOKPOLL KEYBOARD OKPULL KEYBOARD 5POLL NETWORK OKPULL NETWORK 1024FILE bar.1POLL NETWORK NOKPOLL NETWORK OKFILE foo.2...PULL NETWORK 1024FILE foo.2POLL KEYBOARD NOK...Remove idle timeRemove delays
  48. 48. DATA MINIMIZATIONEnvironment data (files):foo.1 foo.2 bar.1Whole entitiesChunksAtoms
  49. 49. DATA MINIMIZATIONEnvironment data (files):foo.2 bar.1Lorem ipsum dolor sitamet, consetetursadipscing elitr,seddiam nonumy eirmodtempor invidunt utlabore et dolore magna aliquyamerat, sed diam voluptua. Atveroeos et accusam et justo duodolores et ea rebum. Stet clitakasd gubergren, no sea takimatasanctus est Lorem ipsum dolorsit amet. Lorem ipsum dolor sitamet, consetetursadipscing elitr, sed diamnonumy eirmod tempor inviduntut labore et dolore magnaaliquyam erat, sed diamvoluptua. At vero eos etWhole entitiesChunksAtoms
  50. 50. DATA MINIMIZATIONEnvironment data (files):foo.2 bar.1Lorem ipsum dolor sitamet, consetetursadipscing elitr,seddiam nonumy eirmodtempor invidunt utlabore et dolore magna aliquyamerat, sed diam voluptua. Atveroeos et accusam et justo duodolores et ea rebum. Stet clitakasd gubergren, no sea takimatasanctus est Lorem ipsum dolorsit amet. Lorem ipsum dolor sitamet, consetetursadipscing elitr, sed diamnonumy eirmod tempor inviduntut labore et dolore magnaaliquyam erat, sed diamvoluptua. At vero eos etWhole entitiesChunksAtoms
  51. 51. DATA MINIMIZATIONEnvironment data (files):foo.2 bar.1Whole entitiesChunksAtoms
  52. 52. DATA MINIMIZATIONEnvironment data (files):bar.1Lorem ipsum dolor sitamet, consetetursadipscing elitr,seddiam nonumy eirmodtempor invidunt utlabore et dolore magna aliquyamerat, sed diam voluptua. Atveroeos et accusam et justo duodolores et ea rebum. Stet clitakasd gubergren, no sea takimatasanctus est Lorem ipsum dolorsit amet. Lorem ipsum dolor sitamet, consetetursadipscing elitr, sed diamnonumy eirmod tempor inviduntut labore et dolore magnaaliquyam erat, sed diamvoluptua. At vero eos etWhole entitiesChunksAtoms
  53. 53. DATA MINIMIZATIONEnvironment data (files):bar.1Lorem ipsum dolor sitamet, consetetursadipscing elitr,seddiam nonumy eirmodtempor invidunt utlabore et dolore magna aliquyamerat, sed diam voluptua. Atveroeos et accusam et justo duodolores et ea rebum. Stet clitakasd gubergren, no sea takimatasanctus est Lorem ipsum dolorsit amet. Lorem ipsum dolor sitamet, consetetursadipscing elitr, sed diamnonumy eirmod tempor inviduntut labore et dolore magnaaliquyam erat, sed diamvoluptua. At vero eos etWhole entitiesChunksAtoms
  54. 54. DATA MINIMIZATIONEnvironment data (files):bar.1Lorem ipsum dolor sitamet, consetetursadipscing elitr,seddiam nonumy eirmodtempor invidunt utlabore et dolore magna aliquyamerat, sed diam voluptua. Atveroeos et accusam et justo duodolores et ea rebum. Stet clitakasd gubergren, no sea takimatasanctus est Lorem ipsum dolorsit amet. Lorem ipsum dolor sitamet, consetetursadipscing elitr, sed diamnonumy eirmod tempor inviduntut labore et dolore magnaaliquyam erat, sed diamvoluptua. At vero eos etWhole entitiesChunksAtoms
  55. 55. DATA MINIMIZATIONEnvironment data (files):bar.1Lorem ipsum dolor sitamet, consetetursadipscing elitr,seddiam nonumy eirmodtempor invidunt utlabore et dolore magna aliquyamerat, sed diam voluptua. Atveroeos et accusam et justo duodolores et ea rebum. Stet clitakasd gubergren, no sea takimatasanctus est Lorem ipsum dolorsit amet. Lorem ipsum dolor sitamet, consetetursadipscing elitr, sed diamnonumy eirmod tempor inviduntut labore et dolore magnaaliquyam erat, sed diamvoluptua. At vero eos etWhole entitiesChunksAtoms
  56. 56. DATA MINIMIZATIONEnvironment data (files):bar.1sadipscing elitr, sed diamnonumy eirmod tempor inviduntut labore et dolore magnaaliquyam erat, sed diamvoluptua. At vero eos etWhole entitiesChunksAtoms
  57. 57. DATA MINIMIZATIONEnvironment data (files):bar.1sadipscing elitr, sed diamnonumy eirmod tempor inviduntut labore et dolore magnaaliquyam erat, sed diamvoluptua. At vero eos etWhole entitiesChunksAtoms
  58. 58. DATA MINIMIZATIONEnvironment data (files):bar.1Whole entitiesChunksAtomssadipscing elitr,eirmod inviduntut labore dolore magnaerat,voluptua.
  59. 59. DATA MINIMIZATIONEnvironment data (files):bar.1Whole entitiesChunksAtomssadipscing elitr,eirmod inviduntut labore dolore magnaerat,voluptua.foo.2
  60. 60. DATA MINIMIZATIONEnvironment data (files):Whole entitiesChunksAtomssadipscing elitr,eirmod inviduntut labore dolore magnaerat,voluptua.foo.2
  61. 61. ANALYSIS
  62. 62. ANALYSIS1. Original and minimized executions produce the same failure2. Minimized execution is not larger than the original execution(assuming a correct oracle)Correctness
  63. 63. ANALYSIS1. Original and minimized executions produce the same failure2. Minimized execution is not larger than the original execution(assuming a correct oracle)Correctnesspolynomial in the size of the captured data(assuming delta debugging)Worst case performance
  64. 64. EVALUATIONCan the technique produce, in a reasonable amountof time, minimized executions that can be used todebug the original failure?
  65. 65. EVALUATIONCan the technique produce, in a reasonable amountof time, minimized executions that can be used todebug the original failure?Pine email and news client• two real field failures• 20 failing executions, 10 per failure
  66. 66. EVALUATIONCan the technique produce, in a reasonable amountof time, minimized executions that can be used todebug the original failure?Pine email and news client• two real field failures• 20 failing executions, 10 per failureMinimized executions generated by• randomly generating interaction scripts• manually performing the scripts (while recording)• minimizing the captured executions
  67. 67. RESULTSHeader-color fault Address book fault0%10%20%30%40%50%60%70%80%90%100%# entities streams size files sizeAveragevalueafterminimization
  68. 68. RESULTSHeader-color fault Address book faultResults are likely to be conservative; recorded executionsonly contain the minimal amount of data needed to perform an action.0%10%20%30%40%50%60%70%80%90%100%# entities streams size files sizeAveragevalueafterminimization
  69. 69. RESULTSHeader-color fault Address book faultResults are likely to be conservative; recorded executionsonly contain the minimal amount of data needed to perform an action.0%10%20%30%40%50%60%70%80%90%100%# entities streams size files sizeAveragevalueafterminimizationInputs can be minimized in a reasonableamount of time (less then 75 minutes)
  70. 70. TECHNICAL CONTRIBUTIONSRecording andreplaying executionsInput anonymizationInput minimization✘Highlighting failure-relevant inputsSupportEnable
  71. 71. Sensitiveinput (I)that causes FInput domainANONYMIZATION
  72. 72. Sensitiveinput (I)that causes FInput domainInputs thatcause FANONYMIZATION
  73. 73. Sensitiveinput (I)that causes FInput domainInputs thatcause FANONYMIZATIONAnonymizedinput (I’)that alsocauses F
  74. 74. Inputs that satisfyF’s path condition Sensitiveinput (I)that causes FInput domainInputs thatcause FANONYMIZATIONAnonymizedinput (I’)that alsocauses F
  75. 75. PATH CONDITION GENERATIONPath condition: set of constraints on a program’sinputs that encode the conditions necessary for aspecific path to be executed.
  76. 76. boolean foo(int x, int y, int z) {if(x <= 5) {int a = x * 2;if(y + a > 10) {if(z == 0) {return true;}}}return false;}PATH CONDITION GENERATION
  77. 77. boolean foo(int x, int y, int z) {if(x <= 5) {int a = x * 2;if(y + a > 10) {if(z == 0) {return true;}}}return false;}PATH CONDITION GENERATION
  78. 78. boolean foo(int x, int y, int z) {if(x <= 5) {int a = x * 2;if(y + a > 10) {if(z == 0) {return true;}}}return false;}PATH CONDITION GENERATION5 3 0
  79. 79. boolean foo(int x, int y, int z) {if(x <= 5) {int a = x * 2;if(y + a > 10) {if(z == 0) {return true;}}}return false;}PATH CONDITION GENERATION5 3 0(sensitive)
  80. 80. Path Condition:Symbolic State:boolean foo(int x, int y, int z) {if(x <= 5) {int a = x * 2;if(y + a > 10) {if(z == 0) {return true;}}}return false;}PATH CONDITION GENERATION5 3 0(sensitive)
  81. 81. Path Condition:Symbolic State:boolean foo(int x, int y, int z) {if(x <= 5) {int a = x * 2;if(y + a > 10) {if(z == 0) {return true;}}}return false;}PATH CONDITION GENERATION5 3 0x→i1y→i2z→i3(sensitive)
  82. 82. Path Condition:Symbolic State:boolean foo(int x, int y, int z) {if(x <= 5) {int a = x * 2;if(y + a > 10) {if(z == 0) {return true;}}}return false;}PATH CONDITION GENERATION5 3 0x→i1y→i2z→i3(sensitive)
  83. 83. Path Condition:i1 <= 5Symbolic State:boolean foo(int x, int y, int z) {if(x <= 5) {int a = x * 2;if(y + a > 10) {if(z == 0) {return true;}}}return false;}PATH CONDITION GENERATION5 3 0x→i1y→i2z→i3(sensitive)
  84. 84. Path Condition:i1 <= 5Symbolic State:boolean foo(int x, int y, int z) {if(x <= 5) {int a = x * 2;if(y + a > 10) {if(z == 0) {return true;}}}return false;}PATH CONDITION GENERATION5 3 0x→i1y→i2z→i3(sensitive)
  85. 85. Path Condition:i1 <= 5Symbolic State:a→i1*2boolean foo(int x, int y, int z) {if(x <= 5) {int a = x * 2;if(y + a > 10) {if(z == 0) {return true;}}}return false;}PATH CONDITION GENERATION5 3 0x→i1y→i2z→i3(sensitive)
  86. 86. Path Condition:i1 <= 5Symbolic State:a→i1*2boolean foo(int x, int y, int z) {if(x <= 5) {int a = x * 2;if(y + a > 10) {if(z == 0) {return true;}}}return false;}PATH CONDITION GENERATION5 3 0x→i1y→i2z→i3(sensitive)
  87. 87. Path Condition:i1 <= 5Symbolic State:a→i1*2boolean foo(int x, int y, int z) {if(x <= 5) {int a = x * 2;if(y + a > 10) {if(z == 0) {return true;}}}return false;}PATH CONDITION GENERATION5 3 0x→i1y→i2z→i3∧ i2+i1*2 > 10(sensitive)
  88. 88. Path Condition:i1 <= 5Symbolic State:a→i1*2boolean foo(int x, int y, int z) {if(x <= 5) {int a = x * 2;if(y + a > 10) {if(z == 0) {return true;}}}return false;}PATH CONDITION GENERATION5 3 0x→i1y→i2z→i3∧ i2+i1*2 > 10(sensitive)
  89. 89. Path Condition:i1 <= 5Symbolic State:a→i1*2boolean foo(int x, int y, int z) {if(x <= 5) {int a = x * 2;if(y + a > 10) {if(z == 0) {return true;}}}return false;}PATH CONDITION GENERATION5 3 0x→i1y→i2z→i3∧ i2+i1*2 > 10∧ i3 == 0(sensitive)
  90. 90. CHOOSING ANONYMIZEDINPUTSPath Condition:i1 <= 5∧ i2+i1*2 > 10∧ i3 == 0
  91. 91. ConstraintSolverCHOOSING ANONYMIZEDINPUTSPath Condition:i1 <= 5∧ i2+i1*2 > 10∧ i3 == 0
  92. 92. ConstraintSolverCHOOSING ANONYMIZEDINPUTSPath Condition:i1 <= 5∧ i2+i1*2 > 10∧ i3 == 0
  93. 93. ConstraintSolverCHOOSING ANONYMIZEDINPUTSPath Condition:i1 <= 5∧ i2+i1*2 > 10∧ i3 == 0 i1 == 5i2 == 3i3 == 0
  94. 94. ConstraintSolverCHOOSING ANONYMIZEDINPUTSPath Condition:i1 <= 5∧ i2+i1*2 > 10∧ i3 == 0 i1 == 5i2 == 3i3 == 0boolean foo(int x, int y, int z) {if(x <= 5) {int a = x * 2;if(y + a > 10) {if(z == 0) {return true;}}}return false;}5 3 0
  95. 95. ConstraintSolverCHOOSING ANONYMIZEDINPUTSPath Condition:i1 <= 5∧ i2+i1*2 > 10∧ i3 == 0
  96. 96. ConstraintSolverCHOOSING ANONYMIZEDINPUTSPath Condition:i1 <= 5∧ i2+i1*2 > 10∧ i3 == 0Input Constraints:i1 != 5∧ i2 != 3∧ i3 != 0
  97. 97. ConstraintSolverCHOOSING ANONYMIZEDINPUTSPath Condition:i1 <= 5∧ i2+i1*2 > 10∧ i3 == 0Input Constraints:i1 != 5∧ i2 != 3∧ i3 != 0(breakable)
  98. 98. ConstraintSolverCHOOSING ANONYMIZEDINPUTSPath Condition:i1 <= 5∧ i2+i1*2 > 10∧ i3 == 0Input Constraints:i1 != 5∧ i2 != 3∧ i3 != 0(breakable)
  99. 99. ConstraintSolverCHOOSING ANONYMIZEDINPUTSPath Condition:i1 <= 5∧ i2+i1*2 > 10∧ i3 == 0Input Constraints:i1 != 5∧ i2 != 3∧ i3 != 0i1 == 4i2 == 10i3 == 0(breakable)
  100. 100. PATH CONDITION RELAXATIONSensitiveinput (I)that causes FInput domain
  101. 101. PATH CONDITION RELAXATIONSensitiveinput (I)that causes FInput domain
  102. 102. PATH CONDITION RELAXATIONSensitiveinput (I)that causes FInput domain
  103. 103. PATH CONDITION RELAXATIONSensitiveinput (I)that causes FInput domain
  104. 104. PATH CONDITION RELAXATIONSensitiveinput (I)that causes FInput domain
  105. 105. EVALUATIONFeasibilityCan the approach generate, in areasonable amount of time, anonymizedinputs that reproduce the failure?StrengthHow much information about theoriginal inputs is revealed?EffectivenessAre the anonymized inputs safe to sendto developers?
  106. 106. SUBJECTS• Columba: 1 fault• htmlparser: 1 fault• Printtokens: 2 faults• NanoXML: 16 faults(20 faults, total)
  107. 107. SUBJECTS• Columba: 1 fault• htmlparser: 1 fault• Printtokens: 2 faults• NanoXML: 16 faultsSelect sensitive failure-inducing inputs• manually generated or included with subject• several 100 bytes to 5MB in size(20 faults, total)
  108. 108. SUBJECTS• Columba: 1 fault• htmlparser: 1 fault• Printtokens: 2 faults• NanoXML: 16 faultsSelect sensitive failure-inducing inputs• manually generated or included with subject• several 100 bytes to 5MB in size(Assume all of each input is potentially sensitive)(20 faults, total)
  109. 109. RQ1: FEASIBILITY0150300450600ExecutionTime(s)05101520columbahtmlparserprinttokens1printtokens2nanoxml1nanoxml2nanoxml3nanoxml4nanoxml5nanoxml6nanoxml7nanoxml8nanoxml9nanoxml10nanoxml11nanoxml12nanoxml13nanoxml14nanoxml15nanoxml16SolverTime(s)
  110. 110. RQ1: FEASIBILITY0150300450600ExecutionTime(s)05101520columbahtmlparserprinttokens1printtokens2nanoxml1nanoxml2nanoxml3nanoxml4nanoxml5nanoxml6nanoxml7nanoxml8nanoxml9nanoxml10nanoxml11nanoxml12nanoxml13nanoxml14nanoxml15nanoxml16SolverTime(s)Inputs can be anonymized in a reasonableamount of time (easily done overnight)
  111. 111. Average % Bits Revealed Average % ResidueRQ2: STRENGTH
  112. 112. Average % Bits Revealed Average % ResidueRQ2: STRENGTHMeasures how many inputsthat satisfy the pathconditionLittleinformation revealed
  113. 113. Average % Bits Revealed Average % ResidueRQ2: STRENGTHMeasures how many inputsthat satisfy the pathconditionLots ofinformation revealed
  114. 114. Average % Bits Revealed Average % ResidueRQ2: STRENGTHMeasures how many inputsthat satisfy the pathconditionMeasures how much of theanonymized input is identicalto the original inputAAAAAAsecretAAAAAA...AAAAAABBBBBBsecretBBBBBB...BBBBBBI’Lots ofinformation revealedI
  115. 115. Average % Bits Revealed Average % ResidueRQ2: STRENGTHMeasures how many inputsthat satisfy the pathconditionMeasures how much of theanonymized input is identicalto the original inputAAAAAAsecretAAAAAA...AAAAAABBBBBBsecretBBBBBB...BBBBBBI’Lots ofinformation revealedI
  116. 116. RQ2: STRENGTH02550751000255075100columbahtmlparserprinttokens1printtokens2nanoxml1nanoxml2nanoxml3nanoxml4nanoxml5nanoxml6nanoxml7nanoxml8nanoxml9nanoxml10nanoxml11nanoxml12nanoxml13nenoxml14nanoxml15nanoxml16Average%BitsRevealedAverage%Residue
  117. 117. RQ2: STRENGTH02550751000255075100columbahtmlparserprinttokens1printtokens2nanoxml1nanoxml2nanoxml3nanoxml4nanoxml5nanoxml6nanoxml7nanoxml8nanoxml9nanoxml10nanoxml11nanoxml12nanoxml13nenoxml14nanoxml15nanoxml16Average%BitsRevealedAverage%ResidueAnonymized inputs reveal, on average, between60% (worst case) and 2% (best case) of theinformation in the original inputs
  118. 118. RQ3: EFFECTIVENESSHTMLPARSER<?xml version="1.0" encoding="UTF-8" ?><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd"><html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en"><head><title>james clause @ gatech | home</title><style type="text/css" media="screen" title=""><!--/*--><![CDATA[<!--*/body {margin: 0px;.../*]]>*/--></style></head><body>...</body>
  119. 119. RQ3: EFFECTIVENESSHTMLPARSER<?xml version="1.0" encoding="UTF-8" ?><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd"><html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en"><head><title>james clause @ gatech | home</title><style type="text/css" media="screen" title=""><!--/*--><![CDATA[<!--*/body {margin: 0px;.../*]]>*/--></style></head><body>...</body>
  120. 120. RQ3: EFFECTIVENESSHTMLPARSER<?xml version="1.0" encoding="UTF-8" ?><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd"><html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en"><head><title>james clause @ gatech | home</title><style type="text/css" media="screen" title=""><!--/*--><![CDATA[<!--*/body {margin: 0px;.../*]]>*/--></style></head><body>...</body>The portions of the inputs that remain afteranonymization tend to be structural in nature andtherefore are safe to send to developers
  121. 121. TECHNICAL CONTRIBUTIONSRecording andreplaying executionsInput anonymizationInput minimization✘Highlighting failure-relevant inputsSupportEnable
  122. 122. Foo512BBar1KBBaz1.5GBOVERVIEW
  123. 123. 1 Taint inputsFoo512BBar1KBBaz1.5GBOVERVIEW
  124. 124. 1 Taint inputsFoo512BBar1KBBaz1.5GBOVERVIEW1234567890
  125. 125. 1 Taint inputs2 Propagatetaint marksFoo512BBar1KBBaz1.5GBOVERVIEW1234567890
  126. 126. 1 Taint inputs2 Propagatetaint marksFoo512BBar1KBBaz1.5GBfoo: 512 ... bar: 1024 ... baz: 150... total: 150...OVERVIEW1234567890
  127. 127. 1 Taint inputs2 Propagatetaint marks3 Identifyrelevant inputsFoo512BBar1KBBaz1.5GBfoo: 512 ... bar: 1024 ... baz: 150... total: 150...OVERVIEW1234567890
  128. 128. 1 Taint inputs2 Propagatetaint marks3 Identifyrelevant inputsFoo512BBar1KBBaz1.5GBfoo: 512 ... bar: 1024 ... baz: 150... total: 150...OVERVIEW1234567890
  129. 129. EVALUATIONStudy 1: Effectiveness for debugging real failuresStudy 2: Comparison with Delta Debugging
  130. 130. EVALUATIONStudy 1: Effectiveness for debugging real failuresStudy 2: Comparison with Delta DebuggingApplication KLoC Fault locationbc 1.06 10.5 more_arrays : 177gzip 1.24 6.3 get_istat : 828ncompress 4.24 1.4 comprexx : 896pine 4.44 239.1 rfc822_cat : 260squid 2.3 69.9 ftpBuildTitleUrl : 1024Subjects:
  131. 131. EVALUATIONStudy 1: Effectiveness for debugging real failuresStudy 2: Comparison with Delta DebuggingApplication KLoC Fault locationbc 1.06 10.5 more_arrays : 177gzip 1.24 6.3 get_istat : 828ncompress 4.24 1.4 comprexx : 896pine 4.44 239.1 rfc822_cat : 260squid 2.3 69.9 ftpBuildTitleUrl : 1024Subjects:We selected a failure-revealing input vector for each subject.
  132. 132. STUDY 1: EFFECTIVENESSIs the information thatPenumbra provides helpful fordebugging real failures?
  133. 133. STUDY 1 RESULTS: GZIP & NCOMPRESSCrash when a file name is longer than 1,024 characters.
  134. 134. STUDY 1 RESULTS: GZIP & NCOMPRESSContents&AttributesContents&AttributesbarContents&Attributesfoo./gzipCrash when a file name is longer than 1,024 characters.# Inputs: 10,000,056longfilename[ ]
  135. 135. STUDY 1 RESULTS: GZIP & NCOMPRESSContents&AttributesContents&AttributesbarContents&Attributesfoo./gzipCrash when a file name is longer than 1,024 characters.# Inputs: 10,000,056 # Relevant (DF): 1longfilename[ ]
  136. 136. STUDY 1 RESULTS: GZIP & NCOMPRESSContents&AttributesContents&AttributesbarContents&Attributesfoo./gzipCrash when a file name is longer than 1,024 characters.# Relevant (DF + CF): 3# Inputs: 10,000,056 # Relevant (DF): 1longfilename[ ]
  137. 137. STUDY 1: CONCLUSIONS1. Data-flow propagation is always effective,data- and control-flow propagation is sometimes effective.➡ Use data-flow propagation first then, if necessary, usecontrol-flow propagation.2. Highlighted inputs correspond to the failure conditions.➡ Our technique is effective in assisting the debugging ofreal failures.
  138. 138. STUDY 2: COMPARISON WITHDELTA DEBUGGINGRQ1: How much manual effortdoes each technique require?RQ2: How long does it take tofix a considered failure giventhe information provided byeach technique?
  139. 139. RQ1: MANUAL EFFORTUse setup-time as a proxy for manual (developer) effort.
  140. 140. RQ1: MANUAL EFFORTUse setup-time as a proxy for manual (developer) effort.5,40012,6001,8001,8001259731470163ncompress bc pineSetup-time(s)gzipPenumbraDelta Debuggingsquid
  141. 141. RQ1: MANUAL EFFORTUse setup-time as a proxy for manual (developer) effort.5,40012,6001,8001,8001259731470163ncompress bc pineSetup-time(s)gzipPenumbraDelta Debuggingsquid
  142. 142. RQ1: MANUAL EFFORTUse setup-time as a proxy for manual (developer) effort.5,40012,6001,8001,8001259731470163ncompress bc pineSetup-time(s)gzipPenumbraDelta Debuggingsquid
  143. 143. RQ1: MANUAL EFFORTUse setup-time as a proxy for manual (developer) effort.5,40012,6001,8001,8001259731470163ncompress bc pineSetup-time(s)gzipPenumbraDelta DebuggingsquidPenumbra requires considerably less setup time than Delta Debugging(although more time time overall for gzip and ncompress).
  144. 144. RQ2: DEBUGGING EFFORTUse number of relevant inputs as a proxy for debugging effort.
  145. 145. RQ2: DEBUGGING EFFORTSubject PenumbraPenumbra Delta DebuggingDF DF + CFbc 209 743 285gzip 1 3 1ncompress 1 3 1pine 26 15,100,344 90squid 89 2,056 —Use number of relevant inputs as a proxy for debugging effort.
  146. 146. RQ2: DEBUGGING EFFORTSubject PenumbraPenumbra Delta DebuggingDF DF + CFbc 209 743 285gzip 1 3 1ncompress 1 3 1pine 26 15,100,344 90squid 89 2,056 —Use number of relevant inputs as a proxy for debugging effort.• Penumbra (DF) is comparable to (slightly better than) Delta Debugging.
  147. 147. RQ2: DEBUGGING EFFORTSubject PenumbraPenumbra Delta DebuggingDF DF + CFbc 209 743 285gzip 1 3 1ncompress 1 3 1pine 26 15,100,344 90squid 89 2,056 —Use number of relevant inputs as a proxy for debugging effort.• Penumbra (DF) is comparable to (slightly better than) Delta Debugging.• Penumbra (DF + CF) is likely less effective for bc, pine, and squid
  148. 148. CONCLUSIONSProgram analysis techniques can enable andsupport the debugging of failures in widely-usedapplications by:1) capturing, replaying, minimizing, and, as muchas possible, anonymizing failing executions2) highlighting subsets of failure-inducing inputsthat are likely to be helpful for debuggingsuch failures

×