A Technique for Enabling andSupporting Debugging ofField FailuresJames Clause and Alessandro OrsoGeorgia Institute of Tech...
3
3Field failures:Anomalous behavior (or crashes) ofdeployed software that occur on user machines
4
Crash logs4
Crash logsUser-providedinformation4
Our solution5
Our solution5Record
Our solution5RecordReplay
Our solution5RecordReplayMinimize
Our solution5RecordReplayMinimizeDebug
Usage Scenario6In house In the fieldReplay / Minimize(off line)Record (on line)DevelopReplay / DebugExecutionrepository✔/✘
Existing record / replay approaches7Regression testing(e.g. Elbaum et al. 06, Orso et al. 06, Orsoand Kennedy 05, Saff et ...
Outline✓Motivation & background• Our technique• record• replay• minimization• Empirical evaluation• Conclusion & future wo...
Record & Replay• Goal: develop an approach that has low overhead and isamenable to minimization• Key insight: avoid focusi...
Record & Replay• Goal: develop an approach that has low overhead and isamenable to minimization• Key insight: avoid focusi...
Environment interactions11
Environment interactionsStreams11
Environment interactionsStreams Files11
Environment interactionsStreams Files11
Environment interactionsStreams Files11Interaction events:FILE — interaction with a filePOLL — checks for availability of d...
Event log:Environment data (files):12Environment data (streams):
Event log:Environment data (files):12Environment data (streams):FILE foo.1foo.1
Event log:Environment data (files):12Environment data (streams):FILE foo.1foo.1
Event log:Environment data (files):12Environment data (streams):FILE foo.1foo.1POLL KEYBOARD NOK
Event log:Environment data (files):12Environment data (streams):FILE foo.1foo.1POLL KEYBOARD NOK
Event log:Environment data (files):12Environment data (streams):FILE foo.1foo.1KEYBOARD: {5680}POLL KEYBOARD OKPOLL KEYBOAR...
Event log:Environment data (files):12Environment data (streams):FILE foo.1foo.1KEYBOARD: {5680}POLL KEYBOARD OKPOLL KEYBOAR...
Event log:Environment data (files):12Environment data (streams):FILE foo.1foo.1KEYBOARD: {5680} helloPOLL KEYBOARD OKPULL K...
Event log:Environment data (files):12Environment data (streams):FILE foo.1foo.1KEYBOARD: {5680} helloPOLL KEYBOARD OKPULL K...
Event log:Environment data (files):12Environment data (streams):FILE foo.1foo.1KEYBOARD: {5680} helloPOLL KEYBOARD OKPULL K...
Event log:Environment data (files):12Environment data (streams):FILE foo.1foo.1KEYBOARD: {5680} helloPOLL KEYBOARD OKPULL K...
Event log:Environment data (files):12Environment data (streams):FILE foo.1foo.1KEYBOARD: {5680} helloPOLL KEYBOARD OKPULL K...
Environment data (files):13Event log:Environment data (streams):KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NETWORK: {3405}<h...
Environment data (files):14Event log:Environment data (streams):KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NETWORK: {3405}<h...
Environment data (files):14Event log:Environment data (streams):KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NETWORK: {3405}<h...
Environment data (files):14Event log:Environment data (streams):KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NETWORK: {3405}<h...
Environment data (files):14Event log:Environment data (streams):KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NETWORK: {3405}<h...
Environment data (files):14Event log:Environment data (streams):KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NETWORK: {3405}<h...
Environment data (files):14Event log:Environment data (streams):KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NETWORK: {3405}<h...
Minimize15Goal: focus debugging effort
Minimize15Goal: focus debugging effortExecutionrecording
Minimize15Goal: focus debugging effortExecutionrecording↺Timeminimization
Minimize15Goal: focus debugging effortExecutionrecordingExecutionrecording↺Timeminimization
Minimize15Goal: focus debugging effortExecutionrecordingExecutionrecording↺Timeminimization✂Dataminimization
Minimize15Goal: focus debugging effortExecutionrecordingExecutionrecordingExecutionrecording↺Timeminimization✂Dataminimiza...
Minimize: timeEnvironment data (files):16Event log:Environment data (streams):KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NET...
Minimize: timeEnvironment data (files):16Event log:Environment data (streams):KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NET...
Minimize: timeEnvironment data (files):17Event log:Environment data (streams):FILE foo.1POLL KEYBOARD OKPULL KEYBOARD 1POLL...
Minimize: timeEnvironment data (files):17Event log:Environment data (streams):FILE foo.1POLL KEYBOARD OKPULL KEYBOARD 1POLL...
Minimize: data18AtomsChunksWholeentitiesData minimization Environment
Minimize: data18AtomsChunksWholeentitiesData minimization Environment
Minimize: data18✔AtomsChunksWholeentitiesData minimization Environment
Minimize: data18AtomsChunksWholeentitiesData minimization Environment
Minimize: data18AtomsChunksWholeentitiesData minimization Environment
Minimize: data18AtomsChunksWholeentitiesData minimization Environment✘
Minimize: data18AtomsChunksWholeentitiesData minimization Environment
Minimize: data18AtomsChunksWholeentitiesData minimization Environment
Minimize: data18AtomsChunksWholeentitiesData minimization Environment
Minimize: data18AtomsChunksWholeentitiesData minimization Environment
Minimize: data18AtomsChunksWholeentitiesData minimization Environment
Minimize: data18AtomsChunksWholeentitiesData minimization Environment
Minimize: data18AtomsChunksWholeentitiesData minimization Environment
Minimize: data18AtomsChunksWholeentitiesData minimization Environment
The tool:ADDAAssisting the Debuggingof Deployed Applications• Record and Replay:• Works on x86 (c-lib based) binaries• Bas...
LimitationsTwo main limitations:• Technique:May not replay non-deterministic failures• Implementation:Does not handle wind...
Empirical evaluation• Research questions• Can ADDA produce minimized executions that can be usedto debug the original fail...
Empirical evaluation• Research questions• Can ADDA produce minimized executions that can be usedto debug the original fail...
Minimization results0%10%20%30%40%50%60%70%80%90%100%# entities streams size files size23AveragevalueafterminimizationHead...
Minimization results0%10%20%30%40%50%60%70%80%90%100%# entities streams size files size23AveragevalueafterminimizationHead...
Minimization results0%10%20%30%40%50%60%70%80%90%100%# entities streams size files size23AveragevalueafterminimizationHead...
Specific Example: Address Book Failure• Complete execution• 34 entities (files and streams)• ≈800kb• Minimized execution• 5 ...
Future work• More studies: additional applications and real users• Extend technique / implementation• Support windowing sy...
Conclusions• Novel approach that supports debuggingfield failures• Prototype implementation for x86 binaries• Preliminary e...
Questions?27
A Technique for Enabling and Supporting Debugging of Field Failures (ICSE 2007)
A Technique for Enabling and Supporting Debugging of Field Failures (ICSE 2007)
A Technique for Enabling and Supporting Debugging of Field Failures (ICSE 2007)
A Technique for Enabling and Supporting Debugging of Field Failures (ICSE 2007)
A Technique for Enabling and Supporting Debugging of Field Failures (ICSE 2007)
Upcoming SlideShare
Loading in …5
×

A Technique for Enabling and Supporting Debugging of Field Failures (ICSE 2007)

282 views

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
282
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

A Technique for Enabling and Supporting Debugging of Field Failures (ICSE 2007)

  1. 1. A Technique for Enabling andSupporting Debugging ofField FailuresJames Clause and Alessandro OrsoGeorgia Institute of TechnologyThis work was supported in part by NSF awardsCCF-0541080 and CCR-0205422 to Georgia Tech.1
  2. 2. 3
  3. 3. 3Field failures:Anomalous behavior (or crashes) ofdeployed software that occur on user machines
  4. 4. 4
  5. 5. Crash logs4
  6. 6. Crash logsUser-providedinformation4
  7. 7. Our solution5
  8. 8. Our solution5Record
  9. 9. Our solution5RecordReplay
  10. 10. Our solution5RecordReplayMinimize
  11. 11. Our solution5RecordReplayMinimizeDebug
  12. 12. Usage Scenario6In house In the fieldReplay / Minimize(off line)Record (on line)DevelopReplay / DebugExecutionrepository✔/✘
  13. 13. Existing record / replay approaches7Regression testing(e.g. Elbaum et al. 06, Orso et al. 06, Orsoand Kennedy 05, Saff et al. 05, MercuryWinRunner)• Replay only a portion of anexecution by recording eventsfor specific subsystemsBoth types of technique are not amenable tominimization and may cause unacceptable overheadDeterministic debugging(e.g. Chen et al. 01, King et al. 05,Narayanasamy et al. 05, Netzer and Weaver94, Srinivasan et al. 04,VMWare)• Replay an entire execution byrecording every component ofan application
  14. 14. Outline✓Motivation & background• Our technique• record• replay• minimization• Empirical evaluation• Conclusion & future work8
  15. 15. Record & Replay• Goal: develop an approach that has low overhead and isamenable to minimization• Key insight: avoid focusing on low-level (internal) events• expensive (large number of events)• not amenable to minimization (high interdependence)9
  16. 16. Record & Replay• Goal: develop an approach that has low overhead and isamenable to minimization• Key insight: avoid focusing on low-level (internal) events• expensive (large number of events)• not amenable to minimization (high interdependence)➡Focus on high-level (external) interactions with theenvironment• efficient (fewer, more “expensive” interactions)• amenable to minimization (low interdependence)10
  17. 17. Environment interactions11
  18. 18. Environment interactionsStreams11
  19. 19. Environment interactionsStreams Files11
  20. 20. Environment interactionsStreams Files11
  21. 21. Environment interactionsStreams Files11Interaction events:FILE — interaction with a filePOLL — checks for availability of data on a streamPULL — read data from a stream
  22. 22. Event log:Environment data (files):12Environment data (streams):
  23. 23. Event log:Environment data (files):12Environment data (streams):FILE foo.1foo.1
  24. 24. Event log:Environment data (files):12Environment data (streams):FILE foo.1foo.1
  25. 25. Event log:Environment data (files):12Environment data (streams):FILE foo.1foo.1POLL KEYBOARD NOK
  26. 26. Event log:Environment data (files):12Environment data (streams):FILE foo.1foo.1POLL KEYBOARD NOK
  27. 27. Event log:Environment data (files):12Environment data (streams):FILE foo.1foo.1KEYBOARD: {5680}POLL KEYBOARD OKPOLL KEYBOARD NOK
  28. 28. Event log:Environment data (files):12Environment data (streams):FILE foo.1foo.1KEYBOARD: {5680}POLL KEYBOARD OKPOLL KEYBOARD NOK
  29. 29. Event log:Environment data (files):12Environment data (streams):FILE foo.1foo.1KEYBOARD: {5680} helloPOLL KEYBOARD OKPULL KEYBOARD 5POLL KEYBOARD NOK
  30. 30. Event log:Environment data (files):12Environment data (streams):FILE foo.1foo.1KEYBOARD: {5680} helloPOLL KEYBOARD OKPULL KEYBOARD 5POLL KEYBOARD NOK
  31. 31. Event log:Environment data (files):12Environment data (streams):FILE foo.1foo.1KEYBOARD: {5680} helloPOLL KEYBOARD OKPULL KEYBOARD 5POLL KEYBOARD NOKPOLL NETWORK OKNETWORK: {3405}
  32. 32. Event log:Environment data (files):12Environment data (streams):FILE foo.1foo.1KEYBOARD: {5680} helloPOLL KEYBOARD OKPULL KEYBOARD 5POLL KEYBOARD NOKPOLL NETWORK OKNETWORK: {3405}❙
  33. 33. Event log:Environment data (files):12Environment data (streams):FILE foo.1foo.1KEYBOARD: {5680} helloPOLL KEYBOARD OKPULL KEYBOARD 5POLL KEYBOARD NOKPOLL NETWORK OKNETWORK: {3405}❙
  34. 34. Environment data (files):13Event log:Environment data (streams):KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NETWORK: {3405}<html><body>... ❙ {202}...FILE foo.1POLL KEYBOARD NOKPOLL KEYBOARD OKPULL KEYBOARD 1POLL NETWORK OKPULL NETWORK 1024FILE bar.1POLL NETWORK NOKPOLL NETWORK OKFILE foo.2...PULL NETWORK 1024FILE foo.2POLL KEYBOARD NOK...foo.1 foo.2 bar.1
  35. 35. Environment data (files):14Event log:Environment data (streams):KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NETWORK: {3405}<html><body>... ❙ {202}...FILE foo.1POLL KEYBOARD NOKPOLL KEYBOARD OKPULL KEYBOARD 1POLL NETWORK OKPULL NETWORK 1024FILE bar.1POLL NETWORK NOKPOLL NETWORK OKFILE foo.2...PULL NETWORK 1024FILE foo.2POLL KEYBOARD NOK...foo.1 foo.2 bar.1
  36. 36. Environment data (files):14Event log:Environment data (streams):KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NETWORK: {3405}<html><body>... ❙ {202}...FILE foo.1POLL KEYBOARD NOKPOLL KEYBOARD OKPULL KEYBOARD 1POLL NETWORK OKPULL NETWORK 1024FILE bar.1POLL NETWORK NOKPOLL NETWORK OKFILE foo.2...PULL NETWORK 1024FILE foo.2POLL KEYBOARD NOK...foo.1 foo.2 bar.1
  37. 37. Environment data (files):14Event log:Environment data (streams):KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NETWORK: {3405}<html><body>... ❙ {202}...FILE foo.1POLL KEYBOARD NOKPOLL KEYBOARD OKPULL KEYBOARD 1POLL NETWORK OKPULL NETWORK 1024FILE bar.1POLL NETWORK NOKPOLL NETWORK OKFILE foo.2...PULL NETWORK 1024FILE foo.2POLL KEYBOARD NOK...foo.1 foo.2 bar.1✔
  38. 38. Environment data (files):14Event log:Environment data (streams):KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NETWORK: {3405}<html><body>... ❙ {202}...FILE foo.1POLL KEYBOARD NOKPOLL KEYBOARD OKPULL KEYBOARD 1POLL NETWORK OKPULL NETWORK 1024FILE bar.1POLL NETWORK NOKPOLL NETWORK OKFILE foo.2...PULL NETWORK 1024FILE foo.2POLL KEYBOARD NOK...foo.1 foo.2 bar.1✔
  39. 39. Environment data (files):14Event log:Environment data (streams):KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NETWORK: {3405}<html><body>... ❙ {202}...FILE foo.1POLL KEYBOARD NOKPOLL KEYBOARD OKPULL KEYBOARD 1POLL NETWORK OKPULL NETWORK 1024FILE bar.1POLL NETWORK NOKPOLL NETWORK OKFILE foo.2...PULL NETWORK 1024FILE foo.2POLL KEYBOARD NOK...foo.1 foo.2 bar.1✔
  40. 40. Environment data (files):14Event log:Environment data (streams):KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NETWORK: {3405}<html><body>... ❙ {202}...FILE foo.1POLL KEYBOARD NOKPOLL KEYBOARD OKPULL KEYBOARD 1POLL NETWORK OKPULL NETWORK 1024FILE bar.1POLL NETWORK NOKPOLL NETWORK OKFILE foo.2...PULL NETWORK 1024FILE foo.2POLL KEYBOARD NOK...foo.1 foo.2 bar.1✔✔✔✔✔✔✔✔✔✔✔✔✔
  41. 41. Minimize15Goal: focus debugging effort
  42. 42. Minimize15Goal: focus debugging effortExecutionrecording
  43. 43. Minimize15Goal: focus debugging effortExecutionrecording↺Timeminimization
  44. 44. Minimize15Goal: focus debugging effortExecutionrecordingExecutionrecording↺Timeminimization
  45. 45. Minimize15Goal: focus debugging effortExecutionrecordingExecutionrecording↺Timeminimization✂Dataminimization
  46. 46. Minimize15Goal: focus debugging effortExecutionrecordingExecutionrecordingExecutionrecording↺Timeminimization✂Dataminimization
  47. 47. Minimize: timeEnvironment data (files):16Event log:Environment data (streams):KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NETWORK: {3405}<html><body>... ❙ {202}...FILE foo.1POLL KEYBOARD NOKPOLL KEYBOARD OKPULL KEYBOARD 1POLL NETWORK OKPULL NETWORK 1024FILE bar.1POLL NETWORK NOKPOLL NETWORK OKFILE foo.2PULL NETWORK 1024FILE foo.2POLL KEYBOARD NOK
  48. 48. Minimize: timeEnvironment data (files):16Event log:Environment data (streams):KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NETWORK: {3405}<html><body>... ❙ {202}...FILE foo.1POLL KEYBOARD NOKPOLL KEYBOARD OKPULL KEYBOARD 1POLL NETWORK OKPULL NETWORK 1024FILE bar.1POLL NETWORK NOKPOLL NETWORK OKFILE foo.2PULL NETWORK 1024FILE foo.2POLL KEYBOARD NOK
  49. 49. Minimize: timeEnvironment data (files):17Event log:Environment data (streams):FILE foo.1POLL KEYBOARD OKPULL KEYBOARD 1POLL NETWORK OKPULL NETWORK 1024FILE bar.1POLL NETWORK OKFILE foo.2PULL NETWORK 1024FILE foo.2KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NETWORK: {3405}<html><body>... ❙ {202}...
  50. 50. Minimize: timeEnvironment data (files):17Event log:Environment data (streams):FILE foo.1POLL KEYBOARD OKPULL KEYBOARD 1POLL NETWORK OKPULL NETWORK 1024FILE bar.1POLL NETWORK OKFILE foo.2PULL NETWORK 1024FILE foo.2KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...NETWORK: {3405}<html><body>... ❙ {202}...
  51. 51. Minimize: data18AtomsChunksWholeentitiesData minimization Environment
  52. 52. Minimize: data18AtomsChunksWholeentitiesData minimization Environment
  53. 53. Minimize: data18✔AtomsChunksWholeentitiesData minimization Environment
  54. 54. Minimize: data18AtomsChunksWholeentitiesData minimization Environment
  55. 55. Minimize: data18AtomsChunksWholeentitiesData minimization Environment
  56. 56. Minimize: data18AtomsChunksWholeentitiesData minimization Environment✘
  57. 57. Minimize: data18AtomsChunksWholeentitiesData minimization Environment
  58. 58. Minimize: data18AtomsChunksWholeentitiesData minimization Environment
  59. 59. Minimize: data18AtomsChunksWholeentitiesData minimization Environment
  60. 60. Minimize: data18AtomsChunksWholeentitiesData minimization Environment
  61. 61. Minimize: data18AtomsChunksWholeentitiesData minimization Environment
  62. 62. Minimize: data18AtomsChunksWholeentitiesData minimization Environment
  63. 63. Minimize: data18AtomsChunksWholeentitiesData minimization Environment
  64. 64. Minimize: data18AtomsChunksWholeentitiesData minimization Environment
  65. 65. The tool:ADDAAssisting the Debuggingof Deployed Applications• Record and Replay:• Works on x86 (c-lib based) binaries• Based on dynamic instrumentation (Pin)• Maps c-library calls to interaction events• Minimization:• Set of extensible scripts19
  66. 66. LimitationsTwo main limitations:• Technique:May not replay non-deterministic failures• Implementation:Does not handle window system events (yet)20
  67. 67. Empirical evaluation• Research questions• Can ADDA produce minimized executions that can be usedto debug the original failure?• How much overhead does ADDA impose?• Subject:• Pine — widely-used email / news client• Data:• Two real field failures from Pine’s history• Set of 20 failing executions, 10 per failure21
  68. 68. Empirical evaluation• Research questions• Can ADDA produce minimized executions that can be usedto debug the original failure?• How much overhead does ADDA impose?• Subject:• Pine — widely-used email / news client• Data:• Two real field failures from Pine’s history• Set of 20 failing executions, 10 per failure22
  69. 69. Minimization results0%10%20%30%40%50%60%70%80%90%100%# entities streams size files size23AveragevalueafterminimizationHeader-color fault Address book fault
  70. 70. Minimization results0%10%20%30%40%50%60%70%80%90%100%# entities streams size files size23AveragevalueafterminimizationHeader-color fault Address book faultMoreover, these results are conservative: recorded executions only contain the minimalamount of data needed to perform an action.
  71. 71. Minimization results0%10%20%30%40%50%60%70%80%90%100%# entities streams size files size23AveragevalueafterminimizationHeader-color fault Address book faultOverhead• Offline: less than 75 minutes for minimization• Online: negligible overhead while recordingMoreover, these results are conservative: recorded executions only contain the minimalamount of data needed to perform an action.
  72. 72. Specific Example: Address Book Failure• Complete execution• 34 entities (files and streams)• ≈800kb• Minimized execution• 5 partial entities (4 files,1 stream)• ≈72kb
  73. 73. Future work• More studies: additional applications and real users• Extend technique / implementation• Support windowing system• Investigate ad-hoc minimization algorithms• Include non-deterministic events (if needed)25
  74. 74. Conclusions• Novel approach that supports debuggingfield failures• Prototype implementation for x86 binaries• Preliminary empirical evaluation: for the casesconsidered, our technique can1. minimize failing executions,2. preserve their failing behavior, and3. impose low overhead on users26
  75. 75. Questions?27

×