The document presents a technique for enabling and supporting debugging of field failures in deployed software. The technique involves recording high-level environment interactions during execution, replaying the execution, and minimizing the recording to focus debugging efforts. An evaluation on the Pine email client found that the technique can produce minimized executions using a small fraction of the original data size, while imposing negligible runtime overhead. The technique aims to help developers debug anomalous behavior that occurs in deployed software.

1. A Technique for Enabling and
Supporting Debugging of
Field Failures
James Clause and Alessandro Orso
Georgia Institute of Technology
This work was supported in part by NSF awards
CCF-0541080 and CCR-0205422 to Georgia Tech.
1

7. 3

8. 3
Field failures:Anomalous behavior (or crashes) of
deployed software that occur on user machines

9. 4

10. Crash logs
4

11. Crash logs
User-provided
information
4

12. Our solution
5

13. Our solution
5
Record

14. Our solution
5
Record
Replay

15. Our solution
5
Record
Replay
Minimize

16. Our solution
5
Record
Replay
Minimize
Debug

17. Usage Scenario
6
In house In the field
Replay / Minimize
(off line)
Record (on line)Develop
Replay / Debug
Execution
repository
✔/✘

18. Existing record / replay approaches
7
Regression testing
(e.g. Elbaum et al. 06, Orso et al. 06, Orso
and Kennedy 05, Saff et al. 05, Mercury
WinRunner)
• Replay only a portion of an
execution by recording events
for specific subsystems
Both types of technique are not amenable to
minimization and may cause unacceptable overhead
Deterministic debugging
(e.g. Chen et al. 01, King et al. 05,
Narayanasamy et al. 05, Netzer and Weaver
94, Srinivasan et al. 04,VMWare)
• Replay an entire execution by
recording every component of
an application

19. Outline
✓Motivation & background
• Our technique
• record
• replay
• minimization
• Empirical evaluation
• Conclusion & future work
8

20. Record & Replay
• Goal: develop an approach that has low overhead and is
amenable to minimization
• Key insight: avoid focusing on low-level (internal) events
• expensive (large number of events)
• not amenable to minimization (high interdependence)
9

21. Record & Replay
• Goal: develop an approach that has low overhead and is
amenable to minimization
• Key insight: avoid focusing on low-level (internal) events
• expensive (large number of events)
• not amenable to minimization (high interdependence)
➡Focus on high-level (external) interactions with the
environment
• efficient (fewer, more “expensive” interactions)
• amenable to minimization (low interdependence)
10

22. Environment interactions
11

23. Environment interactions
Streams
11

24. Environment interactions
Streams Files
11

25. Environment interactions
Streams Files
11

26. Environment interactions
Streams Files
11
Interaction events:
FILE — interaction with a file
POLL — checks for availability of data on a stream
PULL — read data from a stream

27. Event log:
Environment data (files):
12
Environment data (streams):

28. Event log:
Environment data (files):
12
Environment data (streams):
FILE foo.1
foo.1

29. Event log:
Environment data (files):
12
Environment data (streams):
FILE foo.1
foo.1

30. Event log:
Environment data (files):
12
Environment data (streams):
FILE foo.1
foo.1
POLL KEYBOARD NOK

31. Event log:
Environment data (files):
12
Environment data (streams):
FILE foo.1
foo.1
POLL KEYBOARD NOK

32. Event log:
Environment data (files):
12
Environment data (streams):
FILE foo.1
foo.1
KEYBOARD: {5680}
POLL KEYBOARD OK
POLL KEYBOARD NOK

33. Event log:
Environment data (files):
12
Environment data (streams):
FILE foo.1
foo.1
KEYBOARD: {5680}
POLL KEYBOARD OK
POLL KEYBOARD NOK

34. Event log:
Environment data (files):
12
Environment data (streams):
FILE foo.1
foo.1
KEYBOARD: {5680} hello
POLL KEYBOARD OK
PULL KEYBOARD 5
POLL KEYBOARD NOK

35. Event log:
Environment data (files):
12
Environment data (streams):
FILE foo.1
foo.1
KEYBOARD: {5680} hello
POLL KEYBOARD OK
PULL KEYBOARD 5
POLL KEYBOARD NOK

36. Event log:
Environment data (files):
12
Environment data (streams):
FILE foo.1
foo.1
KEYBOARD: {5680} hello
POLL KEYBOARD OK
PULL KEYBOARD 5
POLL KEYBOARD NOK
POLL NETWORK OK
NETWORK: {3405}

37. Event log:
Environment data (files):
12
Environment data (streams):
FILE foo.1
foo.1
KEYBOARD: {5680} hello
POLL KEYBOARD OK
PULL KEYBOARD 5
POLL KEYBOARD NOK
POLL NETWORK OK
NETWORK: {3405}
❙

38. Event log:
Environment data (files):
12
Environment data (streams):
FILE foo.1
foo.1
KEYBOARD: {5680} hello
POLL KEYBOARD OK
PULL KEYBOARD 5
POLL KEYBOARD NOK
POLL NETWORK OK
NETWORK: {3405}
❙

39. Environment data (files):
13
Event log:
Environment data (streams):
KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...
NETWORK: {3405}<html><body>... ❙ {202}...
FILE foo.1
POLL KEYBOARD NOK
POLL KEYBOARD OK
PULL KEYBOARD 1
POLL NETWORK OK
PULL NETWORK 1024
FILE bar.1
POLL NETWORK NOK
POLL NETWORK OK
FILE foo.2
...
PULL NETWORK 1024
FILE foo.2
POLL KEYBOARD NOK
...
foo.1 foo.2 bar.1

40. Environment data (files):
14
Event log:
Environment data (streams):
KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...
NETWORK: {3405}<html><body>... ❙ {202}...
FILE foo.1
POLL KEYBOARD NOK
POLL KEYBOARD OK
PULL KEYBOARD 1
POLL NETWORK OK
PULL NETWORK 1024
FILE bar.1
POLL NETWORK NOK
POLL NETWORK OK
FILE foo.2
...
PULL NETWORK 1024
FILE foo.2
POLL KEYBOARD NOK
...
foo.1 foo.2 bar.1

41. Environment data (files):
14
Event log:
Environment data (streams):
KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...
NETWORK: {3405}<html><body>... ❙ {202}...
FILE foo.1
POLL KEYBOARD NOK
POLL KEYBOARD OK
PULL KEYBOARD 1
POLL NETWORK OK
PULL NETWORK 1024
FILE bar.1
POLL NETWORK NOK
POLL NETWORK OK
FILE foo.2
...
PULL NETWORK 1024
FILE foo.2
POLL KEYBOARD NOK
...
foo.1 foo.2 bar.1

42. Environment data (files):
14
Event log:
Environment data (streams):
KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...
NETWORK: {3405}<html><body>... ❙ {202}...
FILE foo.1
POLL KEYBOARD NOK
POLL KEYBOARD OK
PULL KEYBOARD 1
POLL NETWORK OK
PULL NETWORK 1024
FILE bar.1
POLL NETWORK NOK
POLL NETWORK OK
FILE foo.2
...
PULL NETWORK 1024
FILE foo.2
POLL KEYBOARD NOK
...
foo.1 foo.2 bar.1
✔

43. Environment data (files):
14
Event log:
Environment data (streams):
KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...
NETWORK: {3405}<html><body>... ❙ {202}...
FILE foo.1
POLL KEYBOARD NOK
POLL KEYBOARD OK
PULL KEYBOARD 1
POLL NETWORK OK
PULL NETWORK 1024
FILE bar.1
POLL NETWORK NOK
POLL NETWORK OK
FILE foo.2
...
PULL NETWORK 1024
FILE foo.2
POLL KEYBOARD NOK
...
foo.1 foo.2 bar.1
✔

44. Environment data (files):
14
Event log:
Environment data (streams):
KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...
NETWORK: {3405}<html><body>... ❙ {202}...
FILE foo.1
POLL KEYBOARD NOK
POLL KEYBOARD OK
PULL KEYBOARD 1
POLL NETWORK OK
PULL NETWORK 1024
FILE bar.1
POLL NETWORK NOK
POLL NETWORK OK
FILE foo.2
...
PULL NETWORK 1024
FILE foo.2
POLL KEYBOARD NOK
...
foo.1 foo.2 bar.1
✔

45. Environment data (files):
14
Event log:
Environment data (streams):
KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...
NETWORK: {3405}<html><body>... ❙ {202}...
FILE foo.1
POLL KEYBOARD NOK
POLL KEYBOARD OK
PULL KEYBOARD 1
POLL NETWORK OK
PULL NETWORK 1024
FILE bar.1
POLL NETWORK NOK
POLL NETWORK OK
FILE foo.2
...
PULL NETWORK 1024
FILE foo.2
POLL KEYBOARD NOK
...
foo.1 foo.2 bar.1
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔

46. Minimize
15
Goal: focus debugging effort

47. Minimize
15
Goal: focus debugging effort
Execution
recording

48. Minimize
15
Goal: focus debugging effort
Execution
recording
↺
Time
minimization

49. Minimize
15
Goal: focus debugging effort
Execution
recording
Execution
recording
↺
Time
minimization

50. Minimize
15
Goal: focus debugging effort
Execution
recording
Execution
recording
↺
Time
minimization
✂
Data
minimization

51. Minimize
15
Goal: focus debugging effort
Execution
recording
Execution
recording
Execution
recording
↺
Time
minimization
✂
Data
minimization

52. Minimize: time
Environment data (files):
16
Event log:
Environment data (streams):
KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...
NETWORK: {3405}<html><body>... ❙ {202}...
FILE foo.1
POLL KEYBOARD NOK
POLL KEYBOARD OK
PULL KEYBOARD 1
POLL NETWORK OK
PULL NETWORK 1024
FILE bar.1
POLL NETWORK NOK
POLL NETWORK OK
FILE foo.2
PULL NETWORK 1024
FILE foo.2
POLL KEYBOARD NOK

53. Minimize: time
Environment data (files):
16
Event log:
Environment data (streams):
KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...
NETWORK: {3405}<html><body>... ❙ {202}...
FILE foo.1
POLL KEYBOARD NOK
POLL KEYBOARD OK
PULL KEYBOARD 1
POLL NETWORK OK
PULL NETWORK 1024
FILE bar.1
POLL NETWORK NOK
POLL NETWORK OK
FILE foo.2
PULL NETWORK 1024
FILE foo.2
POLL KEYBOARD NOK

54. Minimize: time
Environment data (files):
17
Event log:
Environment data (streams):
FILE foo.1
POLL KEYBOARD OK
PULL KEYBOARD 1
POLL NETWORK OK
PULL NETWORK 1024
FILE bar.1
POLL NETWORK OK
FILE foo.2
PULL NETWORK 1024
FILE foo.2
KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...
NETWORK: {3405}<html><body>... ❙ {202}...

55. Minimize: time
Environment data (files):
17
Event log:
Environment data (streams):
FILE foo.1
POLL KEYBOARD OK
PULL KEYBOARD 1
POLL NETWORK OK
PULL NETWORK 1024
FILE bar.1
POLL NETWORK OK
FILE foo.2
PULL NETWORK 1024
FILE foo.2
KEYBOARD: {5680}hello ❙ {4056}c ❙ {300}...
NETWORK: {3405}<html><body>... ❙ {202}...

56. Minimize: data
18
Atoms
Chunks
Whole
entities
Data minimization Environment

57. Minimize: data
18
Atoms
Chunks
Whole
entities
Data minimization Environment

58. Minimize: data
18
✔
Atoms
Chunks
Whole
entities
Data minimization Environment

59. Minimize: data
18
Atoms
Chunks
Whole
entities
Data minimization Environment

60. Minimize: data
18
Atoms
Chunks
Whole
entities
Data minimization Environment

61. Minimize: data
18
Atoms
Chunks
Whole
entities
Data minimization Environment
✘

62. Minimize: data
18
Atoms
Chunks
Whole
entities
Data minimization Environment

63. Minimize: data
18
Atoms
Chunks
Whole
entities
Data minimization Environment

64. Minimize: data
18
Atoms
Chunks
Whole
entities
Data minimization Environment

65. Minimize: data
18
Atoms
Chunks
Whole
entities
Data minimization Environment

66. Minimize: data
18
Atoms
Chunks
Whole
entities
Data minimization Environment

67. Minimize: data
18
Atoms
Chunks
Whole
entities
Data minimization Environment

68. Minimize: data
18
Atoms
Chunks
Whole
entities
Data minimization Environment

69. Minimize: data
18
Atoms
Chunks
Whole
entities
Data minimization Environment

70. The tool:ADDA
Assisting the Debugging
of Deployed Applications
• Record and Replay:
• Works on x86 (c-lib based) binaries
• Based on dynamic instrumentation (Pin)
• Maps c-library calls to interaction events
• Minimization:
• Set of extensible scripts
19

71. Limitations
Two main limitations:
• Technique:
May not replay non-deterministic failures
• Implementation:
Does not handle window system events (yet)
20

72. Empirical evaluation
• Research questions
• Can ADDA produce minimized executions that can be used
to debug the original failure?
• How much overhead does ADDA impose?
• Subject:
• Pine — widely-used email / news client
• Data:
• Two real field failures from Pine’s history
• Set of 20 failing executions, 10 per failure
21

73. Empirical evaluation
• Research questions
• Can ADDA produce minimized executions that can be used
to debug the original failure?
• How much overhead does ADDA impose?
• Subject:
• Pine — widely-used email / news client
• Data:
• Two real field failures from Pine’s history
• Set of 20 failing executions, 10 per failure
22

74. Minimization results
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
# entities streams size files size
23
Averagevalueafterminimization
Header-color fault Address book fault

75. Minimization results
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
# entities streams size files size
23
Averagevalueafterminimization
Header-color fault Address book fault
Moreover, these results are conservative: recorded executions only contain the minimal
amount of data needed to perform an action.

76. Minimization results
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
# entities streams size files size
23
Averagevalueafterminimization
Header-color fault Address book fault
Overhead
• Offline: less than 75 minutes for minimization
• Online: negligible overhead while recording
Moreover, these results are conservative: recorded executions only contain the minimal
amount of data needed to perform an action.

77. Specific Example: Address Book Failure
• Complete execution
• 34 entities (files and streams)
• ≈800kb
• Minimized execution
• 5 partial entities (4 files,1 stream)
• ≈72kb

78. Future work
• More studies: additional applications and real users
• Extend technique / implementation
• Support windowing system
• Investigate ad-hoc minimization algorithms
• Include non-deterministic events (if needed)
25

79. Conclusions
• Novel approach that supports debugging
field failures
• Prototype implementation for x86 binaries
• Preliminary empirical evaluation: for the cases
considered, our technique can
1. minimize failing executions,
2. preserve their failing behavior, and
3. impose low overhead on users
26

80. Questions?
27