Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

CrashLocator: Locating Crashing Faults Based on Crash Stacks (ISSTA 2014)

5,639 views

Published on

ISSTA 2014 Presentation.
A winner of the ACM SIGSOFT Distinguished Paper Award.

Published in: Software
  • Be the first to comment

CrashLocator: Locating Crashing Faults Based on Crash Stacks (ISSTA 2014)

  1. 1. CrashLocator: Locating Crashing Faults Based on Crash Stacks Rongxin Wu1, Hongyu Zhang2, Shing-Chi Cheung1 and Sunghun Kim1 The Hong Kong University of Science and Technology1 Microsoft Research2 July 24th , 2014 ISSTA 2014
  2. 2. Background 2 Crash Information with Crash Stack Crash Reporting System Software Crash Bug ReportsDevelopers Crash Buckets
  3. 3. Feedbacks From Mozilla Developers • Locating crashing faults is hard • Ad hoc approach “… and look at the crash stack listed. It shows the line number of the code, and then I go to the code and inspect it. If I am unsure what it does I go to the second line of the stack and code and inspect that, and so on and so forth …” “Some crashes are hard to fix because it is not necessarily indicative of the place where it crashes in the crash stack …” “ I use the top down method of following the crash backwards.” “Sometimes it can be very difficult.” 3
  4. 4. Uncertain Fault Location • The faulty function may not appear in crash stack About 33%~41% of crashing faults in Firefox cannot be located in crash stacks! A B C E F G H Buggy Code D Crash Stack Crash Point 4
  5. 5. • Related Work • Tarantula (J. A. Jones et al., ICSE 2002) (J. A. Jones et al., ASE 2005) • Jaccard (R. Abreu et al., TAICPART-MUTATION 2007) • Ochiai (R. Abreu et al., TAICPART-MUTATION 2007) (S. Art et al., ISSTA 2010) • … • Passing Traces and Failing Traces Spectrum-Based Fault Localization 5
  6. 6. • Are these techniques applicable? Spectrum-Based Fault Localization Instrumented Product Software Failing Traces Passing Traces Privacy Concern Performance Overhead (C. Luk et al., PLDI 2005) x Crash Stack f1 f2 f3 … fn x Test Cases Effectiveness (S. Artzi et al., ISSTA’10) 6
  7. 7. Our Research Goal How to help developers fix crashing faults? – Locate crashing faults based on crash stack 7
  8. 8. Our technique: CrashLocator • Target at locating faulty functions • No instrumentation needed • Approximate failing traces  Based on Crash Stacks  Use static analysis techniques • Rank suspicious functions  Without passing traces  Based on characteristics of faulty functions 8
  9. 9. Approximate Failing Traces • Basic Stack Expansion Algorithm A B C D Crash Stack E J M N Depth-1 F K L Depth-2 G H Depth-3 A B J C K L D E M N F G H Call Graph 9
  10. 10. functionposition File Line D0 file_0 l0 C1 file_1 l1 B2 file_2 l2 A3 file_3 l3 Crash Stack Approximate Failing Traces • Basic Stack Expansion Algorithm  Function call information only • Improved Stack Expansion Algorithm  Source file position information 10
  11. 11. Improved Stack Expansion Algorithm • Control Flow Analysis if J() … B() … Entry Exit In Crash Stack CFG of A A B C D Crash Stack E J M N Depth-1 F K L Depth-2 G H Depth-3 11
  12. 12. Improved Stack Expansion Algorithm • Backward Slicing 1. Obj D(){ 2. Obj s; 3. int a = M(); 4. char b = ‘’; 5. Obj[] c = N(b); 6. s=c[1]; //crash here 7. if(s!=‘’){ 8. … 9. } 8. … 9. } variables {s,c} A B C D Crash Stack E M N Depth-1 F Depth-2 G H Depth-3 Not in slicing 12
  13. 13. After crash stack expansion, there are still a large number of suspicious functions How to rank the suspicious functions? 13
  14. 14. Rank suspicious functions • An empirical study on the characteristics of faulty functions • Quantify the suspiciousness of suspicious functions 14
  15. 15. Observation 1: Frequent Function • Faulty functions appear frequently in the crash traces of the corresponding buckets.  Function Frequency (FF) Crash Report More Frequent, More Suspicious For 89-92% crashing faults, the associated faulty functions appear in all crash execution traces in the corresponding bucket. Crash Bucket 15
  16. 16. Frequent Function • Some frequent functions are unlikely to be buggy  Entry points (main, _RtlUserThreadStart, …)  Event handling routine (CloseHandle) • Information retrieval, some frequent words are useless  stop-words, e.g. “the”, “an”, “a”  Inverse Document Frequency (IDF) • Inverse Bucket Frequency (IBF)  If a function appears in many buckets, it is less likely to be buggy 16
  17. 17. Observation 2: Functions Close to Crash Point • Faulty functions appear closer to crash point  In Mozilla Firefox, for 84.3% of crashing faults, the distance between crash point and the associated faulty functions is less 5. • Inverse Average Distance to Crash Point (IAD) 17
  18. 18. Observation 3: Less Frequently Changed Functions • Functions that do not contain crashing faults are often less frequently changed  94.1% of faulty functions have been changed at least once during the past 12 months  Immune Functions (Y. Dang et al. ICSE 2012) • Less frequently changed functions  Functions that have no changes in past 12 months  Suspicious score is 0 18
  19. 19. Observation 4: Large Functions • Our prior study (H. Zhang. ICSM 2009) showed that large modules are more likely to be defect-prone • Function’s Lines of Code (FLOC) 19
  20. 20. Suspicious Score 𝑆𝑐𝑜𝑟𝑒 𝑓, 𝐵 = 𝐹𝐹 𝑓, 𝐵 ∗ 𝐼𝐵𝐹 𝑓 ∗ 𝐼𝐴𝐷 𝑓, 𝐵 ∗ 𝐹𝐿𝑂𝐶(𝑓) • FF (Function Frequency) 𝐹𝐹 𝑓, 𝐵 = 𝑁𝑓,𝐵 𝑁 𝐵 • IBF(Inverse Bucket Frequency) 𝐼𝐵𝐹 𝑓 = 𝑙𝑜𝑔( #𝐵 #𝐵𝑓 + 1) • IAD(Inverse Distance to Crash Point) 𝐼𝐴𝐷 𝑓, 𝐵 = 𝑁𝑓,𝐵 1 + 𝑗=1 𝑛 𝑑𝑖𝑠𝑗(𝑓) • FLOC(Function Lines of Code) 𝐹𝐿𝑂𝐶 𝑓 = 𝑙𝑜𝑔 (𝐿𝑂𝐶 𝑓 + 1) 20
  21. 21. Evaluation Subjects • Mozilla Products  5 releases of Firefox  2 releases of Thunderbird  1 release of SeaMonkey • 160 crashing faults(buckets) • Large-Scale  More than 2 million LOC  More than 120K functions 21
  22. 22. Evaluation Metrics • Recall@N: Percentage of successfully located faults by examining top N recommended functions • Mean Reciprocal Rank (MRR)  Measure the quality of the ranking results in IR  Range value: 0 ~ 1  Higher value means better ranking 22
  23. 23. Experimental Design • RQ1: How many faults can be successfully located by CrashLocator? • RQ2: Can CrashLocator outperform the conventional stack-only methods? • RQ3: How does each factor contribute to the crash localization performance? • RQ4: How effective is the proposed crash stack expansion algorithm? 23
  24. 24. RQ1: CrashLocator Performance System Recall@1 Recall@5 Recall@10 MRR Firefox 4.0b4 55.6% 66.7% 77.8% 0.627 Firefox 4.0b5 47.1% 70.6% 70.6% 0.566 Firefox 4.0b6 48.0% 64.0% 64.0% 0.540 Firefox14.0.1 52.0% 52.0% 56.0% 0.528 Firefox16.0.1 53.8% 53.8% 53.8% 0.542 Thunderbird17.0 48.5% 66.7% 78.8% 0.568 Thunderbird24.0 50.0% 66.7% 66.7% 0.544 SeaMonkey2.21 55.0% 70.0% 70.0% 0.600 Summary 50.6% 63.7% 67.5% 0.559 24
  25. 25. RQ2: Comparison with Stack-Only methods • Conventional Stack-Only Methods • StackOnlySampling • StackOnlyAverage • StackOnlyChangeDate 25
  26. 26. RQ2: Comparison with Stack-Only methods 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 1 5 10 20 50 100 Recall@N Top N Functions StackOnlySampling StackOnlyAverage StackOnlyChangeDate CrashLocator 26
  27. 27. RQ3: Contribution of Each Factors • Inverse Bucket Frequency (IBF) • Function Frequency (FF) • Function’s Lines of Code (FLOC) • Inverse Average Distance to Crash Point (IAD) 27
  28. 28. RQ3: Contribution of Each Factors 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 ff4.0b4 ff4.0b5 ff4.0b6 ff14.0.1 ff16.0.1 tb17.0 tb24.0 sm2.21 Summary MRR IBF IBF*FF IBF*FF*FLOC IBF*FF*FLOC*IAD 28
  29. 29. RQ4: Stack Expansion Algorithms • Basic Stack Expansion Algorithm  Static Call Graph • Improved Stack Expansion Algorithm  Static Call Graph  Control Flow Analysis  Backward Slicing 29
  30. 30. RQ4: Stack Expansion Algorithms 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Recall@1 Recall@5 Recall@10 Recall@20 Recall@50 MRR Basic Stack Trace Expansion Improved Stack Trace Expansion 30
  31. 31. Conclusions • Propose a novel technique CrashLocator to locate crashing faults based on crash stack only • Evaluate on real and large-scale projects • 50.6%, 63.7%, and 67.5% of crashing faults can be located by examining only top 1,5,10 functions • CrashLocator outperforms Stack-Only methods significantly, with the improvement of MRR at least 32% and the improvement of Recall@10 at least 23% 31

×