Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Predicting Recurring Crash Stacks (ASE 2012)

1,451 views

Published on

Hyunmin's ASE 2012 presentation. A Winner of the ACM SIGSOFT Distinguished Paper Award!

Published in: Technology, Automotive
  • Be the first to comment

  • Be the first to like this

Predicting Recurring Crash Stacks (ASE 2012)

  1. 1. Predicting Recurring Crash StacksHyunmin Seo and Sunghun Kim The Hong Kong University Of Science And Technology September 7th, 2012 Automated Software Engineering 2012 Essen, Germany
  2. 2. Recurring Crashes Bug Report 52831 1 3.6b3 Patch 3.6b4Crash Point Crash PointnsXULTreeAccessible:: nsXULTreeAccessible::GetTreeItemAccessible GetTreeItemAccessible 2
  3. 3. Bad Fixes• Bad fixes comprise as much as 9% of all bugs (Gu et al. ICSE 2010)• 14.8%∼24.4% of fixes for post-release bugs are incorrect (Yin et al. FSE 2011) 3
  4. 4. Motivation• How often do bad fixes occur?• How can we help to prevent it? 4
  5. 5. Crash Reporting System (CRS) 5
  6. 6. Mozilla CRS CRRELEASE CRS CR SERVER 6
  7. 7. Mozilla CRS CRSSERVER 7
  8. 8. Mozilla CRS Bug Report #50001 Patch File NEXT RELEASE 8
  9. 9. How often are bad fixes? Crash Bug Reporting Reporting System System 19 sub-versions of Firefox 3.6 70 Bug Reports 79 Crash Points 9
  10. 10. Have all crashesdisappeared after fixes? ? Crash Report Bug Report #5000 1 Before fix Patch After fix 10
  11. 11. Recurring Crash ExamplesBUGID CRASH POINT ver1 ver2 ver3 nsHtml5ElementName:: 3.6.8 3.6.9 3.6.10538722 initializeStatics 677 0 0 3.6.6 3.6.7 3.6.8554544 nsTextFrame::Reflow 773 186 497 nsXULTreeAccessible:: 3.6b3 3.6b4 3.6b5528311 GetTreeItemAccessible 70 168 0 48.1 % (38/79) 11
  12. 12. Crash Paths• The same crash point but different crash paths• The fix may miss some paths 12
  13. 13. BUGID CRASH POINT ver1 ver2 ver3 nsXULTreeAccessible:: 3.6b3 3.6b4 3.6b5 528311 GetTreeItemAccessible 70 168 0 35 175 30 150 # of Crash Reports# of Crash Reports 25 125 20 100 15 75 10 50 5 25 0 0 #1 #2 #3 #4 #5 #1 #2 #3 #4 #5 Sub-group Sub-group 3.6b3 3.6b4 13
  14. 14. Comment in Bug Report “I don’t know how this bit (crash trace) got lost from the patch I ended up checking in, but it’s pretty essential...” A comment in bug report #523528 14
  15. 15. Incomplete Fixes• We call this as incomplete fixes• “incomplete” in terms of fix locations• How can we help to prevent this? 15
  16. 16. Approach Overview Covered BugReport#5000 1 Patch File Missing 16
  17. 17. Idea behind Classification A fix has nothing to do if it is not executed ✓ Fix Location 17
  18. 18. Stack Expansion L-1 L-2 L-3 EntryA if G H I Path 1 Path 2 Block1 Block2B G ( ) J Y ( ) B ( ) X ( ) C Exit D ✓ CFG of A F E Covered ( if S F ) Crash Stack Missing ( otherwise ) 18
  19. 19. Experimental Design RQ1 - How good is the classification?RQ2 - How can this help developers? 19
  20. 20. SubjectsName DescriptionSubject 19 releases of Firefox 3.6Release Date Oct 2009 ~ Mar 2011Programming Language C / C++LOC 3.2M ~ 4.4MName Value# of crash buckets 33# of total sub-groups 1159# of recurring sub-groups 354 20
  21. 21. Experimental Design RQ1 - How good is the classification?RQ2 - How can this help developers? 21
  22. 22. RQ1 - Prediction Result L4 Prediction Actual 292 167 Precision 0.57 Recall 0.49 F-measure 0.53 22
  23. 23. RQ1 - Expansion Levels 0.9 0.8 0.7 0.6 0.5Value 0.4 0.3 PRECISION 0.2 RECALL 0.1 F-MEASURE 0 L-0 L-1 L-2 L-3 L-4 L-5 L-7 L-10 L-∞ Expansion Level 23
  24. 24. Experimental Design RQ1 - How good is the classification?RQ2 - How can this help developers? 24
  25. 25. IEnumConnectionPoints trace a RQ2 - Case Study_RemoteNext_Thunk trace b IEnumOleUndoUnits _Next_Stub nsAccessibleWrap nsRootAccessible ::Next ::HandleEvent nsXULTreeAccessible ::GetChildAt ✓ nsRootAccessible:: HandleEventWithTarget First Fix (#528311) in 3.6b4 Second Fix (#528311) nsXULTreeAccessible:: ✓ in 3.6b5 GetTreeItemAccessible crash point286 NS_ENSURE_ARG_POINTER(aChild); 545 *aAccessible = nsnull;287 *aChild = nsnull; 546288 547- if (aRow < 0)289+ if (IsDefunct()) 547+ if (aRow < 0 || IsDefunct ())290+ return NS_ERROR_FAILURE; 548 return;291 549292 PRInt32 childCount = 0; 550 PRInt32 rowCount = 0; 25
  26. 26. RQ2 - Developer Feedback Firefox developer emails and mailing lists 21 responses - 3 very useful, 7 useful10“It should be an interesting feature 1 not useful requested more information, and useful like any automation tool. It should make the engineering work easier and keep users less annoyed.” “The first patch fixed the known steps but missed the fact that other routes led to the same state inconsistency. ... If you have a system that automates that process it would indeed be helpful.” 26
  27. 27. Threats to validity• The subject is open source software• Collected crash data might be biased• Oracle data set is incomplete 27
  28. 28. Discussion – Future Work nsJARInputThunk::EnsureJarStream nsZipReaderCache::GetZip nsJAR::Open nsZipArchive::OpenArchive crash point ✓ nsZipArchive::BuildFileList539 //-- Read the central directory headers540 buf = startp + centralOffset;541+ if (endp - buf < sizeof(PRUint32))542+ return NS_ERROR_FILE_CORRUPTED;543 PRUint32 sig = xtolong(buf); // crash point544 while (sig == CENTRALSIG) { 28
  29. 29. Related Work• Crash bucketing (Dang et al., ICSE 2012)• Post-mortem crash analysis (Manevich et al., FSE 2004)• Bug fix verification (Gu et al., ICSE 2010) 29
  30. 30. Conclusions• 48% of fixed crashes in Firefox recurred.• We present an approach to predict recurring crashes• RQ1 - How good is the classification? • Our approach yields reasonable accuracy - 0.57 precision and 0.49 recall• RQ2 - How can this help developers? • Our case studies and developers’ feedback show the idea is useful 30

×