Faults and Regression Testing - Fault interaction and its repercussions

501 views

Published on

Paper: Fault Interaction and its Repercussions

Authors: Nicholas DiGiuseppe and James A. Jones

Seesion: Research Track 1: Faults and Regression Testing

Published in: Technology, News & Politics
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
501
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
8
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Faults and Regression Testing - Fault interaction and its repercussions

  1. 1. Fault Interaction and its RepercussionsNicholas DiGiuseppe James A. Jones UC Irvine Spider Lab
  2. 2. Fault Interaction Defined4 Types of Interaction Pass/Fail Independence Obfuscation Synergy Multi-Type
  3. 3. Fault Interaction Defined4 Types of Interaction Pass/Fail Independence Obfuscation Synergy Multi-Type
  4. 4. Fault Interaction Defined4 Types of Interaction Pass/Fail Independence Obfuscation Synergy Multi-Type
  5. 5. Fault Interaction Defined4 Types of Interaction Pass/Fail Independence Obfuscation Synergy Multi-Type
  6. 6. We Need More Fault Understanding Conferences: ASE, FSE, ICSE, ICSM, ICST, ISSTA Years: 2008–200927 Fault Localization = 2.25 per conference per year4 Fault Behavior = 0.333 per conference per year
  7. 7. Motivating Example X = input [0] - 2 Y = input [0] - 2 If (X == Y) print 1 else print 0 Expected Actual Pass/Fail Results Results StatusTest Case 1: input [0,0] 1 1 PassTest Case 2: input [2,0] 0 1 Fail
  8. 8. Motivating Example X = input [0] - 2 Y = input [1] / 2 If (X == Y) print 1 else print 0 Expected Actual Pass/Fail Results Results StatusTest Case 1: input [0,0] 1 0 FailTest Case 2: input [2,0] 0 1 Fail
  9. 9. Motivating Example X = input [0] * 2 Y = input [1] / 2 If (X == Y) print 1 else print 0 Expected Actual Pass/Fail Results Results StatusTest Case 1: input [0,0] 1 1 PassTest Case 2: input [2,0] 0 0 Pass
  10. 10. Our Experiment Flex Gzip Replace Schedule Sed Space Total1 Fault 21 20 25 20 20 33 1392 Faults 189 182 185 165 188 200 11093 Faults 502 505 598 500 660 200 29654 Faults 1000 1000 700 500 950 200 43505 Faults 1000 1000 700 500 950 200 435017 Faults 1000 1000 700 500 950 200 4350Total Versions 14,712 14,707 10,608 7,685 14,168 3,233 65,113Size of Test Suite 527 214 5,542 2,650 363 13,527 22,733Number of 7,753,224 3,147,298 58,798,536 17,673,600 6,253,984 43,732,791 138,239,433Executions
  11. 11. Our Experiment Flex Gzip Replace Schedule Sed Space Total1 Fault 21 20 25 20 20 33 1392 Faults 189 182 185 165 188 200 11093 Faults 502 505 598 500 660 200 29654 Faults 1000 1000 700 500 950 200 43505 Faults 1000 1000 700 500 950 200 435017 Faults 1000 1000 700 500 950 200 4350 Total VersionsTotal Versions 14,712 14,707 10,608 65,113 7,68514,168 3,233 65,113Size of Test Suite 527 214 5,542 2,650 363 13,527 22,733 Number of Executions 138,239,433Number ofExecutions 7,753,224 3,147,298 58,798,536 17,673,600 6,253,984 43,732,791 138,239,433
  12. 12. a Results 1.0 1.0 Obfuscation Multi Type 1.0 0.8 0.8 0.8Percentage of InteractionPercentage of Interaction Percentage of Interaction 0.6 0.6 Synergy Independence 0.6 0.4 0.4 0.4 0.2 0.2 0.2 0.0 0.0 0.0 00 11 22 33 44 55 66 77 88 99 10 11 12 13 14 15 16 17 10 11 12 13 14 15 16 17 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 The Quantity of Faults The Quantity of Faults The Quantity of Faults 1.0 1.0 0.8 0.8 Percentage of Interaction Percentage of Interaction 0.6 0.6 0.4 0.4 0.2 0.2 0.0 0.0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 The Quantity of Faults The Quantity of Faults 1.0 1.0 0.8 0.8 Percentage of Interaction Percentage of Interaction 0.6 0.6 0.4 0.4 0.2 0.2 0.0 0.0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 The Quantity of Faults The Quantity of Faults
  13. 13. Impact of Fault Interaction Regression TestingFault Localization Two Faults Fix a Fault One Fault Test 1 Pass Fail Test 2 Pass Pass Test 3 Fail Pass Time Failure Clustering Run Test Fix Faults Suite in Clusters Cluster Assign Clusters Failures to Developers
  14. 14. Results Continued Gzip Flex Replace Schedule Sed SpaceLoEC 2013 4020 242 150 2253 3651Branches 1727 2680 182 66 2554 1190Methods 514 1261 63 41 416 717Data Dep 45,291 143,075 2005 1327 94,614 29,883Control Dep 26,766 61,279 1444 1074 21,991 25,047Total Dep 72,057 204,354 3449 2401 116,605 54,930LoEC: 1.165 1.5 3.818 2.273 1.893 3.068BranchesLoEC: Methods 3.916 3.188 3.841 3.659 5.416 5.092LoEC: Data 0.044 0.028 0.121 0.113 0.024 0.122DepLoEC: Control 0.075 0.066 0.168 0.140 0.103 0.146DepLoEC: Total 0.028 0.020 0.070 0.062 0.019 0.067Dep
  15. 15. ConclusionsDefined Fault InteractionDemonstrated that Fault Interaction is prominent (primarilyfault obfuscation)Demonstrated that Fault Interaction can have a realimpact on practice and researchDemonstrated that the size of a program affects FaultInteraction
  16. 16. Conclusions Take Home Message Fault Interaction is prevalent Fault Obfuscation is prevalentWe need to adapt our techniques
  17. 17. Thank You!This work was partially funded by:University of California, Irvine

×