Faults and Regression testing - Localizing Failure-Inducing Program Edits Based on Spectrum Information

  • 223 views
Uploaded on

Paper: Localizing Failure-Inducing Program Edits Based on Spectrum Information. …

Paper: Localizing Failure-Inducing Program Edits Based on Spectrum Information.

Authors: Lingming Zhang, Miryung Kim, Sarfraz Khurshid.

Session: Research Track Session 1: Faults and Regression Testing

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
223
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
4
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Localizing Failure-Inducing ProgramEdits Based on SEdit B d Spectrum I f t Information ti Lingming Zhang, Miryung Kim, Sarfraz Khurshid The University of Texas at Austin ICSM2011, September 27th 2011 1
  • 2. OverviewChange impact analysis is effective at finding suspicious edits but lacks precise ranking.Spectrum-basedSpectrum based fault localization is effective at ranking but does not scale well.Our insight: combine change impact analysis and change-impact spectrum-based fault localization. • Identify suspicious edits using extended call graphs. • Rank suspicious edits using dynamic program spectrum information. L. Zhang: Localizing failure-inducing program edits based on spectrum information 2
  • 3. Summary of our results yFaultTracer localizes failure-inducing edits with failure inducing high precision: • Id tif i suspicious edits: outperforms Identifying i i dit t f Chianti by 19.37%. • Ranking all suspicious edits: ranks real regression faults within top 3 edits for 14 of the 22 studied real-world failures. • R ki method-level suspicious edits: Ranking th d l l i i dit outperforms existing heuristic by 56.25%. L. Zhang: Localizing failure-inducing program edits based on spectrum information 3
  • 4. OutlineFaultTracer ApproachEmpirical EvaluationRelated WorkConclusions L. Zhang: Localizing failure-inducing program edits based on spectrum information 4
  • 5. Example p Program P Program P’ Ppublic class A { public class A { public static int f1=0; public static int f1=1; p public static int f2=0; ; public static int f2=1; f2 1; ... ...} } evolveclass B { class B { int f1=0; int f2=0; int f3=0; int f1=0; int f2=1; int f3=1; ; ; ; public int foo(){return f1;} int f4=1; ... public int foo()} { if(f1>=0) return f1;class C extends B{ else return f4; ... }} ... } Regression test suite T class C extends B{ public int f1=3; public void test1() public void bar(int f) {f3=f+f1;} { A.bar(1); } ... public void test2() { ... } } Test T t public void test3() { ... } public void test4() { Bug! C c = new C(); } int f = c.foo(); Re-Test public void test5() { ... } L. Zhang: Localizing failure-inducing program edits based on spectrum information 5
  • 6. FaultTracer overview Selecting tests g T based on Extended  Call Graph analysis Detecting  changes and  changes and T T’P dependences ② ① ∆P’ ③ Identifying suspicious  Id tif i i i ᵟt edits based  t on Extended  Rank suspicious  Call Graph analysis Call Graph analysis ④ edits  based on edits based on program spectrum  information ’ ᵟt L. Zhang: Localizing failure-inducing program edits based on spectrum information 6
  • 7. Extended Call Graph representation p p public void test1() { A.bar(1); } public void test4() { C c = new C(); int f = c.foo(); }TraditionalCallGraph ExtendedCallGraphusedbyChiantiused by Chianti usedbyFaultTracer used by FaultTracer test1 test4 test1 test4 <C,C.foo()> <C,C.foo()> A.Clinit() A.bar() C.C() C.foo() A.Clinit() A.bar() C.C() C.foo() <SFW,A.f2> <FR,C.f1> B.B() A.f2 B.B() B.f1 L. Zhang: Localizing failure-inducing program edits based on spectrum information 7
  • 8. Step 1. Detecting atomic changes and p g g dependencesChange  DescriptiontypesCM ChangemethodAM AddmethodDM DeletemethodAF AddfieldDF DeletefieldCFI ChangeinstancefieldCSFI Changestaticfield Change static fieldLCm Methodlook-upchangeLCf Fieldlook-up change Changedependencesinferencerules Change dependences inference rules AtomicChangeTypes L. Zhang: Localizing failure-inducing program edits based on spectrum information 8
  • 9. Step 2. Test selection based on ExtendedCall GraphC ll G h (ECG) analysisl iFaultTracer directly matches all changes with test ECGs before edits to select the influenced tests. L. Zhang: Localizing failure-inducing program edits based on spectrum information 9
  • 10. Step 3. Suspicious edit identificationbased on E t d d C ll G h analysisb d Extended Call Graph l iFaultTracer directly selects the non-look-up changes appear on test ECGs after edits as suspicious edits.FaultTracer selects method or field edits that have caused look up look-up changes on test ECGs as suspicious edits edits. L. Zhang: Localizing failure-inducing program edits based on spectrum information 10
  • 11. Step 4. Spectrum-based fault localizationforf program edits ditCorrelation between suspicious edits and tests p Edits test2 test3 test4 test5 CSFI(A.f1) CM(B.foo) CM(B f ) AF(C.f1) AM(C.bar) out Pass Pass Pass FailSuspiciousness score computation Suspiciousness Score Tie Break Edits Tarantula SBI Jaccard Ochiai - CSFI(A.f1) 0.00 0.00 0.00 0.00 - CM(B.foo) 0.75 0.50 0.50 0.71 1 AF(C.f1) 0.75 0.50 0.50 0.71 0 AM(C.bar) 1.00 1.00 1.00 1.00 - L. Zhang: Localizing failure-inducing program edits based on spectrum information 11
  • 12. OutlineFaultTracer ApproachEmpirical EvaluationRelated WorkConclusions L. Zhang: Localizing failure-inducing program edits based on spectrum information 12
  • 13. Research QuestionsRQ1: How does FaultTracer compare to Chianti in identifying id tif i suspicious edits? i i dit ?RQ2: How effective is FaultTracer in ranking suspicious edits? L. Zhang: Localizing failure-inducing program edits based on spectrum information 13
  • 14. Subjects: overview jSubjects from Software-artifact Infrastructure Repository (SIR) (SIR). Project Version Program Size (KLoC) Program Size (KLoC) Number  Number of Test Jtopas 0.0-3.0 1.83 ~ 5.36 95-209 Xml-Security 0.0-3.0 17.44 ~ 18.99 84-106 JMeter 0.0-5.0 31.01 ~ 41.05 70-97 Ant 0.0-8.0 17.20 ~ 80.44 112-878 L. Zhang: Localizing failure-inducing program edits based on spectrum information 14
  • 15. Subjects: change statistics j gNumber of changes for each version pair Ant7.0-8.0 Ant6.0-7.0 Ant5.0 6.0 Ant5 0-6 0 Ant4.0-5.0 Ant3.0-4.0 AM Ant2.0-3.0 Ant1.0-2.0 DM Ant0.0-1.0 CM JMeter4.0-5.0 AF JMeter3.0 4.0 JMeter3.0-4.0 JMeter2.0-3.0 DF JMeter1.0-2.0 CFI JMeter0.0-1.0 CSFIXmlSec2.0-3.0 LCmXmlSec1.0-2.0XmlSec0.0-1.0 LCf Jtopas2.0-3.0 p Jtopas1.0-2.0 Jtopas0.0-1.0 0 1000 2000 3000 4000 5000 6000 7000 L. Zhang: Localizing failure-inducing program edits based on spectrum information 15
  • 16. RQ1: How does FaultTracer compare toChianti in identifyingChi ti i id tif i suspicious edits? i i dit ?FaultTracer achieves 19.37% improvement in the precision of identification suspicious edits.160140120100 80 60 Chianti 40 FaultTracer 20 0 XmlSec0.0-1.0 2.0 3.0 Ant0.0-1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 Jtopas0.0-1.0 2.0 3.0 JMeter0.0-1.0 2.0 3.0 4.0 5.0 XmlSec1.0-2 XmlSec2.0-3 Ant1.0-2 Ant2.0-3 Ant3.0-4 Ant4.0-5 Ant5.0-6 Ant6.0-7 Ant7.0-8 Jtopas1.0-2 Jtopas2.0-3 JMeter1.0-2 JMeter2.0-3 JMeter3.0-4 JMeter4.0-5 X X X L. Zhang: Localizing failure-inducing program edits based on spectrum information 16
  • 17. RQ2: How effective is FaultTracer inranking suspicious edits? ki i i dit ?Ranks all types of edits: • Average performance. Tarantula SBI Jaccard Ochiai Suspicious  Edit edit num. numberAverage 8.50 8.50 10.83 14.66 68.83 3932Percentage To 0.22% 0.22% 0.28% 0.37% 1.75% --edit number • Example (Ant5.0-6.0) Test T t Tarantula T t l SBI Jaccar J Ochiai O hi i Suspicious  Edit S i i d edit num. number ant.taskdefs.optional.EchoPro 1 1 1 10 182 5019 pertiesTest.testEchoToBadFile pertiesTest testEchoToBadFile L. Zhang: Localizing failure-inducing program edits based on spectrum information 17
  • 18. RQ2: How effective is FaultTracer inranking suspicious edits? ki i i dit ?Ranks method edits (FaultTracer v.s. Heuristic) • Achieves 56.25% improvement in the precision of localizing method-level failure-inducing edits L. Zhang: Localizing failure-inducing program edits based on spectrum information 18
  • 19. LimitationsDoes not currently filter out refactorings (e.g., use RefFinder [Prete+2010]).Uses only four spectrum based fault localization spectrum-based techniques.The experimental evaluation is limited by the small number of real regression faults. L. Zhang: Localizing failure-inducing program edits based on spectrum information 19
  • 20. Related workChange impactChange-impact analysis • Chianti [Ren+2004] • Crisp [Chesley+2005] • Heuristic ranking [Ren+2007]Fault localization • Spectrum-based Spectrum based • E.g., Tarantula [Jones+2002], SBI [Liblit+2005], Jaccard [Abreu+2007], Ochiai [Abreu+2007]. • Delta debugging [Zeller1999] • Model-based Model based • E.g., Bayesian diagnosis [Kleer+1987] L. Zhang: Localizing failure-inducing program edits based on spectrum information 20
  • 21. ConclusionFaultTracer combines change impact analysis with g p y dynamic spectra.FaultTracer improves change impact analysis based extended call graph analysis.Experimental evaluation shows FaultTracer: • Performs 19.37% better than Chianti in determining affecting changes changes. • Localizes failure-inducing edits within top 3 edits for 14 of the 22 regression failures failures. • Performs 56.25% better than previous heuristic for localizing f il l li i failure-inducing program edits. i d i dit zhanglm10@gmail.com zhanglm10@gmail com L. Zhang: Localizing failure-inducing program edits based on spectrum information 21