Mutation Analysis for JavaScript Web Applicaiton Testing SEKE2013

2,632 views

Published on

This slide is for a presentation at the 25th International Conference on Software Engineering and Knowledge Engineering (SEKE'13) at Boston.

Abstract of the published paper:
When developers test modern web applications that use JavaScript, challenging issues lie in their event-driven, asynchronous, and dynamic features. Many researchers have assessed the adequacy of test cases with code coverage criteria; however, in this paper, we show that the code coverage-based approach possibly misses some faults in the applications. We propose a mutation analysis approach for estimating the fault-finding capability of test cases. We assume that developers can find overlooked fault instances and improve the test cases with the estimated capability. To create a set of faulty programs, i.e., mutants, we classify the JavaScript features in web applications and then define a comprehensive set of mutation operators. We conducted a case study on a real-world application and found that our approach supported the improvement of test cases to expose hand-seeded faults by an extra ten percent.

You can find the transcript used in the presentation from the link below:
https://docs.google.com/document/d/1CsqcYKyHZQL6Z7OkM_LVVyB-aMURMWl37lc03LOa4fI/edit?usp=sharing

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,632
On SlideShare
0
From Embeds
0
Number of Embeds
537
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Mutation Analysis for JavaScript Web Applicaiton Testing SEKE2013

  1. 1. BostonUSAKazuki NishiuraaYuta MaezawaaHironori WashizakibShinichi Honidena,c06/27/2013Mutation Analysis forJavaScript WebApplication TestingSEKE2013 aThe University of Tokyo, JapanbWaseda University, JapancNational Institute for Informatics, Japan
  2. 2.  Background Mutation Analysis: Test adequacy criteria JavaScript Web Applications (JSWAs) AjaxMutator: Mutation Analysis tool for JSWAs(http://knishiura-lab.github.io/AjaxMutator/) Feature analysis on JSWA Definition of Mutation Operators Bug survey/case study using real-world apps.2Table of ContentsImprovement of test cases; exposed 10% more faults
  3. 3. is a method to evaluate adequacy of test cases3Mutation AnalysisTicket t = issueTicket(true, true);assertEquals(600, t.price);Do we haveenough test ?Ticket issueTicket(bool isStudent,bool extraBanquet){Ticket t;if (isStudent) {t = {price:500, numBanquet:0};} else {t = {price:700, numBanquet:1};}if (extraBanquet) {t.numBanquet = t.numBanquet+1;t.price = t.price + 100;}return t;}Ticket t = issueTicket(false, false);assertEquals(700, t.price);Program under test Test case #1Test case #2
  4. 4. is a method to evaluate adequacy of test cases4Mutation AnalysisTicket t = issueTicket(true, true);assertEquals(600, t.price);Do we haveenough test ?Ticket issueTicket(bool isStudent,bool extraBanquet){Ticket t;if (isStudent) {t = {price:500, numBanquet:0};} else {t = {price:700, numBanquet:1};}if (extraBanquet) {t.numBanquet = t.numBanquet+1;t.price = t.price + 100;}return t;}Ticket t = issueTicket(false, false);assertEquals(700, t.price);Program under test Test case #1Test case #2Test #1 Test #2100% coverage
  5. 5. is a method to evaluate adequacy of test casesMutation AnalysisEx) Program to calculate 𝑥2 − 𝑦2int diffOfSquares(int x, int y){int sum = x + y;int diff = x – y;return sum * diff;}Ticket issueTicket(bool isStudent,bool extraBanquet){Ticket t;if (isStudent) {t = {price:500, numBanquet:0};} else {t = {price:700, numBanquet:1};}if (extraBanquet) {t.numBanquet = t.numBanquet-1;t.price = t.price + 100;}return t;}Mutant #1by injecting artificial faultMutation operator+ -• Mutant guides improvement of test cases• Mutation operator determine mutantTicket t = issueTicket(true, true);assertEquals(600, t.price);Ticket t = issueTicket(false, false);assertEquals(700, t.price);Test case #1Test case #2Unkilled by test #1,2
  6. 6. is a method to evaluate adequacy of test casesMutation AnalysisEx) Program to calculate 𝑥2 − 𝑦2int diffOfSquares(int x, int y){int sum = x + y;int diff = x – y;return sum * diff;}Ticket issueTicket(bool isStudent,bool extraBanquet){Ticket t;if (isStudent) {t = {price:500, numBanquet:0};} else {t = {price:700, numBanquet:1};}if (extraBanquet) {t.numBanquet = t.numBanquet-1;t.price = t.price + 100;}return t;}Mutant #1by injecting artificial faultMutation operator+ -Ticket t = issueTicket(true, true);assertEquals(1, t.numBanquet);Test case #3• Mutant guides improvement of test cases• Mutation operator determine mutantTicket t = issueTicket(true, true);assertEquals(600, t.price);Ticket t = issueTicket(false, false);assertEquals(700, t.price);Test case #1Test case #2Unkilled by test #1,2killed by test #3
  7. 7. 7How JavaScript Web Applications WorkWeb PageAsync. comm.Event invocatione.g., click, mouseoverUpdate pageSend DataDisplayServerClientHTML/CSSXML/XSLTDOMXHRJavaScriptCan we test JavaScript program adequately?Target of our research
  8. 8. 8Motivating Example: E-Commerce app.open(INDEX_URL);wait(untilElementAppeared(“#detailBtn1”));click(“#detailBtn1”);assertVisible(“#detailWindow”);wait(untilElementAppeared(“.discount”));assertEnabled(“#purchaseBtn”);Test scriptNo exception doesn’t mean no fault Insufficiency of coverage
  9. 9. 9Motivating Example: E-Commerce app.open(INDEX_URL);wait(untilElementAppeared(“#detailBtn1”));click(“#detailBtn1”);assertVisible(“#detailWindow”);wait(untilElementAppeared(“.discount”));assertEnabled(“#purchaseBtn”);Test scriptOn detail loadedNo exception doesn’t mean no fault Insufficiency of coverage
  10. 10. 10Motivating Example: E-Commerce app.open(INDEX_URL);wait(untilElementAppeared(“#detailBtn1”));click(“#detailBtn1”);assertVisible(“#detailWindow”);wait(untilElementAppeared(“.discount”));assertEnabled(“#purchaseBtn”);Test scriptOn detail loadedClickNo exception doesn’t mean no fault Insufficiency of coverage
  11. 11. 11Motivating Example: E-Commerce app.open(INDEX_URL);wait(untilElementAppeared(“#detailBtn1”));click(“#detailBtn1”);assertVisible(“#detailWindow”);wait(untilElementAppeared(“.discount”));assertEnabled(“#purchaseBtn”);Test scriptOn detail loadedClickNo exception doesn’t mean no fault Insufficiency of coverage
  12. 12. 12Motivating Example: E-Commerce app.open(INDEX_URL);wait(untilElementAppeared(“#detailBtn1”));click(“#detailBtn1”);assertVisible(“#detailWindow”);wait(untilElementAppeared(“.discount”));assertEnabled(“#purchaseBtn”);Test scriptOn detail loadedClick After a whileNo exception doesn’t mean no fault Insufficiency of coverage
  13. 13. 13Motivating Example: E-Commerce app.open(INDEX_URL);wait(untilElementAppeared(“#detailBtn1”));click(“#detailBtn1”);assertVisible(“#detailWindow”);wait(untilElementAppeared(“.discount”));assertEnabled(“#purchaseBtn”);Test scriptStatement coverage:100%On detail loadedClick After a whileNo exception doesn’t mean no fault Insufficiency of coverage
  14. 14. 14Motivating Example: E-Commerce app.open(INDEX_URL);wait(untilElementAppeared(“#detailBtn1”));click(“#detailBtn1”);assertVisible(“#detailWindow”);wait(untilElementAppeared(“.discount”));assertEnabled(“#purchaseBtn”);Test scriptStatement coverage:100%On detail loadedClick After a whileNo exception doesn’t mean no fault Insufficiency of coverageFault: wrong property name// should be data.priceelm.setText(data.value);
  15. 15. 15Proposal: Mutation Analysis on JSWAsWeb ApplicationsJavaScript(1)Implement Test!
  16. 16. 16Proposal: Mutation Analysis on JSWAsWeb ApplicationsJavaScript(1)Implement TestMutant!AjaxMutator(2) Generate mutants
  17. 17. 17Proposal: Mutation Analysis on JSWAsWeb ApplicationsJavaScript(1)Implement TestMutantUnkilled Mutant!AjaxMutator(2) Generate mutants
  18. 18. 18Proposal: Mutation Analysis on JSWAsWeb ApplicationsJavaScript(1)Implement Test(4)Add TestMutantUnkilled Mutant!AjaxMutator(2) Generate mutants
  19. 19. (3) ExecuteTest19Proposal: Mutation Analysis on JSWAsWeb ApplicationsJavaScript(1)Implement Test(4)Add TestMutantUnkilled Mutant!AjaxMutator1. Analyze feature of JSWA2. Define mutation operatorsApproach(2) Generate mutants
  20. 20. ①②③Web Page Send DataDisplayServerClientHow JSWA worksXHRXML/XSLTDOMHTML, etc..JavaScript20Three Main Characteristics of JSWAs [1][1] Justin Gehtland, et al. “Pragmatic Ajax: A Web 2.0 Primer”, O’REILLY, 2006① ②③Async. comm.Update pageEvent invocatione.g., click, mouseoverJavaScript Web ApplicationAsynchronous communicationsEvent DOM Manipulation① ② ③
  21. 21. JavaScript Web ApplicationAsynchronous communicationsFeature Diagram for Event RegistrationEvent RegistrationIntervalCallback FunctionTimer Event RegistrationRepeat?TargetEvent TypeUser Event RegistrationCallback FunctionWindow DOM Element Body Return ValueClick Etc…Focus21Mandatory OptionalAlternative Or
  22. 22. Event RegistrationTimer Event Regi22detailButton1.click(showDetail1);Original ProgramloginButton.click(showDetail1);Mutant #1Target Event TypeUser Event RegistrationCallback FunctionWindow DOM Element Body Return ValueClick Etc…FocusDefining Mutation Operators Based on FeaturesClick
  23. 23. Event RegistrationTimer Event Regi23detailButton1.click(showDetail1);Original ProgramloginButton.click(showDetail1);Mutant #1Mutant #2Target Event TypeUser Event RegistrationCallback FunctionWindow DOM Element Body Return ValueClick Etc…FocusDefining Mutation Operators Based on FeaturesClickdetailButton1.mouseover(showDetail1);
  24. 24. Event RegistrationTimer Event Regi24detailButton1.click(showDetail1);Original ProgramloginButton.click(showDetail1);Mutant #1Mutant #2detailButton1.click(showDetail2);Mutant #3Target Event TypeUser Event RegistrationCallback FunctionWindow DOM Element Body Return ValueClick Etc…FocusDefining Mutation Operators Based on FeaturesClickdetailButton1.mouseover(showDetail1);
  25. 25. Mutant #1Event RegistrationTimer Event Regi25ClickdetailButton1.click(showDetail1);Original ProgramloginButton.click(showDetail1);detailButton1.mouseover(showDetail1);Mutant #2detailButton1.click(showDetail2);Mutant #3Target Event TypeUser Event RegistrationCallback FunctionWindow DOM Element Body Return ValueClick Etc…FocusDefining Mutation Operators Based on FeaturesTest inmotivating exampleRequire additional test cases to find faults
  26. 26. 26Timer eventMutant #2setTimeout(showDetail, 1000);setTimeout(applyDiscount1, 1000);Original Program Mutant #1setTimeout(applyDiscount1, 2000);Event RegistrationIntervalCallback FunctionRepeat?nt Registration Timer Event Registration
  27. 27. JavaScript Web Application27RequestResponseAsynchronous CommunicationCallback FunctionURL Method Body Return ValueStatus BodyParameterEvent DOM ManipulatiAsynchronous CommunicationMutant #2$.get(“buy.php”, showButtons);$.get(“items.php”, showMsg);$.get(“items.php”, showButtons);Original ProgramFocus on client-side featuresMutant #1
  28. 28. ommunicationWeb Application28Target Selection ManipulationRelativePositionTag NameCreation DeletionInsertionAttributeValue Attribute AssignmentDOM ManipulationMutant #1Mutant #2$(“.detailBtn”).parent();btn.width = 300;$ (“.detailBtn”);Original Program #1Mutant #1btn.height = 100;btn.width = 100;Original Program #2DOM Manipulation
  29. 29. 1. Can JavaScript features really cause faults?2. Can developers improve test caseswith AjaxMutator to find unexposed faults?3. Can AjaxMutator be used in a reasonable time?29Evaluation: Research Questions
  30. 30. 30RQ1: Actual faults in JSWAs related to features[2] Frolin S. Ocariza Jr. et al. “Autoflox: An automatic fault localizer for client-side javascript”Ticket # Related Feature Brief Explanation1895 User Event Don’t handle on-focus event properly2184 Don’t handle on-resize event properly8812 DOMManipulationGenerating span element where p should be generated9740 Selecting some unwanted elementsSearch from bug repository ofJavaScript features can really cause actual faults“JavaScript”, “JS”, or “console” [2]Keyword
  31. 31. Quizzy (5561LOC)JavaScript(311LOC)(2)Add Tests to killmore mutantsMutantUnkilledMutantAjaxMutator(1)Implement Test* Undergraduate students who has been developed JSWAs for two years in industry• Normal• InvalidRQ2, 3: Case study setupscenario
  32. 32. Faultyprogram×20Faultyprogram×2032Quizzy (5561LOC)JavaScript(311LOC)(2)Add Tests to killmore mutantsMutantUnkilledMutantAjaxMutatorFaultyprogram×20(1)Implement TestParticipant*(3) Seedfault* Undergraduate students who has been developed JSWAs for two years in industry• Normal• InvalidRQ2, 3: Case study setupscenario
  33. 33. Faultyprogram×20Faultyprogram×2033Quizzy (5561LOC)JavaScript(311LOC)(2)Add Tests to killmore mutantsMutantUnkilledMutantAjaxMutatorFaultyprogram×20(1)Implement TestParticipant*(3) Seedfault* Undergraduate students who has been developed JSWAs for two years in industry• Normal• Invalid(4) Check if testcan find faultsRQ2, 3: Case study setupscenario
  34. 34. ×20×2034Quizzy (5561LOC)JavaScript(311LOC)(2)Add Test to killmore mutantsMutantUnkilledMutant×20(1)ImplementTestParticipant*(3) Seedfaults• Normal use case• Another invalid scenario(4) Check if testcan find faultsRQ2: Improvement of test casesWe could improve test cases to find unexposed faultsTest suites # of test cases Coverage Killed mutants Fault finding capabilityInitial 2 95% 45.9% 89.5%Improved +5 100% 67.0% 100%
  35. 35. 35Limitations5 5 4 420842 2116111217462220102030405060Semantically Equivalent (5.5%)Unkilled (27.5%)Killed by improved test (21.1%)Killed by initial test (45.9%)Equivalent mutantsBrowser robustness* target program deal with only click-type event*elm.width=100 -1Browser ignore invalidvalue and infer correct onefurther challengeDetails of generated mutantsval = 100elm.width=100 val
  36. 36. 36RQ3: Time for Mutation AnalysisWeb ApplicationsJavaScript109 Mutants59 UnkilledAjaxMutator20 min.109 mutants × 2 tests20 min.1 hourImplement Test3 hourAdd TestReasonable time; automation/optimization can be applied59 mutants × 5 tests
  37. 37. 37Related Work For HTML and JSP [Praphamontripong’10] For JSWA but limited features [Mirshokraie’13] Faults inWeb apps [Marchetto’07][Guo’09] JavaScript run-time errors [Ocariza’11]Survey on real-world faultsMutation analysis on Web AppsAjaxMuator: comprehensive mutation operators for features of JSWADefinition of mutation operators by a top-down wayNo bug reports can be used for defining mutation operators
  38. 38.  Proposal of mutation operators by feature analysis of JSWAs Publicly available implementation: AjaxMutator http://knishiura-lab.github.io/AjaxMutator/ Survey and case study using real-world apps: AjaxMutator helps aa More case study and survey Automated test case generation38Conclusion & Future WorkConclusionFuture WorkImprovement of test cases to expose faults
  39. 39. Q&A39
  40. 40. 40RQ3: Time for Mutation AnalysisInitial Test Additional TestPreparing test cases(manual)1 hour 3 hoursMutation Analysis(Automated)20 minutes(2 Test cases * 109 mutants)20 minutes(5 Test cases×59 mutants)To reduce manual cost→ We can utilize automated-test case generation techniqueTo reduce mutation analysis cost (computational time)→ We can parallelize execution• AjaxMutator can be used in reasonable time• Automation/Optimization can be applied
  41. 41. 41Defined Mutation OperatorsJS Features Operation target Mutation ExampleUser Event Event target cancelBtn.click(requestBuy);Event type buyBtn.mouseover(requestBuy);Callback function buyBtn.click(closeModal);Timer Event Timer interval setTimeout(anotherCallback, 1000);Callback function setTimeout(callback, 2000);AsynchronousCommunicationsRequest target $.get(“server.php”, callback);Request on-successcallback function$.get(“server.php”, callback);DOM Manipulation Selection of element $(“.btn”).id = “btn1”;Assigned property of DOM $(“#btn”).className = “btn1”;Assigned value to DOM $(“#btn”).id = “btn0”;
  42. 42. Category Mutation Target Ours Mirshokraie’s[3]User event Target, type, callback ✔ NATimer event Interval, callback ✔ NAAsynchronouscommunicationsRequest destination URL ✔ ✔Asynchronous or Synchronouns NA(Out of scope) ✔Callback ✔(on-successcallback)✔(condition on whichcallback is invoked)DOM Element selection ✔ ✔Attribute assignment ✔ ✔Element insertion ✔(Implementedafter submission)✔Other Common mistakes forinexperienced programmersNA ✔42Comparison between Mirshokraie et al.’13[3][3] A. Mirshokraie, Shabnam, et al.“Efficient Javascript Mutation Testing,” Proc. of ICST, 2013

×