FASE 2013 in Rome            22/03/2013Waseda University   Kazunori SakamotoGoogle Japan Inc.   Tomohiro KaizuGoogle Japan...
Introduction• Test oracles for web apps in integration testing  – How should we judge test cases are passed or not ?  e.g....
Template engine • Combine fixed and variable fragments in HTML • Many web frameworks employ template engines    – Closure ...
WB unit / BB integration testingWB (White-box) unit testing BB (Black-box) integration testing           Client side      ...
WB / BB Testing example                                {template .addition}  • Testing target              <p>{$l}+{$r}={$...
Problems on BB testingP1: It is unclear what oracles we should test  S1: Extract template variable fragmentsP2: No indicat...
P1: Test oracles are unclearP2: No indicator of oracle quality for web• It’s unclear what fragments we should check• Tradi...
P3,4: High costs to write/maintain tests                                            Maintenance cost• Test code depending ...
Template variable coverage• Based on checked variable fragments  – Ctmp = Vtest / Vall: #checked / #variable fragments  – ...
Coverage definition in detail• Templates can contain same fragments multiple times• Distinguish same fragments because of ...
Gray-box integration testing                      Client side             • Input template files                 Web brows...
Testing process using POGen  Contain methods for                         Add user-specific attributes to                  ...
Architecture of POGenCan addanalyzers                                                  Element location                   ...
Test code generated by POGen <p>      Original HTML template @Test void test() {    Test case                             ...
Research questions• RQ1: Is template variable coverage  correlated with a tests capability to  detect faults?• RQ2: How ca...
Evaluation: Coverage effectivenessApply mutation testing to Seam Framework sample(booking)   H: Coverage is positively cor...
Evaluation: Test code generation                           Apply POGen to online code execution web appsLOC without commen...
Evaluation: Reduction of costs                                   • 8 people wrote 3 test cases 2 times by filling blanks  ...
Limitations• Dependency on template engines:  Our technique cannot elucidate the  dynamic components which are created  us...
Related work• Staats et al., “Programs, tests, and oracles: the  foundations of testing revisited,” ICSE 2011.   – Argued ...
Conclusion• Template variable coverage   – Ratio of checked template variable fragments• POGen: Tool for generating test c...
Upcoming SlideShare
Loading in...5
×

POGen: A Test Code Generator Based on Template Variable Coverage in Gray-Box Integration Testing for Web Applications

1,076

Published on

http://www.etaps.org/2013/fase13
I talked about this slide at FASE 2013.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,076
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
5
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

POGen: A Test Code Generator Based on Template Variable Coverage in Gray-Box Integration Testing for Web Applications

  1. 1. FASE 2013 in Rome 22/03/2013Waseda University Kazunori SakamotoGoogle Japan Inc. Tomohiro KaizuGoogle Japan Inc. Daigo HamuraWaseda University Hironori WashizakiWaseda University Yoshiaki Fukazawa
  2. 2. Introduction• Test oracles for web apps in integration testing – How should we judge test cases are passed or not ? e.g.) Title of web page? Visibility of specific elements? Contributions 1. Defining fragments should be checked 2. New coverage for html templates 3. Tool for generating skeleton test code 4. Evaluation results of coverage and tool 3/22/2013 FASE 2013 in Rome 2
  3. 3. Template engine • Combine fixed and variable fragments in HTML • Many web frameworks employ template engines – Closure Templates, Razor, Smarty, Django, eRuby {template .addition} Calculation page <p> + = Calc {$l}+{$r}={$ans}</p> Result page Result page {/template} Closure Templates (JavaScript) 1+2=3 2+3=5 Contests depend• Should check variable fragments on input values – Expressions embedded in templates – Faults in fixed fragments can be detected easily (**) Ali Mesbah and Arie van Deursen, “Invariant-based automatic testing ofAJAX user interfaces,” ICSE 2009. 3/22/2013 FASE 2013 in Rome 3
  4. 4. WB unit / BB integration testingWB (White-box) unit testing BB (Black-box) integration testing Client side Check working Client side on Android OS Web browser or iOS Web browser Sele Platform (OS) nium Platform (OS) HTML Selenium HTML JUnit test HTML template HTML template Junit test Server program Server program Check working DB Web Auth on real env DB Web Auth server server server server server server Server side Server side 3/22/2013 FASE 2013 in Rome 4
  5. 5. WB / BB Testing example {template .addition} • Testing target <p>{$l}+{$r}={$ans}</p> – Result page {/template} • White-box unit testing We call @Test void test() { WB Test code with JUnit in Java checking code as int ans = add(1, 2); // Test scenariotest oracle assertEquals(ans, is(3)); }; • Black-box integration testing @Test void test() { BB Test code with Selenium in Java // Open calculation result page (abbr.) WebElement p = d.css_selector(“p”); assertEquals(“1+2=3”, p.getText()); }; 3/22/2013 FASE 2013 in Rome 5
  6. 6. Problems on BB testingP1: It is unclear what oracles we should test S1: Extract template variable fragmentsP2: No indicator of oracle quality for web apps S2: Define template variable coverageP3: High costs to write test code S3: Tool for generating skeleton test codeP4: High costs to maintain test code S4: Accessors based on expression strings 3/22/2013 FASE 2013 in Rome 6
  7. 7. P1: Test oracles are unclearP2: No indicator of oracle quality for web• It’s unclear what fragments we should check• Traditional coverage cannot assess oracles Fixed fragment It shows different contents Variable depending on contexts so fragment that it’s hard to detect faultsHow can we extract variable fragments from HTML ?How can we check variable fragments thoroughly ? 3/22/2013 FASE 2013 in Rome 7
  8. 8. P3,4: High costs to write/maintain tests Maintenance cost• Test code depending on HTML structure – It frequently break when changing web design drv.findElement(By.cssSelector(“p > div”));{template .addition} {template .addition}<p>{$l}+{$r}= <p>{$l}+{$r}= <div>{$ans}</div> <span>{$ans}</span></p>{/template} </p>{/template}• Mark elements containing variables Writing cost – It is hard to add marks such as ID by hand drv.findElement(By.id(“ans”));<div id=“ans”>{$ans} <span id=“ans”>{$ans} 3/22/2013 FASE 2013 in Rome 8
  9. 9. Template variable coverage• Based on checked variable fragments – Ctmp = Vtest / Vall: #checked / #variable fragments – E.g. {$left}, {$(x * 2)}, {$(”ab” + “c”)} – Consider only variable fragments to present{template .result} Closure Template<p>{$left}+{$right}=<div>{$answer}</div></p><ul>{call .items} {param list : $histories /}{/call}</ul>{/template}@Test void test() { Test code with Selenium and JUnit WebElement answer = ... // get div elems Coverage assertEquals(“3”, answer.getText()); }; value: 33% 3/22/2013 FASE 2013 in Rome 9
  10. 10. Coverage definition in detail• Templates can contain same fragments multiple times• Distinguish same fragments because of different concatenations between fixed and variable fragments• Not distinguish fragments in loop statements because they have same concatenations {template .result} Closure Template <p>{$value}</p> Distinguish same <ul>{foreach $item in $items} variable fragments <li>{$item}</li> {/foreach}</ul> <div>{$value}</div> Not distinguish variable {/template} fragments in loop statements 3/22/2013 FASE 2013 in Rome 10
  11. 11. Gray-box integration testing Client side • Input template files Web browser – More than BB info Sele nium Platform (OS) – less than WB infoSelenium HTML – Can test on real env test HTML template • Analyze templates Server program – Generate test code • We propose POGen DB Web Auth server server server – Measure coverage Server side • Monitor with AspectJ 3/22/2013 FASE 2013 in Rome 11
  12. 12. Testing process using POGen Contain methods for Add user-specific attributes to ~~----getting DOM elements ~--~~ access DOM elements which and text values ~~~~ --~~~ have template variable fragments depending on variablenames and expressions Modified template Web pass ~~~~ ~~~~ server fail pass ~~~~ Web app POGen Test result HTML Selenium template ~~~~ ~~~~ ~~~~ ~~~~ JUnit Test Users ~~~~ Test case2 log 0110 case1 0011 Skeleton 1000 test code Execution log 75% 0001 Test Monitor code Coverage Coverage information AspectJ 3/22/2013 FASE 2013 in Rome 12
  13. 13. Architecture of POGenCan addanalyzers Element location Insert user- HTML template analyzer ~~~~ ~~---- ~~~~ Templates Expression string ~--~~ specific attrs analyzer ~~~~ ~~~~ ~~~~ Closure --~~~ Accessors transformer Closure Modified HTML dependign on template Templates 00110 template exp strings HTML 01000 00010 generator info ~~~~ Skeleton test code ~~~~ 00110 ~~~~ analyzer ~~~~ ~~~~ 01000 ejs ~~~~ 00010 01000 00010 Skeleton ejs info test code Coverage infoExpression strings in templates (e.g. {$v + 1}) are morerobust than HTML structures (e.g. XPath or CSS selector) 3/22/2013 FASE 2013 in Rome 13
  14. 14. Test code generated by POGen <p> Original HTML template @Test void test() { Test case by hand {$left}+{$right}= WebDriver d = ... <div>{$answer}</div> // Conducting 1+2 Calculation </p> AdditionPage p = ... Call belowTemplate assertEquals(“3”, accessorsin Closure POGen p.getTextOfAnswer()); }Templates<p id=“_po0”> class AdditionPage exnteds AbstractPage { {$left}+{$right}= /* ---- GENERATED CODE START ---- */ @FindBy(id = “_po1”) <div id=“_po1”> WebElement _answer; {$answer}</div> WebElement getElementOfAnswer(){..}<!-- _po {$left} --> String getTextOfAnswer() {..} //abbrev. ... /* ---- GENERATED CODE END ---- */</p> // user code Generated test code designed Modified HTML temaplte } by PageObject pattern 3/22/2013 FASE 2013 in Rome 14
  15. 15. Research questions• RQ1: Is template variable coverage correlated with a tests capability to detect faults?• RQ2: How can our technique improve a tests capability to find faults?• RQ3: How can our technique facilitate writing test code?• RQ4: How can our technique facilitate maintaining test code?3/22/2013 FASE 2013 in Rome 15
  16. 16. Evaluation: Coverage effectivenessApply mutation testing to Seam Framework sample(booking) H: Coverage is positively correlated with #killed mutantsRatio of killed mutants to mutants 25 15 generated by Simple Jester [%] Pearson’s test Pearson’s test cor: 0.930 2 tests cor: 0.657 20 p: 0.0007 p: 0.076 4 tests 6 tests 10 15 8 tests 10 10 tests 5 12 tests 5 Median of 3 coverage 8 coverage values values selecting some 14 tests selecting 2 test 0 test cases randomly 16 tests 0 cases randomly 0 10 20 30 40 0 10 20 30 Template variable coverage [%] 3/22/2013 FASE 2013 in Rome 16
  17. 17. Evaluation: Test code generation Apply POGen to online code execution web appsLOC without comments and spaces 500 Generate 450 68% test code 400 Test code 38 143 350 No need to maintain by hand code when changing 300 250 200 design (HTML structure) 150 13 304 105 304 12 7 100 6 0 Used test 50 82 47 88 44 43 code 0 0 Index Create Edit Solve Result Pages Test Sum generated sum case by POGen 3/22/2013 FASE 2013 in Rome 17
  18. 18. Evaluation: Reduction of costs • 8 people wrote 3 test cases 2 times by filling blanks • Time to write: left, Checked variable fragments: right #Checked variable fragments 10Time to write 3 test cases [min] 70 p: 0.030 p: 0.084 A 9 60 B 8 7.8 50 C 7 6.3 40 35.3 34.8 D 6 5 E 30 4 F 4 3.5 3 20 G 2 10 H 1 p: 0.0011 p: 0.0150 13.3 11.3 0 Avg. 0 Test cases 1 Test cases 2 Test cases 1 Test cases 2 w/ POGen:left, w/ POGen:left, w/ POGen:left, w/ POGen:left, w/o POGen:right w/o POGen:right w/o POGen:right w/o POGen:right 3/22/2013 FASE 2013 in Rome 18
  19. 19. Limitations• Dependency on template engines: Our technique cannot elucidate the dynamic components which are created using web frameworks or the DOM API without template engines.• Robustness of accessor methods’ names: When a template variable occurs more than once in an HTML template, POGen names the corresponding accessor methods with sequential numbers (1,2...). So some changes cause changes of accessor methods’ names.3/22/2013 FASE 2013 in Rome 19
  20. 20. Related work• Staats et al., “Programs, tests, and oracles: the foundations of testing revisited,” ICSE 2011. – Argued test researches should consider test oracles• Schuler et al., Assessing Oracle Quality with Checked Coverage, ICST 2011. – New statement coverage in consideration of oracles• Sreedevi et al., “Coverage Criteria for Testing Web Applications,” TechnicalPaper 2005. No oracle assessment – Control / data flow coverage by a page• Kodaka et al., “Automated testing for Web applications using Aspect-oriented technology,” SIGSE 2007. – Comparing variable fragments with expected strings in JSP Only comparisons 3/22/2013 FASE 2013 in Rome 20
  21. 21. Conclusion• Template variable coverage – Ratio of checked template variable fragments• POGen: Tool for generating test code – Accessors for getting variable fragments• Evaluation – Positive correlation, Reduction of costs• Future work – Evaluate coverage for web apps in real world – Dynamic analysis for extracting variable fragments3/22/2013 FASE 2013 in Rome 21
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×