Seconf2011 database driven combinatorial testing

Get More For Less!Database Driven Combinatorial Testing Aaron Silverman Lead Engineer Applied Predictive Technologies www.predictivetechnologies.com

BackgroundAPT produces business analytic software used by largeorganizations to make decisions worth millions of dollars It is critically important that: • Our software behave as expected • The numbers we display are correct page 2

AgendaWe aim to share the lessons we have learned • Front-end testing can only be made so fast • An alternate way to reduce testing time is to reduce the number of tests—but you don’t want to reduce testing coverage • Combinatorial test selection techniques allow selective test case creation and intelligent test case reduction • Moving test case inputs and expected outputs out of the code and into a testing framework database makes leveraging combinatorial testing easy • The database also makes test case maintenance and debugging transparent and easy page 3

Front End Testing Is SlowEnd-to-end Selenium tests most accurately simulate an actual userusing the product, making them slower than other automated tests March 2011 APT Continuous Integration Automated Testing Stats Test Type Test Cases Run Average Time Per Test Case Unit 434,016 6 seconds Functional 91,912 21 seconds Selenium 13,652 983 seconds page 4

Test Cases Can Not Always Be ShortenedDespite extensive parallelization, some test cases can only be sofast without deviating from real world use cases Large Datasets Complex Computations Slow Tests page 5

Number of Tests Can Often Be ReducedHowever, testing time can still be reduced by running fewer tests Hey! Are you sure about this? Note: No turtles were harmed in the making of this presentation page 6

Number of Tests Can Often Be Reduced Run fewer tests? Doesn’t that mean we are testing less? page 7

Number of Tests Can Often Be Reduced Not if we pick our test cases using combinatorial testing! page 8

Number of Tests Can Often Be Reduced page 9

Intelligently Pick Test CasesCombinatorial testing focuses on testing specific relevantcombinations of inputsThere is a plethora of research andresources about combinatorial testing,especially 2-way, or pairwise testing. • Hundreds of Papers • Dozens of ToolsAPT uses the freely available Microsoft’s Pairwise IndependentCombinatorial Testing, also known as PICT, as well as some tools wedeveloped ourselves Download PICT at: http://msdn.microsoft.com/en-us/testing/bb980925.aspx page 10

Simplified ExampleSuppose we had an attendee survey for this conference… Attendee Survey Name: Do you like Selenium? Yes No Rate the conference: (Scale 1-10) page 11

Simplified Example…and if you do not like selenium, it asks why Attendee Survey Name: Do you like Selenium? Yes No because (check all that apply): I have too many slow tests The devs keep changing the UI My favorite browser is Lynx I like to complain a lot Rate the conference: (Scale 1-10) page 12

Simplified ExampleThere is lots to test! Attendee Survey • No Name Entered Name: • Name Entered Do you like Selenium? • Yes Yes • No No, because (check all that apply) I have too many slow tests 16 permutations of checked The devs keep changing the UI selections when • Valid and My favorite browser is Lynx “No” is selected favorable I like to complain a lot • Valid and unfavorable Rate the conference: (Scale 1-10) • Invalid • Non-integer page 13

Simplified ExampleExhaustive testing would result in 136 permutations of inputs Clearly this is absurd! page 14

Test Case Creation ToolsUsing pairwise tools we can reduce our test cases to a muchsmaller set that maintains coverage quality pict_inputs.txt name: blank, filled likeSelenium: yes, no specify input names and slowTests: checked, unchecked possibilities changingUI: checked, unchecked usesLynx: checked, unchecked complainsALot: checked, unchecked exclude impossible rating: favorable, unfavorable, invalid, non-integer combinations #if user likes selenium, no reasons can be checked IF [likeSelenium] = "yes" THEN [slowTests] = "unchecked" AND [changingUI] = "unchecked" AND [usesLynx] = "unchecked" AND [complainsALot] = "unchecked"; Note: PICT is far more powerful than the options shown here page 15

Test Case Creation ToolsPICT will then generate the optimal set of test cases that coverall possible input pairs meeting our specified conditionsC:Program Files (x86)PICT>pict.exe pict_inputs.txt > pict_outputs.xls pict_outputs.xlsname likeSelenium slowTests changingUI usesLynx complainsALot ratingblank no checked checked checked checked favorableblank no checked checked unchecked checked unfavorableblank no unchecked checked unchecked checked non-integerblank yes unchecked unchecked unchecked unchecked favorableblank yes unchecked unchecked unchecked unchecked invalidfilled no checked checked checked checked invalid Where did all myfilled no checked unchecked checked unchecked non-integer friends go?filled no unchecked checked unchecked unchecked favorablefilled no unchecked unchecked checked checked favorablefilled no unchecked unchecked checked checked unfavorablefilled yes unchecked unchecked unchecked unchecked non-integerfilled yes unchecked unchecked unchecked unchecked unfavorable Note: PICT is far more powerful than the options shown here page 16

We should test outputs this way tooOur survey also has a few different outputs to validate: Attendee Survey Attendee Survey You seem to be having Rating >= 5:Any Invalid input: trouble following Glad you liked the directions, please try conference! again! Attendee Survey Attendee Survey “No” selected, a rating of 1, and Rating < 5: Sorry you did not like You should consider a all checkboxes the conference. career change. selected: page 17

Complete Coverage Is Not Always CompleteThe outputs are related to the inputs but are not inputsthemselves, so we treat them as “metadata” pict_outputs.xls name likeSelenium slowTests changingUI usesLynx complainsALot rating metadata - result blank no checked checked checked checked favorable try again blank no checked checked unchecked checked unfavorable try again blank no unchecked checked unchecked checked non-integer try again blank yes unchecked unchecked unchecked unchecked favorable try again blank yes unchecked unchecked unchecked unchecked invalid try again filled no checked checked checked checked invalid try again filled no checked unchecked checked unchecked non-integer try again filled no unchecked checked unchecked unchecked favorable glad filled no unchecked unchecked checked checked favorable glad filled no unchecked unchecked checked checked unfavorable sorry filled yes unchecked unchecked unchecked unchecked non-integer try again filled yes unchecked unchecked unchecked unchecked unfavorable sorry Even if the unfavorable rating is made a 1, we are still not testing the “career change” result screen page 18

Seeding Helps Achieve Desired CoverageUsing seeding we can intelligently adjust our set of inputsWe know we want at least one test that gets the “careerchange” result. One way to do this is through seeding: pict_seeds.txt name likeSelenium slowTests changingUI usesLynx complainsALot rating filled no checked checked checked checked unfavorableC:Program Files (x86)PICT>pict.exe pict_inputs.txt /e:pict_seeds.txt > pict_outputs.xls Attendee Survey Now we are forcing PICT to include the set of combinations that will You should consider a generate the career change result career change. Note: PICT is far more powerful than the options shown here page 19

Seeding Helps Achieve Desired CoverageWith our seed in place, PICT generates a new set of combinations pict_outputs.xlsname likeSelenium slowTests changingUI usesLynx complainsALot rating metadata - resultblank no checked checked unchecked checked unfavorable try againblank no checked checked unchecked unchecked favorable try againblank no checked unchecked checked checked favorable try againblank no unchecked checked unchecked checked non-integer try againblank no unchecked unchecked checked unchecked unfavorable try againblank yes unchecked unchecked unchecked unchecked invalid try againblank yes unchecked unchecked unchecked unchecked non-integer try againfilled no checked checked checked checked invalid try againfilled no checked checked checked checked unfavorable career changefilled no checked unchecked checked unchecked non-integer try againfilled yes unchecked unchecked unchecked unchecked favorable gladfilled yes unchecked unchecked unchecked unchecked unfavorable sorry Excellent! We can test everything! Too bad the real world isn’t this simple… page 20

Real World Products Are Incredibly ComplexIn the real world, test cases are extremely complicated! • Store location with custom drawn trade area • Map modification • Competition layering • Boundary highlighting • Enabling heat map • Map Detail • Shading attribute selection • Map controls • Product navigation page 21

Practical Test Case SelectionThe approach we’ve found that works best revolves around bothcoverage of inputs and metadata • When there is not much metadata that needs to be considered, the first pass using combinatorial tools works great at test case generation • When there is lots of metadata, we generally intentionally start with too many tests and then trim down using our coverage tools • Using the code to programmatically populate metadata values during test case execution helps keeps inputs and metadata in sync as we adjust our test cases • Seeding is a great way to force in the test cases to ensure the coverage we want • Evaluation of testing coverage is an iterative process during test case development page 22

Using The Test CasesWriting many test cases, where only the inputs and outputschange, results in a lot of code duplication@Testpublic void test_likesSelenium_likesConference() throws Exception { surveyPage.enterName("Aaron"); surveyPage.selectLikesSelenium(true); surveyPage.enterRating("5"); surveyPage.submit(); resultPage.synchronize(); Assert.assertEquals("Glad you liked the conference!", resultPage.getResponseMessage());}@Testpublic void test_dislikes_Selenium_hatesConference() throws Exception { surveyPage.enterName("Aaron"); surveyPage.selectLikesSelenium(false); surveyPage.checkDislikeReason(1); surveyPage.checkDislikeReason(3); surveyPage.enterRating("1"); surveyPage.submit(); resultPage.synchronize(); Assert.assertEquals("Sorry you did not like the conference.", resultPage.getResponseMessage());} DRY! Note: DRY == Do not Repeat Yourself page 23

Database Backed Frameworks Avoid Duplication At APT, we store inputs and outputs in the database, and access them by simple helper methods@Testpublic void test_conferenceSurvey(int testCaseId) throws Exception { setTestCaseId(testCaseId); surveyPage.enterName(getInputValue("name")); surveyPage.selectLikesSelenium(getInputValue("likesSelenium").equals("yes")); String[] reasons = getInputValue("reasonList").split(","); for (String s : reasons) { surveyPage.enterRating(s); } surveyPage.submit(); resultPage.synchronize(); compareAgainstExpectedOutput("Result Page Message", resultPage.getResponseMessage());} Like me, this • Variable length inputs stored as lists function can • Comparisons usually non-fatal handle • Caching used to avoid continuously everything! querying the database • Test cases have names and comments associated with them page 24

The Database Also Improves Testing OverallStoring inputs and outputs in the database opens up manypossibilities for test case creation, evaluation and debugging • Creating new test cases is easier • Database-backed web-based test case editor often easier to use than having inputs in code, excel, wiki, etc. • Developing tools to evaluate coverage is easier • No need to parse files, just query the database • Easy to specify impossible pairs • Easy to specify values not tested • Debugging is easier for engineers • History of inputs and outputs (linked to screenshots) can be tracked • Easy to store reasons for change of input or expected output page 25

Creating Test Cases Is Transparent and EasyFollowing import from pairwise selection tools, database driventest cases can now be easily edited and configured by anyone Test cases easily evaluated and adjusted by: • Testers • Engineers • Product Management Note: Yes we know our internal tools like they are from 1990—but they work great! page 26

Evaluating Coverage is Transparent and Easy With complex tests, coverage reports make identifying both extraneous and missing test cases easyFor this one test:• 38 test cases currently cover 3132 of 7917 pairs• 9 test cases are not adding any additional coverage These test cases• It is still easy to identify gaps in testing—look for the red! need some tuning! page 27

Debugging Results Is Transparent and EasySimple reporting pages can make debugging test failures andupdating test cases much easier Ability to accept new actual outputs as the expected This test case checked 657 outputs; 255 of them failed outputs going forward Ability to see history of output changes Identifiers easily Actual value with Expected value Date output was recognized by link to screenshot with link to set and link to engineers of when it was set screenshot of the information about failure the test where it was set from page 28

Debugging Results Is Transparent and EasyScreenshots can be stored in the database and associated withspecific test case executions It is easy to view all screenshots for this run of test case, the last time this test case passed, etc. page 29

Easy To Integrate With Continuous IntegrationWith lots of information in the database, it is also easy toaggregate up to the level desired for continuous integrationAt APT we aggregate the results and then format them into junit style XMLfiles to allow easy integration with JenkinsHowever each test case still provides a link to drill into the reports shownin the previous slides that contain screenshots, inputs/outputs, logs, etc. Note: Jenkins was formerly know as Hudson page 30

The Big PictureCombinatorial testing is just one part of a well-balancedtesting diet Unit tests Code coverage tools let us know which lines of code we are and are not testing Functional Tests Combinatorial coverage tools let us know which input combinations we are and are not testing Selenium Tests Exploratory testing alerts us to new Manual Tests important inputs and interactions to consider for our automated test cases page 31

The Big PictureDatabase driven combinatorial testing lets you get more for less!• End-to-end selenium tests can be complicated and slow• Combinatorial testing is a great way to reduce the number of these slow tests while maximizing testing coverage• Test metadata must be considered along with inputs when evaluating coverage• For complex tests, tools for test case creation, evaluating coverage, and debugging failures are essential• Storing inputs, outputs, and metadata in a database makes developing test case management and coverage reporting tools easy page 32

Questions Whew that was a lot! Do you think the audience has any questions? page 33

Questions Only one way to find out! Look for hands! page 34

Questions page 35

QuestionsQuestions? page 36

http://www.techgig.com	49
http://www.m.techgig.com	1
http://www.linkedin.com	1

Seconf2011 database driven combinatorial testing

by Zugwalt on Apr 06, 2011

Accessibility

Categories

Upload Details

Usage Rights

3 Embeds 51

Statistics

Seconf2011 database driven combinatorial testing Presentation Transcript