Automated System Testing by Steven Erat
Upcoming SlideShare
Loading in...5
×
 

Automated System Testing by Steven Erat

on

  • 4,111 views

Learn how to build a robust, modular, automated test suite for your website's UI that models real users traversing the application. Built with the free ColdFusion tools CFSelenium and MXUnit, tests ...

Learn how to build a robust, modular, automated test suite for your website's UI that models real users traversing the application. Built with the free ColdFusion tools CFSelenium and MXUnit, tests can pull in parameterized input data from external sources to make them more dynamic and flexible. Using the Page Object Design Pattern, test suites can be made more maintainable and less brittle while allowing for faster test development by abstracting how a user interacts with a website, thus minimizing the effort to fix tests when the website UI changes. Pull it all together with ANT and Jenkins for hands free, reliable test suite execution and test result notification on a daily schedule.

Statistics

Views

Total Views
4,111
Views on SlideShare
4,111
Embed Views
0

Actions

Likes
1
Downloads
80
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n

Automated System Testing by Steven Erat Automated System Testing by Steven Erat Presentation Transcript

  • Automated SystemTestingFor Web ApplicationsUsing CFSelenium, MXUnit, & Jenkins CI Steven Erat FirstComp Insurance talkingtree@gmail.com CF.Objective() 2012
  • About metalkingtree@gmail.comtalkingtree.com/blog@stevenerat
  • There’s no handoutDownloads at talkingtree.com/cfo PDF of Slides The Full Demo Test Suite Readme for Instructions
  • About you
  • What to Test?Functional Testing Unit, Integration, System, Regression, Acceptance does a specific feature work? can the user do {this}?Non Functional Testing Security, Performance, Usability cuts across features
  • Unit TestingA type of Functional TestingWritten & run by software developersFinds problems early in DevelopmentTests components in isolationDependencies usually mockedCan’t test legacy (non-CFC) code
  • System TestingA type of Functional TestingTests the completed software productOften performed manuallyCan be automated with testing tools: QTP - The $6000 Gorilla Selenium - FOSS; Has many language APIs
  • Continuous IntegrationContinuous process of quality control Development - Creates “The Build” QA - Test Suite ExecutionPerformed Frequently after each commit, or scheduled regularly (daily, hourly, etc)CI Software: Jenkins, Bamboo,CruiseControl
  • How to Test?System Testing mimics user interaction with GUIQTP - Expensive, complex, scriptable (VB)TestNG Java Test Framework withSelenium Free. Write tests in Java.MXUnit ColdFusion Framework withCFSelenium
  • The LandscapeTest Suite is independent of AUT AUT 1 Test Suite AUT 2 Framework AUT 3 ColdFusion MXUnit CFSelenium Application(s) Under Test
  • What is Selenium?Selenium IDE record/playback user actions in a browser recorded in Selenese, a special test scripting languageSelenium Client API alternative to Selenese, tests can also be written in various programming languagesSelenium (RC) Server Java-based server that accepts commands for browser
  • Selenium IDEFirefox pluginRecord in HTML,Java, Python, etcPlayback inbrowser
  • What is CFSelenium?Extends theSelenium IDERecord as CFMLin MXUnitTestCase
  • CFSelenium component extends="cfselenium.CFSeleniumTestCase" { public void function beforeTests() {Static input values browserUrl = "http://localhost:8500/"; }like ‘username’, public void function testSimple_blog_admin_login() {‘password’ selenium.open("/blog/admin/index.cfm"); assertEquals("BlogCFC Administrator", selenium.getTitle());Linear & non- selenium.click("css=input[type="submit"]"); selenium.waitForPageToLoad("30000");iterative assertTrue(selenium.isTextPresent("Welcome")); assertEquals("Logout", selenium.getText("link=Logout"));Object identification selenium.click("link=Logout"); selenium.waitForPageToLoad("30000");is mixed with assertEquals("BlogCFC Administrator: Logon",assertions selenium.getTitle()); } }
  • CFSeleniumselenium.cfcSelenium ClientWrapper
  • Selenese CommandsActions open a URL, click a link, wait for page to loadAccessors store results of actions in a variable: storeTitleAssertions assert (failure stops further test method) verify (test method continues after failure)
  • Selenese HTML Format <tr> <td>open</td> <td>/blog/admin/index.cfm?</td> <td></td> </tr> <tr>Selenium <td>assertTitle</td> <td>BlogCFC Administrator: Logon</td>performs the <td></td> </tr> <tr>Assertion <td>click</td> <td>css=input[type=‘submit’;]</td>during playback <td></td> </tr> <tr> <td>waitForPageToLoad</td> <td>30000</td> <td></td> </tr>
  • Selenese CFML Formatselenium.open("/blog/admin/index.cfm?");assertEquals("BlogCFC Administrator: Logon", selenium.getTitle());selenium.click("css=input[type=submit]");selenium.waitForPageToLoad("30000"); MXUnit performs the Assertion during playback Verify() not supported by MXUnit, use Assertions
  • Object LocatorsStrategies for locating objects/elementson pageselenium.isElementPresent(id=username) selenium.isElementPresent(name=username) id selenium.select(‘document.forms[myForm].cb1 name ’) dom selenium.click(“//a[contains(@href,#id1)]”) xpath selenium.click(‘link=Add Comment”) link selenium.click(css=input[type="submit"]) css selenium.getText(‘regexpi:^[A-Z0-9+_.-] +@[A-Z0-9.-]+$’)
  • What is MXUnit?CFML-based UnitTest FrameworkColdFusionBuilder & EclipsepluginAntTask
  • Anatomy of a TestCasebeforeTests()afterTests()setup()tearDown()test functions
  • MXUnit DataprovidersIterate over test methods using native CF datastructuresTest multiple logins with list of usernames &
  • MXUnit PluginTest RunnerPlugin for CFBuilderRun whole TestSuitesRun singleTestCases
  • What are the benefits sofar?Quick testcase creation in familiarlanguageFramework for test execution & reportingSystem Regression Testing
  • What are the problems?Object locators hard coded in eachtestcaseObject locators repeated thru testcase/suiteAssertions interwoven with SeleniumcommandsStatic tests embedded with literal input
  • Solutions: POMThe Page Object Model design patternModels web page or page partindependently navigation bar, sidebar, pods, footerObject identifiers exist in only one placeObject oriented approach to testcasecreation
  • Solutions: POMDante BrionesMust watch
  • What to Model?BannerNavBarSearchBlog Entry CommentsAdd CommentPods
  • Modeling a Part / ScreenAdd Comment pop-upOne function for eachform field, button, orwidgetFunctions isolate theobject locatorMethod chaining
  • Before using POM
  • Using Page Object Model
  • Method ChainingPOM methods named intuitivelyImproved readabilityFaster to type with Code Insight **
  • The Landscape Logical divisions in AUT can be tested Test Suite Framework Applications Under Test
  • TestCase« » Page Test doesn’t know about object locators Page Object doesn’t know about AssertsPage can returninstance of nextpage: loginAsreturns Home
  • Page InheritanceHomePage, SearchPage, andContactPage extend BaseNavItem
  • Parameterized Test DataExternal data read for use as MXUnitdataprovidersEasy to modify data by non-technical usersJExcel let’s you easily use multiple tables
  • WorkflowRecord user interaction in a browser Performed by QA or Business User Use Selenium IDE with CFML Format Produces linear recordings with static dataEnhance static scripts Performed by QA Software Engineer Convert to Page Object Model Convert for Parameterized Data (Dataproviders) Commit to Source Control
  • Recording UserInteractionSelenium IDE records object interactionsCan change priority of which locators ituses id, name, link, XPath, DOM, etcDuring recording you may add Assertionsmanually
  • Recording UserInteraction
  • Running MXUnit viaPlugin
  • Running MXUnit via Ant
  • Ant Build Files Have it your way
  • Ant Properties FilesCan use a global properties file shared aspects across entire test frameworkCan also use local properties files store config values unique to each AUT test suiteFlexible configuration - It’s up to you
  • Global Properties Define a property to switch testing environments Dev or Staging? Tell Selenium which browser to launch
  • Local Properties Properties unique to each UAT being tested Define properties to initialize Selenium - browserURL $ENV swapped to match global env property (Dev or Staging)
  • MXUnit Entry Points MXUnit kicks off Test Suite via a Remote Method call Plugin test runner calls RemoteProxy.cfc Ant test runner calls HttpAntRunner.cfc
  • How are Tests Run SoFar?ManuallyLocally per IndividualExecute Plugin or Ant Test Runner
  • How to Improve?AutomationCentral Test EnvironmentReportingNotifications
  • Continuous IntegrationBambooJenkinsCruiseControl
  • JenkinsCI
  • JenkinsCICreate ‘jobs’ to do workCall Ant TasksRun Source ControlSchedule jobs using cron syntaxChain jobs to run in sequenceStatus / Trends / HistoryExceptions Stack TracesEmail Notifications
  • JobsJobs for discrete Test SuitesStatus and Trend iconsLast Run and Duration
  • Meta JobsJobs that trigger other jobsJobs that restart serversJobs that sync source control
  • Job Trends
  • TestCase History
  • Top Level Dashboard
  • Top Level Dashboard
  • Demo?
  • Lessons LearnedArchitect a meta framework to supportmultiple test suites with better codereuse
  • Lessons LearnedStub out Page Objects up frontOrganize test suite Page and Test directorystructure by logical areas in the AUT Blog Home, Blog Entry, Comment, Contact, Search, RSS ... Blog Admin Login, Home, Edit Entry, Refresh, Stats ...Test writers can then work in parallelwithout duplication
  • Lessons LearnedConvince AUT Developers Testing WorksDevelop AUTs with better object id/naming unique IDs better than CSS, DOM, XPath locatorsDisambiguate significant data in AUTbody
  • Lessons LearnedDesign Page Objects for method chainingLoginPage.enterUserID(‘me’).enterPass(‘pw’).submit();Easier to readTakes advantage of code insightCF Builder’s code insight not quite there yetOther IDEs may provide better code insight(IntelliJ)
  • Lessons LearnedAlways perform LogOut() in beforeTests()methodRun Login() in the setUp() methodRun LogOut() in the tearDown() methodTests no longer need to perform loginactions (you should already have tests just for Login)
  • SnippetsTest Cases, Page Objects, Debugging
  • TroubleshootingResponse of the Selenium RC is invalid:ERROR: Element css=p >input[name="search"] not foundResponse of the Selenium RC is invalid:Current window or frame is closed! Get the AUT’s HTML Source via Selenium Have Selenium grab a screenshot of AUT
  • TroubleshootingResponse of the Selenium RC is invalid:ERROR: Element //input[name="search"]not found
  • TroubleshootingUse custom messages for AssertionsMake them more meaningfulSave more hair
  • Troubleshooting Jenkins will show your custom message for failures
  • Troubleshooting Slow the test execution down Rerun the test via MXUnit Plugin Watch the browser as test runs Add sleep(NNNN) statements at key points
  • Troubleshooting // Runs once before all test methods in file public void function beforeTests() { // dataproviders must be in the variables scope variables.TwoCommenters = application.dataproviders[TwoCommenters]; variables.selenium = application.selenium; variables.HomePage = new HomePage(selenium); variables.defaultSpeed = selenium.getSpeed(); selenium.setSpeed(1000); } // Runs once after all test methods in file public void function afterTests() { HomePage.clickNavButton_Home(); assertEquals(HomePage.getExpectedTitle(), selenium.getTitle()); selenium.setSpeed(variables.defaultSpeed); }
  • Troubleshooting Use Tasks view with keywords //TODO or // FIXME
  • Troubleshooting A TestCase method failed Did the TestCase change? Did a Page Object change? Check TestSuite Source Control for changes since last success Did the AUT change?
  • TroubleshootingA Job in Jenkins Failed to Build Check the Test Suite’s ColdFusion exception.log
  • Thank you!Steven Erattalkingtree@gmail.comGet Slides and Demo TestSuite code at:talkingtree.com/cfo
  • Some Referenceshttp://en.wikipedia.org/wiki/Software_testinghttp://en.wikipedia.org/wiki/Unit_testinghttp://en.wikipedia.org/wiki/Integration_testinghttp://en.wikipedia.org/wiki/System_testinghttp://en.wikipedia.org/wiki/Regression_testinghttp://en.wikipedia.org/wiki/Continuous_integrationhttp://en.wikipedia.org/wiki/Selenium_%28software%29http://seleniumhq.org/docs/02_selenium_ide.html#selenium-commands-selenesehttp://release.seleniumhq.org/selenium-remote-control/0.9.2/doc/dotnet/Selenium.htmlhttp://www.codediesel.com/testing/selenium-ide-pattern-matching/http://wiki.mxunit.org/http://www.mxunit.org/http://wiki.openqa.org/display/SEL/Homehttp://seleniumhq.org/docs/http://release.seleniumhq.org/selenium-remote-control/0.9.2/doc/dotnet/Selenium.htmlhttp://wiki.openqa.org/display/SEL/Help+With+XPathhttp://blog.browsermob.com/2009/04/test-your-selenium-xpath-easily-with-firebug/http://code.google.com/p/selenium/wiki/PageObjectshttp://www.viddler.com/explore/saucelabs/videos/3/http://www.silverwareconsulting.com/index.cfm/2011/2/22/Introducing-CFSelenium--A-Native-ColdFusion-Client-Library-for-SeleniumRChttp://www.thoughtdelimited.org/thoughts/archives.cfm/category/seleniumhttp://wiki.mxunit.org/display/default/MXUnit+Documentationhttp://blog.mxunit.org/2008/03/new-mxunit-ant-automation-video.htmlhttp://selftechy.com/2011/06/02/parameterization-of-selenium-tests-with-microsoft-excel