Automated System Testing by Steven Erat

5,179 views

Published on

Learn how to build a robust, modular, automated test suite for your website's UI that models real users traversing the application. Built with the free ColdFusion tools CFSelenium and MXUnit, tests can pull in parameterized input data from external sources to make them more dynamic and flexible. Using the Page Object Design Pattern, test suites can be made more maintainable and less brittle while allowing for faster test development by abstracting how a user interacts with a website, thus minimizing the effort to fix tests when the website UI changes. Pull it all together with ANT and Jenkins for hands free, reliable test suite execution and test result notification on a daily schedule.

Published in: Spiritual, Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
5,179
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
96
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Automated System Testing by Steven Erat

    1. 1. Automated SystemTestingFor Web ApplicationsUsing CFSelenium, MXUnit, & Jenkins CI Steven Erat FirstComp Insurance talkingtree@gmail.com CF.Objective() 2012
    2. 2. About metalkingtree@gmail.comtalkingtree.com/blog@stevenerat
    3. 3. There’s no handoutDownloads at talkingtree.com/cfo PDF of Slides The Full Demo Test Suite Readme for Instructions
    4. 4. About you
    5. 5. What to Test?Functional Testing Unit, Integration, System, Regression, Acceptance does a specific feature work? can the user do {this}?Non Functional Testing Security, Performance, Usability cuts across features
    6. 6. Unit TestingA type of Functional TestingWritten & run by software developersFinds problems early in DevelopmentTests components in isolationDependencies usually mockedCan’t test legacy (non-CFC) code
    7. 7. System TestingA type of Functional TestingTests the completed software productOften performed manuallyCan be automated with testing tools: QTP - The $6000 Gorilla Selenium - FOSS; Has many language APIs
    8. 8. Continuous IntegrationContinuous process of quality control Development - Creates “The Build” QA - Test Suite ExecutionPerformed Frequently after each commit, or scheduled regularly (daily, hourly, etc)CI Software: Jenkins, Bamboo,CruiseControl
    9. 9. How to Test?System Testing mimics user interaction with GUIQTP - Expensive, complex, scriptable (VB)TestNG Java Test Framework withSelenium Free. Write tests in Java.MXUnit ColdFusion Framework withCFSelenium
    10. 10. The LandscapeTest Suite is independent of AUT AUT 1 Test Suite AUT 2 Framework AUT 3 ColdFusion MXUnit CFSelenium Application(s) Under Test
    11. 11. What is Selenium?Selenium IDE record/playback user actions in a browser recorded in Selenese, a special test scripting languageSelenium Client API alternative to Selenese, tests can also be written in various programming languagesSelenium (RC) Server Java-based server that accepts commands for browser
    12. 12. Selenium IDEFirefox pluginRecord in HTML,Java, Python, etcPlayback inbrowser
    13. 13. What is CFSelenium?Extends theSelenium IDERecord as CFMLin MXUnitTestCase
    14. 14. CFSelenium component extends="cfselenium.CFSeleniumTestCase" { public void function beforeTests() {Static input values browserUrl = "http://localhost:8500/"; }like ‘username’, public void function testSimple_blog_admin_login() {‘password’ selenium.open("/blog/admin/index.cfm"); assertEquals("BlogCFC Administrator", selenium.getTitle());Linear & non- selenium.click("css=input[type="submit"]"); selenium.waitForPageToLoad("30000");iterative assertTrue(selenium.isTextPresent("Welcome")); assertEquals("Logout", selenium.getText("link=Logout"));Object identification selenium.click("link=Logout"); selenium.waitForPageToLoad("30000");is mixed with assertEquals("BlogCFC Administrator: Logon",assertions selenium.getTitle()); } }
    15. 15. CFSeleniumselenium.cfcSelenium ClientWrapper
    16. 16. Selenese CommandsActions open a URL, click a link, wait for page to loadAccessors store results of actions in a variable: storeTitleAssertions assert (failure stops further test method) verify (test method continues after failure)
    17. 17. Selenese HTML Format <tr> <td>open</td> <td>/blog/admin/index.cfm?</td> <td></td> </tr> <tr>Selenium <td>assertTitle</td> <td>BlogCFC Administrator: Logon</td>performs the <td></td> </tr> <tr>Assertion <td>click</td> <td>css=input[type=‘submit’;]</td>during playback <td></td> </tr> <tr> <td>waitForPageToLoad</td> <td>30000</td> <td></td> </tr>
    18. 18. Selenese CFML Formatselenium.open("/blog/admin/index.cfm?");assertEquals("BlogCFC Administrator: Logon", selenium.getTitle());selenium.click("css=input[type=submit]");selenium.waitForPageToLoad("30000"); MXUnit performs the Assertion during playback Verify() not supported by MXUnit, use Assertions
    19. 19. Object LocatorsStrategies for locating objects/elementson pageselenium.isElementPresent(id=username) selenium.isElementPresent(name=username) id selenium.select(‘document.forms[myForm].cb1 name ’) dom selenium.click(“//a[contains(@href,#id1)]”) xpath selenium.click(‘link=Add Comment”) link selenium.click(css=input[type="submit"]) css selenium.getText(‘regexpi:^[A-Z0-9+_.-] +@[A-Z0-9.-]+$’)
    20. 20. What is MXUnit?CFML-based UnitTest FrameworkColdFusionBuilder & EclipsepluginAntTask
    21. 21. Anatomy of a TestCasebeforeTests()afterTests()setup()tearDown()test functions
    22. 22. MXUnit DataprovidersIterate over test methods using native CF datastructuresTest multiple logins with list of usernames &
    23. 23. MXUnit PluginTest RunnerPlugin for CFBuilderRun whole TestSuitesRun singleTestCases
    24. 24. What are the benefits sofar?Quick testcase creation in familiarlanguageFramework for test execution & reportingSystem Regression Testing
    25. 25. What are the problems?Object locators hard coded in eachtestcaseObject locators repeated thru testcase/suiteAssertions interwoven with SeleniumcommandsStatic tests embedded with literal input
    26. 26. Solutions: POMThe Page Object Model design patternModels web page or page partindependently navigation bar, sidebar, pods, footerObject identifiers exist in only one placeObject oriented approach to testcasecreation
    27. 27. Solutions: POMDante BrionesMust watch
    28. 28. What to Model?BannerNavBarSearchBlog Entry CommentsAdd CommentPods
    29. 29. Modeling a Part / ScreenAdd Comment pop-upOne function for eachform field, button, orwidgetFunctions isolate theobject locatorMethod chaining
    30. 30. Before using POM
    31. 31. Using Page Object Model
    32. 32. Method ChainingPOM methods named intuitivelyImproved readabilityFaster to type with Code Insight **
    33. 33. The Landscape Logical divisions in AUT can be tested Test Suite Framework Applications Under Test
    34. 34. TestCase« » Page Test doesn’t know about object locators Page Object doesn’t know about AssertsPage can returninstance of nextpage: loginAsreturns Home
    35. 35. Page InheritanceHomePage, SearchPage, andContactPage extend BaseNavItem
    36. 36. Parameterized Test DataExternal data read for use as MXUnitdataprovidersEasy to modify data by non-technical usersJExcel let’s you easily use multiple tables
    37. 37. WorkflowRecord user interaction in a browser Performed by QA or Business User Use Selenium IDE with CFML Format Produces linear recordings with static dataEnhance static scripts Performed by QA Software Engineer Convert to Page Object Model Convert for Parameterized Data (Dataproviders) Commit to Source Control
    38. 38. Recording UserInteractionSelenium IDE records object interactionsCan change priority of which locators ituses id, name, link, XPath, DOM, etcDuring recording you may add Assertionsmanually
    39. 39. Recording UserInteraction
    40. 40. Running MXUnit viaPlugin
    41. 41. Running MXUnit via Ant
    42. 42. Ant Build Files Have it your way
    43. 43. Ant Properties FilesCan use a global properties file shared aspects across entire test frameworkCan also use local properties files store config values unique to each AUT test suiteFlexible configuration - It’s up to you
    44. 44. Global Properties Define a property to switch testing environments Dev or Staging? Tell Selenium which browser to launch
    45. 45. Local Properties Properties unique to each UAT being tested Define properties to initialize Selenium - browserURL $ENV swapped to match global env property (Dev or Staging)
    46. 46. MXUnit Entry Points MXUnit kicks off Test Suite via a Remote Method call Plugin test runner calls RemoteProxy.cfc Ant test runner calls HttpAntRunner.cfc
    47. 47. How are Tests Run SoFar?ManuallyLocally per IndividualExecute Plugin or Ant Test Runner
    48. 48. How to Improve?AutomationCentral Test EnvironmentReportingNotifications
    49. 49. Continuous IntegrationBambooJenkinsCruiseControl
    50. 50. JenkinsCI
    51. 51. JenkinsCICreate ‘jobs’ to do workCall Ant TasksRun Source ControlSchedule jobs using cron syntaxChain jobs to run in sequenceStatus / Trends / HistoryExceptions Stack TracesEmail Notifications
    52. 52. JobsJobs for discrete Test SuitesStatus and Trend iconsLast Run and Duration
    53. 53. Meta JobsJobs that trigger other jobsJobs that restart serversJobs that sync source control
    54. 54. Job Trends
    55. 55. TestCase History
    56. 56. Top Level Dashboard
    57. 57. Top Level Dashboard
    58. 58. Demo?
    59. 59. Lessons LearnedArchitect a meta framework to supportmultiple test suites with better codereuse
    60. 60. Lessons LearnedStub out Page Objects up frontOrganize test suite Page and Test directorystructure by logical areas in the AUT Blog Home, Blog Entry, Comment, Contact, Search, RSS ... Blog Admin Login, Home, Edit Entry, Refresh, Stats ...Test writers can then work in parallelwithout duplication
    61. 61. Lessons LearnedConvince AUT Developers Testing WorksDevelop AUTs with better object id/naming unique IDs better than CSS, DOM, XPath locatorsDisambiguate significant data in AUTbody
    62. 62. Lessons LearnedDesign Page Objects for method chainingLoginPage.enterUserID(‘me’).enterPass(‘pw’).submit();Easier to readTakes advantage of code insightCF Builder’s code insight not quite there yetOther IDEs may provide better code insight(IntelliJ)
    63. 63. Lessons LearnedAlways perform LogOut() in beforeTests()methodRun Login() in the setUp() methodRun LogOut() in the tearDown() methodTests no longer need to perform loginactions (you should already have tests just for Login)
    64. 64. SnippetsTest Cases, Page Objects, Debugging
    65. 65. TroubleshootingResponse of the Selenium RC is invalid:ERROR: Element css=p >input[name="search"] not foundResponse of the Selenium RC is invalid:Current window or frame is closed! Get the AUT’s HTML Source via Selenium Have Selenium grab a screenshot of AUT
    66. 66. TroubleshootingResponse of the Selenium RC is invalid:ERROR: Element //input[name="search"]not found
    67. 67. TroubleshootingUse custom messages for AssertionsMake them more meaningfulSave more hair
    68. 68. Troubleshooting Jenkins will show your custom message for failures
    69. 69. Troubleshooting Slow the test execution down Rerun the test via MXUnit Plugin Watch the browser as test runs Add sleep(NNNN) statements at key points
    70. 70. Troubleshooting // Runs once before all test methods in file public void function beforeTests() { // dataproviders must be in the variables scope variables.TwoCommenters = application.dataproviders[TwoCommenters]; variables.selenium = application.selenium; variables.HomePage = new HomePage(selenium); variables.defaultSpeed = selenium.getSpeed(); selenium.setSpeed(1000); } // Runs once after all test methods in file public void function afterTests() { HomePage.clickNavButton_Home(); assertEquals(HomePage.getExpectedTitle(), selenium.getTitle()); selenium.setSpeed(variables.defaultSpeed); }
    71. 71. Troubleshooting Use Tasks view with keywords //TODO or // FIXME
    72. 72. Troubleshooting A TestCase method failed Did the TestCase change? Did a Page Object change? Check TestSuite Source Control for changes since last success Did the AUT change?
    73. 73. TroubleshootingA Job in Jenkins Failed to Build Check the Test Suite’s ColdFusion exception.log
    74. 74. Thank you!Steven Erattalkingtree@gmail.comGet Slides and Demo TestSuite code at:talkingtree.com/cfo
    75. 75. Some Referenceshttp://en.wikipedia.org/wiki/Software_testinghttp://en.wikipedia.org/wiki/Unit_testinghttp://en.wikipedia.org/wiki/Integration_testinghttp://en.wikipedia.org/wiki/System_testinghttp://en.wikipedia.org/wiki/Regression_testinghttp://en.wikipedia.org/wiki/Continuous_integrationhttp://en.wikipedia.org/wiki/Selenium_%28software%29http://seleniumhq.org/docs/02_selenium_ide.html#selenium-commands-selenesehttp://release.seleniumhq.org/selenium-remote-control/0.9.2/doc/dotnet/Selenium.htmlhttp://www.codediesel.com/testing/selenium-ide-pattern-matching/http://wiki.mxunit.org/http://www.mxunit.org/http://wiki.openqa.org/display/SEL/Homehttp://seleniumhq.org/docs/http://release.seleniumhq.org/selenium-remote-control/0.9.2/doc/dotnet/Selenium.htmlhttp://wiki.openqa.org/display/SEL/Help+With+XPathhttp://blog.browsermob.com/2009/04/test-your-selenium-xpath-easily-with-firebug/http://code.google.com/p/selenium/wiki/PageObjectshttp://www.viddler.com/explore/saucelabs/videos/3/http://www.silverwareconsulting.com/index.cfm/2011/2/22/Introducing-CFSelenium--A-Native-ColdFusion-Client-Library-for-SeleniumRChttp://www.thoughtdelimited.org/thoughts/archives.cfm/category/seleniumhttp://wiki.mxunit.org/display/default/MXUnit+Documentationhttp://blog.mxunit.org/2008/03/new-mxunit-ant-automation-video.htmlhttp://selftechy.com/2011/06/02/parameterization-of-selenium-tests-with-microsoft-excel

    ×