Continuous Quality Assurance using Selenium WebDriver

2,910 views
2,797 views

Published on

May 17, 2013

San Francisco Selenium Meetup, May 2013

Published in: Technology
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,910
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
56
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Continuous Quality Assurance using Selenium WebDriver

  1. 1. Quality AssuranceContinuous…using Selenium WebDriverSan Francisco Selenium Meetup Group - May 16, 2013
  2. 2. FabrizioBrancaLead System Developer at
  3. 3. San Francisco, CAJanineFionathat’s meIt’s a boy!
  4. 4. E-Commerce:MagentoCMS:TYPO3Portals:ZF, FLOW,…Mobile Searchperience:SOLR120 peoplein 7 officesworld-wideHighPerformance/ScaleGlobalEnterpriseProjects
  5. 5. ContinuousIntegrationDeploymentDevelopmentDeliveryImprovement
  6. 6. Quality AssuranceContinuousEverything sounds betterwith “continuous”
  7. 7. “Continuous”==“Doingstuffoverandoveragain”BuildsDeployment Tests InfrastructureSetupDevboxSetup== “Automation”
  8. 8. timespenttask sizedoes itmanuallydoes itmanuallygets annoyedwrites script toautomateruns scriptwinslosesGeeks vs. Non-GeeksMakes fun of geek’scomplicated method
  9. 9. — Ovidiu PlatonWe are not shipping your machine!if it works on your machine!I don’t care
  10. 10. PictureThe BIG
  11. 11. VagrantBoxes
  12. 12. AwesomestuffEvery singleline potentiallycan breakproduction!Bratwurst(Germans)Donut(Americans)Pizza(Italians)
  13. 13. Version ControlGIT, SVN,…PHP lint
  14. 14. Version Control Code Reviews“Gerrit”
  15. 15. Code Reviews BuildFilesDBInstallerSettingsTriggered bycommit, by timeor manuallyJenkinsminify js/css.tar.gz“virtual package”(revision)
  16. 16. Multiple ReleaseBranches?*Multiple BuildPipelines!………*http://nvie.com/posts/a-successful-git-branching-model/
  17. 17. Version Control Code Reviews BuildFilesDBInstallerSettings
  18. 18. BuildFilesBlerSettingsStatic CodeAnalysisCodeSnifferphp-pmdphp-cpdphp-depend
  19. 19. Static CodeAnalysisInstall on“Integration”First system wherecode from alldevelopers/teamsmeets each otherunstable system.Might be broken.Will be rebuildwithout warning
  20. 20. Install on“Integration”Unit TestsPHPUnitCodecoverage
  21. 21. Unit Tests Integration TestsAOE PMD(for Magento)broken links,HTTP headers,W3C validity,…Test features that rely on otherparts (e.g. framework, externalservices,…)External components(SSL, Redis, Varnish,…)Basically this is alsoPHPUnit, butaddressing other stuff
  22. 22. Integration Tests Acceptance TestsHere we go:Selenium!
  23. 23. Acceptance TestsInstall on“Staging”Behat(Cucumber)Same setup as production,but different machinesClient QA /approval
  24. 24. Install on“Staging”*http://www.slideshare.net/aoemedia/performance-measurement-and-tuningStressTestsJmeterSpin up EC2instances in thecloud to producetraffic*
  25. 25. Again:multiple servers orautoscaling setupStressTestsInstall on“Production”A/BDeploymentson differentscalesPlan B:rollbackZero Downtime
  26. 26. Varnish ArrayEC2Frontend ArrayBackend ArrayEC2 EC2EC2Worker ArrayEC2EC2Deployment xELBLoad BalancerRoute 53DNS-Serviceshop.angrybirds.comDeployment
  27. 27. Varnish ArrayFrontend Array Worker ArrayEC2Deployment xELBLoad BalancerVaEC2FroBackend ArrayEC2 EC2EC2EC2Deployment x+1Release YELBLoad BalancerRoute 53DNS-Serviceshop.angrybirds.comDeployment
  28. 28. StressTestsInstall on“Production”
  29. 29. Deployment !=Feature Release
  30. 30. Install on“Production”Enable Features
  31. 31. VersionControlCodeReviewsBuild Static CodeAnalysisInstall on“Integration”Unit Tests IntegrationTestsAcceptanceTestsInstall on“Staging”Install on“Production”EnableFeatures
  32. 32. Pipeline Visualization#284#283#282#281#280BuildStatic CodeAnalysisUnitTestsInstall on“Integration”IntegrationTestsAcceptanceTests#285for team dashboards
  33. 33. Acceptance Tests
  34. 34. test automationis a developer’s taskwe need to providethem some good tools…us
  35. 35. Selenium1vs.Selenium2
  36. 36. What’s out there?Selenium for PHPPHPUnit_Selenium(Sebastian Bergmann)WebDriver-PHP(Chibimagic)Php-webdriver-bindings(Lukasz Kolczynski)php-webdriverPHPUnit_Selenium(Giorgio Sironi)
  37. 37. php-webdriverSelenium 208/2011 Facebook/php-webdriver (Justin Bishop)Element-34/php-webdriver (Adam Goucher)instaclick/php-webdriver (Anthon Pang)
  38. 38. php-webdriver“A very thin wrapperof WebDriver”
  39. 39. So what?A thin layer is good!But not enough…no test logicno higher levelcomponentsdealing withSelenium specific tasksnot compatible tolegacy test casesNo abstraction /structure
  40. 40. ArchitectureHTTPAUTNo direct access(DB, classes,…)REST*(over HTTP)*http://code.google.com/p/selenium/wiki/JsonWireProtocol
  41. 41. Project test casesPHPUnitProject ComponentsComponents LibraryMentaphp-webdriverreusableprojectspecific
  42. 42. MentaSelenium 2 Testing Framework for PHPWho is“Menta”?What is“Menta”?
  43. 43. Why“Menta”?
  44. 44. FeaturesMentaThe framework.Not the cat…Components Session Management?Bonus Features :)Event / ObserverConfiguration ManagementPHPUnit integration
  45. 45. Why?✔✔✔✔✔✔✔✔✔✔✔✔Data Providers✔✔✔✔✔✔✔✔✔Dependent Tests✔✔✔✔✔✔✔✔✔✔✔✔✔✔✔✔✔✔✔✔✔✔✔✔Test Infrastructure
  46. 46. Menta for PHPUnitAbstract test classPHPUnit_Framework_TestCaseMenta_PHPUnit_Testcase_Selenium2ProjectY_Abstract_TestCaseVery thin layer ontop of PHPUnitProjectY_TestCase_Registration ProjectY_TestCase_Checkoutthe “magic” happensin the components
  47. 47. Legacy Selenium 1 testsPHPUnit_Framework_TestCasePHPUnit_Extensions_SeleniumTestCaseProjectY_TestCase_Registration ProjectY_TestCase_Checkout
  48. 48. Legacy Selenium 1 testsPHPUnit_Framework_TestCaseMenta_PHPUnit_Testcase_Selenium2ProjectY_TestCase_Registration ProjectY_TestCase_CheckoutMenta_PHPUnit_Testcase_Selenium1Selenium1FacadeCompatible withPHPUnit_Extensions_SeleniumTestCase
  49. 49. HTML ReportScreenshots(on demandand on failure)test descriptionfrom phpdocadditionalmessageserror details
  50. 50. HTML Report“DurationHeatmap”shows longrunning tests
  51. 51. Menta for PHPUnitText Result PrinterPHPUnit 3.7.20 by Sebastian Bergmann.Configuration read from /var/www/Tests/Acceptance/conf/devfb.ff.vmhost.xml> SUITE:> SUITE:> SUITE: Acceptance_Tests_Account_Register> TEST: Acceptance_Tests_Account_Register::registerSUCCESS. (Duration: 8.18 sec)> TEST: Acceptance_Tests_Account_Register::loginSUCCESS. (Duration: 6.5 sec)> TEST: Acceptance_Tests_Account_Register::testRegistrationMailFAILURE: Searching for mail with subject Welcome, Test_705b077a962977e1a412696ea42249ae User_705b077a962977e1a412696ea42249ae timed out (Duration: 22< Duration: 36 sec< Duration: 36 sec> SUITE:> SUITE: Acceptance_Tests_Checkout_CheckoutMultiAddressTest> TEST: Acceptance_Tests_Checkout_CheckoutMultiAddressTest::putProductsInCartFAILURE: Element "//table[@id="shopping-cart-table"]//h3[contains(concat( , @class, ), attentionText )]" does not contain text "Bulb Vase" Fail< Duration: 9.87 sec> SUITE: Acceptance_Tests_Checkout_CheckoutTest> TEST: Acceptance_Tests_Checkout_CheckoutTest::categoryViewSUCCESS. (Duration: 1.79 sec)> TEST: Acceptance_Tests_Checkout_CheckoutTest::putPlatterInCartFAILURE: Failed asserting that two strings are equal. (Duration: 3.55 sec)SKIPPED: This test depends on "Acceptance_Tests_Checkout_CheckoutTest::putPlatterInCart" to pass. (Duration: 5.34 sec)
  52. 52. Components?Components!
  53. 53. ManagerComponentMenta_ComponentManager::get(‘PageObject_Cart’) PageObject_Cart
  54. 54. RewritesComponentPageObject_CartProjectZ_PageObject_Cartextendschangelocatorsaddfunctionalitychangebehavioroverwritelabels
  55. 55. ManagerComponentMenta_ComponentManager::get(‘PageObject_Cart’) ProjectZ_PageObject_CartMenta_ComponentManager::addRewrite(‘PageObject_Cart’,‘ProjectZ_PageObject_Cart’);
  56. 56. ComponentsCommon Assert Wait Screenshot Selenium1FacadeImap WebService W3C Validator Curl …Some might not evenrequire a Selenium session
  57. 57. GalleryScreenshot
  58. 58. Components!PageObjects?PageObjectsComponents{ }}⊇{ PageObjects arecomponents
  59. 59. PageObjects?representsservices offered bya page or featureis the only thing witha deep knowledge ofthe HTMLclickCheckoutButton() isInCart($product) getTotals()✔✔✔
  60. 60. ComponentLibrariesMentaMagento(your library)Project-specificMagento(community-provided)General(your library)
  61. 61. SessionManagementLet Menta manageyour sessionsauto-closeauto-startincl. on exceptionsand ctrl+cReusing sessionsCleaning sessions
  62. 62. ConfigurationManagementFallback mechanismfor global settingsCreate multipleconfigurations fordifferent browsers,application instancesor Selenium ServersCompatible toPHPUnit (xml)
  63. 63. Event / ObserverMenta_Events::dispatchEvent(after_session_create);Menta_Events::addObserver(after_session_create, function($session) {$session->window(main)->postSize(array(width => 1024, height => 768));}
  64. 64. Bonus Features
  65. 65. Sauce Labs Support*configuration: http://username:accesskey@ondemand.saucelabs.com:80/wd/hubworks fine*update testresult (REST)
  66. 66. Sauce Labs SupportSpeaking name Test results
  67. 67. Perceptual diffsBuild XBuild X-1PerceptualDiff
  68. 68. github.com / AOEmedia / Mentagithub.com / AOEmedia / Menta_SampleProjectIt’s composer based andyou’ can get startedin a few minutes!
  69. 69. Thank you!Any questions?http://www.aoemedia.comhttp://www.fabrizio-branca.de@fbrncFollow me on twitter!My blog

×