Quality AssuranceContinuous…using Selenium WebDriverSan Francisco Selenium Meetup Group - May 16, 2013
FabrizioBrancaLead System Developer at
San Francisco, CAJanineFionathat’s meIt’s a boy!
E-Commerce:MagentoCMS:TYPO3Portals:ZF, FLOW,…Mobile Searchperience:SOLR120 peoplein 7 officesworld-wideHighPerformance/Sca...
ContinuousIntegrationDeploymentDevelopmentDeliveryImprovement
Quality AssuranceContinuousEverything sounds betterwith “continuous”
“Continuous”==“Doingstuffoverandoveragain”BuildsDeployment Tests InfrastructureSetupDevboxSetup== “Automation”
timespenttask sizedoes itmanuallydoes itmanuallygets annoyedwrites script toautomateruns scriptwinslosesGeeks vs. Non-Geek...
— Ovidiu PlatonWe are not shipping your machine!if it works on your machine!I don’t care
PictureThe BIG
VagrantBoxes
AwesomestuffEvery singleline potentiallycan breakproduction!Bratwurst(Germans)Donut(Americans)Pizza(Italians)
Version ControlGIT, SVN,…PHP lint
Version Control Code Reviews“Gerrit”
Code Reviews BuildFilesDBInstallerSettingsTriggered bycommit, by timeor manuallyJenkinsminify js/css.tar.gz“virtual packag...
Multiple ReleaseBranches?*Multiple BuildPipelines!………*http://nvie.com/posts/a-successful-git-branching-model/
Version Control Code Reviews BuildFilesDBInstallerSettings
BuildFilesBlerSettingsStatic CodeAnalysisCodeSnifferphp-pmdphp-cpdphp-depend
Static CodeAnalysisInstall on“Integration”First system wherecode from alldevelopers/teamsmeets each otherunstable system.M...
Install on“Integration”Unit TestsPHPUnitCodecoverage
Unit Tests Integration TestsAOE PMD(for Magento)broken links,HTTP headers,W3C validity,…Test features that rely on otherpa...
Integration Tests Acceptance TestsHere we go:Selenium!
Acceptance TestsInstall on“Staging”Behat(Cucumber)Same setup as production,but different machinesClient QA /approval
Install on“Staging”*http://www.slideshare.net/aoemedia/performance-measurement-and-tuningStressTestsJmeterSpin up EC2insta...
Again:multiple servers orautoscaling setupStressTestsInstall on“Production”A/BDeploymentson differentscalesPlan B:rollback...
Varnish ArrayEC2Frontend ArrayBackend ArrayEC2 EC2EC2Worker ArrayEC2EC2Deployment xELBLoad BalancerRoute 53DNS-Serviceshop...
Varnish ArrayFrontend Array Worker ArrayEC2Deployment xELBLoad BalancerVaEC2FroBackend ArrayEC2 EC2EC2EC2Deployment x+1Rel...
StressTestsInstall on“Production”
Deployment !=Feature Release
Install on“Production”Enable Features
VersionControlCodeReviewsBuild Static CodeAnalysisInstall on“Integration”Unit Tests IntegrationTestsAcceptanceTestsInstall...
Pipeline Visualization#284#283#282#281#280BuildStatic CodeAnalysisUnitTestsInstall on“Integration”IntegrationTestsAcceptan...
Acceptance Tests
test automationis a developer’s taskwe need to providethem some good tools…us
Selenium1vs.Selenium2
What’s out there?Selenium for PHPPHPUnit_Selenium(Sebastian Bergmann)WebDriver-PHP(Chibimagic)Php-webdriver-bindings(Lukas...
php-webdriverSelenium 208/2011 Facebook/php-webdriver (Justin Bishop)Element-34/php-webdriver (Adam Goucher)instaclick/php...
php-webdriver“A very thin wrapperof WebDriver”
So what?A thin layer is good!But not enough…no test logicno higher levelcomponentsdealing withSelenium specific tasksnot c...
ArchitectureHTTPAUTNo direct access(DB, classes,…)REST*(over HTTP)*http://code.google.com/p/selenium/wiki/JsonWireProtocol
Project test casesPHPUnitProject ComponentsComponents LibraryMentaphp-webdriverreusableprojectspecific
MentaSelenium 2 Testing Framework for PHPWho is“Menta”?What is“Menta”?
Why“Menta”?
FeaturesMentaThe framework.Not the cat…Components Session Management?Bonus Features :)Event / ObserverConfiguration Manage...
Why?✔✔✔✔✔✔✔✔✔✔✔✔Data Providers✔✔✔✔✔✔✔✔✔Dependent Tests✔✔✔✔✔✔✔✔✔✔✔✔✔✔✔✔✔✔✔✔✔✔✔✔Test Infrastructure
Menta for PHPUnitAbstract test classPHPUnit_Framework_TestCaseMenta_PHPUnit_Testcase_Selenium2ProjectY_Abstract_TestCaseVe...
Legacy Selenium 1 testsPHPUnit_Framework_TestCasePHPUnit_Extensions_SeleniumTestCaseProjectY_TestCase_Registration Project...
Legacy Selenium 1 testsPHPUnit_Framework_TestCaseMenta_PHPUnit_Testcase_Selenium2ProjectY_TestCase_Registration ProjectY_T...
HTML ReportScreenshots(on demandand on failure)test descriptionfrom phpdocadditionalmessageserror details
HTML Report“DurationHeatmap”shows longrunning tests
Menta for PHPUnitText Result PrinterPHPUnit 3.7.20 by Sebastian Bergmann.Configuration read from /var/www/Tests/Acceptance...
Components?Components!
ManagerComponentMenta_ComponentManager::get(‘PageObject_Cart’) PageObject_Cart
RewritesComponentPageObject_CartProjectZ_PageObject_Cartextendschangelocatorsaddfunctionalitychangebehavioroverwritelabels
ManagerComponentMenta_ComponentManager::get(‘PageObject_Cart’) ProjectZ_PageObject_CartMenta_ComponentManager::addRewrite(...
ComponentsCommon Assert Wait Screenshot Selenium1FacadeImap WebService W3C Validator Curl …Some might not evenrequire a Se...
GalleryScreenshot
Components!PageObjects?PageObjectsComponents{ }}⊇{ PageObjects arecomponents
PageObjects?representsservices offered bya page or featureis the only thing witha deep knowledge ofthe HTMLclickCheckoutBu...
ComponentLibrariesMentaMagento(your library)Project-specificMagento(community-provided)General(your library)
SessionManagementLet Menta manageyour sessionsauto-closeauto-startincl. on exceptionsand ctrl+cReusing sessionsCleaning se...
ConfigurationManagementFallback mechanismfor global settingsCreate multipleconfigurations fordifferent browsers,applicatio...
Event / ObserverMenta_Events::dispatchEvent(after_session_create);Menta_Events::addObserver(after_session_create, function...
Bonus Features
Sauce Labs Support*configuration: http://username:accesskey@ondemand.saucelabs.com:80/wd/hubworks fine*update testresult (...
Sauce Labs SupportSpeaking name Test results
Perceptual diffsBuild XBuild X-1PerceptualDiff
github.com / AOEmedia / Mentagithub.com / AOEmedia / Menta_SampleProjectIt’s composer based andyou’ can get startedin a fe...
Thank you!Any questions?http://www.aoemedia.comhttp://www.fabrizio-branca.de@fbrncFollow me on twitter!My blog
Continuous Quality Assurance using Selenium WebDriver
Continuous Quality Assurance using Selenium WebDriver
Continuous Quality Assurance using Selenium WebDriver
Upcoming SlideShare
Loading in …5
×

Continuous Quality Assurance using Selenium WebDriver

7,589
-1

Published on

San Francisco Selenium Meetup, May 2013

Published in: Technology

Continuous Quality Assurance using Selenium WebDriver

  1. 1. Quality AssuranceContinuous…using Selenium WebDriverSan Francisco Selenium Meetup Group - May 16, 2013
  2. 2. FabrizioBrancaLead System Developer at
  3. 3. San Francisco, CAJanineFionathat’s meIt’s a boy!
  4. 4. E-Commerce:MagentoCMS:TYPO3Portals:ZF, FLOW,…Mobile Searchperience:SOLR120 peoplein 7 officesworld-wideHighPerformance/ScaleGlobalEnterpriseProjects
  5. 5. ContinuousIntegrationDeploymentDevelopmentDeliveryImprovement
  6. 6. Quality AssuranceContinuousEverything sounds betterwith “continuous”
  7. 7. “Continuous”==“Doingstuffoverandoveragain”BuildsDeployment Tests InfrastructureSetupDevboxSetup== “Automation”
  8. 8. timespenttask sizedoes itmanuallydoes itmanuallygets annoyedwrites script toautomateruns scriptwinslosesGeeks vs. Non-GeeksMakes fun of geek’scomplicated method
  9. 9. — Ovidiu PlatonWe are not shipping your machine!if it works on your machine!I don’t care
  10. 10. PictureThe BIG
  11. 11. VagrantBoxes
  12. 12. AwesomestuffEvery singleline potentiallycan breakproduction!Bratwurst(Germans)Donut(Americans)Pizza(Italians)
  13. 13. Version ControlGIT, SVN,…PHP lint
  14. 14. Version Control Code Reviews“Gerrit”
  15. 15. Code Reviews BuildFilesDBInstallerSettingsTriggered bycommit, by timeor manuallyJenkinsminify js/css.tar.gz“virtual package”(revision)
  16. 16. Multiple ReleaseBranches?*Multiple BuildPipelines!………*http://nvie.com/posts/a-successful-git-branching-model/
  17. 17. Version Control Code Reviews BuildFilesDBInstallerSettings
  18. 18. BuildFilesBlerSettingsStatic CodeAnalysisCodeSnifferphp-pmdphp-cpdphp-depend
  19. 19. Static CodeAnalysisInstall on“Integration”First system wherecode from alldevelopers/teamsmeets each otherunstable system.Might be broken.Will be rebuildwithout warning
  20. 20. Install on“Integration”Unit TestsPHPUnitCodecoverage
  21. 21. Unit Tests Integration TestsAOE PMD(for Magento)broken links,HTTP headers,W3C validity,…Test features that rely on otherparts (e.g. framework, externalservices,…)External components(SSL, Redis, Varnish,…)Basically this is alsoPHPUnit, butaddressing other stuff
  22. 22. Integration Tests Acceptance TestsHere we go:Selenium!
  23. 23. Acceptance TestsInstall on“Staging”Behat(Cucumber)Same setup as production,but different machinesClient QA /approval
  24. 24. Install on“Staging”*http://www.slideshare.net/aoemedia/performance-measurement-and-tuningStressTestsJmeterSpin up EC2instances in thecloud to producetraffic*
  25. 25. Again:multiple servers orautoscaling setupStressTestsInstall on“Production”A/BDeploymentson differentscalesPlan B:rollbackZero Downtime
  26. 26. Varnish ArrayEC2Frontend ArrayBackend ArrayEC2 EC2EC2Worker ArrayEC2EC2Deployment xELBLoad BalancerRoute 53DNS-Serviceshop.angrybirds.comDeployment
  27. 27. Varnish ArrayFrontend Array Worker ArrayEC2Deployment xELBLoad BalancerVaEC2FroBackend ArrayEC2 EC2EC2EC2Deployment x+1Release YELBLoad BalancerRoute 53DNS-Serviceshop.angrybirds.comDeployment
  28. 28. StressTestsInstall on“Production”
  29. 29. Deployment !=Feature Release
  30. 30. Install on“Production”Enable Features
  31. 31. VersionControlCodeReviewsBuild Static CodeAnalysisInstall on“Integration”Unit Tests IntegrationTestsAcceptanceTestsInstall on“Staging”Install on“Production”EnableFeatures
  32. 32. Pipeline Visualization#284#283#282#281#280BuildStatic CodeAnalysisUnitTestsInstall on“Integration”IntegrationTestsAcceptanceTests#285for team dashboards
  33. 33. Acceptance Tests
  34. 34. test automationis a developer’s taskwe need to providethem some good tools…us
  35. 35. Selenium1vs.Selenium2
  36. 36. What’s out there?Selenium for PHPPHPUnit_Selenium(Sebastian Bergmann)WebDriver-PHP(Chibimagic)Php-webdriver-bindings(Lukasz Kolczynski)php-webdriverPHPUnit_Selenium(Giorgio Sironi)
  37. 37. php-webdriverSelenium 208/2011 Facebook/php-webdriver (Justin Bishop)Element-34/php-webdriver (Adam Goucher)instaclick/php-webdriver (Anthon Pang)
  38. 38. php-webdriver“A very thin wrapperof WebDriver”
  39. 39. So what?A thin layer is good!But not enough…no test logicno higher levelcomponentsdealing withSelenium specific tasksnot compatible tolegacy test casesNo abstraction /structure
  40. 40. ArchitectureHTTPAUTNo direct access(DB, classes,…)REST*(over HTTP)*http://code.google.com/p/selenium/wiki/JsonWireProtocol
  41. 41. Project test casesPHPUnitProject ComponentsComponents LibraryMentaphp-webdriverreusableprojectspecific
  42. 42. MentaSelenium 2 Testing Framework for PHPWho is“Menta”?What is“Menta”?
  43. 43. Why“Menta”?
  44. 44. FeaturesMentaThe framework.Not the cat…Components Session Management?Bonus Features :)Event / ObserverConfiguration ManagementPHPUnit integration
  45. 45. Why?✔✔✔✔✔✔✔✔✔✔✔✔Data Providers✔✔✔✔✔✔✔✔✔Dependent Tests✔✔✔✔✔✔✔✔✔✔✔✔✔✔✔✔✔✔✔✔✔✔✔✔Test Infrastructure
  46. 46. Menta for PHPUnitAbstract test classPHPUnit_Framework_TestCaseMenta_PHPUnit_Testcase_Selenium2ProjectY_Abstract_TestCaseVery thin layer ontop of PHPUnitProjectY_TestCase_Registration ProjectY_TestCase_Checkoutthe “magic” happensin the components
  47. 47. Legacy Selenium 1 testsPHPUnit_Framework_TestCasePHPUnit_Extensions_SeleniumTestCaseProjectY_TestCase_Registration ProjectY_TestCase_Checkout
  48. 48. Legacy Selenium 1 testsPHPUnit_Framework_TestCaseMenta_PHPUnit_Testcase_Selenium2ProjectY_TestCase_Registration ProjectY_TestCase_CheckoutMenta_PHPUnit_Testcase_Selenium1Selenium1FacadeCompatible withPHPUnit_Extensions_SeleniumTestCase
  49. 49. HTML ReportScreenshots(on demandand on failure)test descriptionfrom phpdocadditionalmessageserror details
  50. 50. HTML Report“DurationHeatmap”shows longrunning tests
  51. 51. Menta for PHPUnitText Result PrinterPHPUnit 3.7.20 by Sebastian Bergmann.Configuration read from /var/www/Tests/Acceptance/conf/devfb.ff.vmhost.xml> SUITE:> SUITE:> SUITE: Acceptance_Tests_Account_Register> TEST: Acceptance_Tests_Account_Register::registerSUCCESS. (Duration: 8.18 sec)> TEST: Acceptance_Tests_Account_Register::loginSUCCESS. (Duration: 6.5 sec)> TEST: Acceptance_Tests_Account_Register::testRegistrationMailFAILURE: Searching for mail with subject Welcome, Test_705b077a962977e1a412696ea42249ae User_705b077a962977e1a412696ea42249ae timed out (Duration: 22< Duration: 36 sec< Duration: 36 sec> SUITE:> SUITE: Acceptance_Tests_Checkout_CheckoutMultiAddressTest> TEST: Acceptance_Tests_Checkout_CheckoutMultiAddressTest::putProductsInCartFAILURE: Element "//table[@id="shopping-cart-table"]//h3[contains(concat( , @class, ), attentionText )]" does not contain text "Bulb Vase" Fail< Duration: 9.87 sec> SUITE: Acceptance_Tests_Checkout_CheckoutTest> TEST: Acceptance_Tests_Checkout_CheckoutTest::categoryViewSUCCESS. (Duration: 1.79 sec)> TEST: Acceptance_Tests_Checkout_CheckoutTest::putPlatterInCartFAILURE: Failed asserting that two strings are equal. (Duration: 3.55 sec)SKIPPED: This test depends on "Acceptance_Tests_Checkout_CheckoutTest::putPlatterInCart" to pass. (Duration: 5.34 sec)
  52. 52. Components?Components!
  53. 53. ManagerComponentMenta_ComponentManager::get(‘PageObject_Cart’) PageObject_Cart
  54. 54. RewritesComponentPageObject_CartProjectZ_PageObject_Cartextendschangelocatorsaddfunctionalitychangebehavioroverwritelabels
  55. 55. ManagerComponentMenta_ComponentManager::get(‘PageObject_Cart’) ProjectZ_PageObject_CartMenta_ComponentManager::addRewrite(‘PageObject_Cart’,‘ProjectZ_PageObject_Cart’);
  56. 56. ComponentsCommon Assert Wait Screenshot Selenium1FacadeImap WebService W3C Validator Curl …Some might not evenrequire a Selenium session
  57. 57. GalleryScreenshot
  58. 58. Components!PageObjects?PageObjectsComponents{ }}⊇{ PageObjects arecomponents
  59. 59. PageObjects?representsservices offered bya page or featureis the only thing witha deep knowledge ofthe HTMLclickCheckoutButton() isInCart($product) getTotals()✔✔✔
  60. 60. ComponentLibrariesMentaMagento(your library)Project-specificMagento(community-provided)General(your library)
  61. 61. SessionManagementLet Menta manageyour sessionsauto-closeauto-startincl. on exceptionsand ctrl+cReusing sessionsCleaning sessions
  62. 62. ConfigurationManagementFallback mechanismfor global settingsCreate multipleconfigurations fordifferent browsers,application instancesor Selenium ServersCompatible toPHPUnit (xml)
  63. 63. Event / ObserverMenta_Events::dispatchEvent(after_session_create);Menta_Events::addObserver(after_session_create, function($session) {$session->window(main)->postSize(array(width => 1024, height => 768));}
  64. 64. Bonus Features
  65. 65. Sauce Labs Support*configuration: http://username:accesskey@ondemand.saucelabs.com:80/wd/hubworks fine*update testresult (REST)
  66. 66. Sauce Labs SupportSpeaking name Test results
  67. 67. Perceptual diffsBuild XBuild X-1PerceptualDiff
  68. 68. github.com / AOEmedia / Mentagithub.com / AOEmedia / Menta_SampleProjectIt’s composer based andyou’ can get startedin a few minutes!
  69. 69. Thank you!Any questions?http://www.aoemedia.comhttp://www.fabrizio-branca.de@fbrncFollow me on twitter!My blog
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×