Magento Testing on all fronts
Upcoming SlideShare
Loading in...5

Magento Testing on all fronts



San Francisco Magento Meeting 2014-03-12

San Francisco Magento Meeting 2014-03-12



Total Views
Views on SlideShare
Embed Views



1 Embed 31 31



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

Magento Testing on all fronts Magento Testing on all fronts Presentation Transcript

  • On all Fronts Testing Magento San Francisco Magento Meetup Group - March 11, 2014
  • Fabrizio BrancaLead System Developer at
  • E-Commerce: Magento CMS: TYPO3 Portals: ZF, FLOW,… Mobile Searchperience: SOLR 120 people in 7 offices world-wide High Performance /Scale Global Enterprise Projects
  • Unit Service UI Acceptance What the customer thinks the code should do What the developer thinks the code should do Load Test
  • Continuous Integration Deployment Development Delivery Improvement
  • Quality Assurance Continuous Everything sounds better with “continuous”
  • “Continuous” ==“Doingstuffoverandoveragain” Builds Deployment Tests Infrastructure Setup Devbox Setup == “Automation”
  • timespent task size does it manually does it manually gets annoyed writes script to automate runs script wins loses Geeks vs. Non-Geeks Makes fun of geek’s complicated method
  • — Ovidiu Platon We are not shipping your machine! if it works on your machine! I don’t care
  • Picture The BIG
  • Vagrant Boxes
  • Awesome stuff Every single line potentially can break production! Bratwurst (Germans) Donut (Americans) Pizza (Italians)
  • Version Control GIT, SVN,… PHP lint
  • Version Control Code Reviews Gerrit, Crucible, Phabricator,…
  • Code Reviews Build Files DB Installer Settings Triggered by commit, by time or manually Jenkins minify js/css .tar.gz “virtual package” (revision)
  • Version Control Code Reviews Build Files DB Installer Settings
  • Build Files B ler Settings Static Code Analysis CodeSniffer php-pmd php-cpd php-depend
  • Static Code Analysis Install on “Integration” First system where code from all developers/teams meets each other unstable system. Might be broken. Will be rebuild without warning
  • Install on “Integration” Unit Tests PHPUnit Code coverage
  • Unit Tests Integration Tests Test features that rely on other parts (e.g. framework, external services,…) External components (SSL, Redis, Varnish,…) Basically this is also PHPUnit, but addressing other stuff
  • Integration Tests Acceptance Tests
  • Acceptance Tests Install on “Staging” Behat (Cucumber) Same setup as production, but different machines Client QA / approval
  • Install on “Staging” * Stress Tests Jmeter Spin up EC2 instances in the cloud to produce traffic*
  • Stress Tests Install on “Production”
  • Install on “Production” Enable Features
  • Version Control Code Reviews Build Static Code Analysis Install on “Integration” Unit Tests Integration Tests Acceptance Tests Install on “Staging” Install on “Production” Enable Features
  • with EcomDev_PHPUnit Unit Tests
  • EcomDev_PHPUnit use the “dev” branch!
  • Config Model Block Helper Controller fat model, skinny controller…
  • Classes PHPUnit_Framework_TestCaseEcomDev_PHPUnit_Test_Case EcomDev_PHPUnit_Test_Case_Config EcomDev_PHPUnit_Test_Case_Controller
  • Assertions
  • Fixtures
  • Mocks
  • Data Provider
  • Magento Integration Testsuite management Framework integration Create test environment (db+config)
  • Manual download.html Video tutorial code.html Documentation More Information
  • With Selenium Acceptance Testing
  • Acceptance Tests
  • What’s out there? Selenium for PHP PHPUnit_Selenium (Sebastian Bergmann) WebDriver-PHP (Chibimagic) Php-webdriver-bindings (Lukasz Kolczynski) php-webdriver PHPUnit_Selenium (Giorgio Sironi)
  • php-webdriver Selenium 2 08/2011 Facebook/php-webdriver (Justin Bishop) Element-34/php-webdriver (Adam Goucher) instaclick/php-webdriver (Anthon Pang)
  • php-webdriver “A very thin wrapper of WebDriver”
  • So what? A thin layer good! But not enough… no test logic no higher level components dealing with Selenium specific tasks not compatible to legacy test cases No abstraction / structure
  • Architecture HTTP AUT No direct access (DB, classes,…) REST* (over HTTP) *
  • Project test cases PHPUnit Project Components Components Library Menta php-webdriver reusable project specific
  • Menta Selenium 2 Testing Framework for PHP Who is “Menta”? What is “Menta”?
  • Features MentaThe framework. Not the cat… Components Session Management ?Bonus Features :)Event / ObserverConfiguration Management PHPUnit integration
  • Why? ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Data Providers ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Dependent Tests ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Test Infrastructure
  • Menta for PHPUnit Abstract test class PHPUnit_Framework_TestCaseMenta_PHPUnit_Testcase_Selenium2ProjectY_Abstract_TestCase Very thin layer on top of PHPUnit ProjectY_TestCase_Registration ProjectY_TestCase_Checkout the “magic” happens in the components
  • HTML ReportScreenshots (on demand and on failure) test description from phpdoc additional messages error details
  • HTML Report“Duration Heatmap” shows long running tests
  • Menta for PHPUnit Text Result PrinterPHPUnit 3.7.20 by Sebastian Bergmann. Configuration read from /var/www/Tests/Acceptance/conf/devfb.ff.vmhost.xml > SUITE: > SUITE: > SUITE: Acceptance_Tests_Account_Register > TEST: Acceptance_Tests_Account_Register::register SUCCESS. (Duration: 8.18 sec) > TEST: Acceptance_Tests_Account_Register::login SUCCESS. (Duration: 6.5 sec) > TEST: Acceptance_Tests_Account_Register::testRegistrationMail FAILURE: Searching for mail with subject 'Welcome, Test_705b077a962977e1a412696ea42249ae User_705b077a962977e1a412696ea42249ae' timed out (Duration: 22 < Duration: 36 sec < Duration: 36 sec > SUITE: > SUITE: Acceptance_Tests_Checkout_CheckoutMultiAddressTest > TEST: Acceptance_Tests_Checkout_CheckoutMultiAddressTest::putProductsInCart FAILURE: Element "//table[@id="shopping-cart-table"]//h3[contains(concat(' ', @class, ' '), ' attentionText ')]" does not contain text "Bulb Vase" Fail < Duration: 9.87 sec > SUITE: Acceptance_Tests_Checkout_CheckoutTest > TEST: Acceptance_Tests_Checkout_CheckoutTest::categoryView SUCCESS. (Duration: 1.79 sec) > TEST: Acceptance_Tests_Checkout_CheckoutTest::putPlatterInCart FAILURE: Failed asserting that two strings are equal. (Duration: 3.55 sec) SKIPPED: This test depends on "Acceptance_Tests_Checkout_CheckoutTest::putPlatterInCart" to pass. (Duration: 5.34 sec)
  • Components Common Assert Wait Screenshot Selenium1Facade Imap WebService W3C Validator Curl … Some might not even require a Selenium session
  • Components! PageObjects? PageObjectsComponents { }}⊇{ PageObjects are components
  • PageObjects? represents services offered by a page or feature is the only thing with a deep knowledge of the HTML clickCheckoutButton() isInCart($product) getTotals() ✔ ✔ ✔
  • Component Libraries Menta Magento (your library) Project- specific Magento (community-provided) General (your library)
  • Gallery Screenshot
  • Perceptual diffs Build X Build X-1 Perceptual Diff
  • / AOEmedia / Menta / AOEmedia / Menta_SampleProject It’s composer based and you’ can get started in a few minutes!
  • With JMeter Load Testing
  • JMeter Current version: 2.11
  • JMeter HTTP Simple Setup
  • JMeter Slave JMeter Slave JMeter Slave JMeter Master Master/Slave Setup
  • JMeter Slave JMeter Slave JMeter Slave JMeter Master EC2 Setup JMeter Slave JMeter Slave …
  • Creating JMeter tests JMeter Test Manually Replaying Apache Logs Recording Browser interaction using a proxy“Convert” existing acceptance tests CSV Files (Selenium) HTT
  • HTT Behat Selenium Firefox Proxy JMeter Test Converting Behat to JMeter
  • Using a Proxy JMeter
  • Using a Proxy Browser configuration Manual configuration Browser Extension
  • Using a Proxy Browser configuration
  • java -jar selenium-server-standalone-2.40.0.jar -trustAllSSLCertificates -Dwebdriver.firefox.profile=jmeter Using a Proxy Selenium configuration
  • _Recorder Using a Proxy
  • • ThreadGroups • Controllers • Samplers • Logic Controllers • Test Fragments • Timers • Assertions • Configuration Elements • Pre-Processor Elements • Post-Processor Elements Elements of a Test Plan
  • Elements of a Test Plan
  • • Tree structure • Inheritance (Cookie Manager, Header Manager,…) • Timers • Variables • Assertions • Random values • Scripting • Extracting values • Following redirects • (check examples in bin/examples) Basic Concepts
  • Workflows Search / Filters Static Pages 404 What to testConsider conversion rate / funnel!
  • JMeter Plugins
  • Distributed load testing General Master/Slave setup AOE presentations On EC2
  • Server Monitoring
  • Thank you! Any questions? @fbrnc Follow me on twitter! My blog