Magento Testing on all fronts

1,924
-1

Published on

San Francisco Magento Meeting 2014-03-12

Published in: Technology

Magento Testing on all fronts

  1. 1. On all Fronts Testing Magento San Francisco Magento Meetup Group - March 11, 2014
  2. 2. Fabrizio BrancaLead System Developer at
  3. 3. E-Commerce: Magento CMS: TYPO3 Portals: ZF, FLOW,… Mobile Searchperience: SOLR 120 people in 7 offices world-wide High Performance /Scale Global Enterprise Projects
  4. 4. Unit Service UI Acceptance http://www.mountaingoatsoftware.com/blog/the-forgotten-layer-of-the-test-automation-pyramid What the customer thinks the code should do What the developer thinks the code should do Load Test
  5. 5. Continuous Integration Deployment Development Delivery Improvement
  6. 6. Quality Assurance Continuous Everything sounds better with “continuous”
  7. 7. “Continuous” ==“Doingstuffoverandoveragain” Builds Deployment Tests Infrastructure Setup Devbox Setup == “Automation”
  8. 8. timespent task size does it manually does it manually gets annoyed writes script to automate runs script wins loses Geeks vs. Non-Geeks Makes fun of geek’s complicated method
  9. 9. — Ovidiu Platon We are not shipping your machine! if it works on your machine! I don’t care
  10. 10. Picture The BIG
  11. 11. Vagrant Boxes
  12. 12. Awesome stuff Every single line potentially can break production! Bratwurst (Germans) Donut (Americans) Pizza (Italians)
  13. 13. Version Control GIT, SVN,… PHP lint
  14. 14. Version Control Code Reviews Gerrit, Crucible, Phabricator,…
  15. 15. Code Reviews Build Files DB Installer Settings Triggered by commit, by time or manually Jenkins minify js/css .tar.gz “virtual package” (revision)
  16. 16. Version Control Code Reviews Build Files DB Installer Settings
  17. 17. Build Files B ler Settings Static Code Analysis CodeSniffer php-pmd php-cpd php-depend
  18. 18. Static Code Analysis Install on “Integration” First system where code from all developers/teams meets each other unstable system. Might be broken. Will be rebuild without warning
  19. 19. Install on “Integration” Unit Tests PHPUnit Code coverage
  20. 20. Unit Tests Integration Tests Test features that rely on other parts (e.g. framework, external services,…) External components (SSL, Redis, Varnish,…) Basically this is also PHPUnit, but addressing other stuff
  21. 21. Integration Tests Acceptance Tests
  22. 22. Acceptance Tests Install on “Staging” Behat (Cucumber) Same setup as production, but different machines Client QA / approval
  23. 23. Install on “Staging” *http://www.slideshare.net/aoemedia/performance-measurement-and-tuning Stress Tests Jmeter Spin up EC2 instances in the cloud to produce traffic*
  24. 24. Stress Tests Install on “Production”
  25. 25. Install on “Production” Enable Features
  26. 26. Version Control Code Reviews Build Static Code Analysis Install on “Integration” Unit Tests Integration Tests Acceptance Tests Install on “Staging” Install on “Production” Enable Features
  27. 27. with EcomDev_PHPUnit Unit Tests
  28. 28. https://github.com/EcomDev/ EcomDev_PHPUnit use the “dev” branch!
  29. 29. Config Model Block Helper Controller fat model, skinny controller…
  30. 30. Classes PHPUnit_Framework_TestCaseEcomDev_PHPUnit_Test_Case EcomDev_PHPUnit_Test_Case_Config EcomDev_PHPUnit_Test_Case_Controller
  31. 31. Assertions
  32. 32. Fixtures
  33. 33. Mocks
  34. 34. Data Provider
  35. 35. Magento Integration Testsuite management Framework integration Create test environment (db+config)
  36. 36. Manual http://www.ecomdev.org/2011/05/24/ecomdev_phpunit-manual-version-0-2-0-is-available-for- download.html Video tutorial http://www.ecomdev.org/2012/01/31/test-driven-magento-development-seminar-video-and- code.html Documentation More Information
  37. 37. With Selenium Acceptance Testing
  38. 38. Acceptance Tests
  39. 39. What’s out there? Selenium for PHP PHPUnit_Selenium (Sebastian Bergmann) WebDriver-PHP (Chibimagic) Php-webdriver-bindings (Lukasz Kolczynski) php-webdriver PHPUnit_Selenium (Giorgio Sironi)
  40. 40. php-webdriver Selenium 2 08/2011 Facebook/php-webdriver (Justin Bishop) Element-34/php-webdriver (Adam Goucher) instaclick/php-webdriver (Anthon Pang)
  41. 41. php-webdriver “A very thin wrapper of WebDriver”
  42. 42. So what? A thin layer good! But not enough… no test logic no higher level components dealing with Selenium specific tasks not compatible to legacy test cases No abstraction / structure
  43. 43. Architecture HTTP AUT No direct access (DB, classes,…) REST* (over HTTP) *http://code.google.com/p/selenium/wiki/JsonWireProtocol
  44. 44. Project test cases PHPUnit Project Components Components Library Menta php-webdriver reusable project specific
  45. 45. Menta Selenium 2 Testing Framework for PHP Who is “Menta”? What is “Menta”?
  46. 46. Features MentaThe framework. Not the cat… Components Session Management ?Bonus Features :)Event / ObserverConfiguration Management PHPUnit integration
  47. 47. Why? ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Data Providers ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Dependent Tests ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Test Infrastructure
  48. 48. Menta for PHPUnit Abstract test class PHPUnit_Framework_TestCaseMenta_PHPUnit_Testcase_Selenium2ProjectY_Abstract_TestCase Very thin layer on top of PHPUnit ProjectY_TestCase_Registration ProjectY_TestCase_Checkout the “magic” happens in the components
  49. 49. HTML ReportScreenshots (on demand and on failure) test description from phpdoc additional messages error details
  50. 50. HTML Report“Duration Heatmap” shows long running tests
  51. 51. Menta for PHPUnit Text Result PrinterPHPUnit 3.7.20 by Sebastian Bergmann. Configuration read from /var/www/Tests/Acceptance/conf/devfb.ff.vmhost.xml > SUITE: > SUITE: > SUITE: Acceptance_Tests_Account_Register > TEST: Acceptance_Tests_Account_Register::register SUCCESS. (Duration: 8.18 sec) > TEST: Acceptance_Tests_Account_Register::login SUCCESS. (Duration: 6.5 sec) > TEST: Acceptance_Tests_Account_Register::testRegistrationMail FAILURE: Searching for mail with subject 'Welcome, Test_705b077a962977e1a412696ea42249ae User_705b077a962977e1a412696ea42249ae' timed out (Duration: 22 < Duration: 36 sec < Duration: 36 sec > SUITE: > SUITE: Acceptance_Tests_Checkout_CheckoutMultiAddressTest > TEST: Acceptance_Tests_Checkout_CheckoutMultiAddressTest::putProductsInCart FAILURE: Element "//table[@id="shopping-cart-table"]//h3[contains(concat(' ', @class, ' '), ' attentionText ')]" does not contain text "Bulb Vase" Fail < Duration: 9.87 sec > SUITE: Acceptance_Tests_Checkout_CheckoutTest > TEST: Acceptance_Tests_Checkout_CheckoutTest::categoryView SUCCESS. (Duration: 1.79 sec) > TEST: Acceptance_Tests_Checkout_CheckoutTest::putPlatterInCart FAILURE: Failed asserting that two strings are equal. (Duration: 3.55 sec) SKIPPED: This test depends on "Acceptance_Tests_Checkout_CheckoutTest::putPlatterInCart" to pass. (Duration: 5.34 sec)
  52. 52. Components Common Assert Wait Screenshot Selenium1Facade Imap WebService W3C Validator Curl … Some might not even require a Selenium session
  53. 53. Components! PageObjects? PageObjectsComponents { }}⊇{ PageObjects are components
  54. 54. PageObjects? represents services offered by a page or feature is the only thing with a deep knowledge of the HTML clickCheckoutButton() isInCart($product) getTotals() ✔ ✔ ✔
  55. 55. Component Libraries Menta Magento (your library) Project- specific Magento (community-provided) General (your library)
  56. 56. Gallery Screenshot
  57. 57. Perceptual diffs Build X Build X-1 Perceptual Diff
  58. 58. github.com / AOEmedia / Menta github.com / AOEmedia / Menta_SampleProject It’s composer based and you’ can get started in a few minutes!
  59. 59. With JMeter Load Testing
  60. 60. JMeter jmeter.apache.org Current version: 2.11
  61. 61. JMeter HTTP Simple Setup
  62. 62. JMeter Slave JMeter Slave JMeter Slave JMeter Master Master/Slave Setup
  63. 63. JMeter Slave JMeter Slave JMeter Slave JMeter Master EC2 Setup JMeter Slave JMeter Slave …
  64. 64. Creating JMeter tests JMeter Test Manually Replaying Apache Logs Recording Browser interaction using a proxy“Convert” existing acceptance tests CSV Files (Selenium) HTT
  65. 65. HTT Behat Selenium Firefox Proxy JMeter Test Converting Behat to JMeter
  66. 66. Using a Proxy JMeter
  67. 67. Using a Proxy Browser configuration Manual configuration Browser Extension
  68. 68. Using a Proxy Browser configuration
  69. 69. java -jar selenium-server-standalone-2.40.0.jar -trustAllSSLCertificates -Dwebdriver.firefox.profile=jmeter Using a Proxy Selenium configuration
  70. 70. http://jmeter.apache.org/usermanual/jmeter_proxy_step_by_step.pdf https://wiki.apache.org/jmeter/TestRecording210 http://jmeter.apache.org/usermanual/component_reference.html#HTTP%28S%29_Test_Script _Recorder Using a Proxy
  71. 71. https://jmeter.apache.org/usermanual/test_plan.html • ThreadGroups • Controllers • Samplers • Logic Controllers • Test Fragments • Timers • Assertions • Configuration Elements • Pre-Processor Elements • Post-Processor Elements Elements of a Test Plan
  72. 72. http://www.slideshare.net/shubelal/abj-meterbrief-upladed/7 Elements of a Test Plan
  73. 73. • Tree structure • Inheritance (Cookie Manager, Header Manager,…) • Timers • Variables • Assertions • Random values • Scripting • Extracting values • Following redirects • (check examples in bin/examples) Basic Concepts
  74. 74. Workflows Search / Filters Static Pages 404 What to testConsider conversion rate / funnel!
  75. 75. JMeter Plugins jmeter-plugins.org
  76. 76. Distributed load testing General Master/Slave setup https://jmeter.apache.org/usermanual/jmeter_distributed_testing_step_by_step.pdf AOE presentations http://www.slideshare.net/axeljung01/jmeter-and-amazon-cloud-workshop-t3dd2012 http://www.slideshare.net/aoemedia/performance-measurement-and-tuning On EC2 https://github.com/oliverlloyd/jmeter-ec2 http://www.http503.com/2012/run-jmeter-on-amazon-ec2-cloud/
  77. 77. Server Monitoring
  78. 78. Thank you! Any questions? http://www.aoe.com http://www.fabrizio-branca.de @fbrnc Follow me on twitter! My blog
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×