Your SlideShare is downloading. ×
0
On all Fronts
Testing
Magento
San Francisco Magento Meetup Group - March 11, 2014
Fabrizio
BrancaLead System Developer at
E-Commerce:
Magento
CMS:
TYPO3
Portals:
ZF, FLOW,…
Mobile Searchperience:
SOLR
120 people
in 7 offices
world-wide
High
Per...
Unit
Service
UI
Acceptance
http://www.mountaingoatsoftware.com/blog/the-forgotten-layer-of-the-test-automation-pyramid
Wha...
Continuous
Integration
Deployment
Development
Delivery
Improvement
Quality Assurance
Continuous
Everything sounds better
with “continuous”
“Continuous”
==“Doingstuffoverandoveragain”
Builds
Deployment Tests Infrastructure
Setup
Devbox
Setup
== “Automation”
timespent
task size
does it
manually
does it
manually
gets annoyed
writes script to
automate
runs script
wins
loses
Geeks ...
— Ovidiu Platon
We are not shipping your machine!
if it works on your machine!
I don’t care
Picture
The BIG
Vagrant
Boxes
Awesome
stuff
Every single
line potentially
can break
production!
Bratwurst
(Germans)
Donut
(Americans)
Pizza
(Italians)
Version Control
GIT, SVN,…
PHP lint
Version Control Code Reviews
Gerrit, Crucible,
Phabricator,…
Code Reviews Build
Files
DB
Installer
Settings
Triggered by
commit, by time
or manually
Jenkins
minify js/css
.tar.gz
“vir...
Version Control Code Reviews Build
Files
DB
Installer
Settings
Build
Files
B
ler
Settings
Static Code
Analysis
CodeSniffer
php-pmd
php-cpd
php-depend
Static Code
Analysis
Install on
“Integration”
First system where
code from all
developers/teams
meets each other
unstable ...
Install on
“Integration”
Unit Tests
PHPUnit
Code
coverage
Unit Tests Integration Tests
Test features that rely on other
parts (e.g. framework, external
services,…)
External compone...
Integration Tests Acceptance Tests
Acceptance Tests
Install on
“Staging”
Behat
(Cucumber)
Same setup as production,
but different machines
Client QA /
approv...
Install on
“Staging”
*http://www.slideshare.net/aoemedia/performance-measurement-and-tuning
Stress
Tests
Jmeter
Spin up EC...
Stress
Tests
Install on
“Production”
Install on
“Production”
Enable Features
Version
Control
Code
Reviews
Build Static Code
Analysis
Install on
“Integration”
Unit Tests Integration
Tests
Acceptance
T...
with EcomDev_PHPUnit
Unit Tests
https://github.com/EcomDev/
EcomDev_PHPUnit
use the
“dev”
branch!
Config Model Block Helper Controller
fat model,
skinny controller…
Classes
PHPUnit_Framework_TestCaseEcomDev_PHPUnit_Test_Case
EcomDev_PHPUnit_Test_Case_Config EcomDev_PHPUnit_Test_Case_Con...
Assertions
Fixtures
Mocks
Data Provider
Magento Integration
Testsuite
management
Framework
integration
Create test
environment
(db+config)
Manual
http://www.ecomdev.org/2011/05/24/ecomdev_phpunit-manual-version-0-2-0-is-available-for-
download.html
Video tutori...
With Selenium
Acceptance Testing
Acceptance Tests
What’s out there?
Selenium for PHP
PHPUnit_Selenium
(Sebastian Bergmann)
WebDriver-PHP
(Chibimagic)
Php-webdriver-bindings...
php-webdriver
Selenium 2
08/2011 Facebook/php-webdriver (Justin Bishop)
Element-34/php-webdriver (Adam Goucher)
instaclick...
php-webdriver
“A very thin wrapper
of WebDriver”
So what?
A thin layer good!
But not enough…
no test logic
no higher level
components
dealing with
Selenium specific tasks
...
Architecture
HTTP
AUT
No direct access
(DB, classes,…)
REST*
(over HTTP)
*http://code.google.com/p/selenium/wiki/JsonWireP...
Project test cases
PHPUnit
Project Components
Components Library
Menta
php-webdriver
reusable
project
specific
Menta
Selenium 2 Testing Framework for PHP
Who is
“Menta”?
What is
“Menta”?
Features
MentaThe framework.
Not the cat…
Components Session Management
?Bonus Features :)Event / ObserverConfiguration Ma...
Why?
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
Data Providers
✔
✔
✔
✔
✔
✔
✔
✔
✔
Dependent Tests
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
...
Menta for PHPUnit
Abstract test class
PHPUnit_Framework_TestCaseMenta_PHPUnit_Testcase_Selenium2ProjectY_Abstract_TestCase...
HTML ReportScreenshots
(on demand
and on failure)
test description
from phpdoc
additional
messages
error details
HTML Report“Duration
Heatmap”
shows long
running tests
Menta for PHPUnit
Text Result PrinterPHPUnit 3.7.20 by Sebastian Bergmann.
Configuration read from /var/www/Tests/Acceptan...
Components
Common Assert Wait Screenshot Selenium1Facade
Imap WebService W3C Validator Curl …
Some might not even
require ...
Components!
PageObjects?
PageObjectsComponents
{ }}⊇{ PageObjects are
components
PageObjects?
represents
services offered by
a page or feature
is the only thing with
a deep knowledge of
the HTML
clickChe...
Component
Libraries
Menta
Magento
(your library)
Project-
specific
Magento
(community-provided)
General
(your library)
Gallery
Screenshot
Perceptual diffs
Build X
Build X-1
Perceptual
Diff
github.com / AOEmedia / Menta
github.com / AOEmedia / Menta_SampleProject
It’s composer based and
you’ can get started
in ...
With JMeter
Load Testing
JMeter
jmeter.apache.org
Current version:
2.11
JMeter
HTTP
Simple Setup
JMeter
Slave
JMeter
Slave
JMeter
Slave
JMeter
Master
Master/Slave Setup
JMeter
Slave
JMeter
Slave
JMeter
Slave
JMeter
Master
EC2 Setup
JMeter
Slave
JMeter
Slave
…
Creating
JMeter tests
JMeter
Test
Manually
Replaying Apache
Logs
Recording
Browser
interaction using
a proxy“Convert” exis...
HTT
Behat Selenium Firefox Proxy
JMeter
Test
Converting
Behat to JMeter
Using a Proxy
JMeter
Using a Proxy
Browser configuration
Manual
configuration
Browser
Extension
Using a Proxy
Browser configuration
java -jar selenium-server-standalone-2.40.0.jar
-trustAllSSLCertificates
-Dwebdriver.firefox.profile=jmeter
Using a Proxy
...
http://jmeter.apache.org/usermanual/jmeter_proxy_step_by_step.pdf
https://wiki.apache.org/jmeter/TestRecording210
http://j...
https://jmeter.apache.org/usermanual/test_plan.html
• ThreadGroups
• Controllers
• Samplers
• Logic Controllers
• Test Fra...
http://www.slideshare.net/shubelal/abj-meterbrief-upladed/7
Elements of a
Test Plan
• Tree structure
• Inheritance (Cookie Manager, Header Manager,…)
• Timers
• Variables
• Assertions
• Random values
• Scri...
Workflows
Search / Filters
Static Pages
404
What to testConsider
conversion rate /
funnel!
JMeter
Plugins
jmeter-plugins.org
Distributed
load testing
General Master/Slave setup
https://jmeter.apache.org/usermanual/jmeter_distributed_testing_step_b...
Server Monitoring
Thank you!
Any questions?
http://www.aoe.com
http://www.fabrizio-branca.de
@fbrnc
Follow me on twitter!
My blog
Magento Testing on all fronts
Magento Testing on all fronts
Magento Testing on all fronts
Magento Testing on all fronts
Magento Testing on all fronts
Upcoming SlideShare
Loading in...5
×

Magento Testing on all fronts

1,594

Published on

San Francisco Magento Meeting 2014-03-12

Published in: Technology

Transcript of "Magento Testing on all fronts"

  1. 1. On all Fronts Testing Magento San Francisco Magento Meetup Group - March 11, 2014
  2. 2. Fabrizio BrancaLead System Developer at
  3. 3. E-Commerce: Magento CMS: TYPO3 Portals: ZF, FLOW,… Mobile Searchperience: SOLR 120 people in 7 offices world-wide High Performance /Scale Global Enterprise Projects
  4. 4. Unit Service UI Acceptance http://www.mountaingoatsoftware.com/blog/the-forgotten-layer-of-the-test-automation-pyramid What the customer thinks the code should do What the developer thinks the code should do Load Test
  5. 5. Continuous Integration Deployment Development Delivery Improvement
  6. 6. Quality Assurance Continuous Everything sounds better with “continuous”
  7. 7. “Continuous” ==“Doingstuffoverandoveragain” Builds Deployment Tests Infrastructure Setup Devbox Setup == “Automation”
  8. 8. timespent task size does it manually does it manually gets annoyed writes script to automate runs script wins loses Geeks vs. Non-Geeks Makes fun of geek’s complicated method
  9. 9. — Ovidiu Platon We are not shipping your machine! if it works on your machine! I don’t care
  10. 10. Picture The BIG
  11. 11. Vagrant Boxes
  12. 12. Awesome stuff Every single line potentially can break production! Bratwurst (Germans) Donut (Americans) Pizza (Italians)
  13. 13. Version Control GIT, SVN,… PHP lint
  14. 14. Version Control Code Reviews Gerrit, Crucible, Phabricator,…
  15. 15. Code Reviews Build Files DB Installer Settings Triggered by commit, by time or manually Jenkins minify js/css .tar.gz “virtual package” (revision)
  16. 16. Version Control Code Reviews Build Files DB Installer Settings
  17. 17. Build Files B ler Settings Static Code Analysis CodeSniffer php-pmd php-cpd php-depend
  18. 18. Static Code Analysis Install on “Integration” First system where code from all developers/teams meets each other unstable system. Might be broken. Will be rebuild without warning
  19. 19. Install on “Integration” Unit Tests PHPUnit Code coverage
  20. 20. Unit Tests Integration Tests Test features that rely on other parts (e.g. framework, external services,…) External components (SSL, Redis, Varnish,…) Basically this is also PHPUnit, but addressing other stuff
  21. 21. Integration Tests Acceptance Tests
  22. 22. Acceptance Tests Install on “Staging” Behat (Cucumber) Same setup as production, but different machines Client QA / approval
  23. 23. Install on “Staging” *http://www.slideshare.net/aoemedia/performance-measurement-and-tuning Stress Tests Jmeter Spin up EC2 instances in the cloud to produce traffic*
  24. 24. Stress Tests Install on “Production”
  25. 25. Install on “Production” Enable Features
  26. 26. Version Control Code Reviews Build Static Code Analysis Install on “Integration” Unit Tests Integration Tests Acceptance Tests Install on “Staging” Install on “Production” Enable Features
  27. 27. with EcomDev_PHPUnit Unit Tests
  28. 28. https://github.com/EcomDev/ EcomDev_PHPUnit use the “dev” branch!
  29. 29. Config Model Block Helper Controller fat model, skinny controller…
  30. 30. Classes PHPUnit_Framework_TestCaseEcomDev_PHPUnit_Test_Case EcomDev_PHPUnit_Test_Case_Config EcomDev_PHPUnit_Test_Case_Controller
  31. 31. Assertions
  32. 32. Fixtures
  33. 33. Mocks
  34. 34. Data Provider
  35. 35. Magento Integration Testsuite management Framework integration Create test environment (db+config)
  36. 36. Manual http://www.ecomdev.org/2011/05/24/ecomdev_phpunit-manual-version-0-2-0-is-available-for- download.html Video tutorial http://www.ecomdev.org/2012/01/31/test-driven-magento-development-seminar-video-and- code.html Documentation More Information
  37. 37. With Selenium Acceptance Testing
  38. 38. Acceptance Tests
  39. 39. What’s out there? Selenium for PHP PHPUnit_Selenium (Sebastian Bergmann) WebDriver-PHP (Chibimagic) Php-webdriver-bindings (Lukasz Kolczynski) php-webdriver PHPUnit_Selenium (Giorgio Sironi)
  40. 40. php-webdriver Selenium 2 08/2011 Facebook/php-webdriver (Justin Bishop) Element-34/php-webdriver (Adam Goucher) instaclick/php-webdriver (Anthon Pang)
  41. 41. php-webdriver “A very thin wrapper of WebDriver”
  42. 42. So what? A thin layer good! But not enough… no test logic no higher level components dealing with Selenium specific tasks not compatible to legacy test cases No abstraction / structure
  43. 43. Architecture HTTP AUT No direct access (DB, classes,…) REST* (over HTTP) *http://code.google.com/p/selenium/wiki/JsonWireProtocol
  44. 44. Project test cases PHPUnit Project Components Components Library Menta php-webdriver reusable project specific
  45. 45. Menta Selenium 2 Testing Framework for PHP Who is “Menta”? What is “Menta”?
  46. 46. Features MentaThe framework. Not the cat… Components Session Management ?Bonus Features :)Event / ObserverConfiguration Management PHPUnit integration
  47. 47. Why? ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Data Providers ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Dependent Tests ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Test Infrastructure
  48. 48. Menta for PHPUnit Abstract test class PHPUnit_Framework_TestCaseMenta_PHPUnit_Testcase_Selenium2ProjectY_Abstract_TestCase Very thin layer on top of PHPUnit ProjectY_TestCase_Registration ProjectY_TestCase_Checkout the “magic” happens in the components
  49. 49. HTML ReportScreenshots (on demand and on failure) test description from phpdoc additional messages error details
  50. 50. HTML Report“Duration Heatmap” shows long running tests
  51. 51. Menta for PHPUnit Text Result PrinterPHPUnit 3.7.20 by Sebastian Bergmann. Configuration read from /var/www/Tests/Acceptance/conf/devfb.ff.vmhost.xml > SUITE: > SUITE: > SUITE: Acceptance_Tests_Account_Register > TEST: Acceptance_Tests_Account_Register::register SUCCESS. (Duration: 8.18 sec) > TEST: Acceptance_Tests_Account_Register::login SUCCESS. (Duration: 6.5 sec) > TEST: Acceptance_Tests_Account_Register::testRegistrationMail FAILURE: Searching for mail with subject 'Welcome, Test_705b077a962977e1a412696ea42249ae User_705b077a962977e1a412696ea42249ae' timed out (Duration: 22 < Duration: 36 sec < Duration: 36 sec > SUITE: > SUITE: Acceptance_Tests_Checkout_CheckoutMultiAddressTest > TEST: Acceptance_Tests_Checkout_CheckoutMultiAddressTest::putProductsInCart FAILURE: Element "//table[@id="shopping-cart-table"]//h3[contains(concat(' ', @class, ' '), ' attentionText ')]" does not contain text "Bulb Vase" Fail < Duration: 9.87 sec > SUITE: Acceptance_Tests_Checkout_CheckoutTest > TEST: Acceptance_Tests_Checkout_CheckoutTest::categoryView SUCCESS. (Duration: 1.79 sec) > TEST: Acceptance_Tests_Checkout_CheckoutTest::putPlatterInCart FAILURE: Failed asserting that two strings are equal. (Duration: 3.55 sec) SKIPPED: This test depends on "Acceptance_Tests_Checkout_CheckoutTest::putPlatterInCart" to pass. (Duration: 5.34 sec)
  52. 52. Components Common Assert Wait Screenshot Selenium1Facade Imap WebService W3C Validator Curl … Some might not even require a Selenium session
  53. 53. Components! PageObjects? PageObjectsComponents { }}⊇{ PageObjects are components
  54. 54. PageObjects? represents services offered by a page or feature is the only thing with a deep knowledge of the HTML clickCheckoutButton() isInCart($product) getTotals() ✔ ✔ ✔
  55. 55. Component Libraries Menta Magento (your library) Project- specific Magento (community-provided) General (your library)
  56. 56. Gallery Screenshot
  57. 57. Perceptual diffs Build X Build X-1 Perceptual Diff
  58. 58. github.com / AOEmedia / Menta github.com / AOEmedia / Menta_SampleProject It’s composer based and you’ can get started in a few minutes!
  59. 59. With JMeter Load Testing
  60. 60. JMeter jmeter.apache.org Current version: 2.11
  61. 61. JMeter HTTP Simple Setup
  62. 62. JMeter Slave JMeter Slave JMeter Slave JMeter Master Master/Slave Setup
  63. 63. JMeter Slave JMeter Slave JMeter Slave JMeter Master EC2 Setup JMeter Slave JMeter Slave …
  64. 64. Creating JMeter tests JMeter Test Manually Replaying Apache Logs Recording Browser interaction using a proxy“Convert” existing acceptance tests CSV Files (Selenium) HTT
  65. 65. HTT Behat Selenium Firefox Proxy JMeter Test Converting Behat to JMeter
  66. 66. Using a Proxy JMeter
  67. 67. Using a Proxy Browser configuration Manual configuration Browser Extension
  68. 68. Using a Proxy Browser configuration
  69. 69. java -jar selenium-server-standalone-2.40.0.jar -trustAllSSLCertificates -Dwebdriver.firefox.profile=jmeter Using a Proxy Selenium configuration
  70. 70. http://jmeter.apache.org/usermanual/jmeter_proxy_step_by_step.pdf https://wiki.apache.org/jmeter/TestRecording210 http://jmeter.apache.org/usermanual/component_reference.html#HTTP%28S%29_Test_Script _Recorder Using a Proxy
  71. 71. https://jmeter.apache.org/usermanual/test_plan.html • ThreadGroups • Controllers • Samplers • Logic Controllers • Test Fragments • Timers • Assertions • Configuration Elements • Pre-Processor Elements • Post-Processor Elements Elements of a Test Plan
  72. 72. http://www.slideshare.net/shubelal/abj-meterbrief-upladed/7 Elements of a Test Plan
  73. 73. • Tree structure • Inheritance (Cookie Manager, Header Manager,…) • Timers • Variables • Assertions • Random values • Scripting • Extracting values • Following redirects • (check examples in bin/examples) Basic Concepts
  74. 74. Workflows Search / Filters Static Pages 404 What to testConsider conversion rate / funnel!
  75. 75. JMeter Plugins jmeter-plugins.org
  76. 76. Distributed load testing General Master/Slave setup https://jmeter.apache.org/usermanual/jmeter_distributed_testing_step_by_step.pdf AOE presentations http://www.slideshare.net/axeljung01/jmeter-and-amazon-cloud-workshop-t3dd2012 http://www.slideshare.net/aoemedia/performance-measurement-and-tuning On EC2 https://github.com/oliverlloyd/jmeter-ec2 http://www.http503.com/2012/run-jmeter-on-amazon-ec2-cloud/
  77. 77. Server Monitoring
  78. 78. Thank you! Any questions? http://www.aoe.com http://www.fabrizio-branca.de @fbrnc Follow me on twitter! My blog
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×