Cross-­‐browser

              tes*ng     in	
  the



                     real	
  world
Mar*n	
  Kleppmann              ...
http://www.flickr.com/photos/stuart_spivack/2322070560/
Cross-­‐browser

              tes*ng     in	
  the



                     real	
  world
Mar*n	
  Kleppmann              ...
Cross-­‐browser
                                     urgh

              tes*ng     in	
  the



                     real...
Cross-­‐browser

              tes*ng     in	
  the



                     real	
  world
Mar*n	
  Kleppmann              ...
Cross-­‐browser

              tes*ng     in	
  the
     mess
         y
                     real	
  world
Mar*n	
  Klepp...
Cross-­‐browser

              tes*ng     in	
  the



                     real	
  world
Mar*n	
  Kleppmann              ...
Cross-­‐browser                 t?
                                a ?
                              h
                   ...
Tes*ng	
  what?
Tes*ng	
  what?
 Unit	
  tes(ng
 Applica(on
 End-­‐to-­‐end
 (incl.	
  external	
  services)
Tes*ng	
  why?
Tes*ng	
  why?
  Func(onal
    Visual
 Performance
Load/Scalability
   Security
   Usability
Tes*ng	
  how?
Tes*ng	
  how?
   Automated
(TDD,	
  BDD,	
  regression	
  tests,
         smoke	
  tests,	
  ...)


        Manual
   (ex...
Tes*ng	
  dimensions
Unit   Applica(on     End-­‐to-­‐end

Func(onal Visual Performance
Load      Security      Usability
...
Tes*ng	
  dimensions
Unit   Applica(on     End-­‐to-­‐end

Func(onal Visual Performance
Load      Security      Usability
...
Tes*ng	
  dimensions
Unit   Applica(on    End-­‐to-­‐end
                  RSpec/
Func(onal Visual Performance
           ...
Tes*ng	
  dimensions
Unit   Applica(on     End-­‐to-­‐end

Func(onal Visual Performance
Load      Security      Usability
...
Tes*ng	
  dimensions
Unit   Applica(on     End-­‐to-­‐end

Func(onal Visual Performance
Load      Security      Usability
...
(Side-­‐note)
                     	
  wri te
              a n    ’t
    Yo u 	
  c           cs
              R S   p  e...
Tes*ng	
  dimensions
Unit   Applica(on     End-­‐to-­‐end

Func(onal Visual Performance
Load      Security      Usability
...
Tes*ng	
  dimensions
Unit   Applica(on     End-­‐to-­‐end

Func(onal Visual Performance
Load      Security      Usability
...
Me
http://www.flickr.com/photos/71263221@N00/4230334515/
P.S.	
  We’re	
  hiring!
http://www.flickr.com/photos/snake-eyes/410092369/
Anyway.
C++
C++
C++




WebDriver
WebDriver
   C++
WebDriver
WebDriver
Selenium	
  2
WebDriver
Cucumber/RSpec/whatever

      Selenium	
  2
       WebDriver
Cucumber/RSpec/whatever

      Selenium	
  2
       WebDriver
Cucumber/RSpec/whatever

Selenium	
  2
WebDriver
Cucumber/RSpec/whatever



Selenium	
  2
WebDriver
Cucumber/RSpec/whatever



Selenium	
  2
WebDriver
Cucumber/RSpec/whatever

           Capybara

Selenium	
  2
WebDriver
Cucumber/RSpec/whatever

           Capybara

Selenium	
  2
                *Wa(r
WebDriver
Cucumber/RSpec/whatever

           Capybara

Selenium	
  2
                *Wa(r
WebDriver
Cucumber/RSpec/whatever

           Capybara

Selenium	
  2
                *Wa(r
WebDriver
Cucumber/RSpec/whatever

           Capybara
                         Culerity
Selenium	
  2
                *Wa(r
WebDriv...
Cucumber/RSpec/whatever

           Capybara
                         Culerity
Selenium	
  2
                *Wa(r
WebDriv...
Cucumber/RSpec/whatever

           Capybara
                       Culerity
OMGWTFBBQ?!
Selenium	
  2
    *Wa(r
WebDriver...
Cucumber/RSpec/whatever

           Capybara
                         Culerity
Selenium	
  2
                *Wa(r
WebDriv...
Cucumber/RSpec/whatever

           Capybara
                         Culerity
Selenium	
  2
                *Wa(r
WebDriv...
http://www.flickr.com/photos/snake-eyes/449442699/
Cucumber/RSpec/whatever

           Capybara
                         Culerity
Selenium	
  2
                *Wa(r
WebDriv...
Cucumber/RSpec/whatever

           Capybara
                         Culerity
Selenium	
  2
                *Wa(r
WebDriv...
Cucumber/RSpec/whatever

           Capybara
                         Culerity
Selenium	
  2
                *Wa(r
WebDriv...
require 'watir'
Watir::Browser.default = 'firefox'

describe 'Google' do
  before(:each) { @browser = Watir::Browser.new }...
Cucumber/RSpec/whatever

           Capybara
                         Culerity
Selenium	
  2
                *Wa(r
WebDriv...
Cucumber/RSpec/whatever

           Capybara
                         Culerity
Selenium	
  2
                *Wa(r
WebDriv...
require 'selenium-webdriver'

describe 'Google' do
  before(:each){ @browser = Selenium::WebDriver.for :firefox }
  after(...
Cucumber/RSpec/whatever

           Capybara
                         Culerity
Selenium	
  2
                *Wa(r
WebDriv...
Cucumber/RSpec/whatever

           Capybara
                         Culerity
Selenium	
  2
                *Wa(r
WebDriv...
require 'capybara'; require 'capybara/dsl'
Capybara.default_driver = :selenium

Spec::Runner.configure do |config|
  confi...
# API beauty contest!


goto "http://www.google.co.uk"              # Watir
navigate.to "http://www.google.co.uk"       # ...
Cucumber

           Capybara
                         Culerity
Selenium	
  2
                *Wa(r
WebDriver             ...
Feature: Looking up a contact
  In order to understand my contacts better
  As a Gmail user
  I want to see social informa...
Given /^I am logged into Gmail$/ do
  visit 'https://mail.google.com/a/rapportive.com'
  fill_in 'Email', :with => 'test'
...
Ok.
FEI
    (Frequently	
  
Encountered	
  Issues)
10.	
  Time	
  dependence
10.	
  Time	
  dependence
  ‣ Modify	
  date(mes	
  in	
  DB	
  
    using	
  models
  ‣ Create	
  wrapper	
  around	
  
 ...
9.	
  Real-­‐*me	
  comms
8.	
  Managing	
  test	
  VMs
8.	
  Managing	
  test	
  VMs
  ‣ Very	
  tedious	
  (updates	
  etc.)
  ‣ Test	
  management	
  
    infrastructure
  ‣ C...
7.	
  Random	
  DOM	
  IDs
7.	
  Random	
  DOM	
  IDs
 ‣ Ext.js,	
  GWT,	
  Cappuccino
 ‣ Use	
  toolkit	
  APIs
   // Selecting an item in a Ext.js ...
6.	
  Tes*ng	
  layout
6.	
  Tes*ng	
  layout
‣ Automa(cally	
  spoWng	
  
  broken	
  layout?	
  srsly?
‣ Interes(ng	
  experiment
  h"p://code....
$(‘*’).css(‘color’,	
  ‘white’);
$(‘*’).css(‘color’,	
  ‘black’);
image1	
  –	
  image2	
  =	
  ...
5.	
  Unit	
  tes*ng	
  JS
5.	
  Unit	
  tes*ng	
  JS
‣ Command	
  line	
  &&	
  browser
‣ DOM	
  manipula(on	
  →	
  HTML	
  
  fixtures	
  	
  (...a...
5.	
  Unit	
  tes*ng	
  JS
‣ JSpec
 h"p://visionmedia.github.com/jspec/

‣ Blue	
  Ridge/Screw.Unit
 h"p://github.com/rele...
4.	
  Model	
  layer	
  access
4.	
  Model	
  layer	
  access
 Tests
            HTTP/Framework
                   Views
              Controllers
      ...
4.	
  Model	
  layer	
  access
  Tests
            HTTP/Framework
                   Views
              Controllers
Facto...
4.	
  Model	
  layer	
  access
  ‣ Violates	
  abstrac(on
  ‣ BUT:	
  factories	
  =	
  only	
  sane	
  
    way	
  of	
  ...
Factory.define :user do |u|
  u.first_name 'John'
  u.last_name 'Doe'
  u.admin false
end



Given /^I am on John's profil...
3.	
  Star*ng	
  state
3.	
  Star*ng	
  state
‣ Previous,	
  failed	
  tests
‣ Cookies,	
  browser	
  cache
‣ Unexpected	
  persistent	
  state
3.	
  Star*ng	
  state
‣ Long-­‐running	
  transac(on	
  &	
  
  rollback
  (e.g.	
  Cucumber	
  per-­‐scenario	
  transac...
2.	
  Parallel	
  test	
  runs
2.	
  Parallel	
  test	
  runs
‣ End-­‐to-­‐end	
  tests	
  are	
  SLOW
‣ Start	
  up	
  several	
  browsers
  (TestSwarm,...
2.	
  Parallel	
  test	
  runs
One	
  DB	
  per	
  test	
  process
‣ h"p://github.com/grosser/parallel_tests
‣ h"p://githu...
2.	
  Parallel	
  test	
  runs
Caveat:	
  state	
  outside	
  of	
  your	
  
control
‣ e.g.	
  OAuth	
  status
‣ use	
  mo...
1.
Fragility
Fragility
(tests	
  break	
  too	
  easily,	
  
 even	
  if	
  the	
  app	
  is	
  ok)
2	
  schools	
  of	
  thought
1. “Disposable”	
  test	
  scripts
   (invest	
  licle	
  effort,	
  use	
  
   recording	
  ...
Feature: Looking up a contact
  In order to understand my contacts better
  As a Gmail user
  I want to see social informa...
Fragility
No	
  silver	
  bullet,	
  obvious	
  stuff:
‣ Consistent	
  naming	
  of	
  IDs	
  &	
  
  CSS	
  classes
‣ Main...
kthxbai



Mar*n	
  Kleppmann
mar*n@rappor*ve.com
@mar*nkl              rapportive
Cross-browser testing in the real world
Cross-browser testing in the real world
Cross-browser testing in the real world
Cross-browser testing in the real world
Cross-browser testing in the real world
Cross-browser testing in the real world
Cross-browser testing in the real world
Cross-browser testing in the real world
Cross-browser testing in the real world
Cross-browser testing in the real world
Cross-browser testing in the real world
Cross-browser testing in the real world
Cross-browser testing in the real world
Upcoming SlideShare
Loading in...5
×

Cross-browser testing in the real world

9,679

Published on

What's the best way of automating end-to-end, browser-level tests for web apps? In this talk, I compare Selenium, WebDriver, Watir and other libraries, and share experience of automated browser tests on hundreds of different sites. I also give updates on latest developments in open source functional testing tools.

Published in: Technology
1 Comment
12 Likes
Statistics
Notes
No Downloads
Views
Total Views
9,679
On Slideshare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
228
Comments
1
Likes
12
Embeds 0
No embeds

No notes for slide

Cross-browser testing in the real world

  1. 1. Cross-­‐browser tes*ng in  the real  world Mar*n  Kleppmann rapportive
  2. 2. http://www.flickr.com/photos/stuart_spivack/2322070560/
  3. 3. Cross-­‐browser tes*ng in  the real  world Mar*n  Kleppmann rapportive
  4. 4. Cross-­‐browser urgh tes*ng in  the real  world Mar*n  Kleppmann rapportive
  5. 5. Cross-­‐browser tes*ng in  the real  world Mar*n  Kleppmann rapportive
  6. 6. Cross-­‐browser tes*ng in  the mess y real  world Mar*n  Kleppmann rapportive
  7. 7. Cross-­‐browser tes*ng in  the real  world Mar*n  Kleppmann rapportive
  8. 8. Cross-­‐browser t? a ? h w ow h tes*ng in  the real  world Mar*n  Kleppmann rapportive
  9. 9. Tes*ng  what?
  10. 10. Tes*ng  what? Unit  tes(ng Applica(on End-­‐to-­‐end (incl.  external  services)
  11. 11. Tes*ng  why?
  12. 12. Tes*ng  why? Func(onal Visual Performance Load/Scalability Security Usability
  13. 13. Tes*ng  how?
  14. 14. Tes*ng  how? Automated (TDD,  BDD,  regression  tests, smoke  tests,  ...) Manual (exploratory,  scripted, user-­‐centered,  ...)
  15. 15. Tes*ng  dimensions Unit Applica(on End-­‐to-­‐end Func(onal Visual Performance Load Security Usability Automated Manual
  16. 16. Tes*ng  dimensions Unit Applica(on End-­‐to-­‐end Func(onal Visual Performance Load Security Usability Automated Manual
  17. 17. Tes*ng  dimensions Unit Applica(on End-­‐to-­‐end RSpec/ Func(onal Visual Performance Shoulda/ Load Security whatever Usability Automated Manual
  18. 18. Tes*ng  dimensions Unit Applica(on End-­‐to-­‐end Func(onal Visual Performance Load Security Usability Automated Manual
  19. 19. Tes*ng  dimensions Unit Applica(on End-­‐to-­‐end Func(onal Visual Performance Load Security Usability Automated Manual
  20. 20. (Side-­‐note)  wri te a n ’t Yo u  c cs R S p e b i li ty r  u s a fo
  21. 21. Tes*ng  dimensions Unit Applica(on End-­‐to-­‐end Func(onal Visual Performance Load Security Usability Automated Manual
  22. 22. Tes*ng  dimensions Unit Applica(on End-­‐to-­‐end Func(onal Visual Performance Load Security Usability Automated Manual
  23. 23. Me
  24. 24. http://www.flickr.com/photos/71263221@N00/4230334515/
  25. 25. P.S.  We’re  hiring! http://www.flickr.com/photos/snake-eyes/410092369/
  26. 26. Anyway.
  27. 27. C++
  28. 28. C++
  29. 29. C++ WebDriver
  30. 30. WebDriver C++
  31. 31. WebDriver
  32. 32. WebDriver
  33. 33. Selenium  2 WebDriver
  34. 34. Cucumber/RSpec/whatever Selenium  2 WebDriver
  35. 35. Cucumber/RSpec/whatever Selenium  2 WebDriver
  36. 36. Cucumber/RSpec/whatever Selenium  2 WebDriver
  37. 37. Cucumber/RSpec/whatever Selenium  2 WebDriver
  38. 38. Cucumber/RSpec/whatever Selenium  2 WebDriver
  39. 39. Cucumber/RSpec/whatever Capybara Selenium  2 WebDriver
  40. 40. Cucumber/RSpec/whatever Capybara Selenium  2 *Wa(r WebDriver
  41. 41. Cucumber/RSpec/whatever Capybara Selenium  2 *Wa(r WebDriver
  42. 42. Cucumber/RSpec/whatever Capybara Selenium  2 *Wa(r WebDriver
  43. 43. Cucumber/RSpec/whatever Capybara Culerity Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  44. 44. Cucumber/RSpec/whatever Capybara Culerity Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  45. 45. Cucumber/RSpec/whatever Capybara Culerity OMGWTFBBQ?! Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  46. 46. Cucumber/RSpec/whatever Capybara Culerity Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  47. 47. Cucumber/RSpec/whatever Capybara Culerity Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  48. 48. http://www.flickr.com/photos/snake-eyes/449442699/
  49. 49. Cucumber/RSpec/whatever Capybara Culerity Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  50. 50. Cucumber/RSpec/whatever Capybara Culerity Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  51. 51. Cucumber/RSpec/whatever Capybara Culerity Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  52. 52. require 'watir' Watir::Browser.default = 'firefox' describe 'Google' do before(:each) { @browser = Watir::Browser.new } after(:each) { @browser.close } it 'should return search results for "hello world"' do @browser.goto "http://www.google.co.uk" @browser.text_field(:name, "q").set("hello world") @browser.button(:name, "btnG").click @browser.contains_text( "Hello world program - Wikipedia").should be_true end end
  53. 53. Cucumber/RSpec/whatever Capybara Culerity Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  54. 54. Cucumber/RSpec/whatever Capybara Culerity Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  55. 55. require 'selenium-webdriver' describe 'Google' do before(:each){ @browser = Selenium::WebDriver.for :firefox } after(:each) { @browser.quit } it 'should return search results for "hello world"' do @browser.navigate.to "http://www.google.co.uk" @browser.find_element(:name, "q").send_keys("hello world") @browser.find_element(:name, "btnG").submit @browser.find_element(:partial_link_text, "Hello world program - Wikipedia") end end
  56. 56. Cucumber/RSpec/whatever Capybara Culerity Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  57. 57. Cucumber/RSpec/whatever Capybara Culerity Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  58. 58. require 'capybara'; require 'capybara/dsl' Capybara.default_driver = :selenium Spec::Runner.configure do |config| config.include(Capybara, :type => :integration) config.before(:each) { Capybara.reset_sessions! } end describe 'Google' do it 'should return search results for "hello world"' do visit "http://www.google.co.uk" fill_in "q", :with => "hello world" click "Google Search" page.should have_content( "Hello world program - Wikipedia") end end
  59. 59. # API beauty contest! goto "http://www.google.co.uk" # Watir navigate.to "http://www.google.co.uk" # WebDriver visit "http://www.google.co.uk" # Capybara text_field(:name, "q").set("hello world") # Watir find_element(:name, "q").send_keys("hello world") # WD fill_in "q", :with => "hello world" # Capybara button(:name, "btnG").click # Watir find_element(:name, "btnG").submit # WebDriver click "Google Search" # Capybara # You choose...
  60. 60. Cucumber Capybara Culerity Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  61. 61. Feature: Looking up a contact In order to understand my contacts better As a Gmail user I want to see social information next to my email thread Scenario: Load an email conversation Given I am logged into Gmail And Rapportive is loaded When I search for "martin@rapportive.com" And I click on the conversation with subject "Testing 1 2 3" Then I should be on a conversation view And Rapportive should show "Co-founder at Rapportive"
  62. 62. Given /^I am logged into Gmail$/ do visit 'https://mail.google.com/a/rapportive.com' fill_in 'Email', :with => 'test' fill_in 'Passwd', :with => GMAIL_TEST_ACCOUNT_PASSWORD click 'Sign in' end Then /^Rapportive should show "([^"]*)"$/ do |text_to_show| page.within_frame('canvas_frame') do page.within(:css, '#rapportive-sidebar') do page.should have_content(text_to_show) end end end
  63. 63. Ok.
  64. 64. FEI (Frequently   Encountered  Issues)
  65. 65. 10.  Time  dependence
  66. 66. 10.  Time  dependence ‣ Modify  date(mes  in  DB   using  models ‣ Create  wrapper  around   Time.now
  67. 67. 9.  Real-­‐*me  comms
  68. 68. 8.  Managing  test  VMs
  69. 69. 8.  Managing  test  VMs ‣ Very  tedious  (updates  etc.) ‣ Test  management   infrastructure ‣ Commercial  services  (e.g.   Sauce  Labs,  BrowserMob)
  70. 70. 7.  Random  DOM  IDs
  71. 71. 7.  Random  DOM  IDs ‣ Ext.js,  GWT,  Cappuccino ‣ Use  toolkit  APIs // Selecting an item in a Ext.js Combo Box var combo = Ext.getCmp('countryComboBox'); combo.setValue('Germany'); // setValue() doesn't trigger the event combo.fireEvent('select');
  72. 72. 6.  Tes*ng  layout
  73. 73. 6.  Tes*ng  layout ‣ Automa(cally  spoWng   broken  layout?  srsly? ‣ Interes(ng  experiment h"p://code.google.com/p/figh/ng-­‐layout-­‐bugs/
  74. 74. $(‘*’).css(‘color’,  ‘white’);
  75. 75. $(‘*’).css(‘color’,  ‘black’);
  76. 76. image1  –  image2  =  ...
  77. 77. 5.  Unit  tes*ng  JS
  78. 78. 5.  Unit  tes*ng  JS ‣ Command  line  &&  browser ‣ DOM  manipula(on  →  HTML   fixtures    (...and  rollback?)
  79. 79. 5.  Unit  tes*ng  JS ‣ JSpec h"p://visionmedia.github.com/jspec/ ‣ Blue  Ridge/Screw.Unit h"p://github.com/relevance/blue-­‐ridge ‣ JsTestDriver h"p://code.google.com/p/js-­‐test-­‐driver/
  80. 80. 4.  Model  layer  access
  81. 81. 4.  Model  layer  access Tests HTTP/Framework Views Controllers Models
  82. 82. 4.  Model  layer  access Tests HTTP/Framework Views Controllers Factory/ Fixtures? Models
  83. 83. 4.  Model  layer  access ‣ Violates  abstrac(on ‣ BUT:  factories  =  only  sane   way  of  managing  DB  state
  84. 84. Factory.define :user do |u| u.first_name 'John' u.last_name 'Doe' u.admin false end Given /^I am on John's profile page$/ do user = Factory.create(:user) visit user_url(user) end
  85. 85. 3.  Star*ng  state
  86. 86. 3.  Star*ng  state ‣ Previous,  failed  tests ‣ Cookies,  browser  cache ‣ Unexpected  persistent  state
  87. 87. 3.  Star*ng  state ‣ Long-­‐running  transac(on  &   rollback (e.g.  Cucumber  per-­‐scenario  transac/on  rollback) ‣ Clean  up  database h"p://github.com/bmabey/database_cleaner
  88. 88. 2.  Parallel  test  runs
  89. 89. 2.  Parallel  test  runs ‣ End-­‐to-­‐end  tests  are  SLOW ‣ Start  up  several  browsers (TestSwarm,  Selenium  Grid,  ...) ‣ BUT:  concurrent   modifica(on  of  DB  state!
  90. 90. 2.  Parallel  test  runs One  DB  per  test  process ‣ h"p://github.com/grosser/parallel_tests ‣ h"p://github.com/qxjit/deep-­‐test
  91. 91. 2.  Parallel  test  runs Caveat:  state  outside  of  your   control ‣ e.g.  OAuth  status ‣ use  mocks,  or  work  around  it
  92. 92. 1.
  93. 93. Fragility
  94. 94. Fragility (tests  break  too  easily,   even  if  the  app  is  ok)
  95. 95. 2  schools  of  thought 1. “Disposable”  test  scripts (invest  licle  effort,  use   recording  tools) 2. “Engineered”  test  scripts (use  carefully  designed   abstrac(ons)
  96. 96. Feature: Looking up a contact In order to understand my contacts better As a Gmail user I want to see social information next to my email thread Scenario: Load an email conversation Given I am logged into Gmail And Rapportive is loaded When I search for "martin@rapportive.com" And I click on the conversation with subject "Testing 1 2 3" Then I should be on a conversation view And Rapportive should show "Co-founder at Rapportive"
  97. 97. Fragility No  silver  bullet,  obvious  stuff: ‣ Consistent  naming  of  IDs  &   CSS  classes ‣ Maintain  &  refactor  your   tests
  98. 98. kthxbai Mar*n  Kleppmann mar*n@rappor*ve.com @mar*nkl rapportive
  1. Gostou de algum slide específico?

    Recortar slides é uma maneira fácil de colecionar informações para acessar mais tarde.

×