Cross-browser testing in the real world

10,668 views
10,450 views

Published on

What's the best way of automating end-to-end, browser-level tests for web apps? In this talk, I compare Selenium, WebDriver, Watir and other libraries, and share experience of automated browser tests on hundreds of different sites. I also give updates on latest developments in open source functional testing tools.

Published in: Technology
1 Comment
12 Likes
Statistics
Notes
No Downloads
Views
Total views
10,668
On SlideShare
0
From Embeds
0
Number of Embeds
171
Actions
Shares
0
Downloads
243
Comments
1
Likes
12
Embeds 0
No embeds

No notes for slide

Cross-browser testing in the real world

  1. 1. Cross-­‐browser tes*ng in  the real  world Mar*n  Kleppmann rapportive
  2. 2. http://www.flickr.com/photos/stuart_spivack/2322070560/
  3. 3. Cross-­‐browser tes*ng in  the real  world Mar*n  Kleppmann rapportive
  4. 4. Cross-­‐browser urgh tes*ng in  the real  world Mar*n  Kleppmann rapportive
  5. 5. Cross-­‐browser tes*ng in  the real  world Mar*n  Kleppmann rapportive
  6. 6. Cross-­‐browser tes*ng in  the mess y real  world Mar*n  Kleppmann rapportive
  7. 7. Cross-­‐browser tes*ng in  the real  world Mar*n  Kleppmann rapportive
  8. 8. Cross-­‐browser t? a ? h w ow h tes*ng in  the real  world Mar*n  Kleppmann rapportive
  9. 9. Tes*ng  what?
  10. 10. Tes*ng  what? Unit  tes(ng Applica(on End-­‐to-­‐end (incl.  external  services)
  11. 11. Tes*ng  why?
  12. 12. Tes*ng  why? Func(onal Visual Performance Load/Scalability Security Usability
  13. 13. Tes*ng  how?
  14. 14. Tes*ng  how? Automated (TDD,  BDD,  regression  tests, smoke  tests,  ...) Manual (exploratory,  scripted, user-­‐centered,  ...)
  15. 15. Tes*ng  dimensions Unit Applica(on End-­‐to-­‐end Func(onal Visual Performance Load Security Usability Automated Manual
  16. 16. Tes*ng  dimensions Unit Applica(on End-­‐to-­‐end Func(onal Visual Performance Load Security Usability Automated Manual
  17. 17. Tes*ng  dimensions Unit Applica(on End-­‐to-­‐end RSpec/ Func(onal Visual Performance Shoulda/ Load Security whatever Usability Automated Manual
  18. 18. Tes*ng  dimensions Unit Applica(on End-­‐to-­‐end Func(onal Visual Performance Load Security Usability Automated Manual
  19. 19. Tes*ng  dimensions Unit Applica(on End-­‐to-­‐end Func(onal Visual Performance Load Security Usability Automated Manual
  20. 20. (Side-­‐note)  wri te a n ’t Yo u  c cs R S p e b i li ty r  u s a fo
  21. 21. Tes*ng  dimensions Unit Applica(on End-­‐to-­‐end Func(onal Visual Performance Load Security Usability Automated Manual
  22. 22. Tes*ng  dimensions Unit Applica(on End-­‐to-­‐end Func(onal Visual Performance Load Security Usability Automated Manual
  23. 23. Me
  24. 24. http://www.flickr.com/photos/71263221@N00/4230334515/
  25. 25. P.S.  We’re  hiring! http://www.flickr.com/photos/snake-eyes/410092369/
  26. 26. Anyway.
  27. 27. C++
  28. 28. C++
  29. 29. C++ WebDriver
  30. 30. WebDriver C++
  31. 31. WebDriver
  32. 32. WebDriver
  33. 33. Selenium  2 WebDriver
  34. 34. Cucumber/RSpec/whatever Selenium  2 WebDriver
  35. 35. Cucumber/RSpec/whatever Selenium  2 WebDriver
  36. 36. Cucumber/RSpec/whatever Selenium  2 WebDriver
  37. 37. Cucumber/RSpec/whatever Selenium  2 WebDriver
  38. 38. Cucumber/RSpec/whatever Selenium  2 WebDriver
  39. 39. Cucumber/RSpec/whatever Capybara Selenium  2 WebDriver
  40. 40. Cucumber/RSpec/whatever Capybara Selenium  2 *Wa(r WebDriver
  41. 41. Cucumber/RSpec/whatever Capybara Selenium  2 *Wa(r WebDriver
  42. 42. Cucumber/RSpec/whatever Capybara Selenium  2 *Wa(r WebDriver
  43. 43. Cucumber/RSpec/whatever Capybara Culerity Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  44. 44. Cucumber/RSpec/whatever Capybara Culerity Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  45. 45. Cucumber/RSpec/whatever Capybara Culerity OMGWTFBBQ?! Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  46. 46. Cucumber/RSpec/whatever Capybara Culerity Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  47. 47. Cucumber/RSpec/whatever Capybara Culerity Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  48. 48. http://www.flickr.com/photos/snake-eyes/449442699/
  49. 49. Cucumber/RSpec/whatever Capybara Culerity Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  50. 50. Cucumber/RSpec/whatever Capybara Culerity Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  51. 51. Cucumber/RSpec/whatever Capybara Culerity Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  52. 52. require 'watir' Watir::Browser.default = 'firefox' describe 'Google' do before(:each) { @browser = Watir::Browser.new } after(:each) { @browser.close } it 'should return search results for "hello world"' do @browser.goto "http://www.google.co.uk" @browser.text_field(:name, "q").set("hello world") @browser.button(:name, "btnG").click @browser.contains_text( "Hello world program - Wikipedia").should be_true end end
  53. 53. Cucumber/RSpec/whatever Capybara Culerity Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  54. 54. Cucumber/RSpec/whatever Capybara Culerity Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  55. 55. require 'selenium-webdriver' describe 'Google' do before(:each){ @browser = Selenium::WebDriver.for :firefox } after(:each) { @browser.quit } it 'should return search results for "hello world"' do @browser.navigate.to "http://www.google.co.uk" @browser.find_element(:name, "q").send_keys("hello world") @browser.find_element(:name, "btnG").submit @browser.find_element(:partial_link_text, "Hello world program - Wikipedia") end end
  56. 56. Cucumber/RSpec/whatever Capybara Culerity Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  57. 57. Cucumber/RSpec/whatever Capybara Culerity Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  58. 58. require 'capybara'; require 'capybara/dsl' Capybara.default_driver = :selenium Spec::Runner.configure do |config| config.include(Capybara, :type => :integration) config.before(:each) { Capybara.reset_sessions! } end describe 'Google' do it 'should return search results for "hello world"' do visit "http://www.google.co.uk" fill_in "q", :with => "hello world" click "Google Search" page.should have_content( "Hello world program - Wikipedia") end end
  59. 59. # API beauty contest! goto "http://www.google.co.uk" # Watir navigate.to "http://www.google.co.uk" # WebDriver visit "http://www.google.co.uk" # Capybara text_field(:name, "q").set("hello world") # Watir find_element(:name, "q").send_keys("hello world") # WD fill_in "q", :with => "hello world" # Capybara button(:name, "btnG").click # Watir find_element(:name, "btnG").submit # WebDriver click "Google Search" # Capybara # You choose...
  60. 60. Cucumber Capybara Culerity Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  61. 61. Feature: Looking up a contact In order to understand my contacts better As a Gmail user I want to see social information next to my email thread Scenario: Load an email conversation Given I am logged into Gmail And Rapportive is loaded When I search for "martin@rapportive.com" And I click on the conversation with subject "Testing 1 2 3" Then I should be on a conversation view And Rapportive should show "Co-founder at Rapportive"
  62. 62. Given /^I am logged into Gmail$/ do visit 'https://mail.google.com/a/rapportive.com' fill_in 'Email', :with => 'test' fill_in 'Passwd', :with => GMAIL_TEST_ACCOUNT_PASSWORD click 'Sign in' end Then /^Rapportive should show "([^"]*)"$/ do |text_to_show| page.within_frame('canvas_frame') do page.within(:css, '#rapportive-sidebar') do page.should have_content(text_to_show) end end end
  63. 63. Ok.
  64. 64. FEI (Frequently   Encountered  Issues)
  65. 65. 10.  Time  dependence
  66. 66. 10.  Time  dependence ‣ Modify  date(mes  in  DB   using  models ‣ Create  wrapper  around   Time.now
  67. 67. 9.  Real-­‐*me  comms
  68. 68. 8.  Managing  test  VMs
  69. 69. 8.  Managing  test  VMs ‣ Very  tedious  (updates  etc.) ‣ Test  management   infrastructure ‣ Commercial  services  (e.g.   Sauce  Labs,  BrowserMob)
  70. 70. 7.  Random  DOM  IDs
  71. 71. 7.  Random  DOM  IDs ‣ Ext.js,  GWT,  Cappuccino ‣ Use  toolkit  APIs // Selecting an item in a Ext.js Combo Box var combo = Ext.getCmp('countryComboBox'); combo.setValue('Germany'); // setValue() doesn't trigger the event combo.fireEvent('select');
  72. 72. 6.  Tes*ng  layout
  73. 73. 6.  Tes*ng  layout ‣ Automa(cally  spoWng   broken  layout?  srsly? ‣ Interes(ng  experiment h"p://code.google.com/p/figh/ng-­‐layout-­‐bugs/
  74. 74. $(‘*’).css(‘color’,  ‘white’);
  75. 75. $(‘*’).css(‘color’,  ‘black’);
  76. 76. image1  –  image2  =  ...
  77. 77. 5.  Unit  tes*ng  JS
  78. 78. 5.  Unit  tes*ng  JS ‣ Command  line  &&  browser ‣ DOM  manipula(on  →  HTML   fixtures    (...and  rollback?)
  79. 79. 5.  Unit  tes*ng  JS ‣ JSpec h"p://visionmedia.github.com/jspec/ ‣ Blue  Ridge/Screw.Unit h"p://github.com/relevance/blue-­‐ridge ‣ JsTestDriver h"p://code.google.com/p/js-­‐test-­‐driver/
  80. 80. 4.  Model  layer  access
  81. 81. 4.  Model  layer  access Tests HTTP/Framework Views Controllers Models
  82. 82. 4.  Model  layer  access Tests HTTP/Framework Views Controllers Factory/ Fixtures? Models
  83. 83. 4.  Model  layer  access ‣ Violates  abstrac(on ‣ BUT:  factories  =  only  sane   way  of  managing  DB  state
  84. 84. Factory.define :user do |u| u.first_name 'John' u.last_name 'Doe' u.admin false end Given /^I am on John's profile page$/ do user = Factory.create(:user) visit user_url(user) end
  85. 85. 3.  Star*ng  state
  86. 86. 3.  Star*ng  state ‣ Previous,  failed  tests ‣ Cookies,  browser  cache ‣ Unexpected  persistent  state
  87. 87. 3.  Star*ng  state ‣ Long-­‐running  transac(on  &   rollback (e.g.  Cucumber  per-­‐scenario  transac/on  rollback) ‣ Clean  up  database h"p://github.com/bmabey/database_cleaner
  88. 88. 2.  Parallel  test  runs
  89. 89. 2.  Parallel  test  runs ‣ End-­‐to-­‐end  tests  are  SLOW ‣ Start  up  several  browsers (TestSwarm,  Selenium  Grid,  ...) ‣ BUT:  concurrent   modifica(on  of  DB  state!
  90. 90. 2.  Parallel  test  runs One  DB  per  test  process ‣ h"p://github.com/grosser/parallel_tests ‣ h"p://github.com/qxjit/deep-­‐test
  91. 91. 2.  Parallel  test  runs Caveat:  state  outside  of  your   control ‣ e.g.  OAuth  status ‣ use  mocks,  or  work  around  it
  92. 92. 1.
  93. 93. Fragility
  94. 94. Fragility (tests  break  too  easily,   even  if  the  app  is  ok)
  95. 95. 2  schools  of  thought 1. “Disposable”  test  scripts (invest  licle  effort,  use   recording  tools) 2. “Engineered”  test  scripts (use  carefully  designed   abstrac(ons)
  96. 96. Feature: Looking up a contact In order to understand my contacts better As a Gmail user I want to see social information next to my email thread Scenario: Load an email conversation Given I am logged into Gmail And Rapportive is loaded When I search for "martin@rapportive.com" And I click on the conversation with subject "Testing 1 2 3" Then I should be on a conversation view And Rapportive should show "Co-founder at Rapportive"
  97. 97. Fragility No  silver  bullet,  obvious  stuff: ‣ Consistent  naming  of  IDs  &   CSS  classes ‣ Maintain  &  refactor  your   tests
  98. 98. kthxbai Mar*n  Kleppmann mar*n@rappor*ve.com @mar*nkl rapportive

×