Cross-browser testing in the real world

  • 8,751 views
Uploaded on

What's the best way of automating end-to-end, browser-level tests for web apps? In this talk, I compare Selenium, WebDriver, Watir and other libraries, and share experience of automated browser tests …

What's the best way of automating end-to-end, browser-level tests for web apps? In this talk, I compare Selenium, WebDriver, Watir and other libraries, and share experience of automated browser tests on hundreds of different sites. I also give updates on latest developments in open source functional testing tools.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
8,751
On Slideshare
0
From Embeds
0
Number of Embeds
7

Actions

Shares
Downloads
190
Comments
1
Likes
11

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Cross-­‐browser tes*ng in  the real  world Mar*n  Kleppmann rapportive
  • 2. http://www.flickr.com/photos/stuart_spivack/2322070560/
  • 3. Cross-­‐browser tes*ng in  the real  world Mar*n  Kleppmann rapportive
  • 4. Cross-­‐browser urgh tes*ng in  the real  world Mar*n  Kleppmann rapportive
  • 5. Cross-­‐browser tes*ng in  the real  world Mar*n  Kleppmann rapportive
  • 6. Cross-­‐browser tes*ng in  the mess y real  world Mar*n  Kleppmann rapportive
  • 7. Cross-­‐browser tes*ng in  the real  world Mar*n  Kleppmann rapportive
  • 8. Cross-­‐browser t? a ? h w ow h tes*ng in  the real  world Mar*n  Kleppmann rapportive
  • 9. Tes*ng  what?
  • 10. Tes*ng  what? Unit  tes(ng Applica(on End-­‐to-­‐end (incl.  external  services)
  • 11. Tes*ng  why?
  • 12. Tes*ng  why? Func(onal Visual Performance Load/Scalability Security Usability
  • 13. Tes*ng  how?
  • 14. Tes*ng  how? Automated (TDD,  BDD,  regression  tests, smoke  tests,  ...) Manual (exploratory,  scripted, user-­‐centered,  ...)
  • 15. Tes*ng  dimensions Unit Applica(on End-­‐to-­‐end Func(onal Visual Performance Load Security Usability Automated Manual
  • 16. Tes*ng  dimensions Unit Applica(on End-­‐to-­‐end Func(onal Visual Performance Load Security Usability Automated Manual
  • 17. Tes*ng  dimensions Unit Applica(on End-­‐to-­‐end RSpec/ Func(onal Visual Performance Shoulda/ Load Security whatever Usability Automated Manual
  • 18. Tes*ng  dimensions Unit Applica(on End-­‐to-­‐end Func(onal Visual Performance Load Security Usability Automated Manual
  • 19. Tes*ng  dimensions Unit Applica(on End-­‐to-­‐end Func(onal Visual Performance Load Security Usability Automated Manual
  • 20. (Side-­‐note)  wri te a n ’t Yo u  c cs R S p e b i li ty r  u s a fo
  • 21. Tes*ng  dimensions Unit Applica(on End-­‐to-­‐end Func(onal Visual Performance Load Security Usability Automated Manual
  • 22. Tes*ng  dimensions Unit Applica(on End-­‐to-­‐end Func(onal Visual Performance Load Security Usability Automated Manual
  • 23. Me
  • 24. http://www.flickr.com/photos/71263221@N00/4230334515/
  • 25. P.S.  We’re  hiring! http://www.flickr.com/photos/snake-eyes/410092369/
  • 26. Anyway.
  • 27. C++
  • 28. C++
  • 29. C++ WebDriver
  • 30. WebDriver C++
  • 31. WebDriver
  • 32. WebDriver
  • 33. Selenium  2 WebDriver
  • 34. Cucumber/RSpec/whatever Selenium  2 WebDriver
  • 35. Cucumber/RSpec/whatever Selenium  2 WebDriver
  • 36. Cucumber/RSpec/whatever Selenium  2 WebDriver
  • 37. Cucumber/RSpec/whatever Selenium  2 WebDriver
  • 38. Cucumber/RSpec/whatever Selenium  2 WebDriver
  • 39. Cucumber/RSpec/whatever Capybara Selenium  2 WebDriver
  • 40. Cucumber/RSpec/whatever Capybara Selenium  2 *Wa(r WebDriver
  • 41. Cucumber/RSpec/whatever Capybara Selenium  2 *Wa(r WebDriver
  • 42. Cucumber/RSpec/whatever Capybara Selenium  2 *Wa(r WebDriver
  • 43. Cucumber/RSpec/whatever Capybara Culerity Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  • 44. Cucumber/RSpec/whatever Capybara Culerity Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  • 45. Cucumber/RSpec/whatever Capybara Culerity OMGWTFBBQ?! Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  • 46. Cucumber/RSpec/whatever Capybara Culerity Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  • 47. Cucumber/RSpec/whatever Capybara Culerity Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  • 48. http://www.flickr.com/photos/snake-eyes/449442699/
  • 49. Cucumber/RSpec/whatever Capybara Culerity Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  • 50. Cucumber/RSpec/whatever Capybara Culerity Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  • 51. Cucumber/RSpec/whatever Capybara Culerity Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  • 52. require 'watir' Watir::Browser.default = 'firefox' describe 'Google' do before(:each) { @browser = Watir::Browser.new } after(:each) { @browser.close } it 'should return search results for "hello world"' do @browser.goto "http://www.google.co.uk" @browser.text_field(:name, "q").set("hello world") @browser.button(:name, "btnG").click @browser.contains_text( "Hello world program - Wikipedia").should be_true end end
  • 53. Cucumber/RSpec/whatever Capybara Culerity Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  • 54. Cucumber/RSpec/whatever Capybara Culerity Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  • 55. require 'selenium-webdriver' describe 'Google' do before(:each){ @browser = Selenium::WebDriver.for :firefox } after(:each) { @browser.quit } it 'should return search results for "hello world"' do @browser.navigate.to "http://www.google.co.uk" @browser.find_element(:name, "q").send_keys("hello world") @browser.find_element(:name, "btnG").submit @browser.find_element(:partial_link_text, "Hello world program - Wikipedia") end end
  • 56. Cucumber/RSpec/whatever Capybara Culerity Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  • 57. Cucumber/RSpec/whatever Capybara Culerity Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  • 58. require 'capybara'; require 'capybara/dsl' Capybara.default_driver = :selenium Spec::Runner.configure do |config| config.include(Capybara, :type => :integration) config.before(:each) { Capybara.reset_sessions! } end describe 'Google' do it 'should return search results for "hello world"' do visit "http://www.google.co.uk" fill_in "q", :with => "hello world" click "Google Search" page.should have_content( "Hello world program - Wikipedia") end end
  • 59. # API beauty contest! goto "http://www.google.co.uk" # Watir navigate.to "http://www.google.co.uk" # WebDriver visit "http://www.google.co.uk" # Capybara text_field(:name, "q").set("hello world") # Watir find_element(:name, "q").send_keys("hello world") # WD fill_in "q", :with => "hello world" # Capybara button(:name, "btnG").click # Watir find_element(:name, "btnG").submit # WebDriver click "Google Search" # Capybara # You choose...
  • 60. Cucumber Capybara Culerity Selenium  2 *Wa(r WebDriver Celerity HTML Unit
  • 61. Feature: Looking up a contact In order to understand my contacts better As a Gmail user I want to see social information next to my email thread Scenario: Load an email conversation Given I am logged into Gmail And Rapportive is loaded When I search for "martin@rapportive.com" And I click on the conversation with subject "Testing 1 2 3" Then I should be on a conversation view And Rapportive should show "Co-founder at Rapportive"
  • 62. Given /^I am logged into Gmail$/ do visit 'https://mail.google.com/a/rapportive.com' fill_in 'Email', :with => 'test' fill_in 'Passwd', :with => GMAIL_TEST_ACCOUNT_PASSWORD click 'Sign in' end Then /^Rapportive should show "([^"]*)"$/ do |text_to_show| page.within_frame('canvas_frame') do page.within(:css, '#rapportive-sidebar') do page.should have_content(text_to_show) end end end
  • 63. Ok.
  • 64. FEI (Frequently   Encountered  Issues)
  • 65. 10.  Time  dependence
  • 66. 10.  Time  dependence ‣ Modify  date(mes  in  DB   using  models ‣ Create  wrapper  around   Time.now
  • 67. 9.  Real-­‐*me  comms
  • 68. 8.  Managing  test  VMs
  • 69. 8.  Managing  test  VMs ‣ Very  tedious  (updates  etc.) ‣ Test  management   infrastructure ‣ Commercial  services  (e.g.   Sauce  Labs,  BrowserMob)
  • 70. 7.  Random  DOM  IDs
  • 71. 7.  Random  DOM  IDs ‣ Ext.js,  GWT,  Cappuccino ‣ Use  toolkit  APIs // Selecting an item in a Ext.js Combo Box var combo = Ext.getCmp('countryComboBox'); combo.setValue('Germany'); // setValue() doesn't trigger the event combo.fireEvent('select');
  • 72. 6.  Tes*ng  layout
  • 73. 6.  Tes*ng  layout ‣ Automa(cally  spoWng   broken  layout?  srsly? ‣ Interes(ng  experiment h"p://code.google.com/p/figh/ng-­‐layout-­‐bugs/
  • 74. $(‘*’).css(‘color’,  ‘white’);
  • 75. $(‘*’).css(‘color’,  ‘black’);
  • 76. image1  –  image2  =  ...
  • 77. 5.  Unit  tes*ng  JS
  • 78. 5.  Unit  tes*ng  JS ‣ Command  line  &&  browser ‣ DOM  manipula(on  →  HTML   fixtures    (...and  rollback?)
  • 79. 5.  Unit  tes*ng  JS ‣ JSpec h"p://visionmedia.github.com/jspec/ ‣ Blue  Ridge/Screw.Unit h"p://github.com/relevance/blue-­‐ridge ‣ JsTestDriver h"p://code.google.com/p/js-­‐test-­‐driver/
  • 80. 4.  Model  layer  access
  • 81. 4.  Model  layer  access Tests HTTP/Framework Views Controllers Models
  • 82. 4.  Model  layer  access Tests HTTP/Framework Views Controllers Factory/ Fixtures? Models
  • 83. 4.  Model  layer  access ‣ Violates  abstrac(on ‣ BUT:  factories  =  only  sane   way  of  managing  DB  state
  • 84. Factory.define :user do |u| u.first_name 'John' u.last_name 'Doe' u.admin false end Given /^I am on John's profile page$/ do user = Factory.create(:user) visit user_url(user) end
  • 85. 3.  Star*ng  state
  • 86. 3.  Star*ng  state ‣ Previous,  failed  tests ‣ Cookies,  browser  cache ‣ Unexpected  persistent  state
  • 87. 3.  Star*ng  state ‣ Long-­‐running  transac(on  &   rollback (e.g.  Cucumber  per-­‐scenario  transac/on  rollback) ‣ Clean  up  database h"p://github.com/bmabey/database_cleaner
  • 88. 2.  Parallel  test  runs
  • 89. 2.  Parallel  test  runs ‣ End-­‐to-­‐end  tests  are  SLOW ‣ Start  up  several  browsers (TestSwarm,  Selenium  Grid,  ...) ‣ BUT:  concurrent   modifica(on  of  DB  state!
  • 90. 2.  Parallel  test  runs One  DB  per  test  process ‣ h"p://github.com/grosser/parallel_tests ‣ h"p://github.com/qxjit/deep-­‐test
  • 91. 2.  Parallel  test  runs Caveat:  state  outside  of  your   control ‣ e.g.  OAuth  status ‣ use  mocks,  or  work  around  it
  • 92. 1.
  • 93. Fragility
  • 94. Fragility (tests  break  too  easily,   even  if  the  app  is  ok)
  • 95. 2  schools  of  thought 1. “Disposable”  test  scripts (invest  licle  effort,  use   recording  tools) 2. “Engineered”  test  scripts (use  carefully  designed   abstrac(ons)
  • 96. Feature: Looking up a contact In order to understand my contacts better As a Gmail user I want to see social information next to my email thread Scenario: Load an email conversation Given I am logged into Gmail And Rapportive is loaded When I search for "martin@rapportive.com" And I click on the conversation with subject "Testing 1 2 3" Then I should be on a conversation view And Rapportive should show "Co-founder at Rapportive"
  • 97. Fragility No  silver  bullet,  obvious  stuff: ‣ Consistent  naming  of  IDs  &   CSS  classes ‣ Maintain  &  refactor  your   tests
  • 98. kthxbai Mar*n  Kleppmann mar*n@rappor*ve.com @mar*nkl rapportive