1.
Browser-level testing
Martin Kleppmann
Go Test It
Email: martin@go-test.it
Blog: yes-no-cancel.co.uk
Twitter: @martinkl

2.
“But I have unit
tests!”

3.
?

4.
Array literals?
var x = [1,];
x.length == 1 // Firefox 3.5
x.length == 2 // IE 7

5.
Uninitialised vars?
<div id="hello"></div>
typeof(hello) == 'undefined' // Firefox 3.5
typeof(hello) == 'object' // IE 7

6.
Attempt #1:
Simulate events in JavaScript.
(Selenium, Windmill, Sahi)

7.
Attempt #1: Simulate events in JavaScript.
Selenium example.com

8.
Attempt #1: Simulate events in JavaScript.
click
Selenium example.com

9.
Attempt #1: Simulate events in JavaScript.
click
Selenium example.com
http://localhost:8080 http://test.example.com

10.
Attempt #1: Simulate events in JavaScript.
Cross-domain
security policy
click
Selenium example.com
http://localhost:8080 http://test.example.com

11.
Attempt #1: Simulate events in JavaScript.
Selenium example.com

12.
Attempt #1: Simulate events in JavaScript.
Selenium example.com
test.example.com proxy

13.
Attempt #1: Simulate events in JavaScript.
Selenium example.com
test.example.com proxy
/selenium-server/ /*

14.
Attempt #1: Simulate events in JavaScript.
click
Selenium example.com
test.example.com proxy
/selenium-server/ /*

15.
Limitations
• Redirect to other domain or https
• Outside of DOM: pop-ups, Flash, ...
• Slow

16.
Attempt #2:
Browser-specific automation
APIs.
(WebDriver, Watir)

17.
IE
Firefox
Safari
Chrome
Opera
HTMLUnit

18.
IE
Firefox
Safari
WebDriver
Chrome
Opera
HTMLUnit

19.
IE
Firefox
Safari
WebDriver
Chrome
Opera
HTMLUnit

20.
IE
Firefox
Safari
WebDriver
Chrome
Opera
Watir
HTMLUnit

21.
IE
Firefox
Safari
WebDriver
Chrome
Opera
Watir
HTMLUnit

22.
IE
Selenium
Firefox
Safari
WebDriver
Chrome
Opera
Watir
HTMLUnit

23.
IE
Selenium
Firefox
Safari
WebDriver
Chrome
Opera
Watir
HTMLUnit

24.
IE
Selenium
Firefox
Safari
WebDriver
Chrome
Opera
Watir
HTMLUnit

25.
Selenium WebDriver Watir
IE
Firefox
Safari
Chrome
Opera
HTMLUnit

26.
Selenium WebDriver Watir
IE HTA/Proxy ✔ COM ✔ COM
Firefox Ext/Proxy ✔ Extension ✔ Extension
Safari Proxy ✘ ✔ AppleScript
Chrome Proxy ✔ Extension ✔ V8 Debug
Opera Proxy Dragonfly? ✘
HTMLUnit ✘ ✔ ✔ Celerity

27.
Watir
require 'watir'
Watir::Browser.default = 'firefox'
describe 'Google' do
before(:each) { @browser = Watir::Browser.new }
after(:each) { @browser.close }
it 'should return search results for "hello world"' do
@browser.goto "http://www.google.co.uk"
@browser.text_field(:name, "q").set("hello world")
@browser.button(:name, "btnG").click
@browser.contains_text(
"Hello world program - Wikipedia").should be_true
end
end

28.
WebDriver
require 'selenium-webdriver'
describe 'Google' do
before(:each) { @browser = Selenium::WebDriver.for :firefox }
after(:each) { @browser.quit }
it 'should return search results for "hello world"' do
@browser.navigate.to "http://www.google.co.uk"
@browser.find_element(:name, "q").send_keys("hello world")
@browser.find_element(:name, "btnG").submit
@browser.find_element(:partial_link_text,
"Hello world program - Wikipedia")
end
end

29.
Selenium
gem 'selenium-client'; require 'selenium/client'
describe 'Google' do
before(:each) {
@browser = Selenium::Client::Driver.new :browser => "*firefox",
:url => "http://www.google.co.uk"
@browser.start
}
after(:each) { @browser.stop }
it 'should return search results for "hello world"' do
@browser.open "/"
@browser.type "q", "hello world"
@browser.click "btnG", :wait_for => :text, :text => 'Results'
@browser.is_text_present("Hello world program - Wikipedia").
should be_true
end
end

30.
Go Test It (≈Selenium)
require 'gotest'
gotest "http://www.google.co.uk" do
open "/"
type "q", "hello world"
click "btnG"
verify_text_present "Hello world program - Wikipedia"
end

31.
Cucumber
Feature: Google search
In order to learn something new
As an enthusiast for obscure programming languages
I want to find "Hello world" programs on Google
Scenario: Search for "hello world"
Given that I am on google.co.uk
When I enter "hello world" into the search box
And I submit the form
Then I should be shown a list of results
And the page should contain "Hello world program -
Wikipedia"

32.
IE
Selenium
Firefox
Safari
WebDriver
Chrome
Opera
Watir
HTMLUnit

33.
IE
Selenium
Firefox
Safari
WebDriver
Chrome
Opera
Watir
HTMLUnit

34.
IE
Selenium
Firefox
Safari
WebDriver
Chrome
Opera
Watir
HTMLUnit

35.
The future? IE
Selenium
Selenium API
Firefox
Safari
WebDriver
Chrome
Opera
Watir
Watir API
HTMLUnit

36.
Ok.

37.
Fundamental goal:
Robust tests

38.
Choose your locators
carefully

39.
Choose your locators
carefully
id = searchInput
name = search
xpath = //form[@id='searchform']/input[2]

40.
link = EU makes 7bn euro climate pledge

41.
link = EU makes 7bn euro climate pledge
xpath = //td[@class='text']/div[1]/a

42.
link = EU makes 7bn euro climate pledge
xpath = //td[@class='text']/div[1]/a
css = a.tsh

43.
People are complicated
Frontend tests are
messy

44.
2 approaches:
Disposable tests
(record/playback, quick&dirty)
Domain-specific abstractions
(carefully crafted)

45.
Which
approach will
you take?

46.
Thank you!
Martin Kleppmann
Go Test It
Email: martin@go-test.it
Blog: yes-no-cancel.co.uk
Twitter: @martinkl

47.
Fundamental goal:
Parallel tests

48.
Don’t assume a clean database

49.
Don’t assume a clean database
Unit tests:
Load fixtures
Given...
When...
Then...
Reset database

50.
Don’t assume a clean database
Unit tests: Functional tests:
Load fixtures Set up example data
Given... Do stuff Do stuff
When...
Check Check
Then... outcome outcome
Reset database

51.
Tolerate partially run tests

52.
Tolerate partially run tests
(Assume there is no frob)
Create frob
Check frob was created
Delete frob

53.
Tolerate partially run tests
(Assume there is no frob)
Create frob
Check frob was created
Delete frob

54.
Tolerate partially run tests
Bad:
Boom!
(Assume there is no frob)
Create frob
Check frob was created
Delete frob

55.
Tolerate partially run tests
Bad: Better:
Boom!
(Assume there is no frob) Delete frob if it exists
Create frob Create frob
Check frob was created Check frob was created
Delete frob Delete frob (optional)

56.
Avoid race conditions: isolation

57.
Avoid race conditions: isolation
Delete frob if it exists Delete frob if it exists
Create frob Create frob
Check frob was created Check frob was created

58.
Avoid race conditions: isolation
Log in as User A Log in as User B
Delete frob if it exists Delete frob if it exists
Create frob Create frob
Check frob was created Check frob was created

59.
Avoid race conditions: isolation
Log in as User A Log in as User B
Delete frob if it exists Delete frob if it exists
Create frob Create frob
Check frob was created Check frob was created
User A and User B: independent & not used concurrently

60.
Make test runs unique

61.
Make test runs unique

62.
Make test runs unique

63.
Take care with external services

64.
Thank you!
Martin Kleppmann
Go Test It
Email: martin@go-test.it
Blog: yes-no-cancel.co.uk
Twitter: @martinkl