Talk given at Railsconf 2010 in Baltimore, MD. Covers ways of scaling tests.

1. Rocket Fuelled
Cucumbers
Joseph Wilk

2. Dealing with an
Outbreak of Cucumbers
Joseph Wilk

3. No Cucumbers where
harmed in the making of
this presentation.
Just mildly shaken up

4. Scenario: Happy Railsconf attendees
Given you have a vague idea what Cucumber is
When Joseph completes his presentation
Then you should understand the scaling choices
And you should feel better prepared to face them

5. Cucumber
Feature: Filling Cucumbers with
• BDD framework rocket fuel
• Plaintext Scenario: Rocket fuel
Given I have trained monkeys
• Promotes When I click the launch button
communication Then the monkeys should not die
Given /^I have trained monkeys$/ do
@test_pilots << Monkeys.new
end

6. Cucumber provides the venue
Your code makes it slow
Shame on you

7. What are Acceptance Tests?
• Customer-facing
• Cut through the whole stack
• Examples of how the system is intended
to be used

9. You made
me Cuke

11. 1 minute ~ build

12. Local builds
1 minute ~ build

13. Local builds
Autotest
1 minute ~ build

14. Local builds
Autotest Growl
1 minute ~ build

17. Don’t
Panic

18. Slow testing problems
Running all the Cukes
takes too long!
Development is slowed
down by the time to run
focused Cukes

19. Types of Feedback
+ Speed
Development - Confidence
- Speed
Test Build + Confidence

20. Happiness is a
10 min build time

21. Slow builds are
the enemy of
continuous integration

22. Development Feedback
Its too slow to run the cukes.
Just push the code and run away

23. Reuse - Spork
$ spork cucumber
$ cucumber --drb
require 'rubygems'
require 'spork'
Spork.prefork do
puts "I'm loading all the heavy stuff..."
end
Spork.each_run do
puts "I'm loading the stuff just for this run..."
# Cucumber hooks must go here
end
git://github.com/timcharper/spork.git

24. Slow services
• Search
Before('@solr') do
• Solr
Solr.boot unless Solr.running?
• Sphinx end
• Databases
• Mongo Feature: Searching
• Redis
@solr
• Mysql Scenario: Indexed search
• Message Queues Given ...
When ...
• RabbitMQ
Then ...

25. Run Just Enough Tests
Autotest
# * Test files must be stored in test/
# * Test files names must start with test_
# * Test class names must start with Test
# * Implementation files must be stored in lib/
# * Implementation files must match up with a
# test file named
# test_.*implementation.rb
Cucover
@analyzer = Rcov::CodeCoverageAnalyzer.new

26. http://vimeo.com/12500864

27. Slicing Features
@media, @public
Feature: Visitor views artist’s media
@feed, @admin
Feature: Logged in user views views their activity feed
Tagging $ cucumber --tags @media
Filenames $ cucumber visitor_*
Subsystems $ cucumber features/admin/*
Profiles $ cucumber --profile admin

28. Just enough Database

29. Just enough Database
INSERT INTO `cities`
(`name`) VALUES ('Test name')
SELECT * FROM `cities` INSERT INTO
WHERE (`cities`.`id` = 105838) `venues` ...
SELECT * FROM
Cache city data
`venues`
WHERE ...
venue data Cache

30. Test Build Feedback
Commit code
Have a 1 hour nap

31. Options
• Hardware
• Intelligent test selection
• Tighter focused tests
• Divide and Conquer

32. Testjour
Songkick.com
254 Features, 1257 scenarios, 10807 steps
1 build server ~ 4 hours
5 build servers ~ 2 hours

33. 120 sad minutes of my life...

34. $ git clone git://github.com/brynary/testjour.git
$ rake gem install
$ testjour --help

35. $ git clone git://github.com/brynary/testjour.git
$ rake gem install
$ testjour --help
testjour help:

36. Testjour
Worker
Slave
Worker
Redis
Work Worker
Master Slave
Queue Worker
Worker
Slave
Worker

37. Testjour
Worker
rsync - code Slave
Worker
Redis
Work Worker
Master Slave
Queue Worker
Worker
rsync - code Slave
Worker

38. Testjour
Worker
rsync - code Slave
Worker
Redis
Work Worker
Master Slave
Queue Worker
Worker
rsync - code Slave
Worker

39. Testjour
Worker
rsync - code Slave
Worker
Redis
Work Worker
Master Slave
Queue Worker
Worker
rsync - code Slave
Worker

40. Looking to the Cloud
EC2, Rackspace Cloud, Windows Azure

41. Looking to the Cloud
EC2, Rackspace Cloud, Windows Azure
1 build server ~ 4 hours

42. Looking to the Cloud
EC2, Rackspace Cloud, Windows Azure
1 build server ~ 4 hours
20 build servers ~ 11:40 minutes

43. Looking to the Cloud
EC2, Rackspace Cloud, Windows Azure
1 build server ~ 4 hours
20 build servers ~ 11:40 minutes
$2068.99 / $3000

44. Ec2
Disc image
AMI
slim-sumo

45. Ec2
Disc image
AMI
Launch
slim-sumo

46. Ec2
Disc image
ec2 ec2 ec2 ec2
AMI
node node node node
Launch
slim-sumo

47. Ec2
Disc image
ec2 ec2 ec2 ec2
AMI
node node node node
Launch
Build
ENV['EC2_URL'] = ec2_url
require 'slim-sumo'
Sumo.new.running.map{|ec2_node|
ec2_node[:hostname]}
slim-sumo

48. http://vimeo.com/12467100

49. require 'rubygems'
require 'hydra'
require 'hydra/tasks'
Hydra::TestTask.new('hydra') do |t|
t.add_files 'features/**/*.feature'
t.verbose = false
t.autosort = false
end

51. “Buy lots of hardware”

52. “Buy lots of hardware”
“more hardware please”

53. “Buy lots of hardware”
“more hardware please”
“Just a little bit more hardware”

54. “Buy lots of hardware”
“more hardware please”
“Just a little bit more hardware”

55. Just run the tests
that matter...

56. Failure probability
matters
European Computing manufacturer

57. Failure probability
matters
European Computing manufacturer
Automated test suite ~ 18 hours

58. Failure probability
matters
European Computing manufacturer
Automated test suite ~ 18 hours
Tests that were most ~ 2 hours
likely to fail

59. http://vimeo.com/12467122

61. Tests that
never fail

62. Tests that
never fail

63. Tests which Tests that
regularly fail never fail

64. Tests which Tests that
regularly fail never fail
Daily Build

65. Tests which Tests that
regularly fail never fail
Daily Build Nightly Build

66. Tests which Tests that
regularly fail never fail
Daily Build Nightly Build
$cucumber --tags ~@nightly $cucumber --tags @nightly

67. Flickering Confidence

68. Flickering Confidence
FAIL

69. Flickering Confidence
FAIL
FAIL

70. Flickering Confidence
FAIL
FAIL
PASS

71. Flickering Confidence
FAIL
FAIL
PASS
Main @flicker Flicker

72. Divide and conquer
7 7 7
1 hour
7 7 7
7 7 7
• Rails - Engines
• Service Orientated Architecture

73. Divide and Concuquer

74. Divide and Concuquer

75. Divide and Concuquer

76. Divide and Concuquer

77. Divide and Concuquer

78. Divide and Concuquer
Scenario: Related artists
Given an artist “SYGC”
And “M.Bison” is related to “SYGC”
When I visit the artist
Then I should see within Similar
Artists a link to “M.Bison”

79. Divide and conquer
Rails
Scenario: Related artists
Given an artist “SYGC”
And “M.Bison” is related to “SYGC”
When I visit the artist
Then I should see within Similar
Artists a link to “M.Bison”

80. Divide and conquer
Related
Rails Artists
Service
Scenario: Related artists
Given an artist “SYGC”
And “M.Bison” is related to “SYGC”
When I visit the artist
Then I should see within Similar
Artists a link to “M.Bison”

81. Divide and conquer
related(artist)
Related
Rails Artists
Service
Scenario: Related artists
Given an artist “SYGC”
And “M.Bison” is related to “SYGC”
When I visit the artist
Then I should see within Similar
Artists a link to “M.Bison”

82. Divide and conquer
related(artist)
Related
Rails HTML Artists
Service
Scenario: Related artists
Given an artist “SYGC”
And “M.Bison” is related to “SYGC”
When I visit the artist
Then I should see within Similar
Artists a link to “M.Bison”

83. Divide and conquer
related(artist)
Related
Rails HTML Artists
Service
Scenario: Related artists
Given an artist “SYGC”
And “M.Bison” is related to “SYGC”
When I visit the artist
Then I should see within Similar
Artists a link to “M.Bison”

84. Divide and conquer
related(artist)
Related
Rails HTML Artists
Service
unit test unit test
Scenario: Related artists
Given an artist “SYGC”
And “M.Bison” is related to “SYGC”
When I visit the artist
Then I should see within Similar
Artists a link to “M.Bison”

85. Don’t use Acceptance
tests
http://jamesshore.com/Blog/Alternatives-
to-Acceptance-Testing.html
Heresy!

86. Pairwise
Scenario: Testing Cucumber against different environments
Given I have a rails app <Rails version>
And I’m using Ruby <Ruby version>
And I am using the <Test Framework>
And I am using the driver <Driver>
Then Cucumber should install and play nicely
Examples:
| Rails version | Ruby version | test framework | Driver |
| 3.0 | 1.8.7 | Rspec | Webrat |
| 3.0 | 1.8.7 | Testunit | Capybara |
| 3.0 | 1.8.7 | Rspec | Capybara |
| 3.0 | 1.8.7 | Testunit | Webrat |
| 3.0 | 1.9 | Rspec | Webrat |
| 3.0 | 1.9 | Testunit | Webrat |
| 2.3.1 | 1.9 | Rspec | Webrat |
....

87. | Rails version | Ruby version | test framework | Driver |
| 2.3.1 | 1.8.7 | Rspec | Webrat |
| 2.3.1 | 1.8.7 | Rspec | Capybara |
| 2.3.1 | 1.8.7 | Testunit | Webrat |
| 2.3.1 | 1.8.7 | Testunit | Capybara |
| 2.3.1 | 1.9 | Rspec | Capybara |
| 2.3.1 | 1.9 | Rspec | Webrat |
| 2.3.1 | 1.9 | Testunit | Capybara |
| 2.3.1 | 1.9 | Testunit | Webrat |
....
“most faults are caused by interactions of at most two
factors”

88. Pairwise
http://www.screencast.com/t/NGI0NjVk

89. Just when you thought
things were going well
We should add some
funky Ajax features

90. JavaScript
Browser based In-memory
Selenium, Watir Celerity

91. Need for speed
Webrat 0.8
Celerity 12
Watir 12
Selenium 12.9
0 3.75 7.5 11.25 15
start-up time (seconds)

92. Capybara sudo gem install capybara
Like Webrat but bigger
envjs gem install envjs
Johnson gem install johnson --prerelease
spidermonkey

93. http://vimeo.com/12467133

94. Capybara-envjs

96. Find the right
compromise for you

97. Find the right
compromise for you
Slow test builds produce
slow release cycles

98. Find the right
compromise for you
Slow test builds produce
slow release cycles
Your test builds are a
goldmine of data

99. Thanks
Its been
emotional
Joseph Wilk
http://blog.josephwilk.net
@josephwilk