Rocket Fuelled Cucumbers

Rocket Fuelled Cucumbers Joseph Wilk

Dealing with an Outbreak of Cucumbers Joseph Wilk

No Cucumbers where harmed in the making of this presentation. Just mildly shaken up

Scenario: Happy Railsconf attendees Given you have a vague idea what Cucumber is When Joseph completes his presentation Then you should understand the scaling choices And you should feel better prepared to face them

Cucumber Feature: Filling Cucumbers with • BDD framework rocket fuel • Plaintext Scenario: Rocket fuel Given I have trained monkeys • Promotes When I click the launch button communication Then the monkeys should not die Given /^I have trained monkeys$/ do @test_pilots << Monkeys.new end

Cucumber provides the venue Your code makes it slow Shame on you

What are Acceptance Tests? • Customer-facing • Cut through the whole stack • Examples of how the system is intended to be used

You made me Cuke

1 minute ~ build

Local builds 1 minute ~ build

Local builds Autotest 1 minute ~ build

Local builds Autotest Growl 1 minute ~ build

Don’t Panic

Slow testing problems Running all the Cukes takes too long! Development is slowed down by the time to run focused Cukes

Types of Feedback + Speed Development - Conﬁdence - Speed Test Build + Conﬁdence

Happiness is a 10 min build time

Slow builds are the enemy of continuous integration

Development Feedback Its too slow to run the cukes. Just push the code and run away

Reuse - Spork $ spork cucumber $ cucumber --drb require 'rubygems' require 'spork' Spork.prefork do puts "I'm loading all the heavy stuff..." end Spork.each_run do puts "I'm loading the stuff just for this run..." # Cucumber hooks must go here end git://github.com/timcharper/spork.git

Slow services • Search Before('@solr') do • Solr Solr.boot unless Solr.running? • Sphinx end • Databases • Mongo Feature: Searching • Redis @solr • Mysql Scenario: Indexed search • Message Queues Given ... When ... • RabbitMQ Then ...

Run Just Enough Tests Autotest # * Test files must be stored in test/ # * Test files names must start with test_ # * Test class names must start with Test # * Implementation files must be stored in lib/ # * Implementation files must match up with a # test file named # test_.*implementation.rb Cucover @analyzer = Rcov::CodeCoverageAnalyzer.new

http://vimeo.com/12500864

Slicing Features @media, @public Feature: Visitor views artist’s media @feed, @admin Feature: Logged in user views views their activity feed Tagging $ cucumber --tags @media Filenames $ cucumber visitor_* Subsystems $ cucumber features/admin/* Proﬁles $ cucumber --proﬁle admin

Just enough Database

Just enough Database INSERT INTO `cities` (`name`) VALUES ('Test name') SELECT * FROM `cities` INSERT INTO WHERE (`cities`.`id` = 105838) `venues` ... SELECT * FROM Cache city data `venues` WHERE ... venue data Cache

Test Build Feedback Commit code Have a 1 hour nap

Options • Hardware • Intelligent test selection • Tighter focused tests • Divide and Conquer

Testjour Songkick.com 254 Features, 1257 scenarios, 10807 steps 1 build server ~ 4 hours 5 build servers ~ 2 hours

120 sad minutes of my life...

$ git clone git://github.com/brynary/testjour.git $ rake gem install $ testjour --help

$ git clone git://github.com/brynary/testjour.git $ rake gem install $ testjour --help testjour help:

Testjour Worker Slave Worker Redis Work Worker Master Slave Queue Worker Worker Slave Worker

Testjour Worker rsync - code Slave Worker Redis Work Worker Master Slave Queue Worker Worker rsync - code Slave Worker

Looking to the Cloud EC2, Rackspace Cloud, Windows Azure

Looking to the Cloud EC2, Rackspace Cloud, Windows Azure 1 build server ~ 4 hours

Looking to the Cloud EC2, Rackspace Cloud, Windows Azure 1 build server ~ 4 hours 20 build servers ~ 11:40 minutes

Looking to the Cloud EC2, Rackspace Cloud, Windows Azure 1 build server ~ 4 hours 20 build servers ~ 11:40 minutes $2068.99 / $3000

Ec2 Disc image AMI slim-sumo

Ec2 Disc image AMI Launch slim-sumo

Ec2 Disc image ec2 ec2 ec2 ec2 AMI node node node node Launch slim-sumo

Ec2 Disc image ec2 ec2 ec2 ec2 AMI node node node node Launch Build ENV['EC2_URL'] = ec2_url require 'slim-sumo' Sumo.new.running.map{|ec2_node| ec2_node[:hostname]} slim-sumo

require 'rubygems' require 'hydra' require 'hydra/tasks' Hydra::TestTask.new('hydra') do |t| t.add_files 'features/**/*.feature' t.verbose = false t.autosort = false end

“Buy lots of hardware”

“Buy lots of hardware” “more hardware please”

“Buy lots of hardware” “more hardware please” “Just a little bit more hardware”

Just run the tests that matter...

Failure probability matters European Computing manufacturer

Failure probability matters European Computing manufacturer Automated test suite ~ 18 hours

Failure probability matters European Computing manufacturer Automated test suite ~ 18 hours Tests that were most ~ 2 hours likely to fail

Tests that never fail

Tests which Tests that regularly fail never fail

Tests which Tests that regularly fail never fail Daily Build

Tests which Tests that regularly fail never fail Daily Build Nightly Build

Tests which Tests that regularly fail never fail Daily Build Nightly Build $cucumber --tags ~@nightly $cucumber --tags @nightly

Flickering Conﬁdence

Flickering Conﬁdence FAIL

Flickering Conﬁdence FAIL FAIL

Flickering Conﬁdence FAIL FAIL PASS

Flickering Conﬁdence FAIL FAIL PASS Main @flicker Flicker

Divide and conquer 7 7 7 1 hour 7 7 7 7 7 7 • Rails - Engines • Service Orientated Architecture

Divide and Concuquer

Divide and Concuquer Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”

Divide and conquer Rails Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”

Divide and conquer Related Rails Artists Service Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”

Divide and conquer related(artist) Related Rails Artists Service Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”

Divide and conquer related(artist) Related Rails HTML Artists Service Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”

Divide and conquer related(artist) Related Rails HTML Artists Service unit test unit test Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”

Don’t use Acceptance tests http://jamesshore.com/Blog/Alternatives- to-Acceptance-Testing.html Heresy!

Pairwise Scenario: Testing Cucumber against different environments Given I have a rails app <Rails version> And I’m using Ruby <Ruby version> And I am using the <Test Framework> And I am using the driver <Driver> Then Cucumber should install and play nicely Examples: | Rails version | Ruby version | test framework | Driver | | 3.0 | 1.8.7 | Rspec | Webrat | | 3.0 | 1.8.7 | Testunit | Capybara | | 3.0 | 1.8.7 | Rspec | Capybara | | 3.0 | 1.8.7 | Testunit | Webrat | | 3.0 | 1.9 | Rspec | Webrat | | 3.0 | 1.9 | Testunit | Webrat | | 2.3.1 | 1.9 | Rspec | Webrat | ....

| Rails version | Ruby version | test framework | Driver | | 2.3.1 | 1.8.7 | Rspec | Webrat | | 2.3.1 | 1.8.7 | Rspec | Capybara | | 2.3.1 | 1.8.7 | Testunit | Webrat | | 2.3.1 | 1.8.7 | Testunit | Capybara | | 2.3.1 | 1.9 | Rspec | Capybara | | 2.3.1 | 1.9 | Rspec | Webrat | | 2.3.1 | 1.9 | Testunit | Capybara | | 2.3.1 | 1.9 | Testunit | Webrat | .... “most faults are caused by interactions of at most two factors”

Pairwise http://www.screencast.com/t/NGI0NjVk

Just when you thought things were going well We should add some funky Ajax features

JavaScript Browser based In-memory Selenium, Watir Celerity

Need for speed Webrat 0.8 Celerity 12 Watir 12 Selenium 12.9 0 3.75 7.5 11.25 15 start-up time (seconds)

Capybara sudo gem install capybara Like Webrat but bigger envjs gem install envjs Johnson gem install johnson --prerelease spidermonkey

Capybara-envjs

Find the right compromise for you

Find the right compromise for you Slow test builds produce slow release cycles

Find the right compromise for you Slow test builds produce slow release cycles Your test builds are a goldmine of data

Thanks Its been emotional Joseph Wilk http://blog.josephwilk.net @josephwilk

http://blog.josephwilk.net	1364
http://www.g9labs.com	981
http://en.oreilly.com	81
http://www.slideshare.net	46
http://webcache.googleusercontent.com	1
http://paper.li	1
http://ranksit.com	1
http://www.m.techgig.com	1
http://www.linkedin.com	1

Rocket Fuelled Cucumbers

by Joseph Wilk on Jun 11, 2010

Accessibility

Categories

Upload Details

Usage Rights

9 Embeds 2,477

Statistics

Rocket Fuelled Cucumbers Presentation Transcript