Rocket Fuelled
 Cucumbers




            Joseph Wilk
Dealing with an
Outbreak of Cucumbers




                Joseph Wilk
No Cucumbers where
harmed in the making of
   this presentation.

             Just mildly shaken up
Scenario: Happy Railsconf attendees
 Given you have a vague idea what Cucumber is
 When Joseph completes his presentation
...
Cucumber
                     Feature: Filling Cucumbers with
•   BDD framework    rocket fuel

•   Plaintext        Scena...
Cucumber provides the venue
   Your code makes it slow


                  Shame on you
What are Acceptance Tests?

  • Customer-facing
  • Cut through the whole stack
  • Examples of how the system is intended...
You made
me Cuke
1 minute ~ build
Local builds




1 minute ~ build
Local builds
Autotest




   1 minute ~ build
Local builds
Autotest       Growl




   1 minute ~ build
Don’t
Panic
Slow testing problems

          Running all the Cukes
             takes too long!


          Development is slowed
    ...
Types of Feedback
              + Speed
Development   - Confidence




              - Speed
Test Build    + Confidence
Happiness is a
10 min build time
Slow builds are
the enemy of
continuous integration
Development Feedback

   Its too slow to run the cukes.
 Just push the code and run away
Reuse - Spork
                 $ spork cucumber
                 $ cucumber --drb

require 'rubygems'
require 'spork'

Spo...
Slow services
•   Search
                     Before('@solr') do
    •   Solr
                       Solr.boot unless Solr...
Run Just Enough Tests
                       Autotest
#   *   Test files must be stored in test/
#   *   Test files names ...
http://vimeo.com/12500864
Slicing Features
@media, @public
Feature: Visitor views artist’s media

@feed, @admin
Feature: Logged in user views views ...
Just enough Database
Just enough Database
                           INSERT INTO `cities`
                        (`name`) VALUES ('Test name')...
Test Build Feedback

  Commit code
 Have a 1 hour nap
Options

• Hardware
• Intelligent test selection
• Tighter focused tests
• Divide and Conquer
Testjour
Songkick.com
254 Features, 1257 scenarios, 10807 steps
1 build server ~ 4 hours
5 build servers ~ 2 hours
120 sad minutes of my life...
$ git clone git://github.com/brynary/testjour.git
$ rake gem install

$ testjour --help
$ git clone git://github.com/brynary/testjour.git
$ rake gem install

$ testjour --help
 testjour help:
Testjour
                            Worker
                    Slave
                            Worker
         Redis
  ...
Testjour
                                    Worker
         rsync - code       Slave
                                    ...
Testjour
                                    Worker
         rsync - code       Slave
                                    ...
Testjour
                                    Worker
         rsync - code       Slave
                                    ...
Looking to the Cloud

EC2, Rackspace Cloud, Windows Azure
Looking to the Cloud

EC2, Rackspace Cloud, Windows Azure



  1 build server   ~ 4 hours
Looking to the Cloud

EC2, Rackspace Cloud, Windows Azure



  1 build server   ~ 4 hours
  20 build servers ~ 11:40 minut...
Looking to the Cloud

EC2, Rackspace Cloud, Windows Azure



  1 build server   ~ 4 hours
  20 build servers ~ 11:40 minut...
Ec2
Disc image

  AMI




slim-sumo
Ec2
Disc image

  AMI
             Launch




slim-sumo
Ec2
Disc image
                        ec2    ec2    ec2    ec2
  AMI
                       node   node   node   node
   ...
Ec2
Disc image
                            ec2     ec2      ec2     ec2
  AMI
                           node    node     ...
http://vimeo.com/12467100
require 'rubygems'
require 'hydra'
require 'hydra/tasks'

Hydra::TestTask.new('hydra') do |t|
  t.add_files 'features/**/*...
“Buy lots of hardware”
“Buy lots of hardware”


“more hardware please”
“Buy lots of hardware”


   “more hardware please”


“Just a little bit more hardware”
“Buy lots of hardware”


   “more hardware please”


“Just a little bit more hardware”
Just run the tests
  that matter...
Failure probability
         matters
European Computing manufacturer
Failure probability
         matters
European Computing manufacturer

   Automated test suite   ~ 18 hours
Failure probability
         matters
European Computing manufacturer

   Automated test suite   ~ 18 hours

   Tests that ...
http://vimeo.com/12467122
Tests that
never fail
Tests that
never fail
Tests which      Tests that
regularly fail   never fail
Tests which      Tests that
regularly fail   never fail




Daily Build
Tests which       Tests that
regularly fail    never fail




Daily Build      Nightly Build
Tests which                   Tests that
      regularly fail                never fail




       Daily Build            ...
Flickering Confidence
Flickering Confidence
               FAIL
Flickering Confidence
               FAIL

               FAIL
Flickering Confidence
               FAIL

               FAIL

               PASS
Flickering Confidence
                     FAIL

                     FAIL

                     PASS




  Main   @flicker...
Divide and conquer
                        7     7     7
 1 hour
                        7     7     7

                  ...
Divide and Concuquer
Divide and Concuquer
Divide and Concuquer
Divide and Concuquer
Divide and Concuquer
Divide and Concuquer

Scenario: Related artists
 Given an artist “SYGC”
 And “M.Bison” is related to “SYGC”
 When I visit ...
Divide and conquer


   Rails


Scenario: Related artists
 Given an artist “SYGC”
 And “M.Bison” is related to “SYGC”
 Whe...
Divide and conquer

                                      Related
   Rails                              Artists
          ...
Divide and conquer

                 related(artist)
                                      Related
   Rails               ...
Divide and conquer

                 related(artist)
                                      Related
   Rails             HT...
Divide and conquer

          related(artist)
                                        Related
Rails         HTML          ...
Divide and conquer

            related(artist)
                                                   Related
Rails          ...
Don’t use Acceptance
        tests
http://jamesshore.com/Blog/Alternatives-
        to-Acceptance-Testing.html

  Heresy!
Pairwise
Scenario: Testing Cucumber against different environments
  Given I have a rails app <Rails version>
  And I’m us...
| Rails version | Ruby version | test framework   |   Driver     |
|         2.3.1 |        1.8.7 | Rspec            |   W...
Pairwise


http://www.screencast.com/t/NGI0NjVk
Just when you thought
things were going well

We should add some
funky Ajax features
JavaScript


Browser based             In-memory
Selenium, Watir           Celerity
Need for speed
 Webrat 0.8



Celerity                                   12




  Watir                                   ...
Capybara               sudo gem install capybara


Like Webrat but bigger



      envjs              gem install envjs


...
http://vimeo.com/12467133
Capybara-envjs
Find the right
compromise for you
Find the right
 compromise for you

Slow test builds produce
  slow release cycles
Find the right
 compromise for you

Slow test builds produce
  slow release cycles

  Your test builds are a
   goldmine o...
Thanks
                              Its been
                             emotional




        Joseph Wilk
http://blog.j...
Rocket Fuelled Cucumbers
Rocket Fuelled Cucumbers
Rocket Fuelled Cucumbers
Rocket Fuelled Cucumbers
Rocket Fuelled Cucumbers
Rocket Fuelled Cucumbers
Rocket Fuelled Cucumbers
Upcoming SlideShare
Loading in...5
×

Rocket Fuelled Cucumbers

7,180

Published on

Talk given at Railsconf 2010 in Baltimore, MD. Covers ways of scaling tests.

Published in: Technology
2 Comments
11 Likes
Statistics
Notes
No Downloads
Views
Total Views
7,180
On Slideshare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
87
Comments
2
Likes
11
Embeds 0
No embeds

No notes for slide

Transcript of "Rocket Fuelled Cucumbers"

  1. 1. Rocket Fuelled Cucumbers Joseph Wilk
  2. 2. Dealing with an Outbreak of Cucumbers Joseph Wilk
  3. 3. No Cucumbers where harmed in the making of this presentation. Just mildly shaken up
  4. 4. Scenario: Happy Railsconf attendees Given you have a vague idea what Cucumber is When Joseph completes his presentation Then you should understand the scaling choices And you should feel better prepared to face them
  5. 5. Cucumber Feature: Filling Cucumbers with • BDD framework rocket fuel • Plaintext Scenario: Rocket fuel Given I have trained monkeys • Promotes When I click the launch button communication Then the monkeys should not die Given /^I have trained monkeys$/ do @test_pilots << Monkeys.new end
  6. 6. Cucumber provides the venue Your code makes it slow Shame on you
  7. 7. What are Acceptance Tests? • Customer-facing • Cut through the whole stack • Examples of how the system is intended to be used
  8. 8. You made me Cuke
  9. 9. 1 minute ~ build
  10. 10. Local builds 1 minute ~ build
  11. 11. Local builds Autotest 1 minute ~ build
  12. 12. Local builds Autotest Growl 1 minute ~ build
  13. 13. Don’t Panic
  14. 14. Slow testing problems Running all the Cukes takes too long! Development is slowed down by the time to run focused Cukes
  15. 15. Types of Feedback + Speed Development - Confidence - Speed Test Build + Confidence
  16. 16. Happiness is a 10 min build time
  17. 17. Slow builds are the enemy of continuous integration
  18. 18. Development Feedback Its too slow to run the cukes. Just push the code and run away
  19. 19. Reuse - Spork $ spork cucumber $ cucumber --drb require 'rubygems' require 'spork' Spork.prefork do puts "I'm loading all the heavy stuff..." end Spork.each_run do puts "I'm loading the stuff just for this run..." # Cucumber hooks must go here end git://github.com/timcharper/spork.git
  20. 20. Slow services • Search Before('@solr') do • Solr Solr.boot unless Solr.running? • Sphinx end • Databases • Mongo Feature: Searching • Redis @solr • Mysql Scenario: Indexed search • Message Queues Given ... When ... • RabbitMQ Then ...
  21. 21. Run Just Enough Tests Autotest # * Test files must be stored in test/ # * Test files names must start with test_ # * Test class names must start with Test # * Implementation files must be stored in lib/ # * Implementation files must match up with a # test file named # test_.*implementation.rb Cucover @analyzer = Rcov::CodeCoverageAnalyzer.new
  22. 22. http://vimeo.com/12500864
  23. 23. Slicing Features @media, @public Feature: Visitor views artist’s media @feed, @admin Feature: Logged in user views views their activity feed Tagging $ cucumber --tags @media Filenames $ cucumber visitor_* Subsystems $ cucumber features/admin/* Profiles $ cucumber --profile admin
  24. 24. Just enough Database
  25. 25. Just enough Database INSERT INTO `cities` (`name`) VALUES ('Test name') SELECT * FROM `cities` INSERT INTO WHERE (`cities`.`id` = 105838) `venues` ... SELECT * FROM Cache city data `venues` WHERE ... venue data Cache
  26. 26. Test Build Feedback Commit code Have a 1 hour nap
  27. 27. Options • Hardware • Intelligent test selection • Tighter focused tests • Divide and Conquer
  28. 28. Testjour Songkick.com 254 Features, 1257 scenarios, 10807 steps 1 build server ~ 4 hours 5 build servers ~ 2 hours
  29. 29. 120 sad minutes of my life...
  30. 30. $ git clone git://github.com/brynary/testjour.git $ rake gem install $ testjour --help
  31. 31. $ git clone git://github.com/brynary/testjour.git $ rake gem install $ testjour --help testjour help:
  32. 32. Testjour Worker Slave Worker Redis Work Worker Master Slave Queue Worker Worker Slave Worker
  33. 33. Testjour Worker rsync - code Slave Worker Redis Work Worker Master Slave Queue Worker Worker rsync - code Slave Worker
  34. 34. Testjour Worker rsync - code Slave Worker Redis Work Worker Master Slave Queue Worker Worker rsync - code Slave Worker
  35. 35. Testjour Worker rsync - code Slave Worker Redis Work Worker Master Slave Queue Worker Worker rsync - code Slave Worker
  36. 36. Looking to the Cloud EC2, Rackspace Cloud, Windows Azure
  37. 37. Looking to the Cloud EC2, Rackspace Cloud, Windows Azure 1 build server ~ 4 hours
  38. 38. Looking to the Cloud EC2, Rackspace Cloud, Windows Azure 1 build server ~ 4 hours 20 build servers ~ 11:40 minutes
  39. 39. Looking to the Cloud EC2, Rackspace Cloud, Windows Azure 1 build server ~ 4 hours 20 build servers ~ 11:40 minutes $2068.99 / $3000
  40. 40. Ec2 Disc image AMI slim-sumo
  41. 41. Ec2 Disc image AMI Launch slim-sumo
  42. 42. Ec2 Disc image ec2 ec2 ec2 ec2 AMI node node node node Launch slim-sumo
  43. 43. Ec2 Disc image ec2 ec2 ec2 ec2 AMI node node node node Launch Build ENV['EC2_URL'] = ec2_url require 'slim-sumo' Sumo.new.running.map{|ec2_node| ec2_node[:hostname]} slim-sumo
  44. 44. http://vimeo.com/12467100
  45. 45. require 'rubygems' require 'hydra' require 'hydra/tasks' Hydra::TestTask.new('hydra') do |t| t.add_files 'features/**/*.feature' t.verbose = false t.autosort = false end
  46. 46. “Buy lots of hardware”
  47. 47. “Buy lots of hardware” “more hardware please”
  48. 48. “Buy lots of hardware” “more hardware please” “Just a little bit more hardware”
  49. 49. “Buy lots of hardware” “more hardware please” “Just a little bit more hardware”
  50. 50. Just run the tests that matter...
  51. 51. Failure probability matters European Computing manufacturer
  52. 52. Failure probability matters European Computing manufacturer Automated test suite ~ 18 hours
  53. 53. Failure probability matters European Computing manufacturer Automated test suite ~ 18 hours Tests that were most ~ 2 hours likely to fail
  54. 54. http://vimeo.com/12467122
  55. 55. Tests that never fail
  56. 56. Tests that never fail
  57. 57. Tests which Tests that regularly fail never fail
  58. 58. Tests which Tests that regularly fail never fail Daily Build
  59. 59. Tests which Tests that regularly fail never fail Daily Build Nightly Build
  60. 60. Tests which Tests that regularly fail never fail Daily Build Nightly Build $cucumber --tags ~@nightly $cucumber --tags @nightly
  61. 61. Flickering Confidence
  62. 62. Flickering Confidence FAIL
  63. 63. Flickering Confidence FAIL FAIL
  64. 64. Flickering Confidence FAIL FAIL PASS
  65. 65. Flickering Confidence FAIL FAIL PASS Main @flicker Flicker
  66. 66. Divide and conquer 7 7 7 1 hour 7 7 7 7 7 7 • Rails - Engines • Service Orientated Architecture
  67. 67. Divide and Concuquer
  68. 68. Divide and Concuquer
  69. 69. Divide and Concuquer
  70. 70. Divide and Concuquer
  71. 71. Divide and Concuquer
  72. 72. Divide and Concuquer Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
  73. 73. Divide and conquer Rails Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
  74. 74. Divide and conquer Related Rails Artists Service Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
  75. 75. Divide and conquer related(artist) Related Rails Artists Service Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
  76. 76. Divide and conquer related(artist) Related Rails HTML Artists Service Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
  77. 77. Divide and conquer related(artist) Related Rails HTML Artists Service Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
  78. 78. Divide and conquer related(artist) Related Rails HTML Artists Service unit test unit test Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
  79. 79. Don’t use Acceptance tests http://jamesshore.com/Blog/Alternatives- to-Acceptance-Testing.html Heresy!
  80. 80. Pairwise Scenario: Testing Cucumber against different environments Given I have a rails app <Rails version> And I’m using Ruby <Ruby version> And I am using the <Test Framework> And I am using the driver <Driver> Then Cucumber should install and play nicely Examples: | Rails version | Ruby version | test framework | Driver | | 3.0 | 1.8.7 | Rspec | Webrat | | 3.0 | 1.8.7 | Testunit | Capybara | | 3.0 | 1.8.7 | Rspec | Capybara | | 3.0 | 1.8.7 | Testunit | Webrat | | 3.0 | 1.9 | Rspec | Webrat | | 3.0 | 1.9 | Testunit | Webrat | | 2.3.1 | 1.9 | Rspec | Webrat | ....
  81. 81. | Rails version | Ruby version | test framework | Driver | | 2.3.1 | 1.8.7 | Rspec | Webrat | | 2.3.1 | 1.8.7 | Rspec | Capybara | | 2.3.1 | 1.8.7 | Testunit | Webrat | | 2.3.1 | 1.8.7 | Testunit | Capybara | | 2.3.1 | 1.9 | Rspec | Capybara | | 2.3.1 | 1.9 | Rspec | Webrat | | 2.3.1 | 1.9 | Testunit | Capybara | | 2.3.1 | 1.9 | Testunit | Webrat | .... “most faults are caused by interactions of at most two factors”
  82. 82. Pairwise http://www.screencast.com/t/NGI0NjVk
  83. 83. Just when you thought things were going well We should add some funky Ajax features
  84. 84. JavaScript Browser based In-memory Selenium, Watir Celerity
  85. 85. Need for speed Webrat 0.8 Celerity 12 Watir 12 Selenium 12.9 0 3.75 7.5 11.25 15 start-up time (seconds)
  86. 86. Capybara sudo gem install capybara Like Webrat but bigger envjs gem install envjs Johnson gem install johnson --prerelease spidermonkey
  87. 87. http://vimeo.com/12467133
  88. 88. Capybara-envjs
  89. 89. Find the right compromise for you
  90. 90. Find the right compromise for you Slow test builds produce slow release cycles
  91. 91. Find the right compromise for you Slow test builds produce slow release cycles Your test builds are a goldmine of data
  92. 92. Thanks Its been emotional Joseph Wilk http://blog.josephwilk.net @josephwilk
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×