Reducing Build Time

Saturday, 26 th March, 2011

Reducing build time when patience is not a virtue
Aman King
Application Developer
Thought Works

Recognize these?

Rank these

Did you choose this? 1 3 2

Or did you choose this? 1 2 3

Patience is not always a virtue!

Fail fast! quick feedback

Rank these
High functional coverage
Short build time

Did you choose this?
Short build time

Or did you choose this?
Short build time

You can choose both!
Short build time

But before we see how…

Project background

Content driven, community oriented website

Ruby on Rails Cucumber + Capybara + Selenium

4 : 1 ratio of Dev : QA Everyone writes functional tests in a common automation suite Acceptance Test Driven Development (ATDD)

Our problems

Long build time ~ 55 minutes

Non-deterministic failures

Manual reruns needed

Our solution: Parallelization

Basic idea

Reduced build time ~ 5 minutes

Build time chart

What we did

Parallelization on a single multicore machine used Ruby library parallel_tests

Report generation in parallel wrote custom report formatter

Isolating databases each process needs its own database

database.yml test: &test adapter: mysql2 encoding: utf8 reconnect: false database: myapp_test<%= ENV['TEST_ENV_NUMBER'] %> pool: 5 username: root password:

Isolating external dependencies (Solr) each process needs its own Solr instance

sunspot.yml test: solr: hostname: localhost port: <%= 8982 - ENV['TEST_ENV_NUMBER'].to_i %> log_level: INFO #WARNING log_file: <%= File.join(::Rails.root, 'log', "solr_#{ENV['TEST_ENV_NUMBER'].to_i}.log") %> data_path: <%= File.join(::Rails.root, 'solr', 'data', "#{ENV['TEST_ENV_NUMBER'].to_i}") %> pid_path: <%= File.join(::Rails.root, 'solr', 'pids', "#{ENV['TEST_ENV_NUMBER'].to_i}") %>

Handling multiple Firefox instances and Selenium’s use of shared ephemeral ports

Monitoring Memory and CPU usage adjust number of parallel process accordingly

Auto rerunning failed tests non-deterministic failures are likelier with the CPU under stress

Beyond parallelization…

Stubbing external calls reduce dependency-related delays where avoidable

Consolidating scenarios combine similar test scenarios into single runs instead of separate tests

Maintaining conventions when writing automated tests avoid time-based wait statements, use test framework APIs that take lesser time, etc

Conclusion

Resources
http://cukes.info
http://rubygems.org/gems/capybara
http://seleniumhq.org/projects/remote-control
http://rubygems.org/gems/parallel_tests
http://selenium-grid.seleniumhq.org
http://test-load-balancer.github.com
http://github.com/bblimke/webmock
http://testing.thoughtworks.com

Thank you
[email_address]
http://www.wikyblog.com/AmanKing

Reducing Build Time

by Aman King on Apr 03, 2011

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 1

Statistics

Reducing Build Time Presentation Transcript