Scaling Selenium

2,918 views

Published on

8 minute lightning talk from the September 23, 2011 NYC Selenium meetup at Etsy labs. http://bit.ly/p1duO3

Published in: Technology, News & Politics
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,918
On SlideShare
0
From Embeds
0
Number of Embeds
104
Actions
Shares
0
Downloads
17
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Scaling Selenium

  1. 1. Scaling SeleniumNoah SussmanEtsy
  2. 2. 2006: Test JavaScript and CSS locally
  3. 3. 2011: Test the whole Web application stack.
  4. 4. Why would we need to scale our Selenium testing?
  5. 5. MTBF: Mean Time Between FailuresMTTR: Mean Time To RecoverMTTR > MTBF
  6. 6. It’s not whether you get knocked down; it’swhether you get back up. ~Vince Lombardi
  7. 7. Recovery usually requires deploying new code
  8. 8. faster deploys = smaller changesets = fewer change-related outages
  9. 9. End-to-end tests are slow as hell
  10. 10. Scaling communication, concurrency and hardware
  11. 11. Most people have a hard timeinterpreting the results of one failing test
  12. 12. Aggregating and reportingon test results becomesincreasingly challenging.
  13. 13. graphs > emails
  14. 14. Use Jenkins to run tests concurrently.
  15. 15. There are other ways to do it, butJenkins was the simplest (for us).
  16. 16. Jenkins Matrix Build plugin
  17. 17. ######Master Build ###### ########
  18. 18. Most regressions are catastrophic
  19. 19. Walk the happy path
  20. 20. early 2010:Headless Firefox on Linux, raw EC2 instances
  21. 21. late 2010:Physical Hardware
  22. 22. xxxx 2011: SauceLabsxxxx
  23. 23. hybrid cluster
  24. 24. early 2010: about 10 people making 5 deployments a day ~150 Selenium tests run every daymid 2011: about 20 people making 30+ deployments ~700 Selenium tests a day
  25. 25. Lessons LearnedCommunication and discoverability remain our biggest challenges.Jenkins’ concurrent build capacity is a bottleneck.Even with a vendor in front of it, the cloud is still the cloud.
  26. 26. etsy.com/careers
  27. 27. Noah Sussmannoah@etsy.com@noahsussman
  28. 28. photo creditsparkour guy: loki_v http://www.flickr.com/photos/loki_v/268938583/easy button: crd http://www.flickr.com/photos/crd/283156445sleeping cat: binibix http://www.flickr.com/photos/binibix/6090137450etsy bike rack: cubicle spy http://mashable.com/2010/10/13/etsy-office-picsetsy infrastructure diagram: john allspawetsy deployment velocity graph: erik kastnerupside down server: chris munns http://www.flickr.com/photos/teknogeek/5316475923/

×