Journey to 1000 Tests a Day      Bruce McLeodTuesday, 15 November 11
DISCLAIMER      The views and content presented in this presentation      are my own and do not represent the views of any...
Tuesday, 15 November 11
Every journey has a first step                               My introduction to Test AutomationTuesday, 15 November 11
Record and playbackTuesday, 15 November 11
Huge gains compared to     manual execution           • Win32 desktop application           • 5 different clients (95, 98 ...
But there were problems           • Tests were not maintainable           • Spent days in maintenance and development     ...
Tuesday, 15 November 11
Tuesday, 15 November 11
Kindergarten                          The early manual coding yearsTuesday, 15 November 11
Manual coding beginsTuesday, 15 November 11
A 2001 test automation recipe     Ingredients             1 eager client             1 test application             14 tes...
A 2001 test automation recipe     Method           Step 1 : Write an automation “framework”           Step 2 : Create a gu...
Lessons learnt     • A Successful project     • 200 tests automated in 6 months     •   Team was a real dev. team with all...
Tuesday, 15 November 11
Childhood                          The innocent years of keyword testingTuesday, 15 November 11
What if we remove the need to                   code?Tuesday, 15 November 11
Spreadsheets !!!Tuesday, 15 November 11
Keyword driven testing                   Object            Action                  Data            Textfield (domain)      ...
Keyword driven testing           • Advantages           • Reduction in coding effort           • “Anyone” can write tests!...
… Lots of Spreadsheets !!!Tuesday, 15 November 11
Tuesday, 15 November 11
Adolescence                          Embracing open sourceTuesday, 15 November 11
Manual coding returnsTuesday, 15 November 11
Until …Tuesday, 15 November 11
Tuesday, 15 November 11
Unexpected benefits     •   We could now use 1st generation developer tools     •   Visual Studio 2005     •   Version cont...
Unexpected benefits     • Testers were now true peers to developers     • Same tools     • Same development language     • ...
Tuesday, 15 November 11
A discovery & epiphany!                          The “braidy tester” blog postsTuesday, 15 November 11
The “sparkle” stack          • Used to test Microsoft Expression          • Key principles          • Test cases written i...
Tuesday, 15 November 11
How can I execute 1000 tests a day ?Tuesday, 15 November 11
Design goals     •   Maintainability     •   Robustness     •   Performance     •   Maintainability     •   Rapid test dev...
MaintainabilityTuesday, 15 November 11
Devs change stuff     It’s their job !Tuesday, 15 November 11
Each action should occur once     and once onlyTuesday, 15 November 11
Each control should be mapped once     and once onlyTuesday, 15 November 11
Abstract the engine     You shouldn’t know what’s “under the bonnet”Tuesday, 15 November 11
RobustnessTuesday, 15 November 11
Tests should fail, not your tools.Tuesday, 15 November 11
Embrace regexTuesday, 15 November 11
Leverage your vendor, and the       communityTuesday, 15 November 11
PerformanceTuesday, 15 November 11
CPU should be your bottleneckTuesday, 15 November 11
Never wait !                    Wait, don’t sleepTuesday, 15 November 11
Scale-outTuesday, 15 November 11
Rapid test developmentTuesday, 15 November 11
K.I.S.STuesday, 15 November 11
Cheat !           • Leverage an IDE              • Refactoring           • Use code generation           • Write less code...
What about the future?Tuesday, 15 November 11
Tuesday, 15 November 11
Open source frameworks will     consolidate     Driven by adoption of webdriverTuesday, 15 November 11
More focus on mobile automation     iOS, Android, Windows PhoneTuesday, 15 November 11
Behaviour driven development     Will become mainstreamTuesday, 15 November 11
Given I am a logged in user  When I have a valid credit card  And I have 1 click purchasing enabled  Then I should see the...
Tuesday, 15 November 11
Tuesday, 15 November 11
Automation is development.     Hire, plan, develop and execute with that in mind.Tuesday, 15 November 11
Tuesday, 15 November 11
Image attribution     • Creative commons attribution credit is required for the         following users on flickr          ...
Upcoming SlideShare
Loading in …5
×

Journey to 1000 tests a day

2,118 views
2,054 views

Published on

Automation has come a long way in the last 15 years. This presentation tells the story of my journey from no automation, to running in excess of 1000 tests a day. It shares my insights, failures, successes and lessons learned, so hopefully you can get there too.

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,118
On SlideShare
0
From Embeds
0
Number of Embeds
1,062
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Journey to 1000 tests a day

  1. 1. Journey to 1000 Tests a Day Bruce McLeodTuesday, 15 November 11
  2. 2. DISCLAIMER The views and content presented in this presentation are my own and do not represent the views of any of my employers, past or present. All content is copyright 2011 Bruce McLeod Creative commons attribution for images is given at the end of this presentationTuesday, 15 November 11
  3. 3. Tuesday, 15 November 11
  4. 4. Every journey has a first step My introduction to Test AutomationTuesday, 15 November 11
  5. 5. Record and playbackTuesday, 15 November 11
  6. 6. Huge gains compared to manual execution • Win32 desktop application • 5 different clients (95, 98 Beta, NT4, NT 351 and Citrix) • 5 days work for 3 people could be done in 4 hours • 97% efficiency improvementTuesday, 15 November 11
  7. 7. But there were problems • Tests were not maintainable • Spent days in maintenance and development • They were brittle and fragile • And when I left for my next job ...Tuesday, 15 November 11
  8. 8. Tuesday, 15 November 11
  9. 9. Tuesday, 15 November 11
  10. 10. Kindergarten The early manual coding yearsTuesday, 15 November 11
  11. 11. Manual coding beginsTuesday, 15 November 11
  12. 12. A 2001 test automation recipe Ingredients 1 eager client 1 test application 14 test automatorsTuesday, 15 November 11
  13. 13. A 2001 test automation recipe Method Step 1 : Write an automation “framework” Step 2 : Create a gui map Step 3 : Write a test Step 4 : Repeat step 4 another 200 timesTuesday, 15 November 11
  14. 14. Lessons learnt • A Successful project • 200 tests automated in 6 months • Team was a real dev. team with all the same issues • Versioning • Merging and branching • Coding standards • Individual differences in style and designTuesday, 15 November 11
  15. 15. Tuesday, 15 November 11
  16. 16. Childhood The innocent years of keyword testingTuesday, 15 November 11
  17. 17. What if we remove the need to code?Tuesday, 15 November 11
  18. 18. Spreadsheets !!!Tuesday, 15 November 11
  19. 19. Keyword driven testing Object Action Data Textfield (domain) Enter text <domain> Textfield (username) Enter text <username> Textfield (password) Enter text <password> Button (login) Click One left click l Object - Action - data syntax l Automation libraries with excel front endTuesday, 15 November 11
  20. 20. Keyword driven testing • Advantages • Reduction in coding effort • “Anyone” can write tests!! • Challenges • “Anyone” can write tests!! • Each implementation is custom • Maintenance effort moves from code into ...Tuesday, 15 November 11
  21. 21. … Lots of Spreadsheets !!!Tuesday, 15 November 11
  22. 22. Tuesday, 15 November 11
  23. 23. Adolescence Embracing open sourceTuesday, 15 November 11
  24. 24. Manual coding returnsTuesday, 15 November 11
  25. 25. Until …Tuesday, 15 November 11
  26. 26. Tuesday, 15 November 11
  27. 27. Unexpected benefits • We could now use 1st generation developer tools • Visual Studio 2005 • Version control • C# and .net instead of VBATuesday, 15 November 11
  28. 28. Unexpected benefits • Testers were now true peers to developers • Same tools • Same development language • Same recruitment process • Object oriented designTuesday, 15 November 11
  29. 29. Tuesday, 15 November 11
  30. 30. A discovery & epiphany! The “braidy tester” blog postsTuesday, 15 November 11
  31. 31. The “sparkle” stack • Used to test Microsoft Expression • Key principles • Test cases written in customer view • Verification and data separated from tests • Application internals re-created in test codeTuesday, 15 November 11
  32. 32. Tuesday, 15 November 11
  33. 33. How can I execute 1000 tests a day ?Tuesday, 15 November 11
  34. 34. Design goals • Maintainability • Robustness • Performance • Maintainability • Rapid test developmentTuesday, 15 November 11
  35. 35. MaintainabilityTuesday, 15 November 11
  36. 36. Devs change stuff It’s their job !Tuesday, 15 November 11
  37. 37. Each action should occur once and once onlyTuesday, 15 November 11
  38. 38. Each control should be mapped once and once onlyTuesday, 15 November 11
  39. 39. Abstract the engine You shouldn’t know what’s “under the bonnet”Tuesday, 15 November 11
  40. 40. RobustnessTuesday, 15 November 11
  41. 41. Tests should fail, not your tools.Tuesday, 15 November 11
  42. 42. Embrace regexTuesday, 15 November 11
  43. 43. Leverage your vendor, and the communityTuesday, 15 November 11
  44. 44. PerformanceTuesday, 15 November 11
  45. 45. CPU should be your bottleneckTuesday, 15 November 11
  46. 46. Never wait ! Wait, don’t sleepTuesday, 15 November 11
  47. 47. Scale-outTuesday, 15 November 11
  48. 48. Rapid test developmentTuesday, 15 November 11
  49. 49. K.I.S.STuesday, 15 November 11
  50. 50. Cheat ! • Leverage an IDE • Refactoring • Use code generation • Write less code • Use design patternsTuesday, 15 November 11
  51. 51. What about the future?Tuesday, 15 November 11
  52. 52. Tuesday, 15 November 11
  53. 53. Open source frameworks will consolidate Driven by adoption of webdriverTuesday, 15 November 11
  54. 54. More focus on mobile automation iOS, Android, Windows PhoneTuesday, 15 November 11
  55. 55. Behaviour driven development Will become mainstreamTuesday, 15 November 11
  56. 56. Given I am a logged in user When I have a valid credit card And I have 1 click purchasing enabled Then I should see the message “your purchase was successful” And I should be re-directed to the home page.Tuesday, 15 November 11
  57. 57. Tuesday, 15 November 11
  58. 58. Tuesday, 15 November 11
  59. 59. Automation is development. Hire, plan, develop and execute with that in mind.Tuesday, 15 November 11
  60. 60. Tuesday, 15 November 11
  61. 61. Image attribution • Creative commons attribution credit is required for the following users on flickr • seattlemunicipalarchives • mathias-erhart • jaffathecake • 28096801 • loty • brilliantmichaelTuesday, 15 November 11

×