Automation has come a long way in the last 15 years. This presentation tells the story of my journey from no automation, to running in excess of 1000 tests a day. It shares my insights, failures, successes and lessons learned, so hopefully you can get there too.

1. Journey to 1000 Tests a Day
Bruce McLeod
Tuesday, 15 November 11

2. DISCLAIMER
The views and content presented in this presentation
are my own and do not represent the views of any of my
employers, past or present.
All content is copyright 2011 Bruce McLeod
Creative commons attribution for images is given at the end
of this presentation
Tuesday, 15 November 11

3. Tuesday, 15 November 11

4. Every journey has a first step
My introduction to Test Automation
Tuesday, 15 November 11

5. Record and playback
Tuesday, 15 November 11

6. Huge gains compared to
manual execution
• Win32 desktop application
• 5 different clients (95, 98 Beta, NT4, NT 351 and Citrix)
• 5 days work for 3 people could be done in 4 hours
• 97% efficiency improvement
Tuesday, 15 November 11

7. But there were problems
• Tests were not maintainable
• Spent days in maintenance and development
• They were brittle and fragile
• And when I left for my next job ...
Tuesday, 15 November 11

8. Tuesday, 15 November 11

9. Tuesday, 15 November 11

10. Kindergarten
The early manual coding years
Tuesday, 15 November 11

11. Manual coding begins
Tuesday, 15 November 11

12. A 2001 test automation recipe
Ingredients
1 eager client
1 test application
14 test automators
Tuesday, 15 November 11

13. A 2001 test automation recipe
Method
Step 1 : Write an automation “framework”
Step 2 : Create a gui map
Step 3 : Write a test
Step 4 : Repeat step 4 another 200 times
Tuesday, 15 November 11

14. Lessons learnt
• A Successful project
• 200 tests automated in 6 months
• Team was a real dev. team with all the same issues
• Versioning
• Merging and branching
• Coding standards
• Individual differences in style and design
Tuesday, 15 November 11

15. Tuesday, 15 November 11

16. Childhood
The innocent years of keyword testing
Tuesday, 15 November 11

17. What if we remove the need to
code?
Tuesday, 15 November 11

18. Spreadsheets !!!
Tuesday, 15 November 11

19. Keyword driven testing
Object Action Data
Textfield (domain) Enter text <domain>
Textfield (username) Enter text <username>
Textfield (password) Enter text <password>
Button (login) Click One left click
l Object - Action - data syntax
l Automation libraries with excel front end
Tuesday, 15 November 11

20. Keyword driven testing
• Advantages
• Reduction in coding effort
• “Anyone” can write tests!!
• Challenges
• “Anyone” can write tests!!
• Each implementation is custom
• Maintenance effort moves from code into ...
Tuesday, 15 November 11

21. … Lots of Spreadsheets !!!
Tuesday, 15 November 11

22. Tuesday, 15 November 11

23. Adolescence
Embracing open source
Tuesday, 15 November 11

24. Manual coding returns
Tuesday, 15 November 11

25. Until …
Tuesday, 15 November 11

26. Tuesday, 15 November 11

27. Unexpected benefits
• We could now use 1st generation developer tools
• Visual Studio 2005
• Version control
• C# and .net instead of VBA
Tuesday, 15 November 11

28. Unexpected benefits
• Testers were now true peers to developers
• Same tools
• Same development language
• Same recruitment process
• Object oriented design
Tuesday, 15 November 11

29. Tuesday, 15 November 11

30. A discovery & epiphany!
The “braidy tester” blog posts
Tuesday, 15 November 11

31. The “sparkle” stack
• Used to test Microsoft Expression
• Key principles
• Test cases written in customer view
• Verification and data separated from tests
• Application internals re-created in test code
Tuesday, 15 November 11

32. Tuesday, 15 November 11

33. How can I execute 1000 tests a day ?
Tuesday, 15 November 11

34. Design goals
• Maintainability
• Robustness
• Performance
• Maintainability
• Rapid test development
Tuesday, 15 November 11

35. Maintainability
Tuesday, 15 November 11

36. Devs change stuff
It’s their job !
Tuesday, 15 November 11

37. Each action should occur once
and once only
Tuesday, 15 November 11

38. Each control should be mapped once
and once only
Tuesday, 15 November 11

39. Abstract the engine
You shouldn’t know what’s “under the bonnet”
Tuesday, 15 November 11

40. Robustness
Tuesday, 15 November 11

41. Tests should fail, not your tools.
Tuesday, 15 November 11

42. Embrace regex
Tuesday, 15 November 11

43. Leverage your vendor, and the
community
Tuesday, 15 November 11

44. Performance
Tuesday, 15 November 11

45. CPU should be your bottleneck
Tuesday, 15 November 11

46. Never wait !
Wait, don’t sleep
Tuesday, 15 November 11

47. Scale-out
Tuesday, 15 November 11

48. Rapid test development
Tuesday, 15 November 11

49. K.I.S.S
Tuesday, 15 November 11

50. Cheat !
• Leverage an IDE
• Refactoring
• Use code generation
• Write less code
• Use design patterns
Tuesday, 15 November 11

51. What about the future?
Tuesday, 15 November 11

52. Tuesday, 15 November 11

53. Open source frameworks will
consolidate
Driven by adoption of webdriver
Tuesday, 15 November 11

54. More focus on mobile automation
iOS, Android, Windows Phone
Tuesday, 15 November 11

55. Behaviour driven development
Will become mainstream
Tuesday, 15 November 11

56. Given I am a logged in user
When I have a valid credit card
And I have 1 click purchasing enabled
Then I should see the message “your purchase was successful”
And I should be re-directed to the home page.
Tuesday, 15 November 11

57. Tuesday, 15 November 11

58. Tuesday, 15 November 11

59. Automation is development.
Hire, plan, develop and execute with that in mind.
Tuesday, 15 November 11

60. Tuesday, 15 November 11

61. Image attribution
• Creative commons attribution credit is required for the
following users on flickr
• seattlemunicipalarchives
• mathias-erhart
• jaffathecake
• 28096801
• loty
• brilliantmichael
Tuesday, 15 November 11