STPCon presentation

1. …or proving Einstein was right about insanity
Presented by
Jim Hazen

2. Introduction
 Jim Hazen
 Veteran of the Software Testing Trenches
 Experience in software testing, both commercial
and consulting work.
 Working with automation and tools since 1989.
 Implemented Automation Tools on DOS, OS/2,
Windows and Web platforms.
 Experience with tools; including HP/Mercury,
IBM/Rational, and other ‘scripting’ languages.

3. Sage Words to Remember
 “Insanity: doing the same thing over and over again
and expecting different results.” – Albert Einstein

4. Agenda
 What is ‘Test Automation’
 A Brief History
 Different types of test automation, tools and uses
 Automation Myths
 How to correct them
 Resources & Skills needed
 Setting expectations for Management & Others
 Summary
 Q&A

5. What is Test Automation?

6. It’s a Big Topic to Discuss

7. Bigger Than We Think!

8. Is It This?

9. Or More Like This?

10. Who Builds the Automation?

11. Some Are Seen This Way

12. Most Look Like This Instead

13. What ‘Test Automation’ Really Is
 “Automation” definition
 Webster’s – “the technique of making an apparatus, a
process, or a system operate automatically”
 In Software Testing we use ‘tools’ to drive our processes
 People only think of tools that execute a test as
‘Automation’ .
 Other tools can be used for Test Design, Test Data
Generation, Defect Tracking, Test Management, etc.
 “Automation” is using a tool during any part of the
process of testing.

14. Who Test Automator’s Really Are

15. A Brief History
 Test “Execution” tools have been around a long time!
Mainframes (pre
Mid-80’s)
PC DOS
(Mid 80’s –
90’s)
PC GUI &
Web (1990’s
– present)
Mobile
(2008 –
present)
 Mainframe – Home grown ‘macro’ tools1, Hyperion
 PC DOS – AutoTester, SuperKey (macro tool)
 PC GUI – Automated Test Facility (ATF), Rational Robot,
QA Run, Winrunner, etc.
 PC Web – QTP, SilkTest, Selenium, TestComplete, etc.
 Mobile – MonkeyTalk, Robotium, etc.

16. Other Types of Test Automation Tools
 Other Test Execution Tools
 xUnit (JUnit & NUnit), SOAPUI, FitNesse, etc.
 Test Design & Test Data Generation
 Hexawise, AllPairs, Tosca Test Suite, etc.
 Test Execution Management
 HP ALM, MS Team Foundation Server, TestLink, etc.
 Test Environment Management
 VMWare vCloud Director, etc.
 Defect Tracking & Reporting
 Bugzilla, JIRA, etc.

17. Automation Myths & Misconceptions

18. Old Myths
 You can automate 100% of your tests
 Automation will “cure” all problems with testing
 Automation tools are usable by anyone
 Record/Playback is all you need
 Automation allows Management to cut test staff
 One tool is all you need
 Automation will speed up testing process
* Oldies, but goodies!!

19. New Myths
 You don’t have to know how to program
 Opensource tools are “free”
 Commercial tools are “expensive”
 GUI level automation is a waste of time
 “Codeless” and “Scriptless” tools
 Commercial tools can’t support agile
 Developers only need to focus on Unit Tests

20. How to Correct Them
 100% Automation – Certain tests shouldn’t or can’t be
‘automated’. 80/20 Rule; automate 20% of
functionality used 80% of time.
 Automation solves all testing problems – it helps, but
it creates new problems too.
 Automation can be done by anyone – Not everyone has
the ‘mindset’, it is both learned and inherent (logic).
 Record/Playback is all you need – Proven time & time
again to be false, and to fail.

21. How to Correct Them cont.
 One tool is all you need – There isn’t a one-size fits all.
 Automation speeds up testing process – The “Illusion
of Speed” problem. Both execution and creation.
 You don’t need to know programming – Anyone can
Automate problem.
 Opensource tools are free – Costs due to ramp up time
(money) and skilled staff to do it (more expensive).2
 Commercial = Expensive – Not really in grand scheme
of things. 2

22. How to Correct Them cont.
 GUI level automation is a waste of time – Not really

23. How to Correct Them cont.
 Codeless/Scriptless tools – Just a different UI. You’re
still writing a ‘test’ which is a set of ‘instructions’
(code).
 Commercial tools don’t do Agile – Yes they do, it’s how
you use the tool that counts.
 Developers only create Unit Tests – Other tools require
someone to ‘build’ the fixtures/code to ‘talk’ to the
SUT.

24. Why Failures Happen
15%
11%
17%
20%
37%
Failure Reason
Success
Tool Incompatibility
Insufficient Budget
Lack of Experience
Lack of Time
Source: IDT Survey3
85% Failure Rate! Too High!!

25. Einstein was right!
 We keep repeating history, why?
 Always a ‘new’ group of management and testers.
 Companies always looking to do testing faster, cheaper
and with less resources.
 Vendors and others who sell “Automagic”.
 Instant gratification mindset. Automation takes time.
 Not learning from past mistakes.
 Software not designed with “testability” in mind.
 Testers not involved early with decisions, not included
by management.

26. Resources for Automation
 Dedicated Staff
 Skilled & experienced staff
 Dedicated Equipment/Environments
 Reduce risk of competition of resource or teams
stepping on each other
 Dedicated Budget
 Automation isn’t cheap, plan appropriately
 Dedicated Management Support
 Can’t have any of the other 3 without it

27. Skills for Test Automation
 It’s a different view and purpose for the script code.
 Creating ‘software’ to ‘test’ software; i.e.,
PROGRAMMING!
 Need basic understanding of programming methods.
 Understand different types of technology layers
 GUI, API, System, Web, Services, Database, etc.
 You need to know how to program in different
languages when needed.
 C#, Java, VBScript, Jscript, XML, SQL, etc.

28. Test Automation Skills Cont.
 Need to understand how to ‘hook’ the software under
test and interact with it, and at what level.
 API, or Object, or Service, or Protocol, etc.
 Understand how tool ‘interacts with’ the software
under test; what functions/methods are to use.
 GUI Object ‘methods’ (functions)
 Ex: SwfWindow(“LoginForm").SwfEdit(“ID”).SET “Jim”
 Need to understand how to “pick apart” the software
and “drive it” programmatically.

29. Axioms to remember
 SMART
 Standalone
 Test/Script able to run by itself, minimize dependencies
 Modular/Maintainable
 Scripts and code need to be modular in design and easy to
maintain
 Appropriate
 Build according to need for project, don’t try to automate
100%
 Reusable
 Build for reuse; keep script code compact and focused.
 Testable & Traceable
 Make it easy to debug your code, and be able to link together.

30. Axioms to remember cont.
 SEARCH4,5
 Setup
 Test/Script needs to check for pre-test conditions and flag if
needed.
 Execute
 The logic and data needed to run the test itself
 Analyze
 Validations, Error Handling, etc.
 Report
 Progress, Error, Summation of run information (Logfile)
 Cleanup
 Test/Script needs to ‘cleanup’ (temp files, reset data, etc.)
 Help (or Home)

31. Setting Realistic Expectations
 Management & Other Groups
 100% automation is an unreachable goal.6,7
 Automation is its own form of software development.
 Ramp-up time will be required, longer than you think.
 Benefits realized over time, no immediate payback for
investment.

32. Expectations cont.
 Management & Other Groups
 No tool supports all systems & technologies out of the box.
 Not all testers can write scripts. Specialized resources are a
must and need to be dedicated to the work.
 Nothing is “Free”, there are always costs involved.
 Don’t believe the marketing hype! Don’t buy the snake oil.8
There is no Magic!

33. Otherwise, the result

34. Summary
 What is ‘Test Automation’ – A lot more than you think!
 History – Lots of tools over the last 30 plus years.
 Different types of test automation tools – Not just for
execution of a test only.
 Automation Myths – Old and New; lots of failures due
to them.
 How to fix the problems – Be realistic, dispel the myths.
 Resources and Skills needed for test automation –
Again, a lot more than you think. It is Programming!
 Setting expectations for Management & Others –
Without proper expectations you are painted into a
corner.

35. Remember… It’s automation,
Not Automagic!

36. Q&A

37. Contact Info
 Jim Hazen
 Company: Freelance Contractor
 Home email: calkelpdiver@gmail.com
 LinkedIn profile:
http://www.linkedin.com/profile/view?id=2835000&trk=tab_pro

38. References
1) “Experiences of Test Automation: Case Studies of Software Test Automation”,
Chapter 5, Graham & Fewster, Addison-Wesley, 2012
2) “Test Automation - Let's Talk Business”, Igor Gershovich, Connected Testing,
2006.
3) “Implementing Automated Software Testing”, IDT Survey, Elfriede Dustin et
al., Addison-Wesley, 2009.
4) “How We Test Software At Microsoft”, page 187, Alan Page, Microsoft Press,
2009.
5) “How to Automate Testing: The Big Picture“, Keith Stobie & Mark Bergman,
1992.
6) “Seven Steps to Test Automation Success”, Bret Pettichord,
http://www.io.com/~wazmo/papers/seven_steps.html, 2001.
7) “Success with Test Automation”, Bret Pettichord,
http://www.io.com/~wazmo/succpap.htm, 2001.
8) “Test Automation Snake Oil”, James Bach, 1996

39. Thank you for attending this session.
Please fill out an evaluation form.