Test automation is not only about coding. Successful test automation involves critical thinking and clarity of objectives before actually beginning development. This material provides guidance in putting some of the right questions and how to think as for having an efficient and effective test automation in the context of your project.

1. QUESTIONS FOR SUCCESSFUL
TEST AUTOMATION PROJECTS
Daniel Ionita
SEETEST 2016

3. WHAT IS THE MISSION OF THE
PROJECT?
• Think BIG, take a more strategical
approach
• Relate to it with clear and
measurable objectives
• Involve appropriate stakeholders,
including sponsors and beneficiaries
Remember as how software changes so the mission may
change and your test automation must adapt!

4. WHY ARE WE NEEDING TEST
AUTOMATION?
• This is related to the quality expectations and
analysis of existing testing and that translates
into many questions:
• What is the current status of your testing?
• Can you make your testing scalable?
• Can testing keep up with development?
• How fast do you want to deliver?
• How often do you want to have a specific
coverage of testing performed?
… and many more

5. SHOW ME WHY?
ROI ROE
ROI = (CTM- CTA)/ CTA*100
Cost of manual testing (CTM) = Cost
of manually testing as many
times as the automated test will
be executed
Cost of test automation(CTA) = Cost
of hardware + Cost of software + Cost
of developing the test automation +
Cost of executing test automation+
Cost of maintaining the test
automation
ROE= (Efficiency gain –
Investment)/Investment*100
Efficiency gain through test automation
= Manual test execution and analysis
time of tests that will be automated
Investment = Test automation
development, execution, analysis and
maintenance time

6. WHO WILL DO THE AUTOMATION?
• Take in consideration
• Software testing skills
• Software development skills
• Knowledge of test automation
scripting/programming language
• Knowledge of the application under test
• Knowledge of the business

7. WHAT TYPE OF TESTING DO YOU
TARGET WITH TEST AUTOMATION?
• Unit testing
• Regression Testing
• Performance Testing
• Exploratory Testing(mostly to support it)
• Random Testing
• Security Testing
• Usability Testing

8. WHAT TYPE OF AUTOMATION?
• Unit tests layer – testing the code
• Service layer – testing the services of
the application independent from its
user interface
• UI – testing the application through
UI

9. WHAT TO AUTOMATE?
• M - must have this – critical to the current delivery
• S - should have this if all possible – important but
not necessary for delivery
• C - could have this but not necessary – desirable
but not necessary
• W - won’t have this - dropped or reconsidered for
inclusion
• Other factors:
• what takes the most time regarding test execution
• priorities of tests
• coverage expected (breadth or depth)
• tests/requirements age
• what are the most tedious and repetitive tests
…..

10. WHAT IS THE TEST AUTOMATION
INPUT AND OUTPUT DESIRED?
Test design
Test artifacts
Hardware
Software
Interaction with AUT
Traceability
Auditing
Risk information
Artifacts to support
manual testing

11. WHAT IS THE COVERAGE EXPECTED FOR
THE TEST ENVIRONMENT?
• Hardware
• Compiler
• OS
• Databases
• Browser
• Communication protocols
• File system
…

12. WHAT TYPE OF FEEDBACK IS
NEEDED?
• Goal is to offer the fastest possible feedback to the
project stakeholders
• The information should drive actions
• The information needs to target a diverse audience
• Reporting
• focus on test automation results
• communication channels
• visualization of test results
• Metrics(do not use the metrics as objectives)
• costs of test automation
• effectiveness of test automation

13. REPORTING
• Information useful for test automation engineers
• highlighting exceptions and providing alerts
• troubleshooting data results at granular and macro
level
• encouraging actions to improve effectiveness and
efficiency
• supporting maintenance of tests
• Information useful for managers
• aggregated info supporting decision making
• highlight risks especially at product level
• integration data analysis( defect tracker, test case
management, coverage …)
• Information useful for software test engineers
• highlight risks at granular and macro level for tests
and products

14. METRICS
• A useful metrics should be accurate and related to
your goals, objectives.
• “Don’t measure anything unless data helps you make
a better decision or change your actions.”
Seth Godin
• Test automation metrics
• ROI/ROE
• Test automation coverage
• Test automation flakiness
• Mean time of investigation and Resolve the
problem
• Cycle time for test automation
• Project metrics that may be impacted by test
automation
• Defect removal efficiency
• Customer defect trends post-release
• Lead time
Do not use the metrics as
targets and evaluating
performance, use them to
for continuous improvement.

15. HOW?
• The W? questions should drive the How?
Cone of uncertainty Programming process
pyramid
Steve McConnell Code Complete, Second Editionhttp://www.construx.com/

16. METHOD?
• A good method as for putting everything together is
ATAM(Architecture Tradeoff Analysis Method)
• In order to proceed with the architecture analysis and
design we need to focus on the following main things:
architectural significant requirements, constraints and
understanding different architectural tactics.
• The W? questions will give you the business drivers,
architectural significant requirements and constraints.
• The main principle of the method is that is a risk-
mitigation process focusing on finding the trade-offs
and sensitivity points to drive you in choosing the
suitable architecture.

17. ANALYSIS?
• Architectures are mostly driven or shaped by
quality attributes requirements.
• The quality attributes used in general for ATAM are
the following: availability,
interoperability,modifiability,performance,security,t
estability,usability.
• Based on the business drivers, architecturally
significant requirements and this quality attributes
a utility tree will be built as to generate concrete
scenarios. Those will be prioritized from the
business and architectural impact to drive the
architectural design analysis.
• As for that analysis you will need to know
architectural approaches, architectural tactics,
design patterns.
Utility Tree

18. TEST AUTOMATION FRAMEWORKS?
• A test automation framework is an integrated system that
sets the rules of automation of a specific product.
• It is defined as the set of assumptions, concepts, and
practices that constitute a work platform or support for
automated testing.
• Linear – basic record and playback approach
• Modular – abstraction or encapsulation of scripts
• Data driven – externalization of test data from scripts
• Keyword driven – application-independent automation
• Hybrid – combination of several techniques

19. ADVANTAGES AND
DISADVANTAGES?
Test framework Advantages Disadvantages
Linear test framework Low technical knowledge
Requires little training
Ease of automation
Short setup time
Brittle tests
Recording is usually repeated for each test
Not respecting software engineering
standards
Modular test framework Reducing the number of test scripts
Greater flexibility and higher scalability
Easier maintenance
Requires technical expertise
Increased time spent for automation
Data driven test framework
model
Reduced maintenance
Less amount of code written
Low data redundancy
Requires expertise in handling test tool and
test data files
Maintenance of test data
Keyword driven test
model
Less technical expertise in automating test cases
Low maintenance cost
Portable architecture between applications and
tools
Extensive technical and tool skills knowledge
for framework development
High cost of implementation
Hybrid test Framework Minimized maintenance
Takes advantages of the other frameworks
strengths
High complexity
High technical expertise