The document discusses characteristics of good and powerful test automation frameworks. A good framework provides reliability, modularity, error handling, reusability, and reporting. A powerful framework reduces support activities time through features like one touch deployment, zero touch code updates, centralized logging, smart debugging, and hassle-free remote code management. It also improves efficiency through multi-threading, hot pluggable third party scripts, and a results database. The document advocates moving to powerful frameworks rather than just maintaining good frameworks for reduced boredom and sustained innovation.

1. Good
Power

2. Good Test Automation Frameworks
• Reliability
– Framework must reliably work in all supported capabilities otherwise it becomes un
productive
• Modularity
– For easy maintenance, easy extendibility and for easy understandability
• Error Handling and Recovery mechanisms
– Must have recovery mechanisms to handle exceptions or failures or hanging issues.
– This is mandatory in Endurance , performance , stress and persistence testing.
• Reusability
– Every module/library should be reusable in many ways.
• Reporting
– All Testcases marked with PASS/FAIL/ERROR and Failure reason if test fails.
• No Testcase order Dependency
– Every Testcase should be able to run independently without depending on other
Testcase and Each Testcase should not come to initial state at end to achieve this.
– Framework should have intelligence for auto identification of configuration in different
test cases and should configure only delta configuration between test cases.
– It saves test execution time greatly.

3. How good is ‘Good Enough’?
• Prolonged Support activity in long run
– Typical automation developer spends his 40-50% time on automation support activities
(debugging, deployment, Code Updates and code management) after 1 year of script development.
– This percentage might increase if you are not using good frameworks. Either automation developer
dedicates his/her support activities to some other person or himself/herself spends time.
– This simulates saturation for framework features development after certain stage, creates
boredom to automation developer and finally kills innovation after certain period.
– Mindset of JUST MAINTAIN starts. Solution to this is moving to Power Frameworks rather than stopping
at Good Framework
• Power Automation Frameworks not only reduces support activities time but also provides
some great features in addition to Good framework features.
– One Touch Automation deployment
– Zero Touch Automation code update
– Smart Debugging
– Hassle free automation code management for remote device/servers
– Multi Threading to reduce test execution time
– Hot pluggable interface to third party scripts
– Database for results statistics , framework usage statistics
Lets Move To Power

4. Terminology alignment
• Two Broad types of Automation testing
– UI /Web Client Based testing
• Typical setup involves one or more web/UI clients and web/UI servers
• Controlling clients though pixel references or GUI Object references
• Controlling Servers through HTML Posts/Rest API/ SOAP UI e.t.c
• Sometimes one need to maintain some automation code at server side to simulate
some situations at server side. This is referred as remote server automation code
– Embedded Systems testing
• Typical setup involves more than one device or equipment
• Need to control these devices by executing telnet/ssh/ADB/AT e.t.c commands via
serialport/usb/ethernet e.t.c interfaces
• Some times you need to maintain code at remote device to maintain context .
– Few Examples: Serial Port access on remote equipment, To open windows applications from linux box
. This is referred as remote device automation code.
• DUT – Device Under Test

5. Power Test Automation Frameworks
• One Touch Automation Deployment
– Create workspace with single installation script for first time
– Framework should automatically detect and install missing libraries or packages for every new
framework release without user intervention
• Zero Touch Automation Code updates.
– All Automation workspaces/test beds must automatically update code base to new recommended
Framework release.
Note: Not to Latest release. Latest Release may not be recommended
• Centralized logging
– Every Testcase shall have following Logs
• Test case execution flow Logs, low level step flow logs
• Application Debug Logs, Automation Debug Logs
• Server side execution logs (if your application is server based)
• Remote Device Logs (if setup involves some equipment)
– These logs must be collected in one log file or accessible from one log file through links.
– This is very import feature to reduce debugging efforts.( see next slide for more details)

6. Power Test Automation Frameworks-cont
• Smart Debugging
– Framework should be marked with step number for all logs mentioned in previous slide for each
step.
• User must not map different logs time stamps to understand flow.
• This is mandatory if framework can handle multiple steps in parallel
• Should be able to filter all logs for one particular step when users debugging failed step
– Different Users should be able to filter different logs based on their requirement to debug the
issue.
• Framework Basic User is interested only in execution flow Statements to find out which step failed.
• Framework advanced User might be interested in server or remote device logs to understand the issue.
• Framework Developer is interested in debug and low level logs also to debug issue
• One should not re run the test case for collecting logs. Sometimes it is very difficult to reproduce random
bugs.
– Auto calculation of execution time for each step and group of steps .
• This feature is very useful to find out which step is taking longer period and where to optimize the code.
Optimization might require in automation code or in DUT software.
– Test data/instance must be captured for every test run
• Test cases files, Testsuite file, Test configurations parameters files
• Automation framework code revision/version number
• This reduces framework developer debug time greatly
– To know actual test data at that time of test run to debug easily
– Incase of mismatched information between automation user vs developer about test data when debugging issue.

7. Power Test Automation Frameworks-cont
• Hassle free automation code management for remote device/servers
– All automation code should be maintained at one central location.
– Test Controller should push the right code at the time of execution to remote devices/servers
based on parameters.
– Hassles of automation code updates at multiple devices for every new release will vanish.
– Several times automation failures are due to code update mismatches when you have multi
equipment environment.
• Multi threading
– Multiple steps in one Testcase should be able to run in parallel if those steps are not dependent on
each other.
– Multiple Testcases in one test suite should be able to run in parallel if those test cases are not
dependent on each other.
– If they are sharing same resource,
• Need to protect with semaphores while accessing devices
• Need to share Control sessions ( Every remote device/server will have max limit for parallel telnet/SSH e.t.c
access)
– This feature greatly reduces test execution time.
– Some times this feature is mandatory to meet parallel testing requirement. Few Examples below
• Data transfer from multiple mobiles at same time
• Simultaneous stateful server access
– You must store all threads created in framework in global structure to identify threads hanged in
case of error

8. Power Test Automation Frameworks-cont
• Hot pluggable interface to third party scripts
– There could be third party scripts to control some devices.
– Sometimes some legacy scripts exists in working conditions which are not part of framework
– Some scripts written in different scripting language which are not planned to migrate.
– Tester or framework user developed some scripts offline for his/her usage.
– Those can be Hot pluggable into Framework. Modules/scripts Need to undergo minimal changes
to follow few basic rules defined by framework.
• Database for results statistics , framework usage statistics
– All Results need to be stored in database with test data & Results.
• Software version on DUT, Test suite details , Date the test run , User Details, test bed details, Automation
code version, Type of Tests e.t.c .
• Every possible detail need to be stored
• You need to choose right data base( SQL or no SQL) for your requirement.
– This data is very important to understand
• Trend analysis of different DUT Software releases
• Problems areas in DUT Software releases
• Usability of the Automation framework
• Where to enhance the framework

9. How good is “Power”
Next Generation Automation Frameworks –
Power++ comes with A.I.
…..Continues
Why the heck you require A.I in automation frameworks??
No Framework exists with A.I. as of today
See in my next article.