The document evaluates and compares several automated testing tools. It discusses key criteria for selection like supported browsers, advanced features, transparency of pricing, and estimated return on investment. Certain tools like Endtest and Selenium score well overall due to transparency, included functionality, and cost effectiveness. Limitations of tools like Cypress and Mabl are also outlined around browser support, flexibility for complex cases, and pricing clarity.

1. The Best Automated
Testing Tools in 2020
A detailed review

2. Tools that we evaluated

3. The importance of Test Automation
Test automation is an important component of software development.
The benefits cannot be understated.
It saves time, it ensures consistency and it increases efficiency.
The Return On Investment for test automation is remarkable.

4. A wide selection
The abundance of tools on the market can make the selection process
overwhelming.
Furthermore, the Marketing tactics used by some of those tools can
send us on the wrong path.
Actually trying all the test automation solutions on the market and
implementing a POC for each one of them is a costly experiment in
terms of time and resources.

5. Who should be in charge of selecting the tool?
Most developers would likely suggest an open source library.
Tempted by the zero cost of an open source library, they might not
realise that the time they consume to turn that library into a working
solution translates into spendings for their company.
Executives are more qualified to estimate the total cost of adopting a
test automation solution, but they might not take into consideration
certain technical details that could turn into blockers.

6. Why even consider paid tools when there are open source alternatives?
A modern test automation solution will always require infrastructure, so
the zero-cost illusion of the open source solution fades away.
The lack of Technical Support could easily translate into time-
consuming research, debugging, digging on forums and finding hacks.
These aspects need to be taken into consideration when calculating the
effort and Return on Investment, for both open source and paid tools.

7. The hidden cost of infrastructure
In terms of infrastructure, a good example would be the need for a
cross-browser grid with different browsers on Windows and Mac
machines.
In today’s CI/CD-centric world, running the tests on your own machine is
considered to be an outdated and risky practice.
Cross-browser testing is not optional.

8. The corner cases and show-stoppers
The risk of evaluating a solution based on a POC is that your company
might adopt that solution without being aware of all the limitations.
A good example is the handling of multiple browser tabs and iframes.
Another example would be the support for browsers such as Internet
Explorer and Safari.

9. Honest Comparison Criteria
We’ve taken into account all the relevant data.
Most comparison charts that we found online were biased and included
only criteria that would dishonestly place certain solutions on top.
As the saying goes:
Do not judge a fish by its ability to climb a tree.
Comparing a library with a cloud solution is possible and can be done in
an honest way with the correct criteria.

10. Information about the evaluated products

11. Supported Browsers
It’s good to know what browsers a test automation tool can support.
Certain tools are not able to support certain browsers.
This is mostly due to technical limitations of those products.

12. For example, you cannot execute automated tests with Cypress on
Safari, Internet Explorer 11 and mobile browsers.
Another example is Ghost Inspector, it can only work on Chrome and
Firefox, since it uses the CasperJS library.

13. Browsers in the cloud included by default
Certain tools offer you a cross-browser cloud to execute your tests on.
While other tools offer you the option to connect to an external cloud grid.

14. For example, Mabl allows you to execute your tests on Safari only if you
connect your own external cross-browser cloud grid, which is not
included in the price.
Another example is Endtest, which offers all the relevant browsers in
their own cross-browser cloud.

15. Complex use cases
Any solution can handle a POC where you create a basic test for a Login page.
The real challenge tends to come when you hit complex use cases and require
certain functionalities.

16. For example, it’s a well-known fact that Cypress does not support
iframes and multiple browser tabs.
Most modern websites use iframes nowadays, just think of the payment
pages, where the credit card inputs are placed in iframes.
Most modern websites also have workflows where you open a page in a
new browser tab, such as when you click on the Social Media links from
the footer.

17. Flexibility
Since a part of the listed products are codeless, it’s important to check if
they offer the same flexibility as writing your own code.

18. Advanced Features
In 2020, we should expect products to have smart functionalities to help us be
more efficient.
Computer Vision = the ability to perform visual testing
Self-Healing Tests = the tests automatically stay in sync with the application
Video Recordings = you get a video recording with the test execution
Scheduler = the option to schedule a test execution without using other tools
Email Testing = check an email in a test, similar to a disposable email service

19. Advanced Features

20. Other relevant aspects

21. Transparency
The value for Transparency is calculated based on several factors.
For example, Cypress got a low score because they have claims on their
homepage that their solution is completely free and open source, but you later
find out that you need to pay for the Cypress Dashboard service if you want
parallel test executions and integrations with Slack and Jira.
They even went through the trouble of hiding their Pricing page, which doesn’t
have a link anywhere on the homepage.

22. Transparency
Selenium got a high score for Transparency, since it claims to be
completely free and open source and these claims are actually true.
Mabl got a low score for Transparency because they do not admit that
their product is actually using the Selenium library and for not displaying
the prices on their Pricing page.
Endtest and Ghost Inspector got high scores for Transparency for
displaying the detailed prices on their Pricing pages.

23. Data Portability
Endtest and Ghost Inspector are the only cloud solutions that got a high
score for Data Portability, this is due to the fact that you can easily
export your tests in different universal formats.

24. Estimated ROI
The Estimated ROI is calculated by taking into consideration the cost of the
solution and the cost for compensating for what the solution lacks.
For example, Endtest has a High ROI score because it offers all the relevant
functionalities and has a price which is significantly lower than its direct
competitors.
Selenium has a Medium ROI score because it’s completely free and you don’t
need to use other solutions to compensate for the missing functionalities. But it
does not have a High ROI because it requires a lot of man-hours.

25. Estimated ROI
Cypress has a Low ROI score because it lacks an important number of
functionalities, which means that a team would need to use other tools in
parallel.
Functionize, Testim and Mabl have Low ROI scores because their solutions
are considered to be overpriced.