In the past two decades, the software testing industry has been flooded with open source frameworks, including a few at the top such as Selenium that has popularized its adoption. Gartner has recognized that QA teams are increasingly using open source tools and frameworks, and even created an Open source-based testing acceleration use case to highlight the importance of the tools.
QA managers are increasingly asking themselves if they should be leveraging open source tools and validating commercial tools against free, community-based open source options. Although the technical functionality may address the same user needs, it is important to understand the differences between open source and commercial tooling and the advantages and use cases where both can be leveraged to optimize an organization’s testing team’s talent, test approach, and tooling.
1. Making Sense of Selenium:
When to Use It and When Not To
SmartBear Test Team
2. Top Learning Objectives
How to maximize value from open source testing
frameworks
Key differences between open source and commercial
tools
Live sample test scenarios run on both Selenium and
TestLeft
6. Benefits of Selenium
Seleniumisanopen source automation
tool thatcan be downloaded for free and
iscommunity-based.
Free, Open Source Tooling
Multi Language & Framework
Multi Platform Support
Reusability & Integrations
Seleniumcan operate andsupport across
multipleoperating systemsand browsers.
Seleniumsupports a range of languages
includingJava,Perl, Python,C#, Ruby,
Groovy, Java Script.
Reuse Seleniumscripts with other testing
tools to ensure maximumtest coverage;
although,requires code to beinitially
well-crafted
7. Drawbacks of Selenium
There isnobuilt-in classesormethods orobject
repositorythatallowsyoutoquicklyandeasily
buildstabletests.
Requirescarefuldesigning ofwell-craftedtests
andanarchitectural background planned ahead
oftimetotrulyscalewithsuccess.
Notestlogsorsnapshots frompriortestrunsto
quicklydebug issuesorshareresults.
Additional costsoccurfrom3rdpartyintegrations
forSeleniumextendibility, including parallel
execution, debugging, reportingandtraining
No Object Repository
Difficult to Scale
Hidden Costs
Reporting and Debugging
Noprojectandtesttemplates that makeiteasier
foryoutocreateandrunfunctionaltestswith
themostpopularunittesting framework
If you get stuck there is no one to help you,
leading to time wasted on troubleshooting or
having to start from scratch.
No Out-of-Box Frameworks No Support
9. When Speed and Quality Matters
Test Coverage Speed Extendibility
10. The promise of the new software delivery cycle
WaterfallAgile
Design Build Test Implement
DevOps
Week1 Week2 Week3 Week4
11. Automated Test Creation
Creation Analysis Maintenance DeploySetup
Manual
Automated
UI Spy
Application and Page
Object Model
Reports
Snapshots
Scalable
Less brittle
CI the way
you want it
3x3x 6xTime
Savings
$ Savings Example Savings:
600K to maintain your application / 6 = $100K Half a Million in Savings
13. Are you interested in learning more
about a faster Selenium alternative?
14. How does TestLeft help with Speed and Quality?
Create Tests Faster in Your IDE
with Our UI Object Spy
Capture and Share In-Depth
Test Logs for Report Analysis
Test Early, Test Often with
Your DevOps Ecosystem
15. Easy and Quick Two-Click Test Creation
Generate application models for
web or desktop applications in two
clicks. Simply drag and drop objects
to access built-in properties,
methods, and classes in a single file.
Create Tests Faster in Your
IDE with Our UI Object Spy
16. Better Reporting for Faster Debugging
Analyze test results for faster
debugging and share results with
developers and other stakeholders.
Breakdown results by pass/fail,
object, priority, and time, to identify
issues quickly
Capture and Share In-Depth
Test Logs for Report Analysis
17. One Environment and No Back and Forth
Test Early, Test Often with
Your DevOps Ecosystem
TestLeft seamlessly embeds into
your development environment and
DevOps ecosystem. With hundreds
of integrations, TestLeft fits perfectly
with continuous integration process.
19. SmartBear Solutions & Selenium
19
CI Server
Builds App
Unit, API, Functional Testing
CI Server
Triggers Tests &
Sends Them To
CBT To Run
Bugs & defects
go back to Dev
Good code goes to
production to be
monitored
CI/CD Server
(Jenkins, Bamboo)
Build Phase
20. Are you interested in learning more
about how you can maximize your Selenium
scripts with commercial tools?
Earlier this year, Gartner released it’s Critical Capabilities report for Software Test Automation. Compared to prior years, open source-based testing acceleration was actually a new use case, which is Gartner’s recognition that QA teams are increasingly using open source tools and frameworks. It has become evident that no industry is immune to market disruption, including the software development and testing space.
QA managers are increasingly asking themselves if they should be leveraging open source tools and validating commercial tools against free, community-based open source options. Although the technical functionality may address the same user needs, it is important to understand the differences between open source and commercial tooling and the advantages and use cases where one fits your organizations’ need better or both can be leveraged to optimize an organization’s testing team’s talent, test approach, and tooling.
Open Source Benefits
How to Maximize Value from Open Source Tools (Open source vs commercial tools; https://thenewstack.io/maximizing-value-from-open-source-testing-frameworks/)
Open Source vs. Commercial Tools — A False Dichotomy
Benefits of Commercial Tools
TL vs Selenium
TestLeft Value Prop (you to faster test creation, scalable maintenance, reports and analytics (whiteboarding session slide on man vs. auto)
If you are using SE, we have so many tools that can help you and integrates with Se
Developers write test scripts that interact directly with a web browser.
There are clear differences between open source and commercial tools. Not every company, industry, and QA team is the same and how each defines, executes, and maintains tests vary. Aside from a tool’s functionality meeting your software needs, the question comes down to a few key business differentiators:
Options:
Yes, I am a Selenium Expert
Yes, I use Selenium frequently
Yes, but I am just getting started
No, but I am thinking about getting started
Supports languages: Selenium supports a range of languages including Java, Perl, Python, C#, Ruby Groovy, Java Script
Supports Operating systems: Selenium can operate and support across multiple operating systems like Windows, Mac, Linux, UNIX, etc.
Support across browsers: Selenium provides across multiple browsers—Internet Explorer, Chrome, Firefox, Opera, Safari, etc.
Selenium Test Automation Framework uses scripts that can be tested directly across multiple browsers. Concurrently, it is possible to execute multiple tests with Selenium, as it covers almost all aspects of functional testing by implementing add-on tools that broaden the scope of testing.
No Support: Although support for open source tools can be addressed with third party resources, it will usually come at an extra cost or effort. Whereas, most commercial tools include access to 24/7 global support with your purchase. While there may be many online resources for open source frameworks, it is hard to find the right solution or training among all the noise found across disparate websites, user communities, and learning platforms. For example, if you are looking to learn more about Selenium 101, there are training online learning platforms like Coursera to guides provided by commercial vendors that integrate with Selenium. In addition to on-demand resources, commercial tools are increasingly providing access to dedicated customer success managers for on-going questions, transparency, and collaboration to impact product improvements.
No object repository
No out-of-the-box frameworks
Difficult to scale
Hidden costs
No support
No Reporting and debugging
Scalability is the hardest aspect to overcome with open source frameworks especially in the automated testing space. With open source tools, many teams have to manually program descriptions for every object property they need to reference in a test case. Commercial tools come equipped with built-in classes and methods for a variety of controls, allowing software teams to perform different checks across a wide variety of applications elements such as buttons, text boxes, dialogs, and pop-ups. Without having to spend time writing custom methods manually, these tools make it faster to create and maintain tests and scale across different physical and virtual machines for distributed and parallel testing needs.
With the advent of the Agile Manifesto and the “maturation” of DevOps over the past few years, it seems that the promise of continuous deployments at a speed faster than light is almost fulfilled. Everywhere we look, development and ops team brag about how many times they ship a week, a day, or even an hour.
We talk about the ability to continuously deploy with fancy CI tools, but most of the time something is entirely absent in the conversation. Testing. Rarely, if ever, does the conversation about test coverage in their CI process come up – or how much of their application is actually under test.
Look at “Testing” in the waterfall model. We used to get an ENTIRE week in a one month deployment cycle. Now, we have just days to test – sometimes just hours.
We’re not doing any less testing, only deploying more. And that’s not going to stop, so how can we keep up.
Open Source Benefits
How to Maximize Value from Open Source Tools (Open source vs commercial tools; https://thenewstack.io/maximizing-value-from-open-source-testing-frameworks/)
Open Source vs. Commercial Tools — A False Dichotomy
Benefits of Commercial Tools
TL vs Selenium
TestLeft Value Prop (you to faster test creation, scalable maintenance, reports and analytics (whiteboarding session slide on man vs. auto)
If you are using SE, we have so many tools that can help you and integrates with Se
Options:
Yes, please send me information
Yes, I would like to setup a demo
Yes, but not at this time
No, I do not wish to learn more
Easy and quick two-click test creation
Pre-packaged architecture with properties and methods for quick use
More refactoring required in Selenium for non-coordinate based properties
No back and forth between Selenium documentation and scripting/developer environment
Better reporting for faster debugging
Detailed log files with screenshots and usage metrics categorized by configuration, priority and time to be easily exported and shared
Offline Use
There is a world where both open source and commercial tools can exist in your ecosystem. As organizations grow so does the complexity of their applications. Given the differences in open source frameworks and commercial tools, having both in your tool stack not only provides the benefits of both worlds but also drives further advantages.
Software teams can diversify their testing methods by collaborating across experience levels from technical users to non-technical business analysts. In addition, open source frameworks enable teams to have lean testing teams, driving more value from the current resources.
To enable teams to maximize the value from their open source assets, companies have built support for open source into its commercial tools. For example, teams can build on top of their open source projects, converting open source scripts like SoapUI to build more advanced functional tests, load tests, or create virtualizations within commercial tools.
We know that some of the audience are already using Selenium and although it may be easier to maintain scalability in commercial tools. To highlight a few, SmartBear’s TestComplete supports Selenium WebDriver tests created with frameworks you’re already using. These include: JUnit, TestNG, NUnit, PyUnit, Ruby, and PHPUnit.
With Smartbear’s Cross Browser testing, teams can instantly run open source tests from Selenium, for example, on over 1,500 real devices and browsers in the cloud.
Testing and Monitoring can work hand in hand to provide comprehensive insight into your application health, performance, availability, and to better grasp the state of the test environment itself. With Alertiste, you can upload Selenium scripts and run them from AlertSite global monitoring locations, measuring the transaction response time.
Options:
Yes, please send me information
Yes, I would like to setup a demo
Yes, but not at this time
No, I do not wish to learn more