With the advent of the Agile Manifesto and the “maturation” of DevOps over the past few years, it seems that the promise of continuous deployments at a speed faster than light is almost fulfilled. Everywhere we look, development and ops team brag about how many times they ship a week, a day, or even an hour.
There isn’t less testing happening in agile or devops, the phase has simply been broke down, sped up, and spread out.
To add to the innate problem of speed in the software delivery cycle, there are a few speed bumps unique to testing web applications.
As our product matures, we’ll have an increasing number of test cases to run in regression before each deployment. This can add exponential amounts of testing – and test execution time – in just a few months. Precious time most products do not have.
Also, as our product gains in popularity, we’ll have to support an increasing amount of browsers and devices. Customers now want to use their personal iPads to access your web portal that just 5 years ago was only accessed by the latest version of IE on…yep, you guessed it. Windows XP Service Pack 3. This again ads massive time to the time needed to sufficiently test across needed browsers.
LEAD: There are only a few ways to radically change your testing time to keep up with modern development.
There are only a few ways to radically change your testing time to keep up with modern development.
You can just test less. This results in more bugs, and we’d not recommend this strategy.
You can also increase the amount of testers on your team to keep up with demand.
You can diversify your testing suite with a mix of unit, API, and UI tests.
And run tests in parallel, executing tests in continuous batches of 10, 20, 50 tests at one time.
Let’s star high level with the software development lifecycle. While you all are probably very familiar with this let’s talk about some points.
Requirements. Where it all begins. What do we want to make, what will we make, and how should whatever we make really behave. Need to have a solid foundation for success… you need to know what your requirements are and what you want them to accomplish. No confusion. It’s kind of like taking that moment to prepare for your essay before you write it. For Tests you can have multiple types of tests. Test Definition Test set What environments you want to run the test on-operating systems, browsers, resolutions, etc. Defects—it’s inevitable. Your tests are going to find some bugs and there will be some problem. But if you have a defect you want to be able to log it properly and tie it to the correct test and requirement. Releases—all of these things are encapsulated in a release. What are you pushing out for the next major or minor release/build.
There is some feedback you get only from a UI layer test. So while you should minimize the efforts of UI testing, you still need to have these tests to complete end-to-end testing workflows. In this case, the test starts at the browser level: Mozilla, chrome, firefox whatever you’re using. It then touchses on HTML5 or Angular JS depending on what you're using. It then touches on network level testing and services/API/Database test.
Mike Cohn’s Test Automation Pyramid is usually the go-to guide for deciding on how many tests should be automated.
(After discussing this in much detail as well as the recent debate)
The best way to incorporate a myriad of testing practices into your development cycle is to find and implement the right tools. Utilizing source control management tools (SCMS) such as Git or Subversion, CI tools like Jenkins, or defect management tools such as JIRA or Bugzilla, can speed up your processes as they integrate with a wide variety of automated test tools and provide more poignant feedback on which part of a test is failing.
In this image we are showing the evolution of the entire development pipeline.—transforming from manual testing to continuous testing. Even with the benefits a diversified testing process can provide, test automation and easier to debug pieces, there is still a cap on time to test and time to release. To reach the final two phases on the right, atomic testing and continuous testing, teams will have to run their tests in parallel. In most cases, this will require teams to move their testing to the cloud for cost and speed reasons.
Let’s take an example…
Let’s imagine a real world example: in release 3 of your product, you have 8 hours of sequential regression testing to perform before the team feels confident to deploy. By release 5, this may be twice the amount of hours you’ll need to run your tests and as a bonus, your product is getting popular and is being used by more users on an increasing number of different devices. Before you were testing for only Chrome and FireFox, but now you see that you need Android and iOS devices, Safari and multiple versions of Internet Explorer. So you have 16 hours of tests and 10 different devices or browser to cover.
This would take us 160 hours for complete test coverage before our deployment. With parallel testing environments, we can run our 16 hours of tests on 10 different devices at the same time, saving us 146 hours of testing time.
Time is finite. So we need to maximize the time we have. Parallel testing allows you to test faster with a quicker turn around in deployments. No developer wants to spend more time testing their product than they did developing it.
How can parallel testing be done? By setting up multiple VMs and other infrastructure devices or by using a cloud test service.
Cross Browser Tests Regression tests--regression testing is essential between deployments. The process is vital to ensure that a piece of software is still functional after a new update has been made prior to the next shipment. A challenge with running regression tests is the limited amount of time often provided and the growing nature of your test suite as your application matures. Developers looking to push their product out faster may give very little time between when they’ve finished their build process and the production date to conduct any testing. Running through the regression suite quickly with concurrent tests will get build engineers any necessary feedback faster and will keep your DevOps teams happy. Unit Testing: unit tests are a great choice to run in parallel because they are naturally small since they test very particular or niche functions of an application, but can number in the thousands to tens of thousands. Smoke tests or build verification testing: is the final testing practice teams should consider implementing in parallel. Since smoke testing focuses on ensuring that only the most important functions of an application work as expected, it is typically used to decide whether or not to proceed with further testing. A smoke test that passes is an indicator to go ahead with more testing. If it fails, teams will stop testing and ask for a new build with the initial required fixes. Smoke testing is usually done to ensure that the minimal viable amount of testing is done with any quick fixes going into production immediately. Smoke tests can be conducted manually or through automation, but parallel testing will enable the process to go faster. Testing the few business critical processes, such as log-ins and checking out,. To get the product out the door in under five minutes can bring in revenue and make customers happy. This ensures enough time to test as thoroughly as they would like in between deployments—parallel testing accelerates this.
Quick deployments—run as many tests as you want concurrently whether it’s 2, 10, 50, or 100. Faster feedback—the faster you run your tests, the faster the feedback. In older development models and testing practices it could take weeks to receive feedback once a developer built a feature and sent it across to QA. By that time, developers have moved on and may not remember what the function that failed was. Without writing a comment for every piece of code, this can be a real challenge to work with. “shifting left” “continuous testing” Cross Browser Testing—this is drastically more achievable with parallel testing. Typically, expanding coverage to incorporate all necessary environments can become very challenging and time consuming. With parallel testing you can test against multiple browsers and browser types all in a shorter timeframe than if you were running tests sequentially. Better Test Coverage—enabling teams to test more in less time inherently means better test coverage. Environments and platforms Saves valuable time
Accelerating Your Test Execution Pipeline
Accelerating Your Test
Who Am I?
• SmartBear Software
• Automated UI functional testing tools & test
• Stay in Touch!
• Went to Dartmouth: AB in Engineering, BE in
Biomedical Engineering, MEM with a healthcare focus
• What do I love to do?
• Run, dance, play board games (Settlers of Catan
We provide tools for development, testing, and operations teams
to create great software, faster than ever.
AccelerateSDLCWorkflows | ImproveQualityatEveryStage | RealizeRapidTime-to-Value
• HQ in Boston, MA, USA, with 7 offices globally
• Founded in 2009
• Open Source Innovator (Swagger & SoapUI)
Create Great Software, Without Tradeoffs
Perform Code &
Design, Develop, &
DEV TEST OPS
Create Automated UI
(Web, Desktop, Mobile)
Run Tests On Real
Devices in the Cloud
Create Web Load Tests
Create Automated API
(REST, SOAP, and more)
SoapUI Pro Script
Virtualize API &
Create API Load Tests
Monitor Web & API Performance,
Availability, & Functional Correctness
Manage Manual & Automated Tests
Integrations …100 +
| SB Test
What’s Going on in the
There are bottlenecks in today’s development processes.
• Teams today are constantly feeling pressure to deliver software faster, without compromising quality
• There is only a certain point as to how scalable automation can be
• They are very time consuming and costly
The promise of the new software delivery cycle
Design Build Test Implement
Week1 Week2 Week3 Week4
Time Consuming Nature Of Web Testing
More Features = More Testing
Age Of Product
Popularity of Product
Hire More Testers
Diversify Your Testing
Run Tests In Parallel
| SB Test
Now… How Can We Go Faster?
1. Test Frameworks
3. Parallel Testing
The Basics of a Test Framework
Requirements Tests Defects
What do we
make and how
Make sure it
stated in the
Definition Sets Environments
do not equal
What is a Test Framework?
A Test Framework:
Links tests to other SDLC items
Is NOT a Test Automation Framework but often contains one
Allows for rapid creation of tests from reusable components
Separates data from logic (REUSABILITY)
Provides a standardized test “language” and reporting structure for an
application under test
Elements of a Test Framework
• Library: A repository of all your decomposed scripts, separated into their components
• Test Data Sources: A repository of all data sources
• Helper Functions: A repository of all decomposed test scripts, automated or manual,
that are inputs or checks
• Test Environments: A list of all covered testing environments, broken out by type (OS,
• Modules: The combination of library items with any helper functions and test data
• Structure / Hierarchies: The “folder” structure of modules
Level 2: Figure Out What Tests
Should be Automated
There are many types of testing that need to be done…
Browser: Chrome HTML5, Angular JS Network Service/API/Database
Automated Testing Pyramid Approach
A Little Manual v Automated Math
# of Test Cases
# of Browsers Supported
Total Test Cases
Avg Test Run Time
Total Test Time 83 Hrs
With 2 QA engineers,
That is 1 week of testing
# of Test Cases
# of Browsers Supported
Total Test Cases
Avg Test Run Time
Total Test Time 666 Hrs
With 5 manual testers, that is
3.5 weeks of testing
Decide on What to Automate
• Environment Setup/Teardown
• Varyingdatainputs inarepetitive process
• Exposingbackend data(APIs,DBtable,etc.)
• Repetitive/boring tasksthat arepronetoinattention errors
• Taskswithhighreusevalue acrossmanyworkflows
• Capturing Results
Speeding Up Your Pipeline
Time To Test Behind Releases
Manual Testing Record & Replay Unit Testing POM Atomic Testing ContinuousTesting
But my Dev team says I have
days to test, not weeks…
Let’s Go Faster!
Test 2Test 2Test 2
Test 3 Test 3 Test 3
Running tests sequentially, we were able
to run our tests in 1 week
Test 1Test 1 Test 1
With 20 Parallel executions,
we can run our entire test suite
in only 4 hours
Types of test to run in parallel
application. Runmoretests,against morebrowserconfigurations by
According tothetestingpyramid,Unittestsshould beyourmost
abundant testtypeinyourentiretestingsuite.Because ofthis,running
14,000 unittestsinunderanhourisreallyonlypossible withamassive
parallel testinginfrastructure investment.
pushahotfix?Onlywaytodothatistoruntheminparallel, allowing you
Because deployments arehappening atsucharapidpace,regression
makingsurethefunctionality ofthenewbuild, matchesthatofthelast
stablebuild. Running thesetestsinparallel allows moretobetested.
Cross Browser Testing
Most effectivetests tosee ROI fromfor paralleltesting
Benefits of Parallel Testing
1. Quick Deployments
2. Faster Feedback
3. Cross Browser Testing
4. Better Test coverage
5. Saves Valuable Time