The goal of this comparison was to check if there are JMeter based cloud services that would work in the context of our company.

1. Bespoke Software Specialists
Comparison – as of 11.2016
JMeter Performance testing cloud
services

2. Introduction
The goal of this comparison was to check if there are JMeter based cloud services that would work in
the context of our company, e.g. we have JMeter skills, we have our way of structuring scripts, we use
in-house developed plugins etc.
In the comparison we only considered services that utilize JMeter, hence services that are only
loading set of URLs are eliminated from the comparison (e.g. LoadImpact, Loader)

3. Comparison criteria
 Pricing vs number of simulated threads and number of tests to run
 Correct execution of JMeter script
 Extras (additional functionalities) - extra features that simplify service usage
 CI integration / API available
 Kerberos authentication possible - possibility of uploading JMeter’s Kerberos configuration files
 Custom plugin usage - possibility of uploading a custom .jar library
 Selection of JMeter versions - possibility of selecting a JMeter version for test execution
 Useful test analysis features
 Results comparison / trends
 Documentation

4. Area [0 – 3 points] Blazemeter Flood IO RedLine13 OctoPerf CloudLoadIO VSTS
Pricing -> typical plan
99$/month,
1000 threads,
200 tests/year
99$/month,
unlimited # of
threads, 15
node
hours/month
50$/month +
AWS costs,
unlimited # of
threads, tests
and test duration
(because user is
paying the AWS
costs)
300
Euro/quarter,
1000 threads,
15 tests/q, 1
hour test
duration
75 $/month,
1500 threads,
20
tests/month,
1h max test
duration
Azure pricing,
per minute of
each used
thread
Pricing 2 2 3 1 2 3
Additional functionalities 3 1 2 2 0 0
CI integration API 3 3 2 1 0 2
Kerberos authentication 1 1 0 0 0 0
Custom plugins usage 1 1 1 0 0 1
Selection of JMeter versions
available 3 0 1 0 0 0
Presenting results, analysis
capabilities 3 1 1 2 3 1
Results comparison across many
runs 2 0 0 0 1 1
Documentation quality 2 1 0 2 0 1
SUMMARY: 20 10 10 8 6 9

5. Blazemeter
 https://www.blazemeter.com/
 Pros & cons
+ Looks like the most mature JMeter cloud
performance testing service
+ TeamCity integration
+ Extras (e.g. simulation of network bandwidth,
automated splitting of csv files)
+ Very good documentation
+ Possibility to inject custom libraries
+ Many JMeter/Java versions available
+ Results comparison (trends) up to 5 tests
+ Sharable (on request) reports
+ Ready to be printed executive summary report
- Private IPs available only in ‘Pro’ license

6. Blazemeter

7. Flood IO
 https://www.flood.io/
 Pros & cons
+ Executes even advanced JMeter script as is
+ Available API
+ Injection of custom libraries
+ Sharable reports
+ Nice user interface
+ Extras (e.g. JMeter script validation)
- Glitches in functioning of the service (e.g.
problems with signing up for the service – bug
in the form, accessing the account)
- Analysis part of the service seems to be quite
poor in comparison with other services
- No test comparison feature

8. Flood IO

9. RedLine13
 https://www.redline13.com/blog/
 Pros & cons
+ JMeter plugins package already available for
use
+ Injection of custom libraries
+ Extras (e.g. splitting csv data across multiple
nodes, advanced pre-test configuration)
+ API integration
+ Decent test analysis features
- Need to own an AWS account, test are ran on
that account
- Payment for usage of AWS instances is an
additional cost
- Some glitches in the test set-up that are
confusing for the end user
- Redirection http response codes (e.g. 304)
are treated as errors, which leads to having
some crazy statistics from the test runs (see
below)
- Poor documentation

10. RedLine13

11. OctoPerf
 https://octoperf.com/
 Pros & cons
+ some extras (e.g. simulation of network
bandwidth, script validation)
+ API integration
+ Advanced hardware monitoring features built-
in the service
+ Very neat user interface
+ Good documentation
+ Good test analysis features
- JMeter script is not ran as is, it is abstracted
to OctoPerf objects
- Abstraction is provided only for some of the
JMeter components (most common ones)
which is a BIG restriction
- Seem to be usable only for basic HTTP
JMeter tests
- No Kerberos authentication support

12. OctoPerf

13. CloudLoadIO
 http://www.cloudload.io/
 Pros & cons
+ Very good set of results summaries and various
graphs – really helpful during the analysis
+ Shareable link of monitoring a currently
executed test and results of previous tests
+ Simple test comparison mode – possibility
of creating result trends
- No documentation
- No information about API availability or CI
integration
- No information on possibility of using 3rd party
libraries. When a custom .jar library is used,
test just hangs for hours and hours without a
clear error message.
- No information on either JMeter or Java
version

14. CloudLoadIO

15. Visual Studio Team Services
 https://www.visualstudio.com/docs/test/performance-testing/getting-started/get-started-jmeter-test
 Pros & cons
+ executes JMeter script as is through Team
Services on Azure infrastructure
+ can inject custom libraries
+ basic comparison of 2 runs is possible
- Only basic analysis possible using TS
features
- No selection of JMeter or Java versions
available

16. Visual Studio
Team Services

17. Thank you