2. Introduction
The goal of this comparison was to check if there are JMeter based cloud services that would work in
the context of our company, e.g. we have JMeter skills, we have our way of structuring scripts, we use
in-house developed plugins etc.
In the comparison we only considered services that utilize JMeter, hence services that are only
loading set of URLs are eliminated from the comparison (e.g. LoadImpact, Loader)
3. Comparison criteria
Pricing vs number of simulated threads and number of tests to run
Correct execution of JMeter script
Extras (additional functionalities) - extra features that simplify service usage
CI integration / API available
Kerberos authentication possible - possibility of uploading JMeter’s Kerberos configuration files
Custom plugin usage - possibility of uploading a custom .jar library
Selection of JMeter versions - possibility of selecting a JMeter version for test execution
Useful test analysis features
Results comparison / trends
Documentation
4. Area [0 – 3 points] Blazemeter Flood IO RedLine13 OctoPerf CloudLoadIO VSTS
Pricing -> typical plan
99$/month,
1000 threads,
200 tests/year
99$/month,
unlimited # of
threads, 15
node
hours/month
50$/month +
AWS costs,
unlimited # of
threads, tests
and test duration
(because user is
paying the AWS
costs)
300
Euro/quarter,
1000 threads,
15 tests/q, 1
hour test
duration
75 $/month,
1500 threads,
20
tests/month,
1h max test
duration
Azure pricing,
per minute of
each used
thread
Pricing 2 2 3 1 2 3
Additional functionalities 3 1 2 2 0 0
CI integration API 3 3 2 1 0 2
Kerberos authentication 1 1 0 0 0 0
Custom plugins usage 1 1 1 0 0 1
Selection of JMeter versions
available 3 0 1 0 0 0
Presenting results, analysis
capabilities 3 1 1 2 3 1
Results comparison across many
runs 2 0 0 0 1 1
Documentation quality 2 1 0 2 0 1
SUMMARY: 20 10 10 8 6 9
5. Blazemeter
https://www.blazemeter.com/
Pros & cons
+ Looks like the most mature JMeter cloud
performance testing service
+ TeamCity integration
+ Extras (e.g. simulation of network bandwidth,
automated splitting of csv files)
+ Very good documentation
+ Possibility to inject custom libraries
+ Many JMeter/Java versions available
+ Results comparison (trends) up to 5 tests
+ Sharable (on request) reports
+ Ready to be printed executive summary report
- Private IPs available only in ‘Pro’ license
7. Flood IO
https://www.flood.io/
Pros & cons
+ Executes even advanced JMeter script as is
+ Available API
+ Injection of custom libraries
+ Sharable reports
+ Nice user interface
+ Extras (e.g. JMeter script validation)
- Glitches in functioning of the service (e.g.
problems with signing up for the service – bug
in the form, accessing the account)
- Analysis part of the service seems to be quite
poor in comparison with other services
- No test comparison feature
9. RedLine13
https://www.redline13.com/blog/
Pros & cons
+ JMeter plugins package already available for
use
+ Injection of custom libraries
+ Extras (e.g. splitting csv data across multiple
nodes, advanced pre-test configuration)
+ API integration
+ Decent test analysis features
- Need to own an AWS account, test are ran on
that account
- Payment for usage of AWS instances is an
additional cost
- Some glitches in the test set-up that are
confusing for the end user
- Redirection http response codes (e.g. 304)
are treated as errors, which leads to having
some crazy statistics from the test runs (see
below)
- Poor documentation
11. OctoPerf
https://octoperf.com/
Pros & cons
+ some extras (e.g. simulation of network
bandwidth, script validation)
+ API integration
+ Advanced hardware monitoring features built-
in the service
+ Very neat user interface
+ Good documentation
+ Good test analysis features
- JMeter script is not ran as is, it is abstracted
to OctoPerf objects
- Abstraction is provided only for some of the
JMeter components (most common ones)
which is a BIG restriction
- Seem to be usable only for basic HTTP
JMeter tests
- No Kerberos authentication support
13. CloudLoadIO
http://www.cloudload.io/
Pros & cons
+ Very good set of results summaries and various
graphs – really helpful during the analysis
+ Shareable link of monitoring a currently
executed test and results of previous tests
+ Simple test comparison mode – possibility
of creating result trends
- No documentation
- No information about API availability or CI
integration
- No information on possibility of using 3rd party
libraries. When a custom .jar library is used,
test just hangs for hours and hours without a
clear error message.
- No information on either JMeter or Java
version
15. Visual Studio Team Services
https://www.visualstudio.com/docs/test/performance-testing/getting-started/get-started-jmeter-test
Pros & cons
+ executes JMeter script as is through Team
Services on Azure infrastructure
+ can inject custom libraries
+ basic comparison of 2 runs is possible
- Only basic analysis possible using TS
features
- No selection of JMeter or Java versions
available