Mykola Kovsh presented on using JMeter for functional API automation testing at his company. He discussed their problem of not having enough time for full regression testing of a mobile betting application. JMeter was chosen as a solution because it is free, supports parallel test runs, CI integration and comprehensive reports. Key aspects covered included creating modular and data-driven test scripts in JMeter, continuous integration using ANT and Bamboo, and improvements made such as separating test plans and customizing reports. Automating over 200 test cases with JMeter helped find bugs, shorten regression testing time and improve overall application quality.

1. Functional API automation
with JMeter (real case)
by Mykola Kovsh

2. Speaker info
Hi, I’m Mykola Kovsh
Performance Test Engineer
Performance Testing Department at
Testing Center of Excellence,
Ciklum, Ukraine
• 4 years in IT
• Performance / Automation / Manual testing
www.linkedin.com/in/mykolakovsh/

3. Plan
1. Testing Product Description
2. Our Problem
3. JMeter as a Solution
4. Continuous Integration with JMeter. Reporting
5. Demo
6. Our Results
7. Conclusions
8. Useful Links
Questions

4. Testing Product Description
Client: Mobile / Facebook web sport betting application
Server: Amazon cloud-based application, REST API based
Technology stack: Java, Cassandra DB, Unity 3D
Product is in active development for near 3 years
A lot of parsing and integration logic with 3rd-party data providers
Frequent releases driven by business needs (startup)

5. Our Problem
 We should run full regression REST API functional tests each
release
 Manual runs took 4-5 days for 2 QAs what did not satisfy business
 No QA automation specialist on the project, but we use JMeter for
performance testing and have expertise in it
 A lot of GUI/usability testing in addition to functional
 To sum up: often we did not have time for full functional regression
testing before release what resulted in bugs and frequent hotfixes
on Production

6. Solution 1: Postman
Good as supplementary tool for manual testing
CI integration
Reports converting to HTML file
Team work option but NOT FREE
In case of tricky parametrization and custom
logic, some tool’s issues could influence auto
tests runs
JavaScript basic knowledge is required
Hard to maintain when a lot of test-cases
No ability of parallel runs
Reports are not very comprehensive

7. Solution 2: JMeter. Why for API?
Free tool with large community and support
GUI based - fast learning
Customization - through JMeter API and using scripting languages
Ability to reuse functional tests for load purposes
Ability to reuse GUI modules in different tests
Assertions - allow you to "test" whether your application returns the results you expect it to
Parallel test runs
CI integration
Comprehensive HTML reports
Basic knowledge of some scripting language and regular expressions is required
Hard to maintain on large projects
Version control: only JMX files could be tracked

8. JMeter: Functional Tests Creation
1. Thread Group (user) -> Test set/Test suite
2. Logic Controllers:
a) Transaction Controller -> Test suite/Test case
b) Loop/While/ForEach Controller
c) If Controller etc.
3. Samplers -> Requests (HTTP(s), TCP, SOAP, etc.)
4. Assertions -> Test your response/Customize/Parametrize
5. Timers
6. Configuration elements
1
2
3
4
5
6

9. JMeter: Customize/Parametrize/Test
Assertion types:
JSR223 Assertion
Response Assertion
Duration Assertion
Size Assertion
XML Assertion
BeanShell Assertion
MD5Hex Assertion
HTML Assertion
XPath Assertion
XML Schema Assertion
Compare Assertion, etc.

10. Reusable/Modular Scripts Approach
To make functional testing scripts
easy maintainable and reusable
use next JMeter elements:
Test Fragment
Simple / Transaction Controller
Module Controller
Parameterized Controller
Include Controller

11. Data + Keyword Driven Approach
To separate test scripts from test
cases design use next JMeter
elements:
Test Fragment
Switch Controller
Simple / Transaction Controller
Module Controller
ForEach Controller
CSV Data Set Config

12. JMeter: Continuous Integration
Process and tools:
1. JMeter: functional testing scripts
2. ANT build tool: converts results from JTL output files into HTML
reports
- ANT parses JTL logs using XSLT stylesheet file
3. Bamboo CI tool
Development Source Control Build Unit/Integration Testing Functional Testing Report

13. JMeter CI: Approaches
ANT standalone – all steps are
executed in ANT build.xml file,
using 1 Bamboo Task:
• Run JMeter tests
• Convert JTL logs to HTML
report
• Verify test results and pass/fail
build
PowerShell + ANT combination
– 3 Bamboo tasks:
• Task 1 Script: run JMeter
tests
• Task 2 ANT: convert JTL log
to HTML report
• Task 3 Script: verify results
and pass/fail build

14. CI 2nd Approach: Project Structure

15. Bamboo Task 1: Tests Running
del Jmeter_ci_with_antresultstest_report.html
del Jmeter_ci_with_antresultsresultlog.jtl
${bamboo.JMETER_HOME}binjmeter.bat -n -t Jmeter_ci_with_antJmeter_test_plan.jmx -l
Jmeter_ci_with_antresultsresultlog.jtl -q Jmeter_ci_with_antci_jmeter.properties

16. Bamboo Task 2: HTML Reporting

17. Bamboo Task 2: ANT Build File
After tests executed, generate
HTML report using build_report.xml
file (default name build.xml)
We have 3 <target> here:
• report - log process
• xslt-report – convert using
XSLT file
• copy_report – move reports to
separate folder to keep history

18. HTML: XSLT Stylesheet File Update
Our Improvements:
• Statistics by test suites (threads) and request
• Expand/Collapse test suits option
• Show only failed transactions filter
before after
*Origin: http://shanhe.me/node/18/314
*Updated XSLT:
https://drive.google.com/file/d/0B4SCa_HZT_OOQnoyQjc1TGw0a0E/view?usp=sharing

19. HTML Reporting: Investigate Issue

20. HTML Reporting: Limitation
In case different Thread Groups run under the same Test Plan, they are not separated in report
To deal with it, we suggest:
1) Run Thread Groups consequently. It can take some time, if there are a lot of cases
2) or Separate Test Plans (JMX files) by some logic and run them simultaneously

21. Bamboo Task 3: Verify results

23. Recommendations
1) Apply decomposition for faster tests runs:
- One functional area - One separate Test Plan
- Decompose tests runs by jobs in CI tool (Bamboo, Jenkins)
2) Run smoke functional test and, if success, make it trigger full coverage testing run
3) Use Modular approach to create maintainable/reusable scripts
4) Team work: use Include Controllers to run scripts created by each member
5) Use JSR223 Assertion element to customize/parametrize/test your custom logic in flexible way
6) Use User Defined Variables element to create and fast switch between Test/Stage/Prod hosts
7) Share your scripts with developers to debug current and new features
8) Use JMeter scripts for Load testing, if needed

24. Disadvantages
 Inappropriate for large projects/teams
 Hard to maintain scripts, if there are lot of them
 Scripts version control is absent
 Not very convenient for QA team use (if more than 1-2 QA specialists)
 You still need time to learn some scripting language (and regexp) to
create comprehensive scripts with JMeter

25. Our Results
For 5 months ~200 (30%) of API functional high priority test cases were automated
~100 regression and ~80 new features bugs found
Functional API regression testing period shortened from 5 to 2 days
QA team has more time to focus on GUI and cross-platform testing
Overall quality of the application was improved and more stable builds released
to Prod

26. Conclusions
Involve automation QA engineer(s) to implement automation testing on
the project within high standards (Python/Java/etc. frameworks)
IF Your team/project is relatively small (near 5-8 members) and limited
time for full regression runs, to improve product quality you can:
• use JMeter automation testing as complement to manual testing
• automate high priority API test cases to verify main product
functionality - smoke and short acceptance testing
• automate new features testing
• integrate JMeter API testing into product CI pipeline
ELSE

27. Useful Links
• Functional Testing with JMeter: https://www.packtpub.com/books/content/functional-testing-jmeter
• JMeter как относительно удобное и практичное средство для тестирования API:
https://habrahabr.ru/post/243097/
• Test cloud-based applications with Apache JMeter: https://www.ibm.com/developerworks/cloud/library/cl-jmeter-
restful/#listing3
• JMeter – Continuous Performance Testing – JMeter + ANT + Jenkins Integration – Part 1 and 2:
http://www.testautomationguru.com/jmeter-continuous-performance-testing-part1/
• JMeter – Creating Modular / Reusable Test Scripts: http://www.testautomationguru.com/jmeter-modularizing-
test-scripts/
• JMeter – How to create a Data + Keyword Driven Framework for Performance Testing:
http://www.testautomationguru.com/jmeter-how-to-create-a-data-keyword-driven-framework-for-performance-
testing/
• How to Use JMeter Assertions in Three Easy Steps: https://www.blazemeter.com/blog/how-use-jmeter-
assertions-3-easy-steps
• A collection of resources covering different aspects of JMeter usage: https://github.com/aliesbelik/awesome-
jmeter

28. Questions
5 minutes.
You can also ask questions for me in the lounge zone