- JMeter as a solution for API automation on small projects. - Continuous Integration with JMeter (Bamboo). - Bamboo Demo. - HTML Reporting. - Recommendations.

1. 1
Skills | Knowledge | CollaborationSkills | Knowledge | Collaboration
FunctionalAPIautomationwithJMeter(realcase)
by Mykola Kovsh

2. 2
Skills | Knowledge | Collaboration
Speaker info
Hi, I’m Mykola Kovsh
PerformanceTestEngineer
Performance Testing Department at
Testing Centerof Excellence,
Ciklum,Ukraine
•Performance/ Automation / Manualtesting
www.linkedin.com/in/mykolakovsh/

3. 3
Skills | Knowledge | Collaboration
Content
6 Our Results
Demo5
Testing Product Description1
Our Problem2
Continuous Integration with JMeter (Bamboo). Reporting4
JMeter as a Solution3
Conclusions7
Links and Q&A session

4. 4
Skills | Knowledge | Collaboration
Testing Product Description
Client: Mobile / Facebookweb sport betting application
Server:Amazon cloud-based application, REST API based
Technology stack: Java, Cassandra DB, Unity 3D
Product is in active development for near 3years
A lot of parsing and integration logic with 3rd-party data providers
Frequent releases driven by business needs (startup)

5. 5
Skills | Knowledge | Collaboration
Our Problem
 We should run full regression REST API functional tests each release
 Manual runs took 4-5 days for 2 QAs what did not satisfy business
 No QA automation specialist on the project, but we use JMeter for performance testing and have
expertise in it
 A lot of GUI/usability testing in addition to functional
 To sum up:often we did not have time for full functional regression testing before release what
resulted in bugs and frequent hotfixes on Production

6. 6
Skills | Knowledge | Collaboration
Solution 1: Postman
Goodas supplementarytoolfor manualtesting
CIintegration
Reports convertingtoHTML file
Team workoptionbut NOT FREE
Incase oftrickyparametrizationandcustom logic, some tool’s issues
couldinfluence autotests runs
JavaScriptbasicknowledgeis required
Hardtomaintainwhen alot oftest-cases
Noabilityofparallelruns
Reports arenot very comprehensive

7. 7
Skills | Knowledge | Collaboration
Solution 2: JMeter. Why for API?
Free toolwithlargecommunityandsupport
GUI based- fastlearning
Customization-throughJMeter API andusing scripting languages
Abilitytoreuse functionaltestsforload purposes
Abilitytoreuse GUI modulesin different tests
Assertions- allowyouto"test"whetheryourapplicationreturnstheresults youexpectit to
Paralleltest runs
CI integration
ComprehensiveHTML reports
Basicknowledgeofsomescripting languageandregularexpressionsis required
Hardtomaintainonlargeprojects
Version control:onlyJMX filescould betracked

8. 8
Skills | Knowledge | Collaboration
JMeter: Functional Tests Creation
1. ThreadGroup(user) ->Testset/Testsuite
2. LogicControllers:
a) TransactionController->Testsuite/Testcase
b) Loop/While/ForEachController
c) If Controlleretc.
3. Samplers->Requests(HTTP(s), TCP, SOAP,etc.)
4. Assertions->Testyourresponse/Customize/Parametrize
5. Timers
6. Configurationelements
1
2
3
4
5
6

9. 9
Skills | Knowledge | Collaboration
JMeter: Customize/Parametrize/Test
Assertion types:
JSR223Assertion
Response Assertion
Duration Assertion
SizeAssertion
XML Assertion
BeanShell Assertion
MD5Hex Assertion
HTML Assertion
XPath Assertion
XML Schema Assertion
Compare Assertion, etc.

10. 10
Skills | Knowledge | Collaboration
Reusable/Modular Scripts Approach
To makefunctional testing scripts easy
maintainable and reusableuse next JMeter
elements:
Test Fragment
Simple / Transaction Controller
Module Controller
Parameterized Controller
Include Controller

11. 11
Skills | Knowledge | Collaboration
Data + Keyword Driven Approach
To separatetest scriptsfrom test cases design
use next JMeter elements:
Test Fragment
Switch Controller
Simple / Transaction Controller
Module Controller
ForEach Controller
CSV Data Set Config

12. 12
Skills | Knowledge | Collaboration
JMeter: Continuous Integration
Process and tools:
1. JMeter:functional testing scripts
2. ANTbuild tool: converts results from JTL output files into HTML reports
- ANT parses JTL logs using XSLT stylesheet file
3. Bamboo CI tool
Development SourceControl Build Unit/IntegrationTesting FunctionalTesting Report

13. 13
Skills | Knowledge | Collaboration
What is ANT
Apache Ant (Another Neat Tool) is a general purpose build tool.
Ant is primarily used for building and deploying Java projects but can be used for
every possible repetitive tasks, e.g. generating documentation.
A build tool is used to automate repetitive tasks during some process:
- compiling source code,
- running software tests,
- creating files and documentation for the software deployment, etc.
Popular build tools in the Java space are Maven, Gradle and Apache Ant.

14. 14
Skills | Knowledge | Collaboration
JMeter CI: Approaches
ANTstandalone –all steps are executedin
ANT build.xml file, using 1 Bamboo Task:
• Run JMeter tests
• Convert JTL logs to HTML report
• Verify test results and pass/fail build
PowerShell(console) + ANTcombination
–3 Bamboo tasks:
• Task1Script:run JMeter tests
• Task2ANT: convertJTL log to HTML
report
• Task3Script:verify results and pass/fail
build
1 2

15. 15
Skills | Knowledge | Collaboration
CI 2nd Approach: Project Structure

16. 16
Skills | Knowledge | Collaboration
Bamboo Task 1: Tests Running
del Jmeter_ci_with_antresultstest_report.html
del Jmeter_ci_with_antresultsresultlog.jtl
${bamboo.JMETER_HOME}binjmeter.bat -n-tJmeter_ci_with_antJmeter_test_plan.jmx-lJmeter_ci_with_antresultsresultlog.jtl-
qJmeter_ci_with_antci_jmeter.properties

17. 17
Skills | Knowledge | Collaboration
Bamboo Task 2: HTML Reporting

18. 18
Skills | Knowledge | Collaboration
Bamboo Task 2: ANT Build File
After tests executed, generate HTML report using
build_report.xml file (default name build.xml)
Wehave3 <target> here:
• report - log process
• xslt-report –convertusing XSLTfile
• copy_report –move reports to separate folder
tokeephistory

19. 19
Skills | Knowledge | Collaboration
What is XSLT
Wikipedia: XSLT (Extensible Stylesheet Language
Transformations) is a language for transforming XML
documents into other XML documents,
or other formats such as HTML for web pages, plain text or
XSL Formatting Objects, which may subsequently be
converted to other formats, such as PDF, PostScript and PNG

20. 20
Skills | Knowledge | Collaboration
HTML: XSLT Stylesheet File Update
OurImprovements:
• Statistics by test suites (threads) and request
• Expand/Collapse test suits option
• Show only failed transactions filter
before after
*Origin: http://shanhe.me/node/18/314
*UpdatedXSLT: https://drive.google.com/file/d/0B4SCa_HZT_OOQnoyQjc1TGw0a0E/view?usp=sharing

21. 21
Skills | Knowledge | Collaboration
HTML Reporting: Investigate Issue

22. 22
Skills | Knowledge | Collaboration
HTML Reporting: Limitation
In casedifferentThreadGroupsrununderthe sameTestPlan,theyarenotseparatedin report
Todeal withit, wesuggest:
1) RunThreadGroupsconsequently. Itcan takesometime, iftherearea lotofcases
2) orSeparateTest Plans(JMX files) bysomelogic andrun themsimultaneously

23. 23
Skills | Knowledge | Collaboration
Bamboo Task 3: Verify results

24. 24
Skills | Knowledge | Collaboration

25. 25
Skills | Knowledge | Collaboration
Recommendations
1) Apply decomposition for faster tests runs:
-One functional area -One separate Test Plan
-Decompose tests runsby jobs inCItool (Bamboo, Jenkins)
2) Runsmoke functional test and, if success, make it trigger full coverage testing run
3) Use Modular approach to create maintainable/reusable scripts
4) Team work:use IncludeControllers to run scripts created by each member
5) UseJSR223Assertion element to customize/parametrize/test yourcustom logic in flexible way
6) UseUserDefined Variables element to create and fast switch between Test/Stage/Prod hosts
7) Shareyourscripts with developers to debug current and new features
8) UseJMeterscripts for Load testing, if needed

26. 26
Skills | Knowledge | Collaboration
Disadvantages
 Inappropriate for largeprojects/teams
 Hard to maintain scripts, if there are lot of them
 Scripts version control is absent
 Not veryconvenient for large QA team use(if more than 2-3 QAspecialists)
 You still need time to learnsome scripting language (and regexp) to create comprehensive scripts with
JMeter

27. 27
Skills | Knowledge | Collaboration
Our Results
For 5 months ~200 (30%) of API functional high priority test cases were automated
~100 regression and ~80 new features bugs found
Functional API regression testing period shortened from 5 to 2 days
QA team has more time to focus on GUI and cross-platform testing
Overall quality of the application was improved and more stable builds released to Prod

28. 28
Skills | Knowledge | Collaboration
Conclusions
Involve automation QA engineer(s) to implement automation testing on the project within high
standards (Python/Java/etc. frameworks)
IF Your team/project is relatively small (near 5-8 members) and limited time for full regression runs, to
improve product quality you can:
• use JMeter automation testing as complement to manual testing
• automate high priority API test cases to verify main product functionality -smoke and short
acceptance testing
• automate new features testing
• integrate JMeter API testing into product CI pipeline
ELSE

29. 29
Skills | Knowledge | Collaboration

30. 30
Skills | Knowledge | Collaboration
Useful Links
• Functional Testing with JMeter: https://www.packtpub.com/books/content/functional-testing-jmeter
• JMeter как относительно удобноеи практичное средстводля тестирования API: https://habrahabr.ru/post/243097/
• Test cloud-based applications with Apache JMeter: https://www.ibm.com/developerworks/cloud/library/cl-jmeter-restful/#listing3
• JMeter –Continuous Performance Testing –JMeter +ANT +Jenkins Integration –Part1 and 2: http://www.testautomationguru.com/jmeter-continuous-
performance-testing-part1/
• JMeter –Creating Modular/Reusable Test Scripts: http://www.testautomationguru.com/jmeter-modularizing-test-scripts/
• JMeter –How tocreate a Data+Keyword Driven Framework for Performance Testing: http://www.testautomationguru.com/jmeter-how-to-create-a-data-keyword-
driven-framework-for-performance-testing/
• How toUse JMeter Assertions in Three Easy Steps: https://www.blazemeter.com/blog/how-use-jmeter-assertions-3-easy-steps
• Acollection ofresources covering different aspects of JMeter usage: https://github.com/aliesbelik/awesome-jmeter