API testing with Open Source
Code libraries and Cucumber
Ole Lensmar – SmartBear

22
Short about “Shift-Left”
• Enterprise testing and QA is moving from late to
early phases in the development life-cycle
– QA -> Dev -> Design
• Tests are being expressed as code instead of using
visual tools (by both QA and Dev)
– Re-use of existing Test assets / scripts for different types of
tests
• Code-driven approach not just for functional testing
– Load/performance testing
– Security testing
– Mocking / Virtualization

33
Testing at API layer vs Unit tests
• API-layer testing = integration testing of
implementing classes/components
• Unit-testing frameworks often used to
”orchestrate”
– Junit / TestNG / Failsafe etc.
• Tests are managed together with implementing
code - using traditional SCM tools
• Tests are executed as part of existing builds
– Test running instance of API vs implementation

44
Code driven approach to API Testing
• HTTP clients
– Apache HTTPClient, OkHttp, Super-agent, etc.
• Generated clients
– Swagger Codegen, WSDL2Java, etc.
• Special purpose libraries
– REST-Assured (Java/Groovy)
– Chakram, (JS)
– ReadyApi4j (Java)
• Cucumber / BDD - many platforms/languages

55
Approach for comparison
• Simple scenario with single and chained REST
requests testing the SwaggerHub integration API
• Framework specific highlights
• All available on GitHub:
https://github.com/olensmar/apidays-apitesting
• All run with maven (“mvn test”) except javascript
sample

66
Apache HTTP-Client
• Widely used java client library for HTTP
• Exposes fluent API for building / executing
requests and reading responses
• Lacks built in support for JSON / XML parsing or
assertions
– Need to do this manually using Jackson/DOM/etc.

77
Generated clients
• Use code-generation tools to generate client from
API description
• Write tests using generated client
– Automate client generation as part of build
• Pros
– API changes can break tests
– Isolates from network layer – tests at functional level
• Cons
– Relies on correctness of client-generator
– Isolates from network layer – tests at functional level

88
REST-Assured
• Fluent Java API for testing REST APIs
• Support both JSON and XML content assertions
– Uses standalone JSONPath and Xpath implementations
• Built in support for multiple authentication types, XSRF,
etc.
• Very customizable at the network/http/marshalling layers
• http://rest-assured.io/

99
ReadyAPI4j
• Fluent Java API
– Leverages the SoapUI Testing engine
– REST, SOAP, Assertions, orchestration, etc.
• Internally compiles tests into “Test Recipes”
(json)
– Can be reused for performance/security testing in
SoapUI
• Execution either locally or remotely
– Local execution for ease of use (open source)
– Remote execution for access control, performance,
reporting, etc. (commercial)

1010
Maven/Java project
ReadyApi4j execution model
API Test
/src/test/java
ReadyApi4j API
Local Engine ReadyAPI
Testserver
API(s) under test
use
use
call
API calls

1111
Challenges with code-driven API testing
• Running against different environments
– Temporary environments (Docker can help)
– Actual environments
• Testing either with or without network
– Network is more realistic – more work to automate
• External dependencies may need to be mocked
– 3rd party APIs, databases, etc.
– For both functional and non-functional testing
• Simulate errors, response times, invalid responses, etc.

Cucumber for API testing

1313
Cucumber – “Executable Specifications”
• Heavily associated with Behavior Driven Development
• “Plain language” verbalization of requirements
– POs, Business Analysts, Developers, Testers, etc.
• For example
Given a user has logged in with valid credentials
When a deposit of 100 USD is made
Then the account balance increases by 100 USD
And the user gets a confirmation message

1414
Scenarios/Features written in “Gherkin”
• Given X
• When Y
• Then Z
• More keywords: And, Feature, Scenario,
Background, etc.
• Packaged into “feature files”
• Cucumber reads feature files and executes
corresponding “StepDefs” or “GlueCode”

1515
Automated Build / Test executions
Cucumber “execution pipeline”
Cucumber
Runner
StepDefs
Component /
System under test
1. Parse
2. Invoke
.feature files
Ant/Maven/Gra
dle/CLI/etc [@tag]

1616
Cucumber StepDefs
• Code that is executed by the cucumber
framework for each statement
• Must be implemented manually to support the
desired vocabulary
• Cucumber Java example -> code!

1717
Cucumber “styles”
• Declarative style
– Domain-specific vocabulary for describing the business
requirements
– Appeals to non-technical users
– High overhead in creation and maintenance
• Imperative style
– Generic vocabulary for describing requirements
– Appeals more to technical users
– Lower maintenance – but “Misses the point”
• Either requires implementation of StepDefs

1818
Declarative API Testing Example
Feature: SwaggerHub REST API
Scenario: Default API Listing
When a request to the API listing is made
Then a list of APIs should be returned within 500ms
Scenario: TestServer API Retrieval
Given an owner named smartbear
And an api named ready-api-testserver
And a version named 1.0.0
When a request to the API listing is made
Then an API definition should be returned within 500ms

1919
Imperative API Testing Example
Feature: Petstore API
Scenario: Find pet by status
Given the API running at http://petstore.swagger.io/v2
When a GET request to /pet/findByStatus is made
And the status parameter is test
And the Accepts header is application/json
Then a 200 response is returned within 50ms

2020
Imperative API Testing Example with Swagger
Feature: Petstore API
operationId gives
path and method
operation parameter
type not neededa defined response

2121
Managing Feature files and StepDefs
• Keep feature files together with source code
and/or in separate repository
• Use tagging feature of Cucumber to selectively
(re)run tests
• Use datasets for parameterization
– Inline or external
• Bundle StepDefs in separate library
– Automate execution

2222
ReadyApi4j - Cucumber for APIs
• Out-of-the box generic vocabulary for REST API Testing -
integrated Swagger support
• Extensible – built on Cucumber for Java
– Open-source on GitHub
– https://github.com/readyapi/testserver-cucumber
• Packaged as library or Docker image
– https://hub.docker.com/r/smartbear/cucumber4apis
• Uses ReadyApi4j as execution engine (either local or
remote)

2323
That’s it! – Questions?
• @olensmar
• https://github.com/olensmar/apidays-apitesting
• https://hc.apache.org
• http://rest-assured.io/
• http://dareid.github.io/chakram/
• http://frisbyjs.com/
• https://github.com/SmartBear/readyapi4j
• https://cucumber.io/
• https://github.com/readyapi/testserver-cucumber
• https://hub.docker.com/r/smartbear/cucumber4apis