apidays LIVE Paris - Responding to the New Normal with APIs for Business, People and Society December 8, 9 & 10, 2020 Automation API Testing: when Postman collection are generated from OpenAPI specifications Guillaume Jeannic, IT Architect at CBP Group

1. Direction des Technologies et du Digital
Guillaume JEANNIC
IT Architect
APIDays
New York
July 28th&29th 2020
Automation API Testing:
When Postman collection
are generated from
OpenAPI specifications

2. Direction des Technologies et du Digital
Guillaume JEANNIC
IT Architect
APIDays
Paris
December 10th 2020
Automation API Testing:
When Postman collection
are generated from
OpenAPI specifications

3. Direction des Technologies et du Digital
Submitted Abstract
This talk is about our journey into designing test automation for our new API for document management
services (insurance business).
Taking into consideration our testing population (non technician), our specification process
(OpenAPI3/Swagger design not generated from sources), I chose Postman as a tool to automate testing.
First creating Postman collection by hand, then injecting OpenAPI specifications into JSON schema
validation, I finally built a Node.js application that build Postman collections from JSON test scenarios and
iterations data, and run them using newman library as part of the CI.
3

4. Guillaume Jeannic
IT Architect
at Cbp Group
API for Legacy Industries: Banking,
Insurance, Healthcare, and Retail
July 28 & 29, 2020

5. Direction des Technologies et du Digital
Our group
5
● Cbp is a subsidiary of the CEP Financial Group
○ (bought this year by Bridgepoint for around 1,3 billions €)

6. Direction des Technologies et du Digital
● We’re located near Nantes in France
Our location
6

7. Direction des Technologies et du Digital
Our business
7
● Over 26 years of experience in loan protection insurance and personal pensions
● Developing our business model throughout Europe (Spain, Germany, Italy, Austria, …)
● Main activities are:
○ Commissioned Policy Manager
○ Personalised support for our partner
organisations

8. Direction des Technologies et du Digital
Our operations
8
● Our operational business rests on document processing
● Nearly 100 millions documents received
● More than 100 millions documents sent
● Accross around 10 different channels

9. Direction des Technologies et du Digital
Our IT culture
9
● In house development
● Transforming…
○ ...from AS400 mainframe to Java / Grails
○ ...from skill teams to product teams (2017)
○ ...from on premises to AWS or private cloud (since 2018)
○ ...from synchronized and scheduled delivery to continuous delivery
○ ...from manual testing to test automation

10. Direction des Technologies et du Digital
Our IT project
10
● Our in-house Document Management System is more than 20 years old
● Currently building our new DMS API
and migrating our legacy system using the strangler pattern

11. Direction des Technologies et du Digital
● Belief: our API testing shall be automated!
● Context:
○ our testers aren’t computer scientists
○ understaffed for technical specifications
● Problem: how to make test automation friendly with no budget?
Starting point
11

12. Direction des Technologies et du Digital
● Belief: our API testing shall be automated!
● Context:
○ our testers aren’t computer scientists
○ understaffed for technical specification
● Problem: how to make test automation friendly with no budget?
Starting point
12

13. Direction des Technologies et du Digital
● Good reputation tool with a user friendly GUI
● Used before for API endpoints exploration
● Can be used for API testing
● Can be automated from CLI (newman)
● Basic usage is free
● Don’t have time for market research...
Why Postman?
13

14. Direction des Technologies et du Digital
PM lets you define pre-request scripts...
API Testing with Postman
14
...and test scripts executed after the request

15. Direction des Technologies et du Digital
● 2 key features for testing:
○ Chai assertion language
expect([1, 2]).to.be.an('array').that.does.not.include(3);
○ JSON schema validation
API Testing with Postman
15

16. Direction des Technologies et du Digital
● 2 key features for testing:
○ Chai assertion language
expect([1, 2]).to.be.an('array').that.does.not.include(3);
○ JSON schema validation
API Testing with Postman
16
x Not accessible enough for non technicians
x Don’t have JSON schemas of our endpoints

17. Direction des Technologies et du Digital
Meanwhile tester struggles...
17
?
Swagger generated from code
Myself Dev team Tester

18. Direction des Technologies et du Digital
Meanwhile tester struggles...
18
?
Swagger generated from code
Myself Dev team Tester

19. Direction des Technologies et du Digital
Change of paradigm
19
Myself Dev team Tester
Testing
Swagger generated from specifications
✓ Formal description of API
✓ Better documentation

20. Direction des Technologies et du Digital
#1st idea: Injecting OpenAPI specs into PM collections
20
● Open API => JSON Schemas
● Injection into Postman

21. Direction des Technologies et du Digital
#1st idea: Injecting OpenAPI specs into PM collections
✓ Response schema validation
without any more effort
(other than the specifications)
✓ Request schema validation
(response of an invalid request
should be KO)
x Still a lot of manual operations
21
JSON
Schema
Postman
Swagger
- Postman Collection SDK
- Postman API ($)
(1)
(2)
- JS- YAML - UI

22. Direction des Technologies et du Digital
Postman iteration data
22
● Feature to loop over different test data for a given set of requests
iterationData.json
[
{
"var1": val1A,
"var2": val2A
},
…
]
iterationData.csv
var1, var2
val1A, val2A
val1B, val2B
val1C, val2C
val1D, val2D

23. Direction des Technologies et du Digital
#2nd idea: Using Postman iteration data as request data
23
requestData.json
[
{
"key": "PostInputDocumentRequest",
"path": "/private/secure/input/documents",
"method": "post",
"auth": { … }
"queryParams": { … }
"body": { … }
"expectedResponse": {
“status”: 200,
“body”: { … }
}
}
]
iterationData.json
[
{
"var1": val1,
"var2": val2
},
…
]
✓ Test scenario defined in Postman
✓ Request data defined using JSON iteration data
✓ Response body is automatically compared to expected response data
x But...

24. Direction des Technologies et du Digital
But...
24
JSON
Schema
Postman
(1)
(2)
- JS- YAML - UI
x Not sustainable
- Still a lot of copy/paste in Postman
(Lacking template definitions)
- Lots of interactions to inject the
specs
x Some limitations
- Can’t set files in request bodies from
iteration data
x Unpleasant ergonomy
- Must specify manually the Postman
items matching the iterationData in
collection runner
- Can’t use Postman easily when
using iteration data
- Can’t replay tests in collection
runner when using iteration data

25. Direction des Technologies et du Digital
#3rd idea: Building collections from scenarios in JSON
25
scenario.json
{
"name": "Creation of an input document with content",
"steps": [
...,
{
"key": "PostInputDocumentRequest",
"path": "/private/secure/input/documents",
"method": "post",
"auth": {
"type": "bearer",
"bearer": {
"token": "{{dmsAccessToken}}"
}
},
"savings": {
"dmsCreatedDocumentUuid": "pm.response.json().data[0].document.id",
"dmsCreatedReceptionUuid": "pm.response.json().data[0].reception.id",
"dmsUploadUrl": "pm.response.json().data[0].upload.url",
"dmsUploadUuid": "pm.response.json().data[0].upload.formFields.uploadUuid"
}
},
...
✓ Defining Postman items
relationship with JSON convention
- Using Postman variables
- Saving response data into
Postman variables

26. Direction des Technologies et du Digital
ddd
#4th idea: Running the tests from the application
26
Postman- Express
- Postman
Collection SDK
- Newman
Node.js Express app
- generating Postman
collections
- running Newman
- YAML (specifications)
- JS
- JSON (tests)
Import generated collection into
Postman for debugging purpose
only
newman-reporter-htmlextra
- Import from link

27. Direction des Technologies et du Digital
Finally: API testing automation integrated with the CI
27
Postman
- YAML (specifications)
- JS
- JSON (tests)
- Express
- Postman
Collection SDK
- Newman
Run tests from the CI and display
the result in Google Chat
- Import from link

28. Direction des Technologies et du Digital
And now?
28
● The test project is adopted by the team
● Consistent API endpoints with strong documentation
● Postman collections as integration examples
● Devs immediately know when non regression tests fail
● Tester can focus on designing new scenarios
● We can deliver new features of fixes very quickly
Tester
Dev (CI integration) Tester
Dev (False positive fix)

29. Direction des Technologies et du Digital
And now?
29
● The test project is adopted by the team
● Consistent API endpoints with strong documentation
● Postman collections as integration examples
● Devs immediately know when non regression tests fail
● Tester can focus on designing new scenarios
● We can deliver new features of fixes very quickly
Thank you
Any questions?
Tester
Dev (CI integration) Tester
Dev (False positive fix)

30. Cbp filiale du
Groupe Financière CEP
11, rue Royale
75008 PARIS
Tel : + 33 1 80 52 35 00
Cbp Group
www.cbp-group.com
11, rue Royale
75008 PARIS
Tel : + 33 1 80 52 32 90
30