This document proposes an approach to automatically generate test cases for REST APIs using the OpenAPI specification. It leverages the OpenAPI metamodel to extract models and define inference rules to generate nominal and faulty test cases. An implementation was validated on real-world APIs, achieving over 75% coverage of operations, parameters, endpoints and definitions. The main failing points identified were incorrect or invalid OpenAPI definitions. The approach aims to increase coverage levels and support the OpenAPI v3.0 specification.

1. Automatic Generation of Test Cases for REST
APIs: a Specification-Based Approach
Hamza Ed-douibi, Javier L. Cánovas Izquierdo, Jordi Cabot
unsplash-chuttersnap

2. flickr/clark-tibbs
Motivation

3. API-Driven World

4. API-Driven World

5. API-Driven World

6. API-Driven World

7. API-Driven World

8. unsplash-rawpixel

9. Leveraging OpenAPI for testing REST APIs

10. Leveraging OpenAPI for testing REST APIs

11. Leveraging OpenAPI for testing REST APIs

12. OpenAPI Metamodel

13. OpenAPI Model

14. Leveraging OpenAPI for testing REST APIs

15. TestSuite Metamodel

16. Our Approach
• Straightforward from the OpenAPI Metamodel
• Special attention to JSON references
Model Extraction

17. Our Approach
• Simple parameter (examples, default values, enums)
• Dummy parameter
• Complex parameter (response of an operation)
Inference rules

18. Our Approach

19. Generation Rules
GR 1. Nominal test case GR 2. Faulty test case

20. Generation Rules
GR 1. Nominal test case GR 2. Faulty test case
Faulty cases
• Required missing
• Wrong datatypes
• Violated constraints

21. Our Approach

22. Our Approach
https://github.com/SOM-Research/test-generator

23. unsplash-Hello I'm Nick
Validation

24. Validation
RQ1
What is the coverage level of the generated test
cases?
RQ2
What are the main failing points in the definitions
and implementation of real world REST APIs?

25. Validation
RQ1
What is the coverage level of the generated test
cases?
RQ2
What are the main failing points in the definitions
and implementation of real world REST APIs?
Filtering Criteria A ― Free, open and available APIs which provide access to data models

26. Validation
RQ1
What is the coverage level of the generated test
cases?
RQ2
What are the main failing points in the definitions
and implementation of real world REST APIs?
Filtering Criteria A ― Free, open and available APIs which provide access to data models
Filtering Criteria B ― Remove incorrect or invalid OpenAPI definitions

27. Validation
RQ1
What is the coverage level of the generated test
cases?
RQ2
What are the main failing points in the definitions
and implementation of real world REST APIs?
ELEMENTS COUNT
COVERAGE COVERAGE(%)
ALL NOMINAL FAULTY ALL NOMINAL FAULTY
OPERATIONS 367 320 303 233 87% 82% 63%
PARAMETERS 949 595 485 476 62% 51% 50%
ENDPOINTS 356 289 81%
DEFINITIONS 313 239 76%

28. Validation
RQ1
What is the coverage level of the generated test
cases?
RQ2
What are the main failing points in the definitions
and implementation of real world REST APIs?
ELEMENTS COUNT
COVERAGE COVERAGE(%)
ALL NOMINAL FAULTY ALL NOMINAL FAULTY
OPERATIONS 367 320 303 233 87% 82% 63%
PARAMETERS 949 595 485 476 62% 51% 50%
ENDPOINTS 356 289 81%
DEFINITIONS 313 239 76%
TOTAL
NOMINAL TEST CASES FAULTY TEST CASES
4XX/500 SCHEMA 500 2XX
NUMBER OF APIS 37 9 11 11 20
40% 25% 30% 30% 55%

29. Conclusion
• Model-driven approach to automate
REST API testing based on Open API
• Plug-in implementing the approach
• Coverage over 75%
What we have shown
What we want to do next
Increase coverage levels
Support for OpenAPI v3.0
unsplash-Vek Labs

30. Except where otherwise noted, content on this presentation is licensed under a Creative Commons Attribution 4.0 International license.
Thanks!
Javier L. Cánovas Izquierdo
jcanovasi@uoc.edu
@jlcanovas
Hamza Ed-douibi
hed-douibi@uoc.edu
@mazamiz
Jordi Cabot
jordi.cabot@icrea.cat
@softmodeling
https://github.com/SOM-Research/test-generator