This document discusses tools and techniques for functional testing of RESTful applications. It covers different levels of testing from unit to system level. It then describes using blackbox testing as a monitoring tool by generating test data, overcoming latency, and integrating with monitoring services. Graybox testing is discussed as a way to control external dependencies through mocking services. Finally, it recommends using Cucumber to develop business-focused test scenarios that are closer to documentation and easier to maintain.

2. Functional Testing of
RESTful Applications - Tools
and Techniques
October 2015
Nenad Bozic
@NenadBozicNs
SmartCat
www.smartcat.i
o

3. Why this presentation now
• Idea to do End-to-End tests is here for a long time (Selenium)
• Doing tests over presentation layer is hard
• Modern architecture separates client and server
• Servers with REST APIs ease up functional testing by skipping presentation
layer

4. Agenda
• Levels of tests
• Blackbox testing as monitoring tool
• Graybox testing as tool for external dependency control
• Bring functional tests closer to business with Cucumber
• Conclusion – tools and techniques overview

5. Levels of testing
• Unit testing
• Integration testing
• Component testing
• Blackbox
• Whitebox
• Graybox
• System testing
“There are generally four recognized levels of tests: unit testing, integration testing, component interface testing, and
system testing. Tests are frequently grouped by where they are added in the software development process, or by
the level of specificity of the test.” - Wikipedia

6. Blackbox testing as
monitoring tool

7. Blackbox testing as monitoring tool
Overview of the system and goal of test

8. Blackbox testing as monitoring tool
Challenge 1: Generate data
• Presentation from Gerard Meszaros was truly inspiring for our test data
• Hide test data in generators behind DSL
• Isolate data for easy cleanup
• Isolate data for testing in production

9. Blackbox testing as monitoring tool
Challenge 2: Overcome latency in system
• Microservice architecture is bringing latency problems
• Spring Retry – setup number of attempts with pause in between

10. Blackbox testing as monitoring tool
Challenge 3: Monitoring
• Save latest test result in memory
• Endpoint to fetch results of tests at any time
• Integration with Rollbar to log failed executions
• SMS/email notification strategy

11. Blackbox testing as monitoring tool
Result: Tests as monitoring tool in production
• Scheduled tests with isolated data run each 10 minutes
• Health check of 5 applications in production (Rollbar integration and SMS/email notification)
• Test runs accessible via endpoint
• idea for future to have GUI and history of test runs
• Idea for future to save duration of tests so we can compare runs
• Admins sleep better at night

12. Graybox testing as tool
for external dependency
control

13. Graybox testing as tool for external dependency control
Overview of the system and goal of test

14. Graybox testing as tool for external dependency control
Challenge 1: Mock external services
• Mock requests and provide controlled responses
• WireMock – library for stubbing and mocking web services
• Written in Java but provides a JSON API so you can use it with pretty much any language out there
• Application configuration challenges – use Spring Profiles to configure external URLs
• Local and functional environment use WireMock server
• Staging and production use real endpoints

15. Graybox testing as tool for external dependency control
Challenge 1: Mock external services

16. Graybox testing as tool for external dependency control
Challenge 2: Deployment flexibility
• WireMock has two ways of working:
• Standalone – configured via the Java API, JSON over HTTP or JSON files
• Library import – start server as part of the application
• WireMock is running and functional testing app sends rules and responses

17. Graybox testing as tool for external dependency control
Result: Fully tested flow with external dependencies before deployment
• Application is targeting WireMock instead of real external endpoints
• Functional testing application has controlled data which can be used for verification
• This approach can be used for internal dependencies as well

18. Bring functional tests
closer to business with
Cucumber

19. Bring functional tests closer to business with Cucumber
Why we decided to go with Cucumber
• Number of tests grew
• Code base of test application started to be hard to maintain
• Hard to group tests based on features
• Each person in the team had its own vocabulary and business perspective
• Our documentation got outdated faster and faster

20. Bring functional tests closer to business with Cucumber
Overview of Cucumber features
• You write scenarios and they are grouped in features
• Features can be tagged (@smoke-tests, @wire-mock)
• With @Before you can prepare environment for tests
• With @After you can do cleanup
• Everyone can write features which are translated to tests
• Dynamic documentation which is near code

21. Bring functional tests closer to business with Cucumber
Feature example
• Cucumber uses Gherkin notation (Given / When / Then)
• With Given you prepare application for test
• With When you issue command which will be tested
• With Then you state expected outcome
Given products CSV file for organization X
When CSV file is uploaded to FTP server.
Then the API call for organization X returns newly added products

22. Conclusion

23. Conclusion - tools and techniques overview
• Hint 1: try to organize data for testing directly in production
• Hint 2: use tools such as Spring Retry to overcome latency
• Hint 3: use tools such as WireMock to control dependencies
• Hint 4: use tools such as Cucumber to develop business language around
features
• Hint 5: schedule tests and leverage them as powerful monitoring tool

24. Links
• SmartCat Blog post - Blackbox testing microservices
• SmartCat Blog post - Graybox testing - Control your dependencies
• SmartCat Blog post - Bring functional tests closer to business with Cucumber
• Wikipedia testing levels in software
• Gerard Meszaros - Find the right abstraction for your tests
• Tools: Spring Retry, WireMock, Cucumber

25. Q&A

26. Thank you
Nenad Bozic
@NenadBozicNs
SmartCat
www.smartcat.i
o