At Findly we know test automation is key for continuous delivery. However, in the context of a microservices architecture, our monolithic end-to-end test suites have still been limiting our ability to achieve a truly "continuous" pace of delivery. This talk will explain the principles, processes and techniques we are now using to build test suites for microservices and enable continuous delivery at Findly. Presented at Auckland Continuous Delivery meetup, May 2016 (http://www.meetup.com/Auckland-Continuous-Delivery/events/230864194/).

1. Testing Microservices
Nathan Jones
May 2016
Enabling Continuous Delivery at Findly

2. About Me
Nathan Jones
@_ncjones
Software Engineer @ Findly
Previously
Yellow
Air New Zealand
Propellerhead
Orion Health

3. Summary
Multiple test suites per deployable
Prefer tests with faster feedback
Developers involved in testing

4. Contents
Background
Why Testing, Why Microservices, Test Scopes, Test Pyramid,
Common Issues
Our Approach
Types of Test, CD Pipeline, Test Suite Metrics
Future Work
Mock Services, Partial Envs, Non-Functionals

5. Background

6. Continuous Delivery
Local CI QA
Pre
Prod
Prod
Changeset

7. Technical Barriers for CD
auto deploy
high availability
monitoring
clean code
test automationtest automation

8. Microservices
“small, autonomous, services
working together”
- Sam Newman
Building Microservices

9. Microservices Example
Orders
Service
Payment
Service
Catalog
Service
Recommend
Service
Notification
Service

10. Benefits of Microservices
tech choices
resilience
scalability
agility
replaceability
organizational alignment
deployed independentlydeployed independently

11. Independently Deployed
Orders
Service
Payment
Service
Catalog
Service
Recommend
Service
Notification
Service
v1.2

12. Test Scopes
Class
Unit
App
Functional
System
End to end
(E2E)

13. Quantity
Scope
Unit
Functional
Test Pyramid
E2E

14. Fast Feedback
Faster
Unit
Functional
Later in CD
Pipeline
E2E

15. Common Issues

16. E2E
Common Issues
Unit
“We have both kinds of test”

17. E2E
Common Issues
Unit
“...and we value end-to-end more”

18. Orders
Service
Payment
Service
Catalog
Service
Recommend
Service
Notification
Service
Monolithic E2E test suite
Selenium Test
Suite
Common Issues

19. Our Approach

20. Our Approach
Each deployable has multiple test suites:
Unit Functional End-to-end

21. Unit Test
What Test functions/objects in isolation from other
collaborating components.
How Test in-process with mocks + sometimes use real DB
(integration test).
When Impl: same commit as functional change.
Run: every pull request.
Who Dev

22. Test
Class Under
Test
Dependency
Mock / Stub
Dependency
Anatomy of Unit Test System

23. Functional Test
What Test app via its API in isolation from other collaborating
components.
How Cucumber test suite + real DB + real message broker +
mock dependencies.
When Spec: first task when story picked up.
Impl: before story “done”.
Run: every pull request + nightly for real env.
Who Spec: PO + Devs + Tester.
Impl: Devs (mostly) + Tester.

24. Test
App Under
Test
HTTP
DB
Async
Message
Broker
Mock / Stub
Dependency
(async)
Mock / Stub
Dependee
Service
Anatomy of Functional Test System
Real
Downstream
Service

25. Example Functional Test Spec
@MBP-396.AC1
Scenario: Health check
Given The application has been started
When the health check URL is queried
Then it returns a valid status message

26. Functional Test Guidelines
★ Test suite talks to APIs
★ May use real dependencies
★ Create and clean up data
★ Run on every pull request in containers
★ Run against real HA environment nightly
★ Avoid testing orthogonal permutations
★ Collaboration between product owner, tester and developers

27. End-to-End Test
What Test app in fully integrated environment.
How Cucumber tests talking to real neighbouring services.
When Impl: before story “done” if new integration.
Run: nightly + after each deploy.
Who Spec: PO + Devs + Tester.
Impl: Devs + Tester (mostly).

28. Test
App Under
Test
Real
Upstream
Service
Anatomy of E2E Test System
Real
Downstream
Service

29. Example E2E Test Spec
@MBP-1616.AC4 @e2e
Scenario: Imported Log10 profile is available from Profile Service
Given The log10 e2e org is set up
When a profile is posted to the E2E org's Log10 endpoint
Then the profile can be retrieved from the Profile Service

30. CD Pipeline
Local CI QA
Pre
Prod
Prod
Unit Test +
Functional Test
(partial)
Pull
request
Manual
promote
Manual
promote
Manual
promote
Unit Test +
Functional Test
Functional Test E2E Test E2E Test
(also run
nightly)
(also run
nightly)

31. Test Suite Metrics
Test Count Run Duration Lines of Code

32. 0
Test Suite Metrics
125
86
292
109
464
240
E2E
Functional
Unit
Job Scheduler Batch
Processor
ATS
Integrator
1
Test Count
2

33. n/a
Test Suite Metrics
80s
45s
6m10s
50s
3m30s
E2E
Functional
Unit
cf. Monolithic E2E (600 selenium tests): 9 hours
Run Duration
15s
Job Scheduler Batch
Processor
ATS
Integrator
5s5s

34. Test Suite Metrics
2,704 7,708
5,048
10,753
5,902
E2E
Functional
Unit
Lines of Code
Job Scheduler Batch
Processor
ATS
Integrator
Prod 12,4127,0262,566
n/a 312
1,056
88

35. Future Work

36. Mock HTTP Services
Wiremock, MockServer, Martian
Consumer Driven Contracts
UI Testing: use CDN and mock API server

37. Partially Provisioned Environments
Orders
Service
Payment
Service
Catalog
Service
Recommend
Service
Notification
Service
Catalog
Service
Test Suite

38. Testing Non-Functionals
Perf and Security test automation
Recovery from restarts and outages
Inject failure: Saboteur, Chaos Monkey

39. Summary
Multiple test suites per deployable
Prefer tests with faster feedback
Developers involved in testing

40. Books
“How Google Tests Software”
James Whittaker
“Building Microservices”
Sam Newman
Further Reading
Articles
“Testing Strategies in Microservices
Architecture”
Toby Clemson - Thoughtworks
http://martinfowler.com/articles/microservice-testing/
“Just Say No to More End to End Tests”
Mike Wacker - Google
http://googletesting.blogspot.co.nz/2015/04/just-say-
no-to-more-end-to-end-tests.html
“Guidelines for Structuring Automated Tests”
Sunit Parekh - Thoughtworks
https://www.thoughtworks.
com/insights/blog/guidelines-structuring-automated-
tests