The DevOps principle of “Shifting Left” promotes testing early in the development cycle, for improved software quality and system health. At the same time, the rise of containerized microservice applications brings a new challenge: services are developed in isolation. It’s common practice that each service is frequently, thoroughly tested—individually. But they don’t get validated together until deploy time (if at all!). In this session, we’ll explore techniques for running high-fidelity integration tests across multiple services, as part of a Continuous Integration workflow. You will learn how to provision, test, and tear down self-contained Kubernetes environments that replicate complete production systems. This allows you to run full-system tests as part of every build, safely and cost effectively.

1. Fully test your microservices application, early and often
Dave Stanke, Developer Advocate at Google Cloud @davidstanke
Illustrations by Tracey Laguerre
Continuous
Integration Testing

2. @davidstankeContinuous Integration Testing
Continuous All The Things!
Good for business.
Good for process.
Good for people.

3. @davidstankeContinuous Integration Testing
Before Continuous Integration (BCI):

4. @davidstankeContinuous Integration Testing
Before Continuous Integration (BCI):

5. @davidstankeContinuous Integration Testing
Continuous Integration (CI)

6. @davidstankeContinuous Integration Testing
Software Testing
A bug found during development
takes {{totes way more}} time to fix
than a bug found in prod.

7. @davidstankeContinuous Integration Testing
This is good for business
source: DORA Accelerate State of DevOps
Continuous
Testing
Continuous
Delivery
Software
Delivery
Performance
Availability

8. @davidstankeContinuous Integration Testing
Idealized Test Pyramid
Unit
Integration
End-to-End

9. @davidstankeContinuous Integration Testing
Actual Test Pyramid
Unit
Integration
End-to-End

10. @davidstankeContinuous Integration Testing
Production runtimes != developer machines
Integration tests are hard.

11. @davidstankeContinuous Integration Testing
Production runtimes != developer machines
Cost
Integration tests are hard.

12. @davidstankeContinuous Integration Testing
Production runtimes != developer machines
Cost
Speed
Integration tests are hard.

13. @davidstankeContinuous Integration Testing
Production runtimes != developer machines
Cost
Speed
Side Effects
Integration tests are hard.

14. @davidstankeContinuous Integration Testing
Microservices make it harder.

15. @davidstankeContinuous Integration Testing
Microservices make it harder.
PROD
Team A
Team B
Team C
Team D

16. @davidstankeContinuous Integration Testing
Why bother?

17. @davidstankeContinuous Integration Testing
Why bother?
Because Semantic Versioning (SemVer) isn’t enough.

18. @davidstankeContinuous Integration Testing
Why bother?
Because Semantic Versioning (SemVer) isn’t enough.
Because complex systems have emergent properties.

19. @davidstankeContinuous Integration Testing
Why bother?
Because Semantic Versioning (SemVer) isn’t enough.
Because complex systems have emergent properties.
Because users experience an application.

20. @davidstankeContinuous Integration Testing
Don’t bother?
Test in Prod?!?

21. @davidstankeContinuous Integration Testing
Test in Prod.
And before prod.

22. @davidstankeContinuous Integration Testing
How bad could it be?
No, seriously…
how bad could it be?

23. @davidstankeContinuous Integration Testing
Integration Testing limits risk
We can still do it – even on microservice applications.
It’s hard, but (potentially) worth it.
Make informed risk analyses.

24. @davidstankeContinuous Integration Testing
Provisioning
Creating a System Under Test (SUT)

25. @davidstankeContinuous Integration Testing
What are we optimizing for?
Fidelity
Isolation
Speed
Scalability
Ephemerality
Cost

26. @davidstankeContinuous Integration Testing
Fidelity
Integration Testing Goal:

27. @davidstankeContinuous Integration Testing Illustrations by TRACEY LAGUERRE
Isolation
Illustrations by TRACEY LAGUERRE
Integration Testing Goal:

28. @davidstankeContinuous Integration Testing
Speed
Integration Testing Goal:

29. @davidstankeContinuous Integration Testing
Scalability
Illustrations by TRACEY LAGUERRE
Integration Testing Goal:

30. @davidstankeContinuous Integration Testing
Ephemerality
Integration Testing Goal:

31. @davidstankeContinuous Integration Testing
Cost
Integration Testing Goal:

32. @davidstankeContinuous Integration Testing
An example
Kubernetes (“K8s”)
microservice
application
CI pipeline with
integration tests
bit.ly/c-i-t

33. @davidstankeContinuous Integration Testing
Integration test flow
build
web db
provision
web
db
test.sh
curl | grep
deprovision
if pass

34. @davidstankeContinuous Integration Testing
Technique 1: Docker Compose

35. @davidstankeContinuous Integration Testing
Technique 2: K8s Staging Cluster
Kubernetes Engine
Namespace Web DB
Namespace Web DB
Namespace Web DB

36. @davidstankeContinuous Integration Testing
Technique 3: Self-destructing VM

37. @davidstankeContinuous Integration Testing
Self-Destructing VM?
bit.ly/self-destructing-vm

38. @davidstankeContinuous Integration Testing
Technique 3: Self-destructing VM!

39. @davidstankeContinuous Integration Testing
Technique 4: K8s cluster per test
Kubernetes Engine
cluster: `test-123`
Web DB
gcloud gcloud

40. @davidstankeContinuous Integration Testing
Integration testing techniques
Method Fidelity Isolation Scalability Ephemerality Speed Cost
docker-compose ⬤ low ⬤ high ⬤ high ⬤ (depends) ⬤ fast ⬤ medium
Shared “staging” K8s ⬤ high ⬤ medium ⬤ highest ⬤ variable ⬤ fast ⬤ low
Single-node “k8s” ⬤ medium-high ⬤ high ⬤ high ⬤ variable ⬤ medium ⬤ medium
K8s per test ⬤ highest ⬤ high ? ⬤ variable ⬤ slow ⬤ high

41. @davidstankeContinuous Integration Testing
Integration testing techniques
Method Fidelity Isolation Scalability Ephemerality Speed Cost
docker-compose ⬤ low ⬤ high ⬤ high ⬤ (depends) ⬤ fast ⬤ medium
Shared “staging” K8s ⬤ high ⬤ medium ⬤ high ⬤ variable ⬤ fast ⬤ low
Single-node “k8s” ⬤ medium-high ⬤ high ⬤ high ⬤ variable ⬤ medium ⬤ medium
K8s per test ⬤ highest ⬤ high ? ⬤ variable ⬤ slow ⬤ high

42. @davidstankeContinuous Integration Testing
Lots of services?
But Dave, what about...

43. @davidstankeContinuous Integration Testing
Lots of services?
Stateful storage?
But Dave, what about...

44. @davidstankeContinuous Integration Testing
Lots of services?
Stateful storage?
PII?
But Dave, what about...

45. @davidstankeContinuous Integration Testing
Yes, we can (still) do integration testing
There are trade offs between different ways of doing it.
It’s all about risk tolerance vs. reward of speed/cost/etc.
You have to find your own sweet spot. (When you do, tell me about it!)
@davidstanke