Test Smart, not hard

PRESENTED BY Matt Fellows (@matthewfellows)
Test S-M-R-T not HARD
Deploy distributed systems faster with contract tests
and safer!
v

ABOUT ME
2
1. Principal Consultant at DIUS
2. Pact since 2014
3. Would work in sports/fitness if
not IT
@matthewfellows

Everyone is doing microservices APIs
A STORY

Metric Value
No. teams 1
No. components 1
Test environments 1
Time in pipeline (commit to prod) 5 minutes
Risk 2.5%
Deployments per day 10
A STORY > CD HEALTH CHECK

Metric Value
No. teams 2
No. components 2
Test environments 4
Risk 5%

Metric Value
No. teams 3
No. components 6
Test environments 18
Risk 10%

Metric Value
No. teams 10
No. components 20
Test environments 20 200
Time in pipeline (commit to test) 120+ minutes
Risk 20%
Deployments per month 1

So what did we learn?
A STORY > LESSONS

Want: O(1) or O(n)
Got: O(n2
) or worse
A STORY > LESSONS

Where did we go wrong?
A STORY > LESSONS

Cohesive services, loosely coupled (want)
vs
Cohesive teams, tightly coupled by services (got)
A STORY > LESSONS

Previous tooling and strategies were not good enough
A STORY > LESSONS

“Integration tests are a scam”
- JB Rainsberger
A STORY > LESSONS > INTEGRATION TESTS

Scam, you say? Justify!
Integrated tests are:
● Slow
● Fragile
● Hard to manage
When they fail, you can’t point to the problem!

“But my integration tests run in Docker, why can’t I use
them?”
- People

“Because Maths”
- Me

Branches per box vs test cases required
2 code branches = 128 tests
5 code branches = 78,125 tests
10 code branches = 10M tests

Good tests have the exact opposite properties

Mocks to the rescue?

Dictator Driven Contracts
A STORY > LESSONS > API DESIGN

How to: dictator driven contracts
1. Sit in ivory tower and postulate
2. Document perfect API (Swagger/OAS etc.)
3. Create said API
4. Publish said document to consumers
5. Repeat steps 1-4

Dictator Consumer Driven Contracts

Benefits?

You’ll know when you break a consumer

You have a form of documentation

You can test things independently

Evolved from combining these two principles
PACT

● Open source (MIT)
● Multiple languages
○ Javascript
○ Go
○ Python
○ JVM
○ .NET
○ + more
pact.io
● HTTP contracts
● Message contracts
● … protocol agnostic!
PACT

PACT > STEP 1: WRITE A CONSUMER TEST

Given “User A exists”
When I Receive “a GET request for user A”
With “content-type: application/json”
Respond with “200 OK”
And “User A’s details in the body”
PACT > STEP 1: WRITE A CONSUMER TEST

Step 2: Share your pacts
PACT > STEP 2: SHARE CONTRACT

PACT > STEP 3: VERIFY PROVIDER

PACT > NO SILVER BULLET
No Silver Bullet

Are not functional tests
Contracts tests

Are not load
performance tests SLAs
Contracts tests

Are not good for “public”
APIs (many, unknown
consumers)
(Consumer)
Contracts tests
ANOTHER WAY @bethesque

Do I still need integration tests?

Probably, maybe, yes?

Integration test coverage
Consequences of bug
x
Time to find bug
x
Time to release fix

BEST PRACTICES
Integrating with your CICD processes

BEST PRACTICES > CHALLENGES
69
1. Sharing and collaborating on pacts between teams
2. Managing contracts across code branches and environments
3. Orchestrating builds to know when it is safe to deploy

Introducing...
The Pact Broker

BEST PRACTICES > PACT BROKER
71
Integrated collaboration tool to and share and manage pacts* between
teams.
● API documentation that is guaranteed to be up-to date
● Visualisations of the relationships between your services
● Dashboards containing contract verification status
● Pact tagging and versioning
● Webhooks for integration and communication
● View pact diffs
● ...and more!
All powered by a RESTful API

72
Did I mention README badges and integrations?

DEMO
BEST PRACTICES > PACT BROKER > VISUALISATIONNS
73

DEMO
BEST PRACTICES > PACT BROKER > VISUALISATIONNS
74

Replace lengthy integration test pipelines with
“The Matrix”
BEST PRACTICES > PACT BROKER > PIPELINES
75

BEST PRACTICES > PACT BROKER > CAN I DEPLOY?
76
● Verify that a Pacticipant is safe to deploy prior to release
● Enables “matrix” style testing
Consumer Head (1.0.1) Consumer Prod (1.0.0)
Provider Head (2.0.0) ? ?
Provider Prod (1.1.13) ? Already tested

77
Can also do with tags...
Consumer Head (‘dev’) Consumer Prod (‘prod’)
Provider Head (‘dev’) ? ?
Provider Prod (‘prod’) ? Already tested

BEST PRACTICES > PACT BROKER > “THE MATRIX”
78
$ pact-broker can-i-deploy
--app A --version 1
--app B --version 4

79

80
Computer says yes o/

DEMO
81

BEST PRACTICES > CI/CD > PROVIDER
83

BEST PRACTICES > CI/CD > CONSUMER
84

Summary
Best Practices
● Pact Broker is key for scaling
● Use webhooks for team communication
○ But… keep build triggers simple!
● Tag pacts to mirror your git branches
● Use can-i-deploy tool for deployment
pre-flight checks
● Update pact tags after deployment to
environment (e.g. prod)

Recapping...
● Business impact of integrated tests and environments
● Cause and explanation of those effects
● Alternative approach - isolation + contracts
● Contract-testing as an approach, Pact as a tool (in this order!)
● CDC has its own challenges
○ Not suitable for external/public APIS
○ effective collaboration is the key

Thank you
- @matthewfellows
- @pact_up
Given “The presentation is over”
Upon Receiving “A request for an answer”
With “A valid question”
Respond With “A valid answer”

What about Swagger / Open API?

A B
mock/write
C P
pact swagger
verify
verify
Pact+Swagger
OTHER TOOLS

What about systems maintained by other teams?

What about systems built in outdated technologies?

Scary outside
world!
Websphere?
Mainframe

Thank you
- @matthewfellows
Given “The presentation is over”
Upon Receiving “A request for an answer”
With “A valid question”
Respond With “A valid answer”

Test Smart, not hard

More Related Content

What's hot

Similar to Test Smart, not hard

More from DiUS

Recently uploaded

Test Smart, not hard