Escape integration syrup with contract tests

Wading through treacle?
Escape the integration syrup
with contract tests
Stefan Smith, ThoughtWorks
@smith_stefan

A technique to help teams go faster
by decoupling their development and
delivery activities from the external
systems they depend on

A simple delivery pipeline
Developers CI QA Staging Production
Build
Test
Package &
Version Artefact
Manual
Exploratory
Testing
Sanity
Check
Real
Users
Coding
A snapshot in time
v10v16 v13v21

Two services collaborating
Consumer
Provider
Integration test in CI environment
v21 v18 v14 v13
Release
Candidate

Integration Tests: A definition
• Collect modules together into a
subsystem that provides some
larger piece of behaviour
Paraphrasing from Testing Strategies in a
Microservice Architecture (martinfowler.com)
http://bit.ly/1GKK07d
• Check for any incorrect
assumptions the modules make
about the peers they collaborate
with
• Verify interactions between layers
of integration code and the
external components with which
they are integrating
External Component
UI Layer or API Layer
Integration Layer
Domain Layer
Exercise Assert
Exercise Assert
✓
✓

Two services collaborating
Consumer
Provider
Integration test in CI environment
v21 v18 v14 v13
Release
Candidate
Does this scale?

⌛
Multiple collaborating services
Slow build and test phase
v21 v18 v14 v13
Release
Candidate
Queuing
=> development delays
⌛
⌛
⌛

Downstream integration testingDefect in one component fails release candidateMonolithic deployments are slow and riskySlow integration phase increases feature lead time
Separate CI builds
v2
2
v1
1
v9
2
v5
8
v1
4
v1
1
v9
0
v4
0
v42
v1
3
v1
2
v9
0
v4
0
v41
v1
8
v1
1
v9
1
v4
6
v44
v2
1
v1
1
v9
1
v5
8
v47
Developers CI QA Staging ProductionIntegration
Release
Candidate

v26v26v26v26v26
v86
v53
v77
Content
Site
User
Service
Report
Service
Developers
Librarian
Portal
CI QA Staging ProductionIntegration
v40 v37 v36
How can we tell if versions will integrate in Production?
v35
Stepping off the Release Train

v26
Which version are we integrating with?
Does Provider meet Consumer expectations?
Which version are we integrating with?
?
Consumer Provider
____
____
____
____
____
Contract
GET /cities HTTP/1.1
[
{ name: "London", country: "UK" },
{ name: "Paris", country: "France" }
]
Contract
Test
v26
✓

[Test]
public void CitiesEndpointShouldProvideCityNames()
{
var response = httpClient.Get("/cities");
var cities = response.Body.As<JsonArray>();
Assert.That(cities, Has.JsonObjectWith(o => o.name == "London"));
Assert.That(cities, Has.JsonObjectWith(o => o.name == "Paris"));
}
Example
• Write in language of interaction (vs language of consumer)
• Configure Provider URL at runtime
• Develop against shared instance of real system
What will cause this test to fail?
• Maintain one contract test suite per external system

Developers
Flexible contracts
Be conservative in what you send, be
liberal in what you accept (Postel’s Law)
Use the Tolerant Reader pattern
Evolve APIs with backwards compatibility
(no breaking changes)
[
{ name: "London", country: "UK" },
{ name: "Paris", country: "France" }
]

v26v26v26v26v26
v86
v53
v77
v37 v36 v35v40
Content
Site
User
Service
Report
Service
Developers
Librarian
Portal
CI QA Staging Production
Sensing danger: look ahead
_
_
_
_
How easy is it to contract test another environment?
Integration
Contract
Testing

Production
Testing against another environment
If we don’t integration test, how does Provider validate their code?
• Target mirror of Production if possible
• Gain access to environment
• Establish stable “golden dataset” iteratively (per story)
• If Prod, create entities (e.g. users) that are safe to modify
• If Prod, whitelist permitted requests and safe entities
• Consider contention: parallel test runs touch same entities
• Understand availability and stability of environment

Content
Site
User
Service
Report
Service
Developers
Librarian
Portal
_
_
_
_
_
_
_
_
How can Providers track and fetch contracts?
Contract
Testing

QA Staging Production
Sharing contract tests with Providers
_
_
_
_
• Consumer publishes contract tests as
versioned build artefact ("LATEST")
• After deploy, Consumer labels Prod
version of contract tests ("LIVE")
• Provider can fetch and run
Production contract tests at any point
CI
Content
Site
Developers
Librarian
Portal
Artefact Repository
_
_
_
_
"LIVE""LATEST"
• Contract tests should be trivial to run
and easy to understand when they fail
What if a Provider has multiple Consumers?
Contract
Testing
_
_
_
_

Collecting all your consumers’ contracts
_
_
_
_
_
_
_
_
_
_
_
_
Provider Consumer C
Consumer BConsumer A
GET /cities HTTP/1.1[
{
name: "London",
country: "UK"
},
{
name: "Paris",
country: "France"
}
]
[
{
name: "London",
country: "UK"
},
{
name: "Paris",
country: "France"
}
]
[
{
name: "London",
country: "UK"
},
{
name: "Paris",
country: "France"
}
]
[
{
name: "London",
country: "UK"
},
{
name: "Paris",
country: "France"
}
]
Consumer A Contract Consumer B Contract
Consumer C ContractConsumer Driven Contract
✓
• CDC defines Provider’s
obligations to its Consumers
• Provider can use CDC to
understand impact of changes

Content
Site
User
Service
Report
Service
Developers
Librarian
Portal
Contract
Testing
_
_
_
_
_
_
_
_
What if Provider changes before Consumer reaches Production
(or visa versa)?

Contract
Testing
Sensing danger: watch where you’re walking
• Be a good citizen:
check before
deploying to each
shared env.
• Will this consumer
work in Production?
• Will this provider
break Production?
• Hard gate: abort
deployment if
contract test fails
Is this feedback early enough?
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_

Content
Site
User
Service
Report
Service
Developers
Librarian
Portal
CI
Sensing danger: read the warning signs
• If providers do not run
contract tests, consumers
should run them periodically
Are we diverging?
• Run latest contracts against
latest builds
• Failing contract tests do not
necessarily block pipeline,
they provoke conversation
• Early warning system: we’re
diverging
_
_
_
_

v26v26v26v26v26
v11
v53
v77
Quotes
Site
Policies
Service
Core
Domain
Services
Developers
Customer
Portal
v10
v53
v77
v10
v50
v74
v9
v48
v74
v9
v48
v73
The insurance company
_
_
_
_
When did we test Customer Portal’s integration code?
Integration
Contract
Testing

Testing Consumer integration code
Core
Domain
Services
Quotes
Site
Policies
Service
Customer
Portal
Test
Backed by mainframe
Data setup painful
Slow shared
environment
Impossible to generate
some states
Different tech stack
Hard to generate
some errors
Relied on other
shared systems
Slow to start
Sometimes
unavailable

What if this isn’t easy?
Testing Consumer integration code
We verify contract at various stages in pipeline
Is Provider deployed to shared env?
Is the Provider fast to interact with?
Is the Provider usually available?
Is it easy to put the Provider into a desired state?
Is it easy for us to host the Provider?
Does the Provider talk to other services?
Is it easy to make the Provider return errors?
Quotes
Site
Policies
Service
Core
Domain
Services
Customer
Portal
Developers CI

Consumer Provider
Client
Code
Stub
Client
Integration Test
• Tests are only valuable if stubs imitate Provider behaviour
• We observe divergent behaviour
• How do we know when Provider behaviour changes?
• Need to keep stub behaviour up to date
Avoiding the Provider

✗
2. Create feedback loop using contract test
Consumer ProviderIntegration Test
5. Fix Fake
3. Provider changes
Faking it
Fake
__
__
__
__ ✓✗
✓
4. We fix contract test
An out-of-process,
stateful test double
1. Inject golden dataset into Fake
Golde
n Data
6. Run integration test
✓
Integration
Test Data

Some characteristics of Fakes
Fake
• Implement just the functionality
required to pass the contract test
• Transient – no persistent state
• Same network interaction as real
Provider
• Fast to start
• Easy to deploy
• Can be put into desired state easily
• Can simulate failures
• Unlike HTTP stub servers, Fakes are stateful and exhibit dynamic behaviour
• Simple embedded HTTP Servers,
such as:
• Jersey / Scalatra + Jetty
• Nancy + OWIN HttpListener
• Express.js + Node

Leveraging Fakes
Fake
• Use as custom sandboxed
environment for exploratory
testing by BAs, QAs and XDs
• Use in automated tests to
speed them up
• Target Fake from Consumer
simply by configuring a URL
Consumer
v21
____
____
____
____
____
• Version Fake with consumer
and contract test
• Always contract-test Fake before
using in automated tests
• Use in automated tests to
ensure Provider’s initial state is
explicit and maintainable

Integration testing
Core
Domain
Services
Quotes
Site
Policies
Service
Customer
Portal
Test
Core
Domain
Services
Policies
Service
Quotes
Site
______
______
______
______
______
______
______
______
______
______
______
______
______
______
______
______
______
______
______
______
______
Our Source
Repository

Sensing danger throughout the delivery pipeline
Contract
Testing
_
_
_
_
Developers CI
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
_
So everything’s watertight, right?

Supporting Practices: minimising surprises
Contract
Testing
Short lead time
Smoke testing
Deploy small
batches
Canary releases
Design for
failure

No Free Lunch
• Establishing and stabilising golden data
• Learning curve
• Avoiding data contention
• Gaining access to environments
• Living with instable environments
• Infrastructure for sharing contracts
• Packaging contracts for Providers to
execute
• Faking failure modes and
strictly enforcing request
structures
• Dense message structures
(e.g. SAML) that require a lot
of integration logic
• Pass-through systems that
modify outbound or inbound
messages
• Faking systems that share
state/communicate with each
other (e.g. OAuth)

In summary
Contract tests may help you release independently,
with confidence, if either of the following are true:
• The Service Provider is willing to run your
contract tests in their pipeline
• You need to use a realistic fake in your tests
because the external system is unavailable, slow
or hard to configure

Thank you
Stefan Smith, ThoughtWorks
@smith_stefan

Appendices
1. Why not use integration tests?
2. Development lifecycle
3. Faking failure modes
4. Tools

Appendix 1:
Why not use integration tests?

• Integration tests written against real systems may be coupled to fixed data
in those systems, making tests less readable and harder to maintain
Why not use integration tests to check the contract?
• Integration test failures are reported in terms of our domain, not in
the language of the contract: harder for Provider to diagnose errors
• To remain decoupled, other teams would run our integration tests on-
demand in their pipelines, but…
• A single integration test might make calls to multiple systems owned
by different teams, increasing coupling between pipelines.
• Test-driving realistic fakes with integration tests is a case of “the tests
talking to themselves”
• Distributing some of our Production code might constitute a security
or intellectual property concern
Svc 1
Svc 2
AppTest
___
___
___
___
___
___
___
___

Appendix 2:
Development Lifecycle

Story lifecycle: outside-in (ATDD)
Integration Layer
Domain Layer
AcceptanceTest
✗
• Start Fake before all tests
• Inject state into Fake
• Exercise application via UI / API
• Assert on observed behaviour
• Reset Fake state before each test
Start Inject Reset
Exercise Assert

Story lifecycle: unit tests
Integration Layer
Domain Layer
• Drive UI / API code with unit tests
• Drive domain code with unit tests
AcceptanceTest
✗
UnitTests
✓
• Stop when you reach integration layer
Domain Layer

Domain Layer
Story lifecycle: integration tests
Integration Layer
AcceptanceTest
✗
IntegrationTest
✗
• Start Fake before all tests
• Inject state into Fake
• Exercise integration layer
• Assert on response
• Reset Fake state before each test
• Begin implementing integration code
to discover what assumptions we
make about the Provider
Start Inject Reset
Exercise Assert
Integration Layer

Provider
Story lifecycle: contract tests
Integration Layer
Domain Layer
AcceptanceTest
✗
IntegrationTest
✗
• Write contract test and run against
the real Provider (requires access
from developer workstation)
• Assert on responses from Provider in
order to check our assumptions
ContractTest
✓Exercise Assert

Story lifecycle: implementing the Fake
Integration Layer
Domain Layer
AcceptanceTest
✗
IntegrationTest
✗
• Start Fake before all contract tests
ContractTest
• Inject golden dataset into Fake so its
state matches real Provider
• Point contract tests at Fake to drive
out implementation
• Don’t reset Fake state between tests
Start Inject
Exercise Assert
✓

Story lifecycle: all tests green
Integration Layer
Domain Layer
AcceptanceTest
✗
IntegrationTest
✗
• Run integration tests against Fake
• Run acceptance tests against Fake
• Point application at Fake and manually
test
• Start Fake and inject sample state
• Point application at real Provider and
manually test
✓
✓
Provider
Manual/AcceptanceTest
Exercise Assert
Exercise Assert
Start Inject

Appendix 3:
Faking failure modes

Fake.CitiesController
1 [Path("/cities")]
2 public Response GetCities()
3 {
4 return Cities.AsJsonResponse();
5 }
Naïve implementations: a simple example
Consumer Provider
Test
Fake
___
___
___
CitiesEndpointContractTest
1 [Test]
2 public void CitiesEndpointShouldProvideCityNames()
3 {
4 var response = httpClient.Get("/cities?orderBy=name");
5 var cities = response.Body.As<JsonArray>();
6 Assert.That(cities, Has.JsonObjectWith(o => o.name == "London"));
7 }
App.AtlasSource
1 public Atlas GetAtlas()
2 {
3 return httpClient.Get("/cities")
4 .Body.MapToAtlas();
5 }

Naïve implementations: a simple example
CitiesEndpointContractTest
1 [Test]
2 public void CitiesEndpointShouldProvideCityNames()
3 {
4 var response = httpClient.Get("/cities?orderBy=name");
5 var cities = response.Body.As<JsonArray>();
6 Assert.That(cities, Has.JsonObjectWith(o => o.name == "London"));
7 }
8
9 [Test]
10 public void CitiesEndpointShouldRespondWithBadRequestWhenOrderByParamMissing()
11 {
12 var response = httpClient.Get("/cities");
13 Assert.That(response.Status, Is.EqualTo(400));
14 }
Consumer Provider
Test
Fake
___
___
___

Being conservative in what you accept
Consumer Provider
Test
Fake
___
___
___
UsersEndpointContractTest
1 [Test]
2 public void UsersEndpointShouldCreatePostedUser()
3 {
4 var jsonFormat =
5 @"{
6 ""name"" : ""{0}"",
7 ""department"" : { ""name"" : ""Operations"", ""id"" : ""43"" }
8 }";
9 var json = string.Format(jsonFormat, "Bill");
10
11 var response = httpClient.Post("/users", json);
12
14 var user = response.Body.As<JsonObject>()
15 Assert.That(user.name, Is.EqualTo("Bill"));
16 }
Fake.UsersController
1 [Path("/users", Method.Post)]
2 public Response CreateUser(String userJson)
3 {
4 var userDto = userJson.AsDto();
5 Users.Add(userDto);
6 return Response.Created(userDto);
7 }

Fake.UserCreationRequestBody
1 public class UserCreationRequestBody
2 {
3 const String jsonFormat =
4 @"{
5 ""name"" : ""{0}"",
6 ""department"" : { ""name"" : ""Operations"", ""id"" : ""43"" }
7 }";
8
9 public bool IsInvalidFor(String name, String jsonString)
10 {
11 var expected = string.Format(jsonFormat, name).AsJsonObject();
12 var actual = jsonString.AsJsonObject();
13 return !actual.IsEquivalentTo(expected);
14 }
15
16 public String CreateWith(String name)
17 {
18 return string.Format(jsonFormat, name);
19 }
20 }
Consumer Provider
Test
Fake
___
___
___

Consumer Provider
Test
Fake
___
___
___
UsersEndpointContractTest
1 [Test]
2 public void UsersEndpointShouldCreatePostedUser()
3 {
4 var json = UserCreationRequestBody.CreateWith(name: "Bill");
11
14
15 var user = response.Body.As<JsonObject>()
16 Assert.That(user.name, Is.EqualTo("Bill"));
17 }
1 [Test]
2 public void UsersEndpointExpectsRequestBodyStructure()
3 {
var json = @"{ ""name"" : ""Bill"" }";
17 }

Consumer Provider
Test
Fake
___
___
___
Fake.UsersController
1 [Path("/users", Method.Post)]
2 public Response CreateUser(String userJson)
3 {
4 var userDto = userJson.AsDto();
5
6 if (UserCreationRequestBody.IsInvalidFor(userDto.name, userJson))
7 {
8 return Response.BadRequest;
9 }
10
11 Users.Add(userDto);
12 return Response.Created(userDto);
13 }

Tools
https://github.com/realestate-com-au/pact
• Ruby-based (JVM and .NET versions available)
• Write integration and contract tests together
• No need to package for Providers
• Simple HTTP + JSON support
• Complementary tool: Pact Broker
• Define http request and response
• Does not support stateful transaction testing
• Does not produce a separate fake to use in acceptance or manual tests

Tools
http://thoughtworks.github.io/pacto/
• Ruby-only
• Similar to Pact
• Need to edit document to ensure you remain weakly-coupled
• Captures full interaction in JSON-based definition document

Escape integration syrup with contract tests

Recommended

Recommended

More Related Content

What's hot

What's hot (13)

Viewers also liked

Viewers also liked (18)

Similar to Escape integration syrup with contract tests

Similar to Escape integration syrup with contract tests (20)

Recently uploaded

Recently uploaded (20)

Escape integration syrup with contract tests

Editor's Notes