From YOW! night presentation in Melbourne and Sydney Modern distributed architectures are increasingly composed of large numbers of decoupled, asynchronous components. In AWS, these components are plumbed together via services like SQS, Kinesis and S3 often integrated via small and frequent numbers of microservices or lambdas. But how do you test these architectures if they are cloud native? It’s 2019, and we can do better than deploying the entire stack and running a battery of E2E tests against them. In his talk, Matt will demonstrate how you can scale development of large-scale systems across teams, technology and process, and unlock the agility of your cloud-native architecture. See https://www.eventbrite.com.au/e/yow-night-2019-melbourne-modern-testing-jul-23-tickets-63937866881#

1. Serverless microservices
Matt Fellows
@matthewfellows
smarter
Test , not harder

2. 2
The new uService
The New Microservice

3. 3
The New Microservice > After
API
External
Events
API
Outside World
Microservice
Interface
API
Master Data
API
Inside bounded context
Event Emitters
API
Materialised view /
cache
Events and
Processing
Data
Event Handlers
Processing
Offline Processing

4. 4
Provisioning
Amazon ML
The New Microservice > After > Example Team 1 Bounded Context
Place Order API
External
Events
Order Status API
Outside World
Internal Events
Microservice
Interface
Data, Events and
Processing
Suggestions API
Publish Event
to external
subscribers
Insights /
Predictions
Customer /
Other Event
Orders
Push API
Inside bounded context
CRM
Materialised view of
customer
Provisioning Event
External
Events
… API

5. 5
Provisioning
The New Microservice > After > Example Team 1 Bounded Context
External
Events
Outside World
Internal Events
Microservice
Interface
Data, Events and
Processing
Publish Event
to external
subscribers
Insights /
Predictions
Place Order API
Customer /
Other Event
Orders
Inside bounded context
Materialised view of
customer
Provisioning Event
CRM
Marketing
Automation
Customer
Support

6. 6
The New Microservice > After > IoT ExampleThe New Microservice > After > Example 2 > IoT Architecture
API Gateway
Sensor A
Sensor B
Sensor C
IoT Rule A
IoT Rule B
IoT Rule C
AWS IoT Core
Sensor Data A
Sensor Data C
Processed Events
Sensor A
Vendor
Device API
Real-time events
Sensor Data B
Analytics
Device Ingest Process Store

8. 8
Testing The New Microservice
How do we test these?

9. 9
Testing The New Microservice > Test Pyramid
https://martinfowler.com/bliki/TestPyramid.html

10. Testing The New Microservice > Test Pyramid
Mama-mia!

11. 11
Testing The New Microservice > Test Pyramid > Ideal Software Testing Pyramid
https://watirmelon.blog/testing-pyramids/
Don’t forget people - the
ones doing the testing!
Let me make this a bit
clearer for you...

12. 12
Testing The New Microservice > Test Pyramid > Software Testing Trophy
https://kentcdodds.com/blog/write-tests

13. 13
Testing The New Microservice > Test Pyramid > Testing Funnel?
https://medium.com/@copyconstruct/testing-microservices-the-sane-way-9bb31d158c16

14. 14
Testing The New Microservice > Test Pyramid > Round Earth Heuristic
https://www.satisfice.com/blog/archives/4947

15. Today’s focus
Architecture > Microservices
1. Testability
2. Reduce need for UI / end-to-end testing
3. Observability

16. Testability

17. 17
Structure
Testability

18. 18
Testability > Lambda Pattern > Ports and Adaptors > Example
Order
Provisioning
Orders CRM
Orders

19. Testability > Lambda Anti-pattern > Example

20. MQ Handler
WebSocket
handler
Azure Function
Handler
Google Cloud Fn
Handler
Lambda
Handler
Service
Repository
Port
Adapter
Business Logic Domain Model Collaborators
Testability > Lambda Pattern > Ports and Adaptors
...
Test Harness

21. Testability > Lambda Pattern > Ports and Adaptors > Lambda Adapter

22. Testability > Lambda Pattern > Ports and Adaptors > Lambda Adapter

23. Testability > Lambda Pattern > Ports and Adaptors > Lambda Adapter

24. Testability > Lambda Pattern > Ports and Adaptors > Lambda Adapter

25. Testability > Lambda Pattern > Ports and Adaptors > Lambda Adapter

26. Testability > Lambda Pattern > Ports and Adaptors > Lambda Adapter

27. Testability > Lambda Pattern > Ports and Adaptors > Lambda Adapter

28. Summary
● Separate protocol handling from
business logic
● Business logic shouldn’t change with
introduction of a new protocol
● Enables testability - Port, Adapter and
Business Logic are independently
testable
Testability > Lambda Pattern > Ports and Adaptors > Summary

29. WHAT WE ARE KNOWN FORTestability > Modularisation
Adapted from https://martinfowler.com/articles/microservice-testing
Adapters
Services
Domain
Repositories
Collaborators
External
Service
Test
modularity

30. 30
How do you test locally?
Testability

31. WHAT WE ARE KNOWN FOR
31
Options
Testability > Testing Locally
● Use real services
● Stub services (e.g. Localstack)
● Stub SDK (e.g. Moto)
● Local Unit, Integration +
Component tests

32. 32
Testability > Testing Locally

33. WHAT WE ARE KNOWN FOR
33
Options
● Feature arms race
● Trustworthy?
● Integrated = slower / harder
● All cloud providers?
● Point at the real services
● Stub services (e.g. Localstack)
● Stub SDK (e.g. Moto)
● Local Unit, Integration +
Component tests
Testability > Testing Locally
Challenges

34. 34
Local Integration Test
Testability > Testing Locally

35. WHAT WE ARE KNOWN FORTestability > Testing Locally
Show me
where?
Adapters
Domain
Repositories
Collaborators
mock
Services

36. Testability > Testing Locally > Integration Test > Order Service
Look ma, no AWS
Code

37. Testability > Testing Locally > Integration Test > Order Service
Dependencies
injected

38. Testability > Testing Locally > Integration Test > Order Service
Look ma, no AWS
Code

39. Testability > Testing Locally > Integration Test > Order Service
Setup dependencies as
simple interface fakes

40. Testability > Testing Locally > Integration Test > Order Service
Run the unit under
test

41. Testability > Testing Locally > Integration Test > Order Service
Assert

42. 42
Local Component Test
Testability > Testing Locally

43. WHAT WE ARE KNOWN FORTestability > Testing Locally > Component Test > Order Provisioning Service
Show me
where?
Adapters
Services
Domain
Repositories
Collaborators
mock
In-memory database

44. Testability > Testing Locally > Component Test > Order Provisioning Service
Component test of
handler

45. Setup dependencies,
inputs and expected
output
Testability > Testing Locally > Component Test > Order Provisioning Service

46. Testability > Testing Locally > Component Test > Order Provisioning Service
Invoke handler with
kinesis-shaped input

47. Testability > Testing Locally > Component Test > Order Provisioning Service
Assert the handler
response

48. Reduce # of e2e tests

49. Challenges
Integration
● More “white space” - integrations
● Async
○ who is my consumer?
○ delay in feedback
● Third Parties

50. 50
Contract tests
Integration > Contract Tests

51. Problems with
automated e2e
tests
Integration > Contract Tests > Contracts
● Slow
● Easy to break
● Hard to fix
● Scales badly across teams
● Lots of set up maintenance
● $$ potentially costly

52. Integration > Contract Tests > Contracts
Mocks to the rescue?

53. Integration > Contract Tests > Contracts
Mocks to the rescue?

54. WHAT WE ARE KNOWN FOR
54
Mocks
Solved problems New problems
● Hard to keep both sides in
sync
● Fast feedback
● Few dependencies
● No dedicated environment
● Reliable
● Easy to debug
Integration > Contract Tests > Contracts

55. 55
Contract
{
“id”: 1234,
“items”:[
...
],
}
Consumer
Provider
Integration > Contract Tests > Contracts
Order API
GET /orders/1234

56. 56
Provisioning
Amazon ML
Integration > Contract Tests > Contracts
Place Order API
External
Events
Order Status API
Outside World
Internal Events
Microservice
Interface
Data, Events and
Processing
Suggestions API
Publish Event
to external
subscribers
Insights /
Predictions
Customer /
Other Event
Orders
Push API
Inside bounded context
CRM
Materialised view of
customer
Provisioning Event
External
Events
… API

57. 57
Provisioning
Amazon ML
Integration > Contract Tests > Contracts Everywhere!
Place Order API
External
Events
Order Status API
Outside World
Internal Events
Microservice
Interface
Data, Events and
Processing
Suggestions API
Publish Event
to external
subscribers
Insights /
Predictions
Customer /
Other Event
Orders
Push API
Inside bounded context
CRM
Materialised view of
customer
Provisioning Event
External
Events
… API
Contract
Contract
Contract
Contract
Contract
Contract
Contract
Contract

58. Integration > Contract Tests > Contracts

59. Integration > Contract Tests > Contracts
Mock

60. Integration > Contract Tests > Contracts
Mock
GET /orders/1234

61. Integration > Contract Tests > Contracts
Mock
{
“id”: 1234,
“items”:[
...
],
}
GET /orders/1234

62. Integration > Contract Tests > Contracts
Mock

63. Integration > Contract Tests > Contracts
Mock
GET /orders/1234

64. Integration > Contract Tests > Contracts
Mock
{
“id”: 1234,
“items”:[
...
],
}
GET /orders/1234

65. Integration > Contract Tests > Contracts
Mock ✔

66. WHAT WE ARE KNOWN FORTestability > Modularisation > Consumer Test
Adapted from https://martinfowler.com/articles/microservice-testing
Adapters
Services
Domain
Repositories
Collaborators
External
Service
Show me
where?

67. WHAT WE ARE KNOWN FORTestability > Modularisation > Provider Test
Adapted from https://martinfowler.com/articles/microservice-testing
Show me
where?
Adapters
Services
Domain
Repositories
Collaborators
mock
In-memory database

68. 68
Integration > Contract Tests > Contracts
https://martinfowler.com/bliki/TestPyramid.html
Contract
Show me
where?

69. Bug turnaround -
minutes
Contracts tests
Integration > Contract Tests > Contracts

70. Know before you
commit
Contracts tests
Integration > Contract Tests > Contracts

71. Make changes with
speed and confidence
Contracts tests
Integration > Contract Tests > Contracts

72. Deploy independently
Contracts tests
Integration > Contract Tests > Contracts

73. Better API design
(Consumer)
Contracts tests
Integration > Contract Tests > Contracts

74. 74
● Open source
● Multiple languages
○ Javascript
○ Go
○ Python
○ JVM
○ .NET
○ + more
pact.io
● HTTP contracts
● Message contracts
● … protocol agnostic!
Integration > Contract Tests > Contracts

75. 75
Contract
{
“id”: 1234,
“items”:[
...
],
}
Consumer
Provider
Integration > Contract Tests > Consumer Contract Test
Order API
GET /orders/1234

76. Integration > Contract Tests > Contracts > Fetch Order Example
Setup API Host

77. Integration > Contract Tests > Contracts > Fetch Order Example
Expectations on
Order API made clear

78. Integration > Contract Tests > Contracts > Fetch Order Example
Expectations on
Order API made clear

79. Integration > Contract Tests > Contracts > Fetch Order Example
Call the OrderService
+ Assert

80. Integration > Contract Tests > Contracts > Fetch Order Example
Test state setup

81. Integration > Contract Tests > Contracts > Fetch Order Example
Test state setup

82. ● Contract tests are faster, easier to
maintain and more reliable than e2e
tests
● Works for both synchronous and async
communication
● Use contract tests to reduce your
end-to-end test reliance
● End-to-end tests closer represent the
user experience
● Contract testing doesn’t replace
functional testing!
Summary
Integration > Contract Tests

83. Observe the Platform

84. 84
Push responsibility into Platform
Observability

85. 85
Push responsibility into Platform
Observability

86. 86
#1 Log all the things!
Observability

87. 87
#2 Use Middleware
Observability

88. Handler
Observability > Approach #2 > Middleware
{
"Records": [
{
"EventVersion": "1.0",
"EventSubscriptionArn": "somearn",
"EventSource": "aws:sns",
"Sns": {
...
}
}
]
}
{
"Success": true
}
HandlerHandlerHandler
before after
Request Response
1st middleware
2nd middleware
3rd middleware

89. Observability > Approach #2 > Middleware

90. Observability > Approach #2 > Middleware

91. Observability > Approach #2 > Middleware

92. 92
#3 Push responsibility into Platform
Observability

93. 93
Observability > Approach #1 > IT Ops Automation
Decide
Alarm
Schedule
Notify Pagerduty
Act
Isolate instance from
ASG
Offsite backup
Ship logs to Splunk
Orient
Security Analysis
Application Anomaly /
Canary Detection
Log forwarder
Policy Violation
Observe
Flow Logs
CloudTrail
CloudWatch Logs
Application Event
Configured Rule
Event
subscription

94. 94
Observability > Approach #1 > IT Ops Automation
Decide
Alarm
Act
Ship logs to Splunk
Orient
Log forwarder
Observe
CloudWatch Logs
Event
subscription

95. WHAT WE ARE KNOWN FOR
95
Examples
● Automatically subscribe log groups and forward to ___
● Allow dynamic DEBUG monitoring (runtime sampling)
● Enable platform tracing (e.g. X-ray, IOPipe, Dashbird)
● Automatically inject correlation IDs
● Anomaly detection
Observability > Approach #1 > IT Ops Automation

96. ● Invest in platform
● Structured + semantic logging is crucial
● Middleware to simplify and template out
specific shaped functions
Summary
Observability > Summary

97. Wrap it up

98. Summary
98
Summary
Problems
● Microservices + architecture has evolved
● ...and so must the pyramid!
● More diverse integrations
● More glue, cloud-native code
We’re all digital plumbers!

99. ● Use Ports + Adapters (Hexagonal
architecture) to increase modularity
● Enable better testability and test
modularity
● Mock SDK for local tests
● Component tests can help to reduce
need for e2e tests
There is no single test strategy that will
work for all cases!
Structure...
Summary > Structure

100. ● Contract tests are faster and more
reliable than E2E tests
● Use them to reduce your E2E footprint
● Contract tests enable independent
deployment
● They are not functional tests
Never forget forget the end user!
Reduce E2E...
Summary > Reduce reliance on E2E

101. ● Is the property of understanding a
system as illuminated by its outputs
● Allows us to ask questions
● Helps us discover and respond to
unknown, unknowns
It’s not monitoring, but there is significant
implementation overlap.
Observability...
Summary > Observability

102. Matt Fellows
@matthewfellows
Serverless microservices
smarter
Pact
@pact_up
@pactflow
slack.pact.io
Test , not harder