1.
Deploying large-scale, serverless
and asynchronous systems
Matt Fellows
@matthewfellows
...without integrated tests!

2.
Architecture

3.
3
Evolution
Architecture

4.
4
Release
Architecture > Evolution
MoreEvolved
Time
Monolith
SOA
Serverless
Microservices
Testing
Culture
Unit tests,
TDD, BDD
Contract
tests
Waterfall
Outsourcing
API
Testing
Scenario
Tests
Fn
Not all primitives
are invisible!
Co-evolved
practices:
XP
AgileLean
Multiple E2E
Test Env.
Chaos
Engineering
DevOps
CD
CI

5.
5
If you can’t build a good monolith you
shouldn’t be doing microservices
Architecture > Microservices

6.
Entry criteria to
make µServices
● Ability to write a modular monolith
● Strong grasp on your domain model
● Ability to design loosely coupled services
○ = state + transactions
● Ability to scale and decouple teams
● Ability to deploy continuously
● A strong “DevOps” culture
● Be comfortable with decentralised,
complex systems + uncertainty
Architecture > Microservices

7.
...the µServices
ride
● Ability to write a modular monolith
● Strong grasp on your domain model
● Ability to design loosely coupled services
○ = state + transactions
● Ability to scale and decouple teams
● Ability to deploy continuously
● A strong “DevOps” culture
● Be comfortable with decentralised,
complex systems + uncertainty
Architecture > Microservices

8.
...the µServices
ride
● Ability to write a modular monolith
● Strong grasp on your domain model
● Ability to design loosely coupled services
○ = state + transactions
● Ability to scale and decouple teams
● Ability to deploy continuously
● A strong “DevOps” culture
● Be comfortable with decentralised,
complex systems + uncertainty
Architecture > Microservices

9.
...the µServices
ride
● Ability to write a modular monolith
● Strong grasp on your domain model
● Ability to design loosely coupled services
○ = state + transactions
● Ability to scale and decouple teams
● Ability to deploy continuously
● A strong “DevOps” culture
● Be comfortable with decentralised,
complex systems + uncertainty
Architecture > Microservices

10.
WHAT WE ARE KNOWN FOR
10
Microservices
Evolution Drivers for levelling up
● Increase time-to-value
● Increase utilisation / cost
reduction
● Raise abstraction; focus on
composition
● Scale product teams
● Massively distributed,
real-time apps
● API contracts (e.g. OAS)
● Ubiquity of cloud computing
● Rise of containers
● Unreliable networking /
components
● Rise of the Service Mesh
● Improved observability
Architecture > Microservices

11.
Functions + Serverless

12.
12
Architecture > Functions + Serverless (Cloud Native)
https://aws.amazon.com/retail/data-analytics/

13.
13 https://www.slideshare.net/AmazonWebServices/new-launch-data-driven-apps-with-graphql-aws-appsync-deep-dive-mbl402-reinvent-2017
Architecture > Functions + Serverless (Cloud Native)

14.
14 https://aws.amazon.com/blogs/big-data/analyze-a-time-series-in-real-time-with-aws-lambda-amazon-kinesis-and-amazon-dynamodb-streams/
Architecture > Functions + Serverless (Cloud Native)

15.
15 https://aws.amazon.com/blogs/big-data/automating-analytic-workflows-on-aws/
Architecture > Functions + Serverless (Cloud Native)

16.
16 https://www.slideshare.net/nateware/aws-gdc2015-gameanalyticsfinal
Architecture > Functions + Serverless (Cloud Native)

17.
17
Architecture > Functions + Serverless (Cloud Native)
https://aws.amazon.com/retail/data-analytics/

18.
18
Much Compose! So speed!
Architecture > Functions + Serverless (Cloud Native)Architecture > Functions + Serverless (Cloud Native)

19.
19
BUT.
Architecture > Functions + Serverless (Cloud Native)

20.
20
If you can’t build good microservices
you shouldn’t be doing serverless
Architecture > Functions + Serverless (Cloud Native)

21.
21
If you can’t build a good Fn you
shouldn’t be doing FaaS
Architecture > Functions + Serverless (Cloud Native)

22.
22
So, what practices do we need?
Architecture > Functions + Serverless (Cloud Native)

23.
AWS Meetup Lounge

24.
24
Demo: AWS Meetup Lounge
CloudFront
API Gateway
Authentication
You!
awslounge.onegeek.com.au
Twitter Fetch
Twitter feed
Twitter Push
IoT Message
Broker
Sentiment
Analysis
Checkpoint
Amazon ML
#awsmelb

25.
Challenges w Serverless
Blasphemy!

26.
26
Vendor lock-in
Challenges > Vendor lock-in

27.
27 https://twitter.com/alambert/status/958050456101642241
Challenges > Vendor lock-in

28.
WHAT WE ARE KNOWN FOR
28
Options
● Multi-cloud = lowest
common denominator
● There is always a vendor
(even if it’s you!)
● You’ll never be completely
vendor agnostic
● Multi-cloud abstraction
● Avoid altogether
● Structure your code better
Challenges
Challenges > Vendor lock-in

29.
WHAT WE ARE KNOWN FOR
29
Options
● Multi-cloud = lowest
common denominator
● There is always a vendor
(even if it’s you!)
● You’ll never be completely
vendor agnostic
● Multi-cloud abstraction
● Avoid altogether
● Structure your code better
Challenges
Challenges > Vendor lock-in

30.
Lambda
Handler
Challenges > Vendor lock-in > Lambda Anti-pattern
{
"Records": [
{
"EventVersion": "1.0",
"EventSubscriptionArn": "somearn",
"EventSource": "aws:sns",
"Sns": {
...
}
}
]
}
{
"Success": true
}

31.
MQ Handler
WebSocket
handler
MQ Adapter WS Adapter
Azure Function
Adapter
Google Cloud Fn
Adapter
Azure Function
Handler
Google Cloud Fn
Handler
Lambda
Handler
Lambda
Adapter
Service
Repository
Adapter
(Anti-corruption layer)
Ports
Business Logic
Collaborators ...
Challenges > Vendor lock-in > Ports and Adaptors

32.
32
Challenges > Vendor lock-in > Ports and Adaptors > Sentiment Analysis Example
Twitter feed
Sentiment
Analysis
IoT Message
Broker

33.
Challenges > Vendor lock-in > Ports and Adaptors > Lambda Port

34.
Challenges > Vendor lock-in > Ports and Adaptors > SNS Adapter

35.
Challenges > Vendor lock-in > Ports and Adaptors > Lambda Sentiment Service

36.
Summary
Architecture > Microservices
● Separate protocol handling from
business logic
● Business logic shouldn’t change with
introduction of a new protocol
● Port, Adapter and Business Logic are
independently testable

37.
37
How do you test locally?
Challenges > Testing Locally

38.
WHAT WE ARE KNOWN FOR
38
Options
Challenges > Testing Locally
● Stub services (e.g. Localstack)
● Stub API (e.g. Moto)
● Multiple handlers
● Unit tests

39.
39
Challenges > Testing Locally

40.
WHAT WE ARE KNOWN FOR
40
Options
● Feature arms race
● Trustworthy?
● Integration style = slower
● All cloud providers?
● Stub services (e.g. Localstack)
● Stub API (e.g. Moto)
● Multiple handlers
● Unit tests
Challenges > Testing Locally
Challenges

41.
WHAT WE ARE KNOWN FOR
41
Writing good unit tests
● Follow test pyramid
● Stub out AWS APIs
● Test your business logic
separately from any
serverless features (e.g.
lambda handler)
● Callback pyramid of death
● Tests are hard to write
● Your business logic knows
about AWS
● You have to stub multiple
services per test
● Large methods
● Confusing control flow
Challenges > Testing Locally > Unit Test
Code Smells

42.
42
Challenges > Testing Locally > Unit Test > Twitter Fetch
Checkpoint
Twitter Fetch

43.
Challenges > Testing Locally > Unit Test > Lambda Service
Look ma, no AWS
Code

44.
Challenges > Testing Locally > Unit Test > Lambda Service
Look ma, no AWS
Code

45.
45
Challenges > Testing Locally > Unit Test > Sentiment Analysis Example
Twitter feed
Sentiment
Analysis

46.
Integration-style test
of handler
Challenges > Testing Locally > Unit Test > Lambda Handler

47.
Summary
Architecture > Microservices
● Business logic should be tested
separately from port + adapter
● Majority of tests, should be fast unit
tests
● Stub AWS Services where required
● (Lambda) Handlers can be locally
integration tested with pre-canned JSON
events

48.
48
Async - who is my consumer?
Challenges > Async

49.
49
Async - delay in feedback
Challenges > Async

50.
WHAT WE ARE KNOWN FOR
50
Options
● Local integration testing
● Cloud env. integration testing
● Contract testing
Challenges > Async

51.
WHAT WE ARE KNOWN FOR
51
Options
● Can’t run complete local
environment
● Many moving parts
● Problems with integration
testing...
● Local integration testing
● Cloud env. integration testing
● Contract testing
Challenges
Challenges > Async

52.
WHAT WE ARE KNOWN FOR
52
Options
● Can’t run complete local
environment
● Many moving parts
● Problems with integration
testing...
● Local integration testing
● Cloud env. integration testing
● Contract testing
Challenges
Challenges > Async

53.
53
Contract tests
Challenges > Async

54.
54
IoT Message
Broker
Sentiment
Analysis
Contract
{
'Positive': 0.234,
'Negative': 10.123,
'Neutral': 7.334,
'Mixed': 0
}
Consumer
Provider
Challenges > Async > Contract Tests

55.
55
CloudFront
API Gateway
Authentication
You!
awslounge.onegeek.com.au
Twitter Fetch
Twitter feed
Twitter Push
IoT Message
Broker
Sentiment
Analysis
Checkpoint
Amazon ML
#awsmelb
Challenges > Async > Contract Tests

56.
56
Authentication Twitter Fetch
Twitter feed
Twitter Push
IoT Message
Broker
Sentiment
Analysis
Checkpoint
#awsmelb
Challenges > Async > Contract Tests
Contract
Contract
Contract
Contract
Contract
Contract
Contract
Contract

57.
WHAT WE ARE KNOWN FOR
57
Problems with automated e2e tests
● Slow
● Easy to break
● Hard to fix
● Scales badly across teams
● Lots of set up maintenance
● $$ potentially costly
Challenges > Async > Contract Tests

58.
58
C P
mock
Challenges > Async > Contract Tests
Understanding Test Symmetry

59.
WHAT WE ARE KNOWN FOR
59
Test symmetry
Solved problems New problems
● Hard to keep both sides in
sync
● Fast feedback
● Few dependencies
● No dedicated environment
● Reliable
● Easy to debug
Challenges > Async > Contract Tests

60.
60
C P
mock
Challenges > Async > Contract Tests

61.
61
C P
mock
Challenges > Async > Contract Tests

62.
PPCC
mock
contract
Challenges > Async > Contract Tests

63.
Bug turnaround -
minutes
Contracts tests
Challenges > Async > Contract Tests

64.
Know before you
commit
Contracts tests
Challenges > Async > Contract Tests

65.
Make changes with
speed and confidence
Contracts tests
Challenges > Async > Contract Tests

66.
Deploy independently
Contracts tests
Challenges > Async > Contract Tests

67.
Better API design
(Consumer)
Contracts tests
Challenges > Async > Contract Tests

68.
68
● Open source
● Multiple languages
○ Javascript
○ Go
○ Python
○ JVM
○ .NET
○ + more
pact.io
● HTTP contracts
● Message contracts

69.
How to be taller
...and ride that Fn

70.
Summary
70
Summary
Problems Solutions
● Microservices+
● Create cloud and protocol
anti-corruption layer
● Mock out AWS SDK in unit tests
● Use contract tests
● Reduce reliance on end-to-end
environments
● Ad-hoc exploratory testing
● Serverless complexity >
uServices
● Testing cloud native services
● Avoiding vendor lock-in
● Handling asynchronous Fn’s

71.
Deploying large-scale, serverless
and asynchronous systems
Matt Fellows
@matthewfellows
...without integrated tests!