The document summarizes lessons learned about microservices from Vincent Kok's experience at Trello. It covers 6 areas: 1) Basics of designing minimal, stateless microservices; 2) Deployments should be automated and take less than 15 minutes; 3) Services need thorough testing, including mocking dependencies; 4) Security requires standards like OAuth 2.0 and service-to-service authentication; 5) Operations require resilience patterns like circuit breakers and request tracing; 6) Decomposing monoliths into domain-aligned services with independent teams owning each service. The overall message is that microservices impact must be understood, and services should optimize for rapid, sustainable value delivery.

1. Microservices; 5 things I wish I’d known
Vincent Kok | @vincentkok
CODEMOTION MILAN - SPECIAL EDITION
10 - 11 NOVEMBER 2017

2. Microservices; 6 things I wish I’d known
Vincent Kok | @vincentkok
CODEMOTION MILAN - SPECIAL EDITION
10 - 11 NOVEMBER 2017

3. Part-time speaker
For fun and zero profit
About me: @vincentkok
Trello
Engineering Manager on the
Trello team
Dutch
You probably heard that already ;)

4. Microservices
Everybody seems to want them. Do we
really know the impact of our choices?
Why do we want them so badly?
Microservices are messy!
https://flic.kr/p/9u5pDA

5. http://geek-and-poke.com/geekandpoke/2013/7/13/foodprints

6. Grow Fat
Code base grows. All
the things slow
down.
Age
Your code base will
become a jurassic
park introducing new
tech becomes hard
Ownership
Who is responsible
for which part and
more important: who
has the pager
Economies of
Scale
The bigger the team
the more they
interrupt each other
Monolithical issues

8. 8100
Build jobs ran last week

9. 31992
Automated tests

10. Cause of issues can be extremely hard

11. Who is having the pager?
INCIDENT RESPONSE

12. Remember, we’re not all
webscale

14. Optimise for rapid and
sustainable flow of value.
DAN NORD

16. Small
The size will be
reasonable and
manageable
Independent
lifecycle
Nothing will hold the
team back. Go as
fast as you can
Optimise for
the problem
Pick solution and
tech based on the
problem at hand
Replaceable
It is easier to replace
if there is a need for
it
The microservice promise

17. Patterns
Basics
Deployments
Testing
Security
Operations
https://flic.kr/p/9t2138
Decomposition

18. #1: Basics
https://flic.kr/p/5E9ZF

19. Creating a call-out
Watch the tutorial in the
Presentation Guidelines to learn
how to create call-outs on
screenshots within this template.

20. MINIMAL SERVICE
Health check
200 app is alive. 500 app is unhealthy,
destroy the node
Stateless*
Run as many nodes as you need
Expose a port
Only access to the service

21. DEEPCHECK
Deep check
Quickly discover if a service
fails to connect to a dependency

22. DEEPCHECK EXAMPLE
{
"avatar": {
"details": {
"avatarRepository": {
"isHealthy": true
},
"crowd": {
"isHealthy": true
},
"deadlock": {
"isHealthy": true

23. CODE & BUILDS
1 repository 1 build

24. Libraries
Feel free to use
shared libraries but
keep them loose
Config
Be aware of the
configuration
lifecycle
Schemas
Make sure that
services are resilient
to schema changes
-> Postel’s law
Testing
Test in isolation.
Keep them decoupled

25. Strict separation of config
from code.
12 FACTOR APP

26. Redeploy
Part of the service
configuration.
Configuration lifecycles
Instant change
Switches you would like to
enable/disable straight away
Rebuild
Rebuild to apply changes

27. Treat them as cattle, not
pets.
BILL BAKER

28. #2: Deployments
https://flic.kr/p/qP31Tf

29. Only one person
There is only one person in
the team that owns it
Deployment smells
Takes more then 15
mins
Setting it up should be quick
and initial deployment should
quick
Requires a ticket
A ticket for the deployment
team

30. Always deploy an empty
service into production
ME AND PROBABLY OTHERS

31. Developers in control
Artifact
What is the artifact we’re running.
We’re mostly standardising on Docker
Resources
What resources are requires: RDS,
SQS, Dynamo etc..
Compute
What EC2 instance do we want how
many of those and when to scale
Alarms
What are the alarm thresholds for this
service
Ownership
Who is owning the service
Configuration
We will be adding more icons as need
arises. Speak up if in need!

32. DECLARATIVE DEPLOYMENT
name: Confluence
description: Confluence Cloud
links:
binary:
type: docker
name: docker.atlassian.io/confluence
tag: latest
healthcheck:
uri: /wiki/internal/healthcheck
deepcheck:
uri: /wiki/internal/deepcheck
semanticCheck:

33. CONFIGURATION
config:
environmentVariables:
ASAP_AUDIENCE: "foo"
ASAP_ISSUER: "foo"
CONFLUENCE_VERTIGO_SMTP_HOST: "smtp.foo.com"
CONFLUENCE_VERTIGO_SMTP_PORT: "587"
LOG4J_EXTRA_RULES: "log4j.logger.org.hiberate=DEBU
environmentOverrides:
staging:
config:
environmentVariables:
ASAP_PUBLIC_KEY_FALLBACK_REPOSITORY_URL:

34. RESOURCES
resources:
- type: sqs
name: default
attributes:
MaxReceiveCount: 20
VisibilityTimeout: 60
scaling:
instance: m3.xlarge
min: 7

35. SIDECARS
compose:
httpfrontend:
image: nginx
tag: ‘1.13.6’
ports:
- 8080:80

36. 500
Services in production

37. #3: Testing
https://flic.kr/p/hn4K4b

38. Testing microservices

39. TESTING MONOLITHS IS EASY

40. Unit
Integration
UI

41. TESTING
Live service
Test agains a real serviceMock service
Test against a mock service

42. In process
A local implementation of
your client
Out of process
Use tools like WireMock and
MockServer
Two options

43. MOCKING SERVICES - IN PROCESS
<beans profile=“integration-test">
<bean id="attachmentService"
class=“c.a.attachment.AttachmentStub”/>
</beans>

44. MOCKING SERVICES - WIREMOCK
{
"request": {
"url": “/rest/api/content“,
"method": “POST”
"Accept": {
"matches": “application/json”
}
},
"response": {
"status": 200
}
}

45. Stable API
If it is external it already
should have a CTK so rely on
it
How to trust your mock?
Contract testing
Internal fast moving API’s an
benefit from this
Rely on monitoring
Small service, low MTTR
therefore low impact

46. Semantic Check
Automated test that runs against a
node before it will be added to the
load balancer

47. #4: Security
https://flic.kr/p/7LcF2W

48. OAuth 2.0
Grant a client access to
resources based on a newly
created set of credentials
Common standards
OpenID Connect
Identity on top of OAuth 2
OpenID
Allows identity and some
metadata only

49. How to secure a set of many
services?
SECURING SERVICES

50. ASAP
Atlassian Service Authentication Protocol

51. HOW DOES IT WORK?
Foo BarJWT

52. WHAT’S INSIDE?
Foo Bar
{
"typ": "JWT",
"kid": "foo/key1",
"alg": "RS256"
}
{
"sub": “32769:87e…”
"aud": "bar",
"nbf": 1494284564,
"iss": "foo",
"exp": 1494284624,
"iat": 1494284564,
"jti": “961253cf-ac…”
}

53. {
"kid": "foo/key1",
}
{
"sub": “32769:87e…”
"aud": "bar",
"iss": "foo"
}

54. s2sauth.bitbucket.io
AVAILABLE ON BITBUCKET

55. #5: Operations
https://flic.kr/p/npbxAm

56. 100 lbs
99% water
dehydrate 98%
Guess the weight!
https://flic.kr/p/npbxAm

57. 50
lbs

58. Uptime of a system with 30
services of 99.99?
TRANSLATING THIS TO A MICROSERVICE ARCHITECTURE

59. 2 hours
99.99 = 99.7
30

60. Failure is imminent
RESILIENCE

61. Circuit breakers
Write code with failure in
mind
Three must haves
Request tracing
Don’t spend hours debugging
Log aggregations
Stream all logs into one
place.

62. DO YOU KNOW YOUR SYSTEM?

63. CREATE INSIGHT: AGGREGATED LOGGING

64. Response times
How much time do services
spend calling other services.
Back pressure
Stop putting pressure on a
system that is in trouble
and fail fast
Fallback
How do you handle failure. A
mandatory step in the
programming model.
Circuit breakers

65. CREATE INSIGHT: CIRCUIT BREAKERS

66. Request TracingX-B3-TraceId : 1
X-B3-SpanId : 1
X-B3-TraceId : 1
X-B3-SpanId : 2
X-B3-ParentSpanId : 1
X-B3-TraceId : 1
X-B3-SpanId : 3
X-B3-ParentSpanId : 2
X-B3-TraceId : 1
X-B3-SpanId : 4
X-B3-ParentSpanId : 3

67. TRACE ID’S

68. You Build It You Run It
The team who builds it looks after it.
Ops Team
Handover your services and let them
deal with the fun. Don’t do this.

69. #6: Decomposition
https://flic.kr/p/4hAC16

70. The monolith is deprecated
MAKE A STATEMENT

71. A CONFLUENCE EXAMPLE
Core functionality
Scheduler
Attachments
Operational
Transformation
Platform Services
Front end

72. Code
Team is responsible for the codebase.Focus on
ownership
Pipeline
Team responsible for CI and Deployment
Incidents
You built it you run it

73. Decomposing core functionality
GraphQL service

74. What should you take home?
Basics
Services are cattle not pets.
Testing
Testing a monolith is “easy” think
about your service testing strategy
Deployment
Deploying a service shouldn’t take
longer then 15 minutes
Operations
You build it you run it.
Security
Think how you would like to secure
service to service communications
Focus on value
Optimise for rapid and sustainable
flow of value

75. VINCENT KOK | ENGINEERING MANAGER, TRELLO | @VINCENTKOK
Thanks!