Microservices: 
lessons from the trenches 
http://tiny.cc/microservices-lessons 
@MehdiKhalili
Mehdi Khalili 
@MehdiKhalili
http://giphy.com/gifs/parkour-tJLFLygAbaxYk
http://giphy.com/gifs/fail-parkour-dZeSENj3pXT6o
Microservices is 
awesome
Microservices is 
challenging
agenda 
• Introduction 
• Benefits 
• Challenges and tips 
• Resources 
• Questions
Introduction
Monolith
Monolith 
one 
repo
Monolith 
one 
language
Monolith 
one 
database
Monolith 
one 
deployment
Microservices
Microservices 
one 
repo 
per service
Microservices 
one 
language 
per service
Microservices 
one 
persistence engine 
per service
Microservices 
one 
data store 
per service
Microservices 
one 
deployment 
per service
Benefits 
AKA the cool parts!
Scalability
Decentralized 
Data Management
Loose Coupling
Fault Tolerance
Scalable Development
Lower 
Cognitive Load
Technical 
Diversification
Autonomous Teams
Continuous Delivery
Evolutionary Design
http://giphy.com/gifs/win-parkour-12GGmxlQU22fxS
http://giphy.com/gifs/parkour-hardcore-SdtFpqUnHtlbq
http://giphy.com/gifs/fail-parkour-jCi66sK0E770s
Challenges
Fault Tolerance
30 services with 99.9% SLA 
= 432 mins 
= 21.6 hours
99.9% SLA 
30 services 
one day of downtime a month
tightly coupled services
Netflix: 
assumed broken components
design for failure
circuit breaker pattern
circuit breaker
http://martinfowler.com/bliki/images/circuitBreaker/state.png
tightly coupled services
loosely coupled services
contain the failure
fallback 
and 
graceful degradation
https://github.com/Netflix/Hystrix
Hangfire 
hangfire.io
be proactive 
about failure
monitoring endpoints
synthetic monitoring
extensive logging
troubleshooting
correlation id
log aggregation
Gif-y break 
http://www.buzzfeed.com/richardhjames/are-these-the-funniest-gifs-of-all-time
http://www.buzzfeed.com/richardhjames/are-these-the-funniest-gifs-of-all-time
http://www.buzzfeed.com/richardhjames/are-these-the-funniest-gifs-of-all-time
http://www.buzzfeed.com/richardhjames/are-these-the-funniest-gifs-of-all-time
http://www.buzzfeed.com/richardhjames/are-these-the-funniest-gifs-of-all-time
http://giphy.com/gifs/fail-parkour-movement-7SqLFyxYHal6o
http://giphy.com/gifs/fail-parkour-2BuleA8z0C4uY
Decentralized Data Management
Werner Vogels: 
"No direct database access is 
allowed from outside the service, 
and there’s no data sharing 
among the services." 
http://queue.acm.org/detail.cfm?id=1142065
don’t take 
inter-database dependency
you will lose 
loose coupling
you will lose 
tech diversification
that includes BI too!
use aggregator services 
for cross-cutting logic
don’t use 
distributed transactions
CAP theorem 
http://www.allthingsdistributed.com/2007/12/eventually_consistent.html
eventual consistency
compensating actions
Loose Coupling
consumer 
provider 
consumer
loose coupling could lead to 
violation of contract
runtime failure
consumer driven contracts 
consumer 
provider 
consumer 
contract 
contract
pact 
https://github.com/realestate-com-au/pact
pactNet 
https://github.com/SEEK-Jobs/pact-net
consumer driven contracts 
consumer 
provider 
consumer 
mock contract 
mock 
contract 
<- DSL -> 
<- DSL ->
consumer 
provider 
consumer 
service libraries 
service lib 
service lib
you need the big picture
don’t draw diagrams
use logging for 
populating the big picture
correlation id 
plus 
transaction code
document your services
swagger.io 
http://petstore.swagger.wordnik.com/
swagger and curies for 
self documenting services
"_links": { 
"curies": [ 
{ 
"name": "doc", 
"href": "http://domain/docs/{rel}” 
} 
] 
}
Continuous Delivery
Netflix: 
” you code it, you'll deploy it 
and you'll be on pager-duty.”
DevOps culture
test 
automation
deployment 
automation
infrastructure 
automation
Size Matters
It’s microservices; 
not nanoservices
Arnon Rotem-Gal-Oz: 
“A nanoservice is a service whose 
overhead outweighs its utility.”
nanoservice 
kill me now 
monolith 
Yaaaay! 
Size 
pain
organized around 
Business Capabilities
Ian Cartwright: 
“There should be business and 
architecture isomorphism”
The Culture
“Our release process has 43 
steps”
“We need a DevOps team”
“Test automation might work for 
others but our system is different.”
“We need to stop obsessing 
about quality and focus on 
getting this out soon.”
Conway’s Law: 
“Organizations which design 
systems are constrained to produce 
systems which are copies of the 
communication structures of these 
organizations.”
monolithic apps thrive 
at monolithic organizations
microservices thrive 
at agile organizations
fix your organization first
Wrap up
Microservices is 
awesome
Microservices is 
NOT a silver bullet
apply it to the extent 
where benefits 
outweigh the overheads
takeaway 
Culture, culture, culture!
takeaway 
Size matters
takeaway 
Fault Tolerance
Resources
Thanks 
Q&A 
@MehdiKhalili

Microservices: lessons from the trenches

Editor's Notes

  • #5 We see something awesome and we get really excited about it and think I want to be like that. I want to do that
  • #12 One repo One programming language One database One deployment
  • #13 One repo One programming language One database One deployment
  • #14 One repo One programming language One database One deployment
  • #15 One repo One programming language One database One deployment
  • #16 One repo One programming language One database One deployment
  • #17 Each service has One repo One programming language One data store And one deployment
  • #18 Each service has One repo One programming language One data store And one deployment
  • #19 Each service has One repo One programming language One data store And one deployment
  • #20 Each service has One repo One programming language One data store And one deployment
  • #21 Each service has One repo One programming language One data store And one deployment
  • #22 Each service has One repo One programming language One data store And one deployment
  • #24 Scale the bits that require scale.
  • #25 Encapsulate data with the services No locks =&amp;gt; Scalability Ease of maintenance: 4000 sps and 1000 tables!
  • #26 allows services and consumers to evolve independently of each other. reimplement a service as long as from outside it looks and behaves the same
  • #27 It&amp;apos;s not all or nothing unlike monolith
  • #28 Monolith is one repo not as many merge conflicts not as much consistency force/requirement
  • #29 Less code =&amp;gt; easier to understand =&amp;gt; faster to run =&amp;gt; easier to maintain
  • #30 Eliminates any long-term commitment to a technology stack. Use right tool, standards and stack for every service and change it whenever you like.
  • #31 from scoping out the functionality, to architecting it, to building it, and operating it
  • #32 Each service can be deployed independently of other services - easier to deploy new versions of services frequently. Less to release each time =&amp;gt; CD =&amp;gt; all the benefits of CD
  • #33 Smaller services are easier to change and evolve and that facilitates experimentation Create a new microservice to experiment with an idea in isolation of the rest of the system. If it flies, keep it. If it doesn&amp;apos;t, kill it.
  • #36 This actually shows one of the bigger risks with microservices: when it hurts, it really hurts!!
  • #41 Akamai NetStorage
  • #43 Assume everything your system talks to is broken and write it so it can cope!
  • #44 when one of your dependencies stops responding, you need to stop calling it for a little while.
  • #45 circuit breaker
  • #48 Fire &amp; forget Pub/sub Contain the failure
  • #50 Hitting a different URL Caching Returning a canned response (null)
  • #51 circuit breaker with fallbacks and graceful degradation monitoring
  • #52 Guaranteed fire &amp; forget background processing
  • #53 Don&amp;apos;t wait for it to fail before you fix it
  • #54 to expose the error count, response times, up time, transaction per second
  • #55 simulate an action or path that a customer or end-user would take. continuously monitor functionality, availability, and response time measures
  • #56 Helps with troubleshooting and also with health monitoring
  • #57 We were getting exception on an endpoint because it was missing a param Where did the traffic came in from?
  • #70 you’ll lose most of microservices benefits
  • #73 We changed a stored procedure’s param and it broke BI
  • #75 In a monolithic app you have transactions You will have slower transactions, and might create system wide locking!
  • #81 We were introducing soft deletes and we didn’t know what would be impacted
  • #84 Pact: DSL, generates mock and plays back the request for the provider
  • #86 Consumer codes against a mock and provider complies with contracts Unit testing
  • #87 circuit breaker Fallback graceful degradation caching
  • #88 so when you want to change the behavior of something you know what will be impacted
  • #90 Extensive logging Correlation id Transaction code Log aggregation
  • #91 Extensive logging Correlation id Transaction code Log aggregation
  • #96 It will be resisted by operations and change management Resistance by ops
  • #106 And you should have a bounded context representing each capability keep to your bounded context
  • #107 A business person should be able to look at a high level map of the architecture and see the business represented there and similarly as technologists we should be able to look at the business and see the architecture represented there.
  • #108 The microservices culture
  • #109 Explain what’s wrong with this
  • #115 Not organizations doing agile; but agile organization
  • #121 Micro management Long release process Hard to sell unit testing