Big Sky Dev Con 2018 Matt West & Kyle McIntrye

1. Microservices:
Drinking the Kool-Aid
Matt West Kyle McIntyre
1

2. Review of Microservice Architectures
This is everything you need to know.
2

3. 3

4. Review of Microservice Architectures
Microservice Characteristics
● Like SOA but “smaller”
● Many small services
● Versioned independently
● Deployed independently
● Scale independently
Possible Microservice Benefits
● Resilience (through clustered services)
● Zero downtime
● Horizontal scaling
● Independent scaling of components
● Ease of making changes
● Ease of uptaking new technology
● Canary testing
● Independent release cycles
“Building Microservices” - Sam Newman
4

5. The taste of the kool-aid (part 1)
● It’s all about the contract! That’s not exactly revolutionary
● Microservices mess with everything. Including testing strategies.
● Hard things become easier (releasing changes, scaling)
● Easy things become hard (e.g. multi-lingual support, referential integrity checks)
● Service boundaries don’t necessarily follow the data model.
● Some ‘scary’ stuff like a repository/db per service really isn’t a big deal. Seriously.
5

6. The taste of the kool-aid (part 2)
● Communication and state exchange between microservices gets complicated (more to follow)
● Zero-downtime upgrades: Not actually that bad once you get the hang of it
● Horizontal scalability: Not a simple yes or no question but a spectrum. Also, not free.
● Canaries are essential for both quality and velocity (more to follow)
● Polyglot stack: More compelling at the database level than at the language/runtime level
● As a developer, you’re empowered. With power comes responsibility
6

7. Deep-Dive #1
Continuous Deployment and Canary Testing….
7

8. “Monolithic” Deployment
OperationsDevelopment
Customers
Months
Support
8

9. “Monolithic” Deployment (Hotfix)
OperationsDevelopment
Customers
Minutes
Support
9

10. Continuous Deployment
OperationsDevelopment
Customers
Days/Hours
Support
10

11. Continuous Deployment - ala Microservices
Customers
Days/Hours
11

12. Continuous Deployment - ala Microservices
Customers
Days/Hours
12

13. “Monolithic” Deployment
OperationsDevelopment
Customers
Months
Support
13

14. Guiding Principles
1. You can’t “contrive” production in a
laboratory
○ It’s probably impossible or at a minimum
not worth the cost
2. Reality is our friend
○ Time doesn’t make everything “stronger”
○ Are devs thinking about production?
3. Build “crisis” reflexes
○ Do everyday what you will do in a crisis
14

15. How?
Pop Quiz Hotshot:
You MUST push a change out to
production in the next hour - what do
you do?
15

16. Continuous Deployment - ala Microservices
Customers
Days/Hours
16

17. Canary Testing
?
17

18. ● Code level
○ Easy to do
○ Basically some “if checks” and a global registry per tenant or %
○ Larger blast radius
● Binary Level
○ Harder to do
○ Utilize service routing techniques (per tenant/% based)
○ Smaller blast radius
● Example (Chat change)
○ Reverse works on this as well
Canary Testing -- It’s not that hard
18

19. Guiding Principles Played Out @ Quiq
1. You can’t “contrive” production in a
laboratory
○ It’s probably impossible or at a minimum
not worth the cost
2. Reality is our friend
○ Time doesn’t make everything “stronger”
○ Are devs thinking about production?
3. Build “crisis” reflexes
○ Do everyday what you will do in a crisis
19

20. Deep-Dive #2 - Microservice Communication
20

21. 21

22. More Protocols, More Problems
● Achieving the benefits of microservices is non-trivial
● Using both HTTP & AMQP made each more daunting
○ HTTP necessitates service discovery
○ Canary testing - must cutover & direct two traffic types
○ Resilience - Need clustered/redundant AMQP
○ Horizontal scalability - AMQP server a possible bottleneck
● AMQP worked directly against some of the hallmarks
○ Becomes a point of centralization, like a shared database
○ Polyglot stack - Not as ubiquitous as HTTP
But here’s the worst part…
We were actually afraid to horizontally scale some of our services!
22

23. 1. What if Downstream2 finishes first?
a. Even DB locking does not help
because the race condition is in
the queue itself
2. The more instances of Downstream1,
the more likely the corruption
3. We need an AMQP extension, TCP-
style message ordering, or merge
strategies, or...
User A Offline
User A Online
Downstream2Downstream1
SessionService1
23
Knock, knock: race condition (who’s there?)

24. Our Solution
● All inter-service communication over HTTP
● Store the transactionally ordered side effects in the same database that mutated records
during the transaction (e.g. in redis)
● Ordered executor in (e.g. SessionService) publishes side effect via HTTP
○ Ordered executor leverages locking mechanisms of host service’s primary DB
● Downstream1 receives side effect, pushes it into its own queue, kicks off a future &
immediately returns (still async!)
● Now we have ordered, durable downstream state updates and/or side effects!
24

25. 25

26. Some Observations
1. Achieving the benefits of microservices is challenging. Do yourself a favor and limit
the number of communication protocols
2. Don’t use producer/consumer queues for general communication
3. In the event of problems like process death, always err on the side of sending the
same message twice instead of dropping it. Idempotency is king.
26

27. Questions?
27