Many presentations on microservices offer a high-level view of the architecture; rarely do you hear what it’s like to work in such an environment. Stephen Pember shares his experience migrating from a monolith to microservices across several companies, highlighting the mistakes made along the way and offering advice.

1. Surviving In A Microservices
Environment
Steve Pember
Principal Engineer @ Toast
Software Architecture Conf, San Jose, 2019
@svpember

2. @svpember
Microservice Blog Posts & Presentations

4. <3

6. What Are Microservices?

8. @svpember
Why Choose Microservices?
• Reduce Coupling!
• Team Autonomy
• Forced Boundaries between Functional Areas
• Right Tool for the Job
• Continuous Delivery
• Smaller codebases are easier to reason about
• Easy to replace old services
• Efficient Scaling
• Can move quickly (once you’re up and running)

11. Help?

12. This talk is not a Good Talk.

13. Please let me know if I missed
anything

14. @svpember
Topics
• Is it time for Microservices?
• Infrastructure / Operations
• Architecture
• Developer Productivity / Happiness
• People Communication
• Breaking off from the Monolith
• Random

15. @svpember
Topics
• Is it time for Microservices?

16. @svpember
Is it time for Microservices?
• … is it?

18. Not before 8 Engineers.
After You have known
Defined Functional Areas.

20. You may not be ready, and
that’s OK.

21. @svpember
Topics
• Is it time for Microservices?
• Infrastructure / Operations

22. One Microservice is Easy
One Microservice Platform is
HARD

23. @svpember
Infrastructure / Operations
• Do we have Centralized Logging?

24. Centralized Log Aggregation is
Critical to Success

25. @svpember

26. @svpember

27. @svpember
Infrastructure / Operations
• Do we have Centralized Logging?
• What do we use for Metrics?

29. @svpember

31. Your Infrastructure needs to
Scale along with your Platform

32. Your Infrastructure needs to
Scale along with your Platform

33. @svpember

34. –Johnny Appleseed
“Type a quote here.”

35. @svpember
Infrastructure / Operations
• Do we have Centralized Logging?
• What do we use for Metrics?
• How are our builds performed?

36. @svpember
Infrastructure / Operations
• Do we have Centralized Logging?
• What do we use for Metrics?
• How are our builds performed?
• When are our builds performed?

37. Continuous Integration!

38. Frequent merging provides fast
feedback. Faster Iterations.

39. CI should be run on each ‘main’
branch and PR commit

40. CI should produce some
correct Artifact

41. @svpember

42. @svpember

43. @svpember
Infrastructure / Operations
• Do we have Centralized Logging?
• What do we use for Metrics?
• How are our builds performed?
• When are our builds performed?
• Can our CI keep up with the the team’s build demand?

44. @svpember

45. @svpember

46. @svpember
Infrastructure / Operations
• Do we have Centralized Logging?
• What do we use for Metrics?
• How are our builds performed?
• When are our builds performed?
• Can our CI keep up with the the team’s build demand?
• Do we have any coding conventions or quality gates?

47. @svpember

48. @svpember

49. @svpember
Infrastructure / Operations
• Do we have Centralized Logging?
• What do we use for Metrics?
• How are our builds performed?
• When are our builds performed?
• Can our CI keep up with the the team’s build demand?
• Do we have any coding conventions or quality gates?
• How are our services deployed?

50. Continuous Deployment is your
Ultimate Goal

52. A Certain level of
Operational-Readiness is
required

53. @svpember

54. @svpember

55. @svpember
Service Mesh
• Likely don’t need in a small org with few services / Essential as you grow
• Aids in resiliency, can reroute traffic around failing nodes
• App Devs focus on building services vs wiring them up
• Sidecars can handle metric, log collection and auth between services
• Easy Canary Deploys

56. @svpember
Infrastructure / Operations
• Do we have Centralized Logging?
• What do we use for Metrics?
• How are our builds performed?
• When are our builds performed?
• Can our CI keep up with the the team’s build demand?
• Do we have any coding conventions or quality gates?
• How are our services deployed?
• Do we have multiple environments? How are our environments
managed?

57. @svpember

58. Goal: standup a new
environment with a
single command

59. @svpember
Topics
• Is it time for Microservices?
• Infrastructure / Operations
• Architecture

60. @svpember
Architecture
• Do we have an overall design vision?

62. @svpember
Architecture
• Do we have an overall design vision?
• What technologies do we use? How much freedom do we have in
choosing them?

63. Right Tool for the Job!

64. @svpember

65. Be wary of new adopting new
technology

66. Boring is Good

67. @svpember
Architecture
• Do we have an overall design vision?
• What technologies do we use? How much freedom do we have in choosing
them?
• How do we share code between services?

69. Shared Code can increase
coupling and decrease
Autonomy

70. @svpember
Shared code?
• Internal Commons (e.g. a custom String Utils) - OK!
• Styling, CSS - Go for it!
• Business Logic - You’re gonna have a bad time
• Domain Objects - Never!

71. @svpember

72. @svpember
Architecture
• Do we have an overall design vision?
• What technologies do we use? How much freedom do we have in choosing
them?
• How do we share code between services?
• How do we test an individual service?

73. Integration tests are the best tests

75. @svpember

76. @svpember
Architecture
• Do we have an overall design vision?
• What technologies do we use? How much freedom do we have in choosing
them?
• How do we share code between services?
• How do we test an individual service?
• How do we test the platform as a whole?

77. @svpember
End-to-End tests are Difficult
1. Reset fixture / default state data on each service’s datastore
2. Execute test by performing some user action at the edge
3. Wait for test to complete
4. Make assertions on each service (e.g. look out for unknown side effects)
5. Repeat

78. Consider… Not doing that.

79. @svpember

80. @svpember
Architecture
• Do we have an overall design vision?
• What technologies do we use? How much freedom do we have in choosing
them?
• How do we share code between services?
• How do we test an individual service?
• How do we test the platform as a whole?
• Overall architectural style for services?

81. Are there common, repeatable,
patterns?

82. Does everyone in the org
know?

84. Asynchronous, Event Driven
Communications
+
Domain Driven Design

85. 1) Async Comms

87. Services that communicate
Synchronously are a
Distributed Monolith

88. @svpember

89. @svpember

90. Embrace Eventual Consistency

92. @svpember

93. @svpember

94. Decoupled Services allow for
team Autonomy

95. Decoupled Services scale
easily and efficiently

96. @svpember

97. @svpember

98. –Johnny Appleseed
“Type a quote here.”
Source: https://dzone.com/articles/distributed-sagas-for-microservices

99. 2) Domain Driven Design

100. Please
Read =>

101. And also
=>

102. @svpember
Some DDD concepts
• Ubiquitous Language
• Entities
• Aggregates
• Bounded Contexts
• No Direct communications across Boundaries, communicate through some
single point of contact

104. @svpember

105. One Last Thing: Always have a
way to recover

106. @svpember

107. @svpember
Architecture
• Do we have an overall design vision?
• What technologies do we use? How much freedom do we have in choosing
them?
• How do we share code between services?
• How do we test an individual service?
• How do we test the platform as a whole?
• Overall architectural style for services?
• When do we decide to create new services?

108. @svpember
New Service?
• Don’t make services ‘just because’
• Don’t make CRUD services
• Look for places where Eventual Consistency is acceptable
• A service should be thought as wrapping some functional business logic, not
the domain objects inside of it
• When new functionality is proposed: does it fall within the responsibilities of an
existing service?

109. # of services < # of developers

110. @svpember
Architecture
• Do we have an overall design vision?
• What technologies do we use? How much freedom do we have in choosing
them?
• How do we share code between services?
• How do we test an individual service?
• How do we test the platform as a whole?
• Overall architectural style for a service?
• When do we decide to create new services?
• How do we bootstrap a new service?

111. An Event Log makes this trivial

112. @svpember
Topics
• Is it time for Microservices?
• Infrastructure / Operations
• Architecture
• Developer Productivity / Happiness

113. @svpember
Developer Productivity / Happiness
• Do we have a service template?

114. git clone git@github.com:
your_org/services-template.git

115. @svpember
Developer Productivity / Happiness
• Do we have a service template?
• What does it take to run our services?

116. What’s our Developer
experience like?

117. Running the Monolith on your
laptop will eventually become
problematic

118. Running a single Microservice
is easy.
Running all of them is
impossible.

119. @svpember
Developing against other Services
1. Every team publishes a mock service along with the real service
2. Only run services you need at the time
3. On-demand Environments
4. Keep buying bigger laptops

120. @svpember

121. @svpember
Developer Productivity / Happiness
• Do we have a service template?
• What does it take to run our services?
• Do our Engineers know how our services are deployed?

122. Let App Devs App Dev

123. @svpember
Developer Productivity / Happiness
• Do we have a service template?
• What does it take to run our services?
• Do our Engineers know how our services are deployed?
• How do our Engineers know about other Services?

124. @svpember
Topics
• Is it time for Microservices?
• Infrastructure / Operations
• Architecture
• Developer Productivity / Happiness
• People Communication

125. Conway’s Law is Real

126. People are the reason for
Microservices

129. @svpember
People Communication
• How large are our teams?

130. @svpember

131. @svpember

132. @svpember

133. @svpember

134. Team larger than 4?
Break them up

135. @svpember

136. @svpember
People Communication
• How large are our teams?
• How are they structured?

137. @svpember
People Communication
• How large are our teams?
• How are they structured?
• Are there clear functional boundaries between teams?

139. @svpember
People Communication
• How large are our teams?
• How are they structured?
• Are there clear functional boundaries between teams?
• Do we encourage cross-team socializing?

141. @svpember
People Communication
• How large are our teams?
• How are they structured?
• Are there clear functional boundaries between teams?
• Do we encourage cross-team socializing?
• Do we have clear methods for engineers to share & publicize services
they’ve built?

142. @svpember
People Communication
• How large are our teams?
• How are they structured?
• Are there clear functional boundaries between teams?
• Do we encourage cross-team socializing?
• Do we have clear methods for engineers to share & publicize services they’ve
built?
• Do we add more process if something goes wrong?

143. Process is like Scar Tissue

144. @svpember
People Communication
• How large are our teams?
• How are they structured?
• Are there clear functional boundaries between teams?
• Do we encourage cross-team socializing?
• Do we have clear methods for engineers to share & publicize services they’ve
built?
• Do we add more process if something goes wrong?
• Do we have company ‘Core Values’? Do they include non-technical
skills?

145. Empathy should be your #1
Core Value

147. @svpember
Topics
• Is it time for Microservices?
• Infrastructure / Operations
• Architecture
• Developer Productivity / Happiness
• People Communication
• Breaking off from the Monolith

148. @svpember
Breaking off from the Monolith
• Do we have a process / guideline for breaking functionality out of the
monolith?

149. @svpember
Breaking off from the Monolith
• Do we have a process / guideline for breaking functionality out of the monolith?
• Do we have a process / guideline for new services to interact w/ the
monolith?

150. @svpember
Breaking off from the Monolith
• Do we have a process / guideline for breaking functionality out of the monolith?
• Do we have a process / guideline for new services to interact w/ the monolith?
• So… how much functionality have we actually split off?

151. @svpember
Breaking off from the Monolith
• Do we have a process / guideline for breaking functionality out of the monolith?
• Do we have a process / guideline for new services to interact w/ the monolith?
• So… how much functionality have we actually split off?
• How’s the monolith doing?

152. All of your dependent systems
need to scale to meet the
demands of your services

153. @svpember

154. @svpember
Slicing code from the Monolith
• Not that straightforward!
• Create programmatic boundaries around the departing code
• Measure traffic and operational load across those boundaries
• Any dependencies on original system?
• Has the monolith accounted for the backpressure from the new service?
• Roll out slowly
• How are the metrics looking?

155. The First Service will be HARD

156. Beware Being the First

157. @svpember
Topics
• Is it time for Microservices?
• Infrastructure / Operations
• Architecture
• Developer Productivity / Happiness
• People Communication
• Breaking off from the Monolith
• Random

158. @svpember
Random Advice
• API & Message versioning is a real issue

159. @svpember
Random Advice
• API & Message versioning is a real issue
• Always have more than one cluster

160. @svpember
Random Advice
• API & Message versioning is a real issue
• Always have more than one cluster
• Don’t get cute with the naming of services

161. Any idea
what these
do?

162. @svpember
Random Advice
• API & Message versioning is a real issue
• Always have more than one cluster
• Don’t get cute with the naming of services
• New Feature? Walk backwards from the End User UI

163. Be Customer Focused

164. @svpember
Random Advice
• API & Message versioning is a real issue
• Always have more than one cluster
• Don’t get cute with the naming of services
• New Feature? Walk backwards from the End User UI
• Release when a feature is ready, don’t be afraid of bugs

165. @svpember
Random Advice
• API & Message versioning is a real issue
• Always have more than one cluster
• Don’t get cute with the naming of services
• New Feature? Walk backwards from the End User UI
• Release when a feature is ready, don’t be afraid of bugs
• But don’t release Friday afternoon.

166. To Summarize:

167. Microservices are hard because
of Infrastructure and People

168. Thank you!
@svpember

169. @svpember
Images
• Factory Workers: https://www.stallardkaneassociates.com/hr-employment-law/factory-workers/
• Bad Demolition: https://media.giphy.com/media/zf3skxc4BFBqo/giphy.mp4
• Tom Hanks, Desert: http://www.trbimg.com/img-57ba8bc0/turbine/sdut-lost-in-the-desert-waiting-for-the-king-2016apr21
• Say Anything / John Cusak: http://www.glamour.com/story/happy-birthday-john-cusack-and
• Wilderness Survival Guide: https://www.kobo.com/us/en/ebook/the-wilderness-survival-guide-1
• XKCD: Code Review: https://xkcd.com/1513/
• Monolithic vs Microservices: @alvaro_sanchez
• Drago (break you): https://giphy.com/gifs/GWD5nSpiHxs3K
• Michael Jackson, Bad: https://www.imdb.com/title/tt0124288/mediaviewer/rm4118942720
• Rocky Stairs: http://pixgood.com/rocky-stairs.html
• Assembly Line: http://www.solidsmack.com/culture/humans-need-apply-new-short-film-explores-future-robots-manufacturing-automation/
• Site Reliability Engineer Shirt: https://www.sunfrog.com/102539321-160396457.html
• Watson, “Wait”: https://giphy.com/gifs/B8qMV3f7PH7nq
• Service Mesh: https://www.redhat.com/en/topics/microservices/what-is-a-service-mesh
• Trick question: https://media.giphy.com/media/ZcaSXPd7di8O4ZfsnH/giphy.mp4
• Dumpster fire: https://sentinelksmo.org/lawmakers-spark-dumpster-fire-with-tax-school-funding-bill/shutterstock_49161094/
• Work meeting: https://pxhere.com/en/photo/910324
• People in Conversation: https://preply.com/en/blog/2018/07/06/5-main-principles-of-small-talk/
• Indiana Jones: https://giphy.com/explore/indiana-jones-and-the-raiders-of-the-lost-ark
• Simpsons Mob: https://www.bobleesays.com/2017/06/18/halt-angry-fan-mob/
• Picard: https://fanfilmfactor.com/2019/03/07/the-right-captain-at-the-right-time-why-we-like-pike-discovery-editorial/
• Megaphone: https://imgflip.com/i/332n0j

170. @svpember
Links
• Eric Evans: DDD and Microservices: https://www.youtube.com/watch?
v=yPvef9R3k-M
•