Recommended
Recommended
More Related Content
Similar to Design & Implementation of Microservices - James Lewis
Similar to Design & Implementation of Microservices - James Lewis (20)
More from JAXLondon_Conference
More from JAXLondon_Conference (20)
Recently uploaded
Recently uploaded (20)
Design & Implementation of Microservices - James Lewis
- 3. Schedule • 09:00 - 10:20 Session 1 • 10:20 - 10:35 Break • 10:35 - 12:00 Session 2 • 12:00 - 13:00 Lunch • 13:00 - 14:20 Session 3 • 14:20 - 14:35 Break • 14:35 - 16:00 Session 4
- 4. 09:00 - 10:20 Session 1 • Introduction • Why Services • Principles & Constraints • Evolutionary Architecture
- 5. 10:35 - 12:00 Session 2 • Integration
- 6. 13:00 - 14:20 Session 3 • Splitting Services • CAP • Testing & CDCs • Operational Concerns
- 7. 14:35 - 16:00 Session 4 • Deployment • Conway’s Law • Conclusions
- 10. Why Services?
- 11. or: are we building systems that are too big?
- 12. 90% of the TCO of an application is incurred post launch
- 14. IT projectrelated losses are an embarrassment for the industry fund backers – AustralianSuper, Cbus, HOSTPLUS, HESTA and MTAA Super – which pride themselves on low fees and improving member services. Illustration: Karl Hilzinger A group of industry superannuation funds has revealed in accounts lodged with the Australian Securities and Investments Commission that the cost of implementing a key IT project has blown out by another $43 million. This means that a project that started in 2008 and was meant to be completed by 2010 will cost super fund members at least $250 million and will be delivered at least four years late. Superpartners, a super administration company owned by five industry retirement schemes, posted a $7.4 million loss on revenues of $257 million for the 12 months ended June 30, after being forced to take a $20.4 million impairment Superpartners’ botched IT project costs industry super funds millions Published 26 November 2013 01:17, Updated 27 November 2013 07:46 Sally Patten
- 15. we have to rewrite entire ecosystems every few years
- 16. we have to rewrite entire ecosystems every few years this doesn’t make many CFO’s happy
- 17. real life 1
- 21. Retail Site Departure Control 48 Cores 256 GB RAM (NUMA) ~ $1 x 106 per machine
- 22. 21
- 23. Airline Tightly coupled “Golden Hammer Syndrome” Single point of failure Expensive to scale High operational cost High cost of failure
- 24. 23
- 29. We can organise services along organisational boundaries
- 30. We can organise services along organisational boundaries
- 31. V1 By sub-divinding our systems, we can speed the release of new features
- 32. V2 By sub-divinding our systems, we can speed the release of new features
- 33. It allows us different options in terms of scaling
- 34. It allows us different options in terms of scaling
- 36. Go JVM NodeJS Ruby and we can use different tools and tech
- 37. Go NodeJS Ruby Clojure! and we can use different tools and tech
- 38. Why now?
- 41. standing on the shoulders of giants
- 43. Engineering for: reliability and availability Devops circuit breakers, health checks, status pages
- 44. Strategic Design Bounded Contexts “town planning”
- 46. infrastructure automation patterns for push button deployments the lean software value stream
- 48. Summary We understand more about building reliable distributed systems cloud compute and programmable infrastructure has matured organisations need to adapt and change quickly to survive we spend too much money on building monoliths
- 49. plus… it’s cool
- 57. 45 building software is not like building a house
- 58. 46 its much more like how towns evolve
- 59. 47 evolutionary architecture is in the gaps
- 60. 48 emergent design is within the zones
- 61. 49 ALLOW BUSINESS CAPABILITIES TO EVOLVE HAVE A ROUGH IDEA ABOUT WHAT YOU WANT TO BUILD, AND DEFER DECISIONS UNTIL YOU KNOW MORE
- 62. Summary do just enough up front evolutionary architecture is in the gaps emergent design is in the boxes
- 63. Bounded Contexts
- 64. What makes a good service?
- 65. High Cohesion
- 66. Loose Coupling
- 67. The Trifle
- 78. Bounded Context
- 79. "The delimited applicability of a particular model. BOUNDING CONTEXTS gives team members a clear and shared understanding of what has to be consistent and what can develop independently."
- 80. A specific responsibility enforced by explicit boundaries http://www.sapiensworks.com/blog/post/ 2012/04/17/DDD-The-Bounded-Context- Explained.aspx
- 88. Add to cart
- 90. Add to cart Checkout View Latest Releases
- 91. Add to cart Checkout View Latest Releases Search
- 92. Add to cart Checkout View Latest Releases Search Listen To Previews
- 93. Add to cart Checkout View Latest Releases Search Listen To Previews
- 94. Add to cart Checkout View Latest Releases Search Listen To Previews Shopping Cart Catalog Music Library
- 95. Hexagonal Architectures 66 bounded contexts all the way down (and back up again)
- 99. Object James’ conjecture “objects should be no bigger than my head”
- 100. Object
- 104. as we chunk up domains of abstraction, each domain should be small enough to fit in my head
- 106. you want to maximise the degrees of freedom of your system
- 107. to build systems is to make trade-offs
- 108. to build systems is to make trade-offs throughput vs cost
- 109. to build systems is to make trade-offs throughput vs cost
- 110. to build systems is to make trade-offs throughput vs cost portability vs deployability
- 111. to build systems is to make trade-offs throughput vs cost portability vs deployability
- 112. to build systems is to make trade-offs throughput vs cost portability vs deployability replacability vs maintainability
- 113. to build systems is to make trade-offs throughput vs cost portability vs deployability replacability vs maintainability evolutionary architecture and emergent design are approaches that maximise flex
- 114. The idea of architecture principles is to try and balance these tradeoffs
- 115. to try and balance short term gain with longer term strategic goals The idea of architecture principles is to try and balance these tradeoffs
- 116. to try and balance short term gain with longer term strategic goals The idea of architecture principles is to try and balance these tradeoffs
- 117. to try and balance short term gain with longer term strategic goals Where trade offs have to be made they should be done so visibility and consciously The idea of architecture principles is to try and balance these tradeoffs
- 118. to try and balance short term gain with longer term strategic goals Where trade offs have to be made they should be done so visibility and consciously The idea of architecture principles is to try and balance these tradeoffs
- 119. to try and balance short term gain with longer term strategic goals Where trade offs have to be made they should be done so visibility and consciously They should move you towards a state where the tradeoffs don’t happen so often or have such large impact The idea of architecture principles is to try and balance these tradeoffs
- 120. to try and balance short term gain with longer term strategic goals Where trade offs have to be made they should be done so visibility and consciously They should move you towards a state where the tradeoffs don’t happen so often or have such large impact The idea of architecture principles is to try and balance these tradeoffs They should be driven by the goals of the business
- 121. to try and balance short term gain with longer term strategic goals Where trade offs have to be made they should be done so visibility and consciously They should move you towards a state where the tradeoffs don’t happen so often or have such large impact The idea of architecture principles is to try and balance these tradeoffs They should be driven by the goals of the business
- 122. to try and balance short term gain with longer term strategic goals Where trade offs have to be made they should be done so visibility and consciously They should move you towards a state where the tradeoffs don’t happen so often or have such large impact The idea of architecture principles is to try and balance these tradeoffs They should be driven by the goals of the business for the next 18-24 months
- 123. to try and balance short term gain with longer term strategic goals Where trade offs have to be made they should be done so visibility and consciously They should move you towards a state where the tradeoffs don’t happen so often or have such large impact The idea of architecture principles is to try and balance these tradeoffs They should be driven by the goals of the business for the next 18-24 months any longer and, well, holographic watches
- 129. Heroku’s 12 factors are a mixture of principles and constraints http://www.12factor.net/
- 131. Integration
- 132. or…
- 133. Avoiding Babel
- 134. Two Key Attributes Of A Good Service
- 135. 1. High Cohesion 2. Loose Coupling
- 136. 1. High Cohesion 2. Loose Coupling
- 137. © 2013 Electronic Arts Inc.
- 139. An Evolutionary View Data Oriented Procedure Oriented Document Oriented Resource Oriented
- 140. Databases
- 141. DB MusikShopMono
- 145. DB Schema Recomendation ServiceMusicShopMono No loose coupling!
- 147. which is ok until...
- 148. HR UI "Middleware DB" ? ? ? Data Warehouse ? canned reports cubes / ad-hoc UIUI UI Finance UI Views of External Data Read only data Read only data Direct db access Direct db access Direct db access Direct db access Direct db access Direct db access Direct db access Direct db access SSO UI / Service AD Direct db access Direct db access which is ok until... and yes, this is a real world example...
- 149. HR UI "Middleware DB" ? ? ? Data Warehouse ? canned reports cubes / ad-hoc UIUI UI Finance UI Views of External Data Read only data Read only data Direct db access Direct db access Direct db access Direct db access Direct db access Direct db access Direct db access Direct db access SSO UI / Service AD Direct db access Direct db access changing anything is really really hard
- 150. HR UI "Middleware DB" ? ? ? Data Warehouse ? canned reports cubes / ad-hoc UIUI UI Finance UI Views of external Data Read only external data Read only external dataDirect db access Direct db access Direct db access Direct db access Direct db access Direct db access Direct db access Direct db access SSO UI / Service AD Direct db access Direct db access ? different types of data are smeared about
- 151. HR UI "Middleware DB" ? ? ? Data Warehouse ? canned reports cubes / ad-hoc UIUI UI Finance UI Views of external Data Read only external data Read only external dataDirect db access Direct db access Direct db access Direct db access Direct db access Direct db access Direct db access Direct db access SSO UI / Service AD Direct db access Direct db access ?
- 152. systems like this are brittle HR UI "Middleware DB" ? ? ? Data Warehouse ? canned reports cubes / ad-hoc UIUI UI Finance UI Views of external Data Read only external data Read only external dataDirect db access Direct db access Direct db access Direct db access Direct db access Direct db access Direct db access Direct db access SSO UI / Service AD Direct db access Direct db access ?
- 153. systems like this are brittle difficult to reason aboutHR UI "Middleware DB" ? ? ? Data Warehouse ? canned reports cubes / ad-hoc UIUI UI Finance UI Views of external Data Read only external data Read only external dataDirect db access Direct db access Direct db access Direct db access Direct db access Direct db access Direct db access Direct db access SSO UI / Service AD Direct db access Direct db access ?
- 154. systems like this are brittle difficult to reason about difficult to change HR UI "Middleware DB" ? ? ? Data Warehouse ? canned reports cubes / ad-hoc UIUI UI Finance UI Views of external Data Read only external data Read only external dataDirect db access Direct db access Direct db access Direct db access Direct db access Direct db access Direct db access Direct db access SSO UI / Service AD Direct db access Direct db access ?
- 155. systems like this are brittle difficult to reason about difficult to change difficult to maintain HR UI "Middleware DB" ? ? ? Data Warehouse ? canned reports cubes / ad-hoc UIUI UI Finance UI Views of external Data Read only external data Read only external dataDirect db access Direct db access Direct db access Direct db access Direct db access Direct db access Direct db access Direct db access SSO UI / Service AD Direct db access Direct db access ?
- 156. Next!
- 157. RPC
- 158. method calls across a process boundary
- 159. consider a client invoking the createUser endpoint
- 160. consider a client invoking the createUser endpoint
- 161. createUser(id, firstName, lastName, address) consider a client invoking the createUser endpoint
- 162. other clients can use the same call as the first createUser(id, firstName, lastName, address)
- 163. other clients can use the same call as the first createUser(id, firstName, lastName, address) so far so good
- 164. but what happens when you want to change how one of the clients calls your service?
- 165. but what happens when you want to change how one of the clients calls your service? maybe I don’t want to use first name and last name anymore
- 166. I want to use the ‘fullname’ createUser(id, firstName, lastName, address) createUserByFullname( id, fullName, address)
- 167. or I want to specify address individually createUser(id, firstName, lastName, address) createUserByFullname( id, fullName, address) createUserByFullnameAnd Address( id, fullName, street1, street2, zipcode)
- 168. one of two things tends to happen with systems of this type
- 169. one of two things tends to happen with systems of this type 1. you end up with very long service definitions
- 170. one of two things tends to happen with systems of this type 1. you end up with very long service definitions 2. coordination of changes to clients becomes difficult
- 171. 1. specifications quickly become very very long and a nightmare to maintain
- 172. 1. specifications quickly become very very long and a nightmare to maintain createUserWithFullname(...)
- 173. 1. specifications quickly become very very long and a nightmare to maintain createUserWithFullname(...) createUser(...)
- 174. 1. specifications quickly become very very long and a nightmare to maintain createUserWithFullname(...) createUser(...) createUserWithFullnameAndAddress (...)
- 175. 1. specifications quickly become very very long and a nightmare to maintain createUserWithFullname(...) createUser(...) createUserWithFullnameAndAddress (...) createUserWithAddress(...)
- 176. 1. specifications quickly become very very long and a nightmare to maintain createUserWithFullname(...) createUser(...) createUserWithFullnameAndAddress (...) createUserWithAddress(...) every time I want to change some logic, I have to change every method call
- 177. 2. you have to coordinate the release cycles of your clients createUser(id, firstName, lastName, address)
- 178. createUser(id, firstName, lastName, address) createUser(id, fullname, address) @deprecated 2. you have to coordinate the release cycles of your clients
- 179. createUser(id, firstName, lastName, address) createUser(id, fullname, address) @deprecated and if you have many clients, thats no easy task 2. you have to coordinate the release cycles of your clients
- 180. A Brief Aside…
- 181. Who can tell me about RFC 761?
- 183. Postel’s Law: “Be liberal in what you do, conservative in what you expect”
- 184. Practical impact of this - only bind to what you need to - this way you minimise breaking service consumption
- 185. Service B Service A Shared Lib v1 Shared Lib v1
- 186. Service B Service A Shared Lib v1 Shared Lib v1
- 187. Service B Service A Shared Lib v1 Shared Lib v1
- 188. Service B Service A Shared Lib v1 Shared Lib v1 Shared Lib v2
- 189. Service B Service A Shared Lib v1 Shared Lib v1 Shared Lib v2 Beware of shared serialization protocols
- 190. Service B Service A Shared Lib v1 Shared Lib v1 Shared Lib v2 Beware of shared serialization protocols WSDL-binding
- 191. Service B Service A Shared Lib v1 Shared Lib v1 Shared Lib v2 Beware of shared serialization protocols WSDL-binding JAXB
- 192. Service B Service A Shared Lib v1 Shared Lib v1 Shared Lib v2 Beware of shared serialization protocols WSDL-binding JAXB Java Serialization
- 193. Messaging
- 194. (AMC / Associated Press)
- 195. a bit like going back to the 50’s enterprise (AMC / Associated Press)
- 196. a bit like going back to the 50’s enterprise except without the smoking and the rampant misogyny (AMC / Associated Press)
- 197. back in the day, if you wanted to book a holiday, you didn’t go onto your corporate intranet to do it right?
- 198. back in the day, if you wanted to book a holiday, you didn’t go onto your corporate intranet to do it right? you went to the cupboard
- 199. back in the day, if you wanted to book a holiday, you didn’t go onto your corporate intranet to do it right? you went to the cupboard and you pulled out one of these
- 200. back in the day, if you wanted to book a holiday, you didn’t go onto your corporate intranet to do it right? you went to the cupboard and you pulled out one of these and you filled it in
- 201. james’ holiday request form and then you sent it to the HR department
- 202. james’ holiday request form and then you sent it to the HR department where it was processed, and eventually you got another envelope back containing the approval
- 203. and messaging is a bit like that
- 204. and messaging is a bit like that asynchronous
- 205. and messaging is a bit like that asynchronous after all, you wouldn’t want to block waiting for internal mail right?
- 206. incidentally, I wasn’t actually there in the 50’s. I just have this on good authority and messaging is a bit like that asynchronous after all, you wouldn’t want to block waiting for internal mail right?
- 207. generally you create a message composed of a document
- 208. and you push it onto some kind of queue
- 209. and you push it onto some kind of queue
- 210. systems interested in your documents can pop those documents and act on them
- 211. systems interested in your documents can pop those documents and act on them
- 212. and return them should that be the semantics of the exchange
- 214. the documents allowed additive changes to be made without breaking existing clients
- 215. the documents allowed additive changes to be made without breaking existing clients If you want to add a field, you can do so as long as clients are late bound to the documents
- 216. the documents allowed additive changes to be made without breaking existing clients If you want to add a field, you can do so as long as clients are late bound to the documents and if you want to rename something, you can do that easily too (add another one with the same name)
- 217. and the asynchronous nature decouples the applications from each other
- 218. and the asynchronous nature decouples the applications from each other
- 219. and the asynchronous nature decouples the applications from each other you can change this
- 220. and the asynchronous nature decouples the applications from each other you can change this without breaking this
- 221. of course, there is a teensy bit more to it than that...
- 222. Channel Adapter Channel Datatype Channel Dead Letter Channel Guaranteed Delivery ! Invalid Message Channel Message Bus Messaging Bridge Publish Subscribe Channel Aggregator Content Based Router Composed Message Message FIlter Message Router Recipient List Process Manager Splitter Routing SlipResequencer Competing Consumers Message Endpoint Durable Subscriber Event-Driven Consumer Message Dispatcher ? Selective Consumer Service Activator Polling Consumer Transactional Client Messaging Gateway quite a lot more to be perfectly frank
- 223. Getting async comms right can be hard!
- 224. And can require the dreaded ‘middleware’
- 225. The ESB!
- 227. Next!
- 228. Let’s take a look at the world’s most successful, distributed, scalable computing system for some tips
- 232. Client cache Proxy cache CDN Infrastructure caches Reverse proxy cache
- 233. HTTP -> REST!
- 238. http://martinfowler.com/articles/richardsonMaturityModel.html level 2: handle similar situations the same way
- 240. http://martinfowler.com/articles/richardsonMaturityModel.html level 3: application protocol discoverable by clients
- 241. scottwshaw / gist:ee875a6018e1676b36ad Last active 22 minutes ago 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 var jsonpath = require('JSONPath'); var version1 = {"results": {"link": {"uri": "http://musikshop.com/search/artist=the%20brakes", "rel": "self"}, "releases": [{"release": "Tale of Two Cities", "year": "2008", "link": {"uri": "http://musikshop.com/api/cmVsZWFzZXMvMQ==", "rel": "http://musikshop.com/rels/release"}}, {"release": "The Brakes", "year": "2006", "link": {"uri": "http://musikshop.com/api/cmVsZWFzZXMvMg==", "rel": "http://musikshop.com/rels/release"}}]}}; jsonpath.eval(version1, "$..[?(@.rel=='http://musikshop.com/rels/release')]"); gistfile1.json A Document with Hypermedia Links
- 245. Basket!
- 246. What if consumers of a service could act in the same way?
- 247. scottwshaw / gist:ee875a6018e1676b36ad Last active 22 minutes ago 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 var jsonpath = require('JSONPath'); var version1 = {"results": {"link": {"uri": "http://musikshop.com/search/artist=the%20brakes", "rel": "self"}, "releases": [{"release": "Tale of Two Cities", "year": "2008", "link": {"uri": "http://musikshop.com/api/cmVsZWFzZXMvMQ==", "rel": "http://musikshop.com/rels/release"}}, {"release": "The Brakes", "year": "2006", "link": {"uri": "http://musikshop.com/api/cmVsZWFzZXMvMg==", "rel": "http://musikshop.com/rels/release"}}]}}; jsonpath.eval(version1, "$..[?(@.rel=='http://musikshop.com/rels/release')]"); gistfile1.json A Document with Hypermedia Links
- 248. A non-breaking change to the service contract scottwshaw / gist:d531a3f35de01ea45644 Last active 20 minutes ago 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 var jsonpath = require('JSONPath'); var version2 = {"results": { "link": {"uri": "http://musikshop.com/search/artist=the%20brakes", "rel": "http://musikshop.com/rels/self"}, "fullLengthReleases": [{"release": "Tale of Two Cities", "year": "2008", "link": {"uri": "http://musikshop.com/api/cmVsZWFzZXMvMQ==", "rel": "http://musikshop.com/rels/release"}}], "EPReleases": [{"release": "The Brakes", "year": "2006", "link": {"uri": "http://musikshop.com/api/cmVsZWFzZXMvMg==", "rel": "http://musikshop.com/rels/release"}}]}} jsonpath.eval(version2, "$..[?(@.rel=='http://musikshop.com/rels/release')]"); gistfile1.json
- 249. In practice most people end up here Aim high!
- 251. Summary • REST over HTTP is best - aim high! • Messaging is cool, but can be hard • Whatever you do, think lazy binding!
- 252. How To Split Services
- 254. Add to cart
- 255. Add to cart Checkout
- 256. Add to cart Checkout View Latest Releases
- 257. Add to cart Checkout View Latest Releases Search
- 258. Add to cart Checkout View Latest Releases Search Listen To Previews
- 259. Add to cart Checkout View Latest Releases Search Listen To Previews
- 260. Add to cart Checkout View Latest Releases Search Listen To Previews Shopping Cart Catalog Music Library
- 263. MusikShopMono
- 265. That was the easy bit
- 266. Enter the DB
- 268. DB MusikShopMono
- 279. Catalog Finance Line Items Ledger MusikShopMono
- 280. Catalog Finance Line Items Ledger MusikShopMono
- 281. Catalog Finance Line Items Ledger MusikShopMono
- 282. Catalog Finance Line Items Ledger MusikShopMono
- 283. Catalog Finance Line Items Ledger MusikShopMono
- 286. MusikShop System Line Items Finance ServiceCatalog Service
- 287. MusikShop System Line Items Ledger Finance ServiceCatalog Service
- 289. Catalog Finance Warehouse Country Codes Country Codes Country Codes MusikShopMono
- 292. reify
- 298. Catalog Warehouse Item MusikShop Bee Gees Hits | $4.99 | 45
- 299. Catalog Warehouse Item MusikShop Bee Gees Hits | $4.99 | 45
- 300. Catalog Warehouse Item MusikShop Bee Gees Hits | $4.99 | 45
- 301. Catalog Warehouse Catalog Item MusikShop Stock Levels
- 303. Do some up front thinking. Just enough!
- 304. Do some up front thinking. Just enough!
- 305. Do some up front thinking. Just enough!
- 306. DB Do some up front thinking. Just enough!
- 307. DB Cost Of Change Do some up front thinking. Just enough!
- 308. @samnewman Summary • Split around bounded contexts • Make small, incremental changes • Split inside the process boundary before splitting out services • Start coarse-grained
- 309. CAP Theory
- 311. What is CAP theory?
- 312. It is impossible for a distributed computer system to simultaneously provide all three of the following guarantees: http://en.wikipedia.org/wiki/CAP_theorem • Consistency (all nodes see the same data at the same time) • Availability (a guarantee that every request receives a response about whether it was successful or failed) • Partition tolerance (the system continues to operate despite arbitrary message loss or failure of part of the system)
- 313. Partition Tolerance The system continues to operate despite arbitrary message loss or failure of part of the system Typically, we need this - so end up trading off the other two
- 314. Node 1 Inventory Service Master DB dc1 Node 2 Slave DB dc2 Load Balancer A standard setup - slave DB for backup purposes
- 315. Node 1 Inventory Service Master DB dc1 Node 2 Slave DB dc2 Load Balancer A standard setup - slave DB for backup purposes
- 316. Option 1: Keep Node 2 serving traffic Node 1 Master DB Node 2 Slave DB Load Balancer Inventory Service
- 317. Option 1: Keep Node 2 serving traffic Data is potentially stale, but, we keep Node 2 up Node 1 Master DB Node 2 Slave DB Load Balancer Inventory Service
- 318. Option 1: Keep Node 2 serving traffic Data is potentially stale, but, we keep Node 2 up We have sacrificed consistency for availability Node 1 Master DB Node 2 Slave DB Load Balancer Inventory Service
- 319. Option 2: Remove Node 2 from service Node 1 Master DB Node 2 Slave DB Load Balancer Inventory Service
- 320. Option 2: Remove Node 2 from service Node 1 Master DB Node 2 Slave DB Load BalancerNow we have had to degrade availability to ensure consistency Inventory Service
- 321. Which is right?
- 322. What about sacrificing Partition Tolerance?
- 323. Node 1 Inventory Service Master DB dc1 Node 2 Slave DB dc2 Load Balancer
- 324. Node 1 Inventory Service Master DB dc1 Node 2 Slave DB dc2 Load Balancer
- 325. So in general, we talk about CP or AP systems
- 327. Node 1 Node 1 Catalog Service Web Shop Node 1 & 2 will have the same catalog ‘eventually’ Each node pulls back a copy of the catalog, and caches it for speed reasons
- 328. Node 1 Node 1 Catalog Service Web Shop ttl: 5 mins 12:00 Node 1 & 2 will have the same catalog ‘eventually’ Each node pulls back a copy of the catalog, and caches it for speed reasons
- 329. Node 1 Node 1 Catalog Service Web Shop ttl: 5 mins 12:00 Update12:02 Node 1 & 2 will have the same catalog ‘eventually’ Each node pulls back a copy of the catalog, and caches it for speed reasons
- 330. Node 1 Node 1 Catalog Service Web Shop ttl: 5 mins 12:00 Update12:02 ttl: 5 mins 12:03 Node 1 & 2 will have the same catalog ‘eventually’ Each node pulls back a copy of the catalog, and caches it for speed reasons
- 332. You can’t beat CAP Theory
- 335. You entire system doesn’t need to be CP or AP!
- 336. Catalog Web Shop Payment ttl: 5 mins Customer
- 337. Catalog Web Shop Payment ttl: 5 mins Customer
- 338. Catalog Web Shop Payment ttl: 5 mins Customer
- 339. Catalog Web Shop Payment ttl: 5 mins Customer
- 340. Catalog Web Shop Payment ttl: 5 mins Customer
- 341. Web Shop
- 342. Web Shop Web Shop Web Shop Web Shop
- 343. Web Shop Web Shop Web Shop Master DB Web Shop
- 344. Web Shop Web Shop Web Shop Master DB Web Shop Slave Slave
- 345. Web Shop Web Shop Web Shop Master DB Web Shop Slave Slave
- 346. Consistency = locks locks in distributed systems are hard and they are the enemy of scaling
- 348. Transaction Club • The first rule is…don’t! • If you really, really, really have to, consider merging services first
- 349. Summary • Understand if consistency or availability is important - and this is normally a business decision! • It isn’t all or nothing • Avoid distributed transactions if you can
- 353. 59
- 354. 59
- 355. 59 DB
- 356. 60 Small Medium Large TEST PYRAMID Increasing Scope More Confidence Faster! Better Isolation
- 368. 66 Small Medium Large Build Tests Tests Source Control Faster Feedback Binary Artifact(s) … BUILD PIPELINE
- 369. 66 Small Medium Large Build Tests Tests Source Control Faster Feedback Binary Artifact(s) … BUILD PIPELINE
- 370. 67 DB
- 371. 67 DB
- 372. 68 S/M TestsBuild Large Tests UAT Prod
- 373. 68 S/M TestsBuild Large Tests UAT Prod DB Machine CI Node Large Tests Environment Large Tests
- 374. 69 S/M TestsBuild Large Tests UAT Prod DB Machine UAT Environment Machine
- 375. 70 S/M TestsBuild Large Tests UAT Prod Master DB Machine Production Environment Machine Machine Machine Slave DB
- 376. 71 S/M TestsBuild Large Tests UAT Prod More Production Like
- 377. 71 S/M TestsBuild Large Tests UAT ProdLarge Tests More Production Like
- 378. 71 S/M TestsBuild Large Tests UAT ProdLarge Tests Faster Feedback More Production Like
- 379. 72 Customer Service
- 380. 72 Customer Service S/M TestsBuild Large Tests
- 381. 72 Customer Service S/M TestsBuild Large Tests
- 382. 72 Customer Service S/M TestsBuild Large Tests
- 389. 76 S/M TestsBuild Large Tests
- 390. 76 S/M TestsBuild Large Tests Customer Service Fulfilment Service Stub
- 395. 78 S/M TestsBuild Large Tests Customer Service Customer Service v1 Web Shop v1 Production
- 396. 78 S/M TestsBuild Large Tests Integration Test Customer Service Customer Service v1 Web Shop v1 Production
- 397. 78 S/M TestsBuild Large Tests Integration Test Customer Service Customer Service v1 Web Shop v1 Production Customer Service v2 Web Shop v1 Integration Test
- 399. 79 Customer Service v1 Web Shop v1 Production S/M TestsBuild Large Tests Integration Test Customer Service Customer Service v2
- 400. 79 S/M TestsBuild Large TestsWeb Shop Customer Service v1 Web Shop v1 Production S/M TestsBuild Large Tests Integration Test Customer Service Customer Service v2
- 401. 79 S/M TestsBuild Large TestsWeb Shop Customer Service v1 Web Shop v1 Production S/M TestsBuild Large Tests Integration Test Customer Service Customer Service v2 Web Shop v2
- 402. 79 S/M TestsBuild Large TestsWeb Shop Customer Service v1 Web Shop v1 Production S/M TestsBuild Large Tests Integration Test Customer Service Customer Service v2 Web Shop v2 ???
- 403. 80 S/M TestsBuild Large TestsWeb Shop S/M TestsBuild Large Tests Customer Service
- 404. 80 S/M TestsBuild Large TestsWeb Shop S/M TestsBuild Large Tests Customer Service Integration Test
- 405. 80 S/M TestsBuild Large TestsWeb Shop S/M TestsBuild Large Tests Customer Service Integration Test S/M TestsBuild Large Tests Invoice Service
- 406. 80 S/M TestsBuild Large TestsWeb Shop S/M TestsBuild Large Tests Customer Service Integration Test S/M TestsBuild Large Tests Invoice Service S/M TestsBuild Large TestsBasket
- 407. 80 S/M TestsBuild Large TestsWeb Shop S/M TestsBuild Large Tests Customer Service Integration Test S/M TestsBuild Large Tests Invoice Service S/M TestsBuild Large TestsBasket S/M TestsBuild Large TestsFulfilment
- 408. 81
- 409. 81 Browsers
- 410. 81 Timing Browsers
- 415. 82
- 416. 82 Integration Test
- 422. 84 Show Tangle
- 423. 85 Golden Rule: Get good at releasing services independently
- 424. 86 SO NO INTEGRATION TESTS?
- 425. 87
- 426. 87
- 437. 90
- 445. 92 S/M TestsBuild Large Tests UAT ProdLarge Tests Faster Feedback More Production Like
- 446. 92 S/M TestsBuild Large Tests UAT ProdLarge Tests Faster Feedback More Production Like
- 447. 92 S/M TestsBuild Large Tests UAT ProdLarge Tests Faster Feedback More Production Like
- 448. 93 DB Machine CI Node Large Tests Environment S/M TestsBuild Large Tests UAT ProdLarge TestsLarge Tests
- 449. 93 DB Machine CI Node Large Tests Environment DB Machine UAT Environment Machine S/M TestsBuild Large Tests UAT ProdLarge Tests UAT
- 450. 93 DB Machine CI Node Large Tests Environment DB Machine UAT Environment Machine Master DB Machine Production Environment Machine Machine Machine Slave DB S/M TestsBuild Large Tests UAT ProdLarge Tests Prod
- 451. 94 Faster Feedback More Production Like S/M TestsBuild Large Tests UAT ProdLarge Tests remember, this is what we are optimising for
- 452. 94 Faster Feedback More Production Like S/M TestsBuild Large Tests UAT ProdLarge Tests remember, this is what we are optimising for
- 453. 95
- 454. 96
- 463. 97 Ansible Puppet Chef AWS Digital Ocean OpenStack VMWare Vagrant Immutable Servers Fast Spin-up
- 464. 97 Ansible Puppet Chef AWS Digital Ocean OpenStack VMWare Vagrant Immutable Servers Fast Spin-up Provider Agnostic
- 465. 97 Ansible Puppet Chef AWS Digital Ocean OpenStack VMWare Vagrant Immutable Servers Fast Spin-up Provider Agnostic Feedback Can Suffer
- 466. 97 Ansible Puppet Chef AWS Digital Ocean OpenStack VMWare Vagrant Immutable Servers Fast Spin-up Provider Agnostic Feedback Can Suffer Cycle Time
- 467. 98 Prod Prod Prod Prod optimise for a single service on a single image
- 468. 98 Prod Prod Prod Prod Packer Images optimise for a single service on a single image
- 469. 98 Prod Prod Prod Prod “Machine” Service Packer Images optimise for a single service on a single image
- 470. 98 Prod Prod Prod Prod “Machine” Service “Machine” Service Service Packer Images optimise for a single service on a single image
- 471. 98 Prod Prod Prod Prod “Machine” Service “Machine” Service Service Packer Images optimise for a single service on a single image
- 472. 99 S/M TestsBuild Large Tests UAT ProdLarge Tests
- 473. 99 S/M TestsBuild Large Tests UAT ProdLarge Tests AWS
- 474. 99 S/M TestsBuild Large Tests UAT ProdLarge Tests AWS VMWare
- 475. 99 S/M TestsBuild Large Tests UAT ProdLarge Tests AWS VMWare Vagrant
- 476. 99 S/M TestsBuild Large Tests UAT ProdLarge Tests AWS VMWare Vagrant
- 477. 99 S/M TestsBuild Large Tests UAT ProdLarge Tests AWS VMWare Vagrant
- 478. 99 S/M TestsBuild Large Tests UAT ProdLarge Tests AWS VMWare Vagrant
- 480. 100 “Machine” Service Much Easier To Reason About
- 481. 100 “Machine” Service Much Easier To Reason About Easier To Provision (Or Decommission)
- 482. 100 “Machine” Service Much Easier To Reason About Easier To Provision (Or Decommission) Fewer Side-effects
- 483. 100 “Machine” Service Much Easier To Reason About Easier To Provision (Or Decommission) Fewer Side-effects Cost & Management Overhead!
- 484. 100 “Machine” Service Much Easier To Reason About Easier To Provision (Or Decommission) Fewer Side-effects Cost & Management Overhead! AWS Digital Ocean OpenStack
- 485. 100 “Machine” Service Much Easier To Reason About Easier To Provision (Or Decommission) Fewer Side-effects Cost & Management Overhead! AWS Digital Ocean OpenStack
- 486. 101
- 487. 101 Be aware of - and balance - your test Pyramid
- 488. 101 Be aware of - and balance - your test Pyramid Understand the balance between testing & rapid remediation
- 489. 101 Be aware of - and balance - your test Pyramid Understand the balance between testing & rapid remediation Deploy one thing at a time
- 490. 101 Be aware of - and balance - your test Pyramid Understand the balance between testing & rapid remediation Deploy one thing at a time Consider consumer-driven contracts over integration tests
- 491. 101 Be aware of - and balance - your test Pyramid Understand the balance between testing & rapid remediation Deploy one thing at a time Consider consumer-driven contracts over integration tests Explore image-based deployments to reduce environment differences
- 499. Alert!!!
- 507. UP
- 508. DOWN
- 509. DOWN
- 510. DOWN Too slow?
- 513. UP?
- 514. UP? DOWN?
- 515. UP? DOWN? Too slow? Where is the problem?
- 517. “it pushes the accidental complexity into the infrastructure” Martin Fowler
- 518. “it pushes the accidental complexity into the infrastructure” Martin Fowler monitoring and logging are essential
- 519. Monitoring
- 524. Response time tracking Log aggregation
- 525. Response time tracking Log aggregation Monitor down-stream dependencies
- 526. Response time tracking Log aggregation Monitor down-stream dependencies
- 527. Response time tracking Log aggregation Monitor down-stream dependencies UP? DOWN? Track health
- 529. every instance of every service…
- 530. every instance of every service… And you want to standardize …needs to be status aware
- 537. Logstash & Kibana
- 539. ID 123
- 540. ID 123 ID 123 ID 123 ID 123
- 541. ID 123 ID 123 ID 123 ID 123
- 542. ID 123 ID 123 ID 123 ID 123 Correlation IDs
- 549. “Every socket, process, pipe, or remote procedure call can and will hang. Even database calls [...]” M. Nygard,“Release It”
- 550. Cascading Failures • Happen when a problem in a service causes a problem in one or more consumers of that service • Become a bigger problem with more services (cross more process boundaries) • Can a failure in one back-end application take down the entire system (including the parts that don’t depend on that back-end)?
- 551. Failure Types • Rejected connections • Dropped ACKs • Slow responses (these are the nasty ones!)
- 552. Defense Mechanisms • Resource pools • Timeouts • Circuit breakers • Bulkheads • Fail fast
- 553. Circuit Breakers • If a service consistently fails, stop calling it for a while!
- 554. MusikShop v1 Rekomender v124 www.MusikShop Take That Queens Of The Stone Age Snoop Dogg We Rekomend ‘The Brakes’!
- 555. MusikShop v1 Rekomender v124 www.MusikShop Take That Queens Of The Stone Age Snoop Dogg We Rekomend ‘The Brakes’!
- 556. MusikShop v1 Rekomender v124 www.MusikShop Take That Queens Of The Stone Age Snoop Dogg We Rekomend ‘The Brakes’! We Can’t Rekomend Right Now!
- 557. MusikShop v1 Rekomender v124 www.MusikShop Take That Queens Of The Stone Age Snoop Dogg We Rekomend ‘The Brakes’! We Can’t Rekomend Right Now!
- 558. MusikShop v1 Rekomender v124 www.MusikShop Take That Queens Of The Stone Age Snoop Dogg We Rekomend ‘The Brakes’!
- 559. Bulkheads • = Damage containment • Separate resource pools • Partitioned servers
- 560. Fail Fast • Check (and perhaps reserve) required resources before processing a request • Reject immediately if, say, a circuit breaker has been tripped • Allow consumers to query state of service before proceeding (see monitoring later)
- 561. How Small?
- 562. This might be the wrong question
- 563. How many >> size
- 564. So how many is too many?
- 565. From an architectural & operations view...
- 566. How many >> how small From an architectural & operations view...
- 571. Pain?
- 572. Pain? Pain?
- 574. Summary • Complexity doesn’t vanish, but with help it can be more evident • Monitoring & architectural safety measures are essential! • Start with a few services and understand what your appetite is for this new sort of complexity
- 576. "organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations" - Melvin Conway, Dude (HBR rejected the original paper as the thesis wasn’t proved)
- 577. “If seven people create a compiler, you get a seven pass compiler” - Dan North, not quite a dude
- 586. = Pain!
- 592. #123 As a despot when I press the big red button I want...
- 593. #123 As a despot when I press the big red button I want...
- 594. Splitting Stories • When splitting, try and synchronise the work • Consider re-assigning service ownership temporarily • Splitting stories across multiple teams is painful... • ...so what about shared services?
- 597. Conway’s law!
- 598. #123 As a despot when I press the big red button I want...
- 599. #123 As a despot when I press the big red button I want... Problems:
- 600. #123 As a despot when I press the big red button I want... Consistency of XD Problems:
- 601. #123 As a despot when I press the big red button I want... Consistency of XD Sequencing Problems:
- 602. #123 As a despot when I press the big red button I want... Consistency of XD Sequencing Bottlenecks Problems:
- 603. #123 As a despot when I press the big red button I want... Consistency of XD Sequencing Bottlenecks Testing Problems:
- 604. #123 As a despot when I press the big red button I want... Consistency of XD Sequencing Bottlenecks TL, QA, PM Testing Problems:
- 605. Summary • In general assign services to team... • ...where team means a co-located group of people • Strongly splitting services around organizational boundaries • Avoid shared services, instead temporarily re-assign ownership to reduce the need for fine-grained orchestration of work
- 609. 158
- 612. 161 Go Incremental
- 614. 163
- 615. 164 1. Rule of Modularity: Write simple parts connected by clean interfaces. 2. Rule of Clarity: Clarity is better than cleverness. 3. Rule of Composition: Design programs to be connected to other programs. 4. Rule of Separation: Separate policy from mechanism; separate interfaces from engines. 5. Rule of Simplicity: Design for simplicity; add complexity only where you must. 6. Rule of Parsimony: Write a big program only when it is clear by demonstration that nothing else will do. 7. Rule of Transparency: Design for visibility to make inspection and debugging easier. 8. Rule of Robustness: Robustness is the child of transparency and simplicity. 9. Rule of Representation: Fold knowledge into data so program logic can be stupid and robust. 10.Rule of Least Surprise: In interface design, always do the least surprising thing. 11.Rule of Silence: When a program has nothing surprising to say, it should say nothing. 12.Rule of Repair: When you must fail, fail noisily and as soon as possible. 13.Rule of Economy: Programmer time is expensive; conserve it in preference to machine time. 14.Rule of Generation: Avoid hand-hacking; write programs to write programs when you can. 15.Rule of Optimization: Prototype before polishing. Get it working before you optimize it. 16.Rule of Diversity: Distrust all claims for “one true way”. 17.Rule of Extensibility: Design for the future, because it will be here sooner than you think.
- 616. 164 1. Rule of Modularity: Write simple parts connected by clean interfaces. 2. Rule of Clarity: Clarity is better than cleverness. 3. Rule of Composition: Design programs to be connected to other programs. 4. Rule of Separation: Separate policy from mechanism; separate interfaces from engines. 5. Rule of Simplicity: Design for simplicity; add complexity only where you must. 6. Rule of Parsimony: Write a big program only when it is clear by demonstration that nothing else will do. 7. Rule of Transparency: Design for visibility to make inspection and debugging easier. 8. Rule of Robustness: Robustness is the child of transparency and simplicity. 9. Rule of Representation: Fold knowledge into data so program logic can be stupid and robust. 10.Rule of Least Surprise: In interface design, always do the least surprising thing. 11.Rule of Silence: When a program has nothing surprising to say, it should say nothing. 12.Rule of Repair: When you must fail, fail noisily and as soon as possible. 13.Rule of Economy: Programmer time is expensive; conserve it in preference to machine time. 14.Rule of Generation: Avoid hand-hacking; write programs to write programs when you can. 15.Rule of Optimization: Prototype before polishing. Get it working before you optimize it. 16.Rule of Diversity: Distrust all claims for “one true way”. 17.Rule of Extensibility: Design for the future, because it will be here sooner than you think. the 17 rules of UNIX programming
- 617. 165 The Rule of Diversity the 16th rule DISTRUST ALL CLAIMS FOR “ONE TRUE WAY” and finally