Downloaded 117 times
![…and the database is just
an optimization.[1]
[1] https://christophermeiklejohn.com/lasp/erlang/2015/10/27/tendency.html](https://image.slidesharecdn.com/natsmeetup1-160323164111/85/Simple-Solutions-for-Complex-Problems-6-320.jpg)
![“There will be no further print editions
[of the Merck Manual]. Publishing a
printed book every five years and
sending reams of paper around the
world on trucks, planes, and boats is
no longer the optimal way to provide
medical information.”
Dr. Robert S. Porter
Editor-in-Chief, The Merck Manuals](https://image.slidesharecdn.com/natsmeetup1-160323164111/85/Simple-Solutions-for-Complex-Problems-7-320.jpg)
![Distributed computation is
inherently asynchronous
and the network is
inherently unreliable[2]…
[2] http://queue.acm.org/detail.cfm?id=2655736](https://image.slidesharecdn.com/natsmeetup1-160323164111/85/Simple-Solutions-for-Complex-Problems-13-320.jpg)
![Guaranteed, ordered,
exactly-once delivery
is expensive (if not impossible[3]).
[3] http://bravenewgeek.com/you-cannot-have-exactly-once-delivery/](https://image.slidesharecdn.com/natsmeetup1-160323164111/85/Simple-Solutions-for-Complex-Problems-19-320.jpg)
![End-to-end system semantics matter
much more than the semantics of an
individual building block[4].
[4] http://web.mit.edu/Saltzer/www/publications/endtoend/endtoend.pdf](https://image.slidesharecdn.com/natsmeetup1-160323164111/85/Simple-Solutions-for-Complex-Problems-38-320.jpg)
![Simple doesn’t mean easy.
[5] https://blog.wearewizards.io/some-a-priori-good-qualities-of-software-development](https://image.slidesharecdn.com/natsmeetup1-160323164111/85/Simple-Solutions-for-Complex-Problems-43-320.jpg)
![[6] http://bravenewgeek.com/benchmarking-message-queue-latency/](https://image.slidesharecdn.com/natsmeetup1-160323164111/85/Simple-Solutions-for-Complex-Problems-55-320.jpg)
Uploaded byApcera
Simple Solutions for Complex Problems
The document is a presentation by Tyler Treat discussing the complexities of distributed systems and the importance of simplicity in addressing these challenges. It highlights how Workiva utilizes NATS for high-performance messaging while emphasizing the need for realistic expectations around message delivery guarantees in complex environments. The speaker advocates for embracing chaos and prioritizing availability and simplicity in system design to enhance overall performance and scalability.
More Related Content
Similar to Simple Solutions for Complex Problems
![[WSO2Con EU 2017] Building Next Generation Banking Middleware at ING: The Rol...](https://cdn.slidesharecdn.com/ss_thumbnails/ingmodelbank-final-171107154840-thumbnail.jpg?width=640&height=640&fit=bounds)
Simple Solutions for Complex Problems
- 1. Simple Solutions for ComplexProblems Tyler Treat / Workiva Bay Area NATS Meetup 3/22/2016
- 2. • Messaging techlead at Workiva • Platform infrastructure • Distributed systems • bravenewgeek.com @tyler_treat tyler.treat@workiva.com ABOUT THE SPEAKER
- 3. • Embracing thereality of complex systems • Using simplicity to your advantage • Why NATS? • How Workiva uses NATS ABOUT THIS TALK
- 4. There are alot of parallels between real-world systems and distributed software systems.
- 5. The world iseventually consistent…
- 6. …and the databaseis just an optimization.[1] [1] https://christophermeiklejohn.com/lasp/erlang/2015/10/27/tendency.html
- 7. “There will beno further print editions [of the Merck Manual]. Publishing a printed book every five years and sending reams of paper around the world on trucks, planes, and boats is no longer the optimal way to provide medical information.” Dr. Robert S. Porter Editor-in-Chief, The Merck Manuals
- 8. Programmers find asynchronyhard to reason about, but the truth is…
- 9. Life is mostlyasynchronous.
- 10. What does thismean for us as programmers?
- 11.
- 12.
- 13. Distributed computation is inherentlyasynchronous and the network is inherently unreliable[2]… [2] http://queue.acm.org/detail.cfm?id=2655736
- 14. …but the naturaltendency is to build distributed systems as if they aren’t distributed at all because it’s easy to reason about. strong consistency - reliable messaging - predictability
- 15. • Complicated algorithms •Transaction managers • Coordination services • Distributed locking What’s in a guarantee?
- 17. • Message handedto the transport layer? • Enqueued in the recipient’s mailbox? • Recipient started processing it? • Recipient finished processing it? What’s a delivery guarantee?
- 18. Each of thesehas a very different set of conditions, constraints, and costs.
- 19. Guaranteed, ordered, exactly-once delivery isexpensive (if not impossible[3]). [3] http://bravenewgeek.com/you-cannot-have-exactly-once-delivery/
- 20.
- 21.
- 22. Difficult to deploy& operate
- 23.
- 24.
- 25. At large scale,guarantees will give out.
- 26. 0.1% failure atscale is huge.
- 29.
- 30. Replayable > Guaranteed Idempotent> Exactly-once
- 31. Replayable > Guaranteed Idempotent> Exactly-once Commutative > Ordered
- 32. But delivery !=processing
- 33. Also, what doesit even mean to “process” a message?
- 34. It depends onthe business context!
- 35. If you needbusiness-level guarantees, build them into the business layer.
- 37. We can alwaysbuild stronger guarantees on top, but we can’t always remove them from below.
- 38. End-to-end system semanticsmatter much more than the semantics of an individual building block[4]. [4] http://web.mit.edu/Saltzer/www/publications/endtoend/endtoend.pdf
- 39.
- 40. “Simplicity is theultimate sophistication.”
- 41. EMBRACING THE CHAOSMEANS LOOKING AT THE NEGATIVE SPACE.
- 42. A simple technology ina sea of complexity.
- 43. Simple doesn’t meaneasy. [5] https://blog.wearewizards.io/some-a-priori-good-qualities-of-software-development
- 44. “Simple can beharder than complex. You have to work hard to get your thinking clean to make it simple. But it’s worth it in the end because once you get there, you can move mountains.”
- 45. • Wdesk: platformfor enterprises to collect, manage, and report critical business data in real time • Increasing amounts of data and complexity of formats • Cloud solution: - Data accuracy - Secure - Highly available - Scalable - Mobile-enabled About Workiva
- 48. • First solutionbuilt on Google App Engine • Scaling new solutions requires service-oriented approach • Scaling new services requires a low-latency communication backplane About Workiva
- 49.
- 50.
- 51. • Always on,always available • Protects itself at all costs—no compromises on performance • Disconnects slow consumers and lazy listeners • Clients have automatic failover and reconnect logic • Clients buffer messages while temporarily partitioned Availability over Everything
- 52. Simplicity as afeature.
- 53. • Single, lightweightbinary • Embraces the “negative space”: - Simplicity —> high-performance - No complicated configuration or external dependencies (e.g. ZooKeeper) - No fragile guarantees —> face complexity head-on, encourage async • Simple pub/sub semantics provide a versatile primitive: - Fan-in - Fan-out - Request/response - Distributed queueing • Simple text-based wire protocol Simplicity as a Feature
- 54.
- 55.
- 57. • Fast, predictableperformance at scale and at tail • ~8 million messages per second • Auto-pruning of interest graph allows efficient routing • When SLAs matter, it’s hard to beat NATS Fast as Hell
- 58. • Low-latency servicebus • Pub/Sub • RPC How We Use NATS
- 59.
- 60.
- 61.
- 62.
- 63.
- 64.
- 65.
- 66.
- 67. “Just send thisthing containing these fields serialized in this way using that encoding to this topic!”
- 68. “Just subscribe tothis topic and decode using that encoding then deserialize in this way and extract these fields from this thing!”
- 70. Pub/Sub is meantto decouple services but often ends up coupling the teams developing them.
- 71. How do weevolve services in isolation and reduce development overhead?
- 72. • Extension ofApache Thrift • IDL and cross-language, code-generated pub/sub APIs • Allows developers to think in terms of services and APIs rather than opaque messages and topics • Allows APIs to evolve while maintaining compatibility • Transports are pluggable (we use NATS) Frugal RPC
- 73. struct Event { 1:i64 id, 2: string message, 3: i64 timestamp, } scope Events prefix {user} { EventCreated: Event EventUpdated: Event EventDeleted: Event } subscriber.SubscribeEventCreated( "user-1", func(e *event.Event) { fmt.Println(e) }, ) . . . publisher.PublishEventCreated( "user-1", event.NewEvent()) generated
- 74. • Service instancesform a queue group • Client “connects” to instance by publishing a message to the service queue group • Serving instance sets up an inbox for the client and sends it back in the response • Client sends requests to the inbox • Heartbeats used to check health of server and client • Connecting is cheap—no service discovery and no sockets to create, just a request/response • Very early prototype code: https://github.com/workiva/thrift-nats RPC over NATS
- 76. • Store JSONcontaining cluster membership in S3 • Container reads JSON on startup and creates routes w/ correct credentials • Services only talk to the NATS daemon on their VM via localhost • Don’t have to worry about encryption between services and NATS, only between NATS peers NATS per VM
- 77. • Only messagesintended for a process on another host go over the network since NATS cluster maintains interest graph • Greatly reduces network hops (usually 0 vs. 2-3) • If local NATS daemon goes down, restart it automatically NATS per VM
- 78. • Doesn’t scaleto large number of VMs • Fairly easy to transition to floating NATS cluster or running on a subset of machines per AZ • NATS communication abstracted from service • Send messages to services without thinking about routing or service discovery • Queue groups provide service load balancing NATS per VM
- 79. • We’re aSaaS company, not an infrastructure company • High availability • Operational simplicity • Performance • First-party clients: Go Java C C# Python Ruby Elixir Node.js NATS as a Messaging Backplane
- 80. –Derek Landy, SkulduggeryPleasant “Every solution to every problem is simple… It's the distance between the two where the mystery lies.”
- 81.