This document discusses the evolution of messaging solutions at Workiva from RPC to messaging. It describes how Workiva transitioned from a monolithic architecture to microservices, and needed to standardize communication between services. It details how Workiva used Thrift for RPC initially over HTTP, then moved to NATS for asynchronous messaging to improve scalability without guarantees. For guaranteed delivery, Workiva is testing Frugal over SQS. The document outlines lessons learned around complexity, product management, and being willing to change plans.

1. Messaging
Kevin Sookocheff
cc: andrewrennie - https://www.flickr.com/photos/29712408@N02

2. Messaging
Mistakes Were Made
cc: andrewrennie - https://www.flickr.com/photos/29712408@N02

3. About Me
• Kevin Sookocheff
• Software Architect - Workiva
• Messaging Systems
• API Platform
• sookocheff.com
cc: andrewrennie - https://www.flickr.com/photos/29712408@N02

4. Wdesk

5. Wdesk
Became a Monolith
cc: bhavya999 - https://www.flickr.com/photos/144421990@N02

6. Wdesk
Split into Microservices
cc: drubuntu - https://www.flickr.com/photos/51980416@N03

8. Microservices
How to Communicate?
cc: Wiertz Sébastien - https://www.flickr.com/photos/36169570@N08

9. Messaging Platform
Standardize How Services Communicate
cc: Wiertz Sébastien - https://www.flickr.com/photos/36169570@N08

10. How?
• Interface
• Protocol
• Transport
cc: Wiertz Sébastien - https://www.flickr.com/photos/36169570@N08

11. Apache Thrift
Remote Procedure Call Framework
cc: kellyhogaboom - https://www.flickr.com/photos/11597293@N00

12. Thrift
• Cross-Language Type System
• Byte Transport
• Serialization Protocol
• Versioning System
• Processors
cc: kellyhogaboom - https://www.flickr.com/photos/11597293@N00

13. Thrift Server
1. Define an Interface
2. Generate Server Code
3. Choose a Protocol
4. Choose a Transport
5. Implement Processor Logic
cc: bigpresh - https://www.flickr.com/photos/62806367@N00

14. Thrift Client
1. Generate client code
2. Choose Protocol
3. Choose Transport
4. Make RPC calls
cc: kjetikor - https://www.flickr.com/photos/43642098@N06

16. Problem Solved
Thrift + Compact Protocol + HTTP Transport = RPC
cc: Brian Sawyer - https://www.flickr.com/photos/45609637@N00

17. cc: kirtaph - https://www.flickr.com/photos/13006614@N07

18. Remote Procedure Call
cc: Fey Ilyas - https://www.flickr.com/photos/28968923@N08

19. RPC
• one-to-one messaging
• hides network cost
• hides failure conditions
• not a procedure call

20. cc: Leo Reynolds - https://www.flickr.com/photos/49968232@N00

22. Microservices
How to Scale Communication?
cc: Wiertz Sébastien - https://www.flickr.com/photos/36169570@N08

23. Messaging

24. Messaging
• Asynchronous
• Decoupled
• Scalable

25. Messaging
Radio > Phone Call
cc: cogdogblog - https://www.flickr.com/photos/37996646802@N01

26. Messaging
• Subscriber listens to topic
• Publisher publishes to topic
• Messaging system delivers to
subscribers

27. Messaging

28. Request-Reply

29. Publish-Subscribe

30. Queueing

31. NATS
Click to add more text here
cc: andrewfhart - https://www.flickr.com/photos/83711730@N06

32. NATS
• Availability over everything
• Fast (~8 million
messages/second)
cc: andrewfhart - https://www.flickr.com/photos/83711730@N06

33. Problem Solved
Thrift + Compact Protocol + NATS Transport = Messaging
cc: Brian Sawyer - https://www.flickr.com/photos/45609637@N00

34. cc: kirtaph - https://www.flickr.com/photos/13006614@N07

35. Thrift != Pub/Sub
Frugal (Thrift Extension)
cc: kirtaph - https://www.flickr.com/photos/13006614@N07

36. Problem Solved
Frugal + Compact Protocol + NATS Transport = Messaging
cc: Brian Sawyer - https://www.flickr.com/photos/45609637@N00

37. cc: kirtaph - https://www.flickr.com/photos/13006614@N07

38. NATS != Guaranteed
cc: kirtaph - https://www.flickr.com/photos/13006614@N07

39. Assumptions Made
• Replayable > Guaranteed
• Idempotent > Exactly-once
• Commutative > Ordered
cc: kirtaph - https://www.flickr.com/photos/13006614@N07

40. Guarantees Have a Costcc: mag3737 - https://www.flickr.com/photos/50318388@N00

41. Guarantees Are
Slow
Fragile
Complex
Will Still Fail
But People Want Them
cc: mag3737 - https://www.flickr.com/photos/50318388@N00

43. Guarantees
Persistence + Durability = Guaranteed Delivery
cc: chrisinplymouth - https://www.flickr.com/photos/21450297@N06

44. Persistence
• Store messages
• Backup for message system
failure
cc: hermitsmoores - https://www.flickr.com/photos/35137530@N00

45. Durability
• Store subscriptions
• Backup for consumer system
failure
cc: Jaymis - https://www.flickr.com/photos/49048289@N00

46. Guaranteed Delivery
Persistence + Durability
cc: chrisinplymouth - https://www.flickr.com/photos/21450297@N06

47. Simple Queue Service (SQS)
• Guaranteed Delivery
• Queueing
• No publish-subscribe
• "Just a queue"
cc: gadl - https://www.flickr.com/photos/24183489@N00

48. Cloud Pub/Sub
• Guaranteed Delivery
• Queueing
• Publish-subscribe
• No wild-card subscriptions
cc: Maldita la hora - https://www.flickr.com/photos/92635326@N00

49. Recipient List
Enterprise Integration Patterns
cc: HeyRocker - https://www.flickr.com/photos/74785688@N00

50. Recipient List
1. Define a channel for each
recipient
2. Inspect each message
3. Determine the list of desired
recipients
4. Forward message to all channels
associated with the recipients
cc: HeyRocker - https://www.flickr.com/photos/74785688@N00

53. Problem Solved
Frugal + Compact Protocol + SQS Transport = Guaranteed Messaging
cc: Brian Sawyer - https://www.flickr.com/photos/45609637@N00

54. Problem Solved
• Thrift + Frugal
• Serialization
• Standardization
• Cross-platform
cc: Brian Sawyer - https://www.flickr.com/photos/45609637@N00

55. Problem Solved
• HTTP Transport
• RPC
• One-to-one communication
cc: Brian Sawyer - https://www.flickr.com/photos/45609637@N00

56. Problem Solved
• NATS Transport
• Messaging
• Low-latency
• Not guaranteed
cc: Brian Sawyer - https://www.flickr.com/photos/45609637@N00

57. Problem Solved
• Durable Transport (SQS)
• Guaranteed delivery
• Added latency
cc: Brian Sawyer - https://www.flickr.com/photos/45609637@N00

58. Problem Solved
• Supports > 60 services
• Multi-cloud
• > 10 million msgs/day (NATS)
• ~ 20 million msgs/day (HTTP)
• Durable messaging in beta
testing
cc: Brian Sawyer - https://www.flickr.com/photos/45609637@N00

59. Lessons Learned
cc: symphony of love - https://www.flickr.com/photos/85608594@N00

61. Lessons Learned
Complexity is Everywhere
cc: symphony of love - https://www.flickr.com/photos/85608594@N00

62. Lessons Learned
Be a Product Manager
cc: symphony of love - https://www.flickr.com/photos/85608594@N00

63. Lessons Learned
Plan It — Then Throw it Away
cc: symphony of love - https://www.flickr.com/photos/85608594@N00

64. Questions?
sookocheff.com
@soofaloofa
cc: Eleaf - https://www.flickr.com/photos/12348847@N00