Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Message Driven Architecture enables Elasticity, Resilience and Availability

431 views

Published on

In which I talked about our experience implementing a Message Driven architecture at SpinGo, and the benefits thereof. I sprinkled in some FRP concepts, and a handful of terrible memes.

Published in: Technology
  • Do you have any plans to present this somewhere?
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Message Driven Architecture enables Elasticity, Resilience and Availability

  1. 1. Message / Event Driven Arch Tim Harper, Software Craftsman
  2. 2. Me
  3. 3. Books
  4. 4. Books + anything out of this guy’s mouth
  5. 5. Messaging / Event Oriented Arch Tim Harper, Software Craftsman
  6. 6. Why use privates in code? /** * INTERNAL API */ private[akka] object StreamLayout { // compile-time constant final val Debug = false final def validate(m: Module, level: Int = 0, doPrint: Boolean = false, idMap: mutable.Map[AnyRef, Int] = mutable.Map.empty): Unit = { val ids = Iterator from 1 def id(obj: AnyRef) = idMap get obj match { case Some(x) => x
  7. 7. Why use privates in code? /** * INTERNAL API */ private[akka] object StreamLayout { // compile-time constant final val Debug = false final def validate(m: Module, level: Int = 0, doPrint: Boolean = false, idMap: mutable.Map[AnyRef, Int] = mutable.Map.empty): Unit = { val ids = Iterator from 1 def id(obj: AnyRef) = idMap get obj match { case Some(x) => x
  8. 8. Why use privates in code? /** * INTERNAL API */ private[akka] object StreamLayout { // compile-time constant final val Debug = false final def validate(m: Module, level: Int = 0, doPrint: Boolean = false, idMap: mutable.Map[AnyRef, Int] = mutable.Map.empty): Unit = { val ids = Iterator from 1 def id(obj: AnyRef) = idMap get obj match { case Some(x) => x
  9. 9. Why use privates in code?
  10. 10. • Good API design Why use privates in code?
  11. 11. • Good API design • It’s easier to add functionality than remove it / change it Why use privates in code?
  12. 12. • Good API design • It’s easier to add functionality than remove it / change it • Less guarantees = less coupling Why use privates in code?
  13. 13. • Good API design • It’s easier to add functionality than remove it / change it • Less guarantees = less coupling Don’t “pay the cost of a rich API without any of the benefits” Why use privates in code?
  14. 14. “Private” return value? trait Publisher[T] { def subscribe(sub: Subscriber[T]): Unit } trait Subscription { def requestMore(n: Int): Unit def cancel(): Unit } trait Subscriber[T] { def onSubscribe(s: Subscription): Unit def onNext(elem: T): Unit def onError(thr: Throwable): Unit def onComplete(): Unit }
  15. 15. “Private” return value? trait Publisher[T] { def subscribe(sub: Subscriber[T]): Unit } trait Subscription { def requestMore(n: Int): Unit def cancel(): Unit } trait Subscriber[T] { def onSubscribe(s: Subscription): Unit def onNext(elem: T): Unit def onError(thr: Throwable): Unit def onComplete(): Unit }
  16. 16. “Private” return value? trait Publisher[T] { def subscribe(sub: Subscriber[T]): Unit } trait Subscription { def requestMore(n: Int): Unit def cancel(): Unit } trait Subscriber[T] { def onSubscribe(s: Subscription): Unit def onNext(elem: T): Unit def onError(thr: Throwable): Unit def onComplete(): Unit }
  17. 17. Private result?
  18. 18. Private result? • Schedule now or then
  19. 19. Private result? • Schedule now or then • Run it here or there
  20. 20. Private result? • Schedule now or then • Run it here or there • Recovery, prioritization, etc.
  21. 21. Private result? • Schedule now or then • Run it here or there • Recovery, prioritization, etc. Don’t “pay the cost of a synchronous API without any of the benefits”
  22. 22. Reactive Manifesto
  23. 23. Reactive Manifesto
  24. 24. Reactive Manifesto It all starts here.
  25. 25. Event Oriented • Flip the arrows
  26. 26. Event Oriented Concert added to database
  27. 27. Event Oriented Concert added to database Send Email to User
  28. 28. Event Oriented Concert added to database Send Email to User Schedule Ingestion Task
  29. 29. Event Oriented Concert added to database Send Email to User Notify SalesForce Schedule Ingestion Task
  30. 30. Event Oriented Concert added to database Send Email to User Record in ElasticSearch Notify SalesForce Schedule Ingestion Task
  31. 31. Event Oriented Concert added to database Send Email to User Record in ElasticSearch Notify SalesForce Schedule Ingestion Task
  32. 32. Event Oriented Concert added to database Send Email to User Record in ElasticSearch Notify SalesForce Schedule Ingestion Task Such knowledge
  33. 33. Event Oriented Concert added to database Send Email to User Record in ElasticSearch Notify SalesForce Schedule Ingestion Task Such knowledge Very couple
  34. 34. Event Oriented Concert added to database Send Email to User Record in ElasticSearch Notify SalesForce Schedule Ingestion Task Such knowledge Very couple Wow
  35. 35. Event Oriented Concert added to database Record in ElasticSearch Send Email to User Schedule Ingestion Task Notify SalesForce
  36. 36. Event Oriented Concert added to database Record in ElasticSearch Send Email to User Schedule Ingestion Task Notify SalesForce
  37. 37. Event Oriented Concert added to database Record in ElasticSearch Send Email to User Schedule Ingestion Task Notify SalesForce Announce Listen for announce Listen for announce Listen for announce Listen for announce
  38. 38. Event Oriented Notify SalesForce Listen for promotion event Listen for new concert event Listen for concert update event
  39. 39. Functional Reactive Programming Event X happened
 @ time Y
  40. 40. Functional Reactive Programming Event X happened
 @ time Y AS TRUE TODAY AS WHEN IT WAS WRITTEN
  41. 41. Functional Reactive Programming A @ 1:00 B @ 1:01 C @ 1:05 D @ 1:06
  42. 42. Functional Reactive Programming A @ 1:00 B @ 1:01 C @ 1:05 D @ 1:06 F @ 1:00 G @ 1:01 H @ 1:05 I @ 1:06
  43. 43. FRP A @ 1:00 B @ 1:01 C @ 1:05 D @ 1:06 REDUCE (cache layer)
  44. 44. FRP A @ 1:00 B @ 1:01 C @ 1:05 D @ 1:06 REDUCE (cache layer) It’s beginning to look
 a lot like CQRS
  45. 45. FRP / CQRS • “Travel through time” • Retain all data (UPDATE = DELETE PRIOR STATE) • Full audit log • Separation of read / write concerns; et. al
  46. 46. Messaging Patterns • Queue • Topic • Exchange
  47. 47. Orange 1 Messaging Patterns: Direct Orange 1 Publish “oranges” “oranges” queue Orange 3 Orange 2
  48. 48. Orange 1 Messaging Patterns: Direct Publish “oranges” “oranges” queue Orange 3 Orange 2 Orange 1
  49. 49. Orange 1 Messaging Patterns: Broadcast / Fanout Orange 1 Publish “fruit” exchange “fruit” Orange 3 Orange 2 “products” Orange 3 Orange 2
  50. 50. Orange 1 Messaging Patterns: Broadcast / Fanout Publish “fruit” exchange “fruit” Orange 3 Orange 2 “products” Orange 3 Orange 2 Orange 1 Orange 1
  51. 51. Orange 1 Messaging Patterns: Broadcast / Fanout Publish “fruit” exchange “fruit” Orange 3 Orange 2 “products” Orange 3 Orange 2 Orange 1 Orange 1 AKA PubSub
  52. 52. Orange 1 Messaging Patterns: Topic / PubSub Orange 1 Publish “fruit” exchange topic: fruit.orange “all-fruit” fruit.* Apple 6 Apple 4 “apples” fruit.apples Apple 6 Apple 4
  53. 53. Orange 1 Messaging Patterns: Topic / PubSub Publish “fruit” exchange topic: fruit.orange “all-fruit” fruit.* Apple 6 Apple 4 “apples” fruit.apples Apple 6 Apple 4 Orange 1
  54. 54. Orange 1 Messaging Patterns: Topic / PubSub Publish “fruit” exchange topic: fruit.orange “all-fruit” fruit.* Apple 6 Apple 4 “apples” fruit.apples Apple 6 Apple 4 Orange 1 “Yo dawgs, this thing happened”
  55. 55. Message Broker Benefits • Messages are routes and enqueued while app not booting • Centralized point of coordination • Replication, persistence, etc. • If you don’t use one then you will build one.
  56. 56. Delivery guarantees • At least once • At most once
  57. 57. Delivery guarantees Network Process acksend receive Broker
  58. 58. Delivery guarantees
  59. 59. Idempotence • fn(fn(x)) = fn(x) • Great idea in general • (NOT impotency)
  60. 60. Idempotence • The operation to perform is a function of the current state and next command.
  61. 61. Cons (in which it is acknowledged that everything has a cost)
  62. 62. Cons • Complexity moved, not entirely solved. • (although, functional discipline aids in reducing state complexity)
  63. 63. • Difficulty grokking message flow (DOCS!!!) Cons
  64. 64. • Tracing errors can be difficult. Cons
  65. 65. • Rate-limiting with high- availability can be complex. Cons DDOS YOUR
 DATABASE WITH THIS ONE WEIRD TRICK
  66. 66. Thanks! ( ⋂‿⋂’) @timcharper http://tim.theenchanter.com/

×