Real world akka recepies v3

8,484 views

Published on

Jamie Allen, Björn Antonsson and Patrik Nordwall discuss patterns of building Akka systems, including the Sentinel for handling failure above a supervisor, flow control and distributed workers. Patrik also built a template for Typesafe Activator v0.2.1 allowing you to try out this distributed workers pattern on your own machine.

Published in: Technology, Business
0 Comments
54 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
8,484
On SlideShare
0
From Embeds
0
Number of Embeds
417
Actions
Shares
0
Downloads
0
Comments
0
Likes
54
Embeds 0
No embeds

No notes for slide

Real world akka recepies v3

  1. 1. Real World Akka RecipesJamie AllenBjörn AntonssonPatrik Nordwall
  2. 2. The Fallacy ofGuaranteed DeliveryJamie Allen@jamie_allen
  3. 3. Guaranteed Delivery• From Enterprise Integration Patterns• Messaging system uses built-in storeto persist• ACK everywhere– Producer to sender– Sender to receiver– Receiver to consumer@jamie_allen3
  4. 4. Akka Guarantees• Not much, intentionally• At most once, with no reordering• Pick your poison:– At most once– At least once– Exactly once• You have to add it on top@jamie_allen4
  5. 5. How Do I Do It?• Handle “at least” semantics on receiverto deal with duplicates– Idempotent behavior in receiver– Check message ID• Handle “at most” semantics on thesender via retries– ACK every time message is handled– Cancel repeated send@jamie_allen5
  6. 6. Durable Mailboxes?@jamie_allen6Uh,  no.
  7. 7. @jamie_allen• Doesn’t work with future-based messagesending (ask, ?)• No guarantee is there that the messageeven got to the mailbox in distributedsystems• Asking for guarantees in an uncertainworld7Durable Mailboxes
  8. 8. Event Sourcing?• Wonderful pattern for compiling a list oftime-series events• Separation of concerns from actormailboxes• Still lots of things that can go wrong– Disk writing– Replication consistency– Network partitions– Application latency@jamie_allen8
  9. 9. External Durable Message Queue• You still have to ACK• No certainty the message you neededeven got this far• Additional dependencies in yourarchitecture@jamie_allen9
  10. 10. Guaranteed Delivery Doesn’t Exist• We don’t know what we don’t know• Increased effort• Increased complexity• Increased latency• No guarantees of consistency• Doesn’t guarantee ordering@jamie_allen10
  11. 11. So What Do We Do?• This falls outside of actor supervision;nothing the actors know about hasgone wrong• Listen to Roland Kuhn:“Recovery ... should ideally beautomatic in order to restore normalservice as quickly as possible.”@jamie_allen11
  12. 12. Sentinels12@jamie_allen
  13. 13. Sentinels• Supervisors handle failure BELOW them.Sentinels handle failure ABOVE.• Responsible for querying a “source of truth” andgetting latest state• Sends data to supervisor, who resolvesdifferences in which instances of actors shouldexist versus those that do• Supervisor forwards data to instances thatshould exist for them to resolve their internalstate@jamie_allen13
  14. 14. Missed Events@jamie_allenCustomer  2Customer  1Customer  SupervisorAdd  Customer  3Delete  Customer  1Update  Customer  2
  15. 15. Sentinels@jamie_allenCustomer  2Customer  1Customer  SupervisorCustomer  SentinelCustomer  2Customer  3
  16. 16. Sentinels@jamie_allenCustomer  2Customer  1Customer  SupervisorCustomer  SentinelCustomer  2Customer  3
  17. 17. Sentinels@jamie_allenCustomer  2Customer  1Customer  SupervisorCustomer  SentinelXCustomer  2Customer  3
  18. 18. Sentinels@jamie_allenCustomer  2Customer  SupervisorCustomer  SentinelCustomer  2Customer  3Customer  2
  19. 19. Sentinels@jamie_allenCustomer  2Customer  SupervisorCustomer  SentinelCustomer  2Customer  3Customer  3
  20. 20. Sentinels• Localize them for each kind of datathat must be synchronized in yoursupervisor hierarchy• Do not create one big one and try toresolve the entire tree at once@jamie_allen20
  21. 21. Drawbacks• Doesn’t work well with localized eventsourcing - time series can be lost• Does introduce additional complexityand tunable latency over applicationswith no guarantees• Pattern only works when there is aqueryable source of truth@jamie_allen21
  22. 22. Inconsistent Views?• Using Sentinels at multiple levels of asupervisory hierarchy can lead totemporarily inconsistent views whenchild actors are resolved beforeparents on delete (no atomicity)• But is this necessarily bad?@jamie_allen22
  23. 23. Sentinels in Hierarchy@jamie_allenC2C1CSA2A1ASD2D1DSA2SSS
  24. 24. Sentinels in Hierarchy@jamie_allenC2C1CSA2A1ASD2D1DSA2SSSX
  25. 25. Sentinels in Hierarchy@jamie_allenC2C1CSA2A1ASD2D1DSA2SSSX
  26. 26. Sentinels in Hierarchy@jamie_allenC2C1CSASD2D1DS SSSX
  27. 27. Sentinels in Hierarchy@jamie_allenC2C1CSD2D1DS SSXAS S
  28. 28. Sentinels in Hierarchy@jamie_allenC1CS S
  29. 29. A Huge Win• Your system is resilient to externalfailures• You can tune sentinel updatefrequency to meet changingrequirements• Your system is considerably lesscomplex than attempting to guaranteeno message loss@jamie_allen29
  30. 30. Flow ControlBjörn Antonsson@bantonsson
  31. 31. Pure Push Applications• Often the first Actor application youwrite– Once you start telling and stop asking• Easy to implement and reason about• Fits nicely with short lived jobs thatcome at a fixed rate31@bantonsson
  32. 32. 32@bantonsson
  33. 33. Why do you need anything else?• Produce jobs faster than you can finishthem• Jobs are expensive compute/memorywise• External resources impose limits• Unpredictable job patterns33@bantonsson
  34. 34. What can you do instead?• Push with rate limiting– A fixed number of jobs per time unit arepushed• Push with acknowledgment– A fixed number of jobs can be in progress.– New jobs are pushed after old jobs finish• Pull– Jobs are pulled from the master at the ratethat they are completed34@bantonsson
  35. 35. Push with rate limiting• A timer sends the master ticks at fixedintervals• When a tick arrives, the master fills upits token count• If a job arrives and there are no tokens,it gets queued• When the master has tokens, it pullsjobs off the queue and pushes them35@bantonsson
  36. 36. 36@bantonsson
  37. 37. Push with acknowledgement• The master push a fixed number of jobsbefore waiting for an acknowledgement• If a job arrives and the master cantpush, it gets queued• To keep workers busy, push more thanone job per worker– You can use a high water mark to stop anda low water mark to start pushing37@bantonsson
  38. 38. 38@bantonsson
  39. 39. Pull• The master actor queues incoming jobs• Worker actors ask the master for a joband receives jobs when available• The workers dont need to do activepolling• Can lead to lag if jobs are smallcompared to the time it takes to get anew one– Use batching to counteract lag39@bantonsson
  40. 40. 40@bantonsson
  41. 41. References• Push with rate limiting– Kaspar Fischerhttp://letitcrash.com/post/28901663062/throttling-messages-in-akka-2• Pull– Derek Wyatthttp://letitcrash.com/post/29044669086/balancing-workload-across-nodes-with-akka-2– Michael Pollmeierhttp://www.michaelpollmeier.com/akka-work-pulling-pattern-to-throttle-work/41@bantonsson
  42. 42. DistributedWorkersPatrik Nordwall@patriknw
  43. 43. 43
  44. 44. Goal• elastic addition/removal of front endnodes• elastic addition/removal of workers• thousands of workers• jobs should not be lost44
  45. 45. 45
  46. 46. 46
  47. 47. 47
  48. 48. 48
  49. 49. 49
  50. 50. 50
  51. 51. 51DistributedPubSubMediator
  52. 52. 52
  53. 53. 53
  54. 54. 54
  55. 55. 55
  56. 56. 56

×