• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Real world akka recepies v3

Real world akka recepies v3



Jamie Allen, Björn Antonsson and Patrik Nordwall discuss patterns of building Akka systems, including the Sentinel for handling failure above a supervisor, flow control and distributed workers. ...

Jamie Allen, Björn Antonsson and Patrik Nordwall discuss patterns of building Akka systems, including the Sentinel for handling failure above a supervisor, flow control and distributed workers. Patrik also built a template for Typesafe Activator v0.2.1 allowing you to try out this distributed workers pattern on your own machine.



Total Views
Views on SlideShare
Embed Views



4 Embeds 259

https://twitter.com 207
http://www.scoop.it 50
https://web.tweetdeck.com 1
https://www.rebelmouse.com 1



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

    Real world akka recepies v3 Real world akka recepies v3 Presentation Transcript

    • Real World Akka RecipesJamie AllenBjörn AntonssonPatrik Nordwall
    • The Fallacy ofGuaranteed DeliveryJamie Allen@jamie_allen
    • Guaranteed Delivery• From Enterprise Integration Patterns• Messaging system uses built-in storeto persist• ACK everywhere– Producer to sender– Sender to receiver– Receiver to consumer@jamie_allen3
    • Akka Guarantees• Not much, intentionally• At most once, with no reordering• Pick your poison:– At most once– At least once– Exactly once• You have to add it on top@jamie_allen4
    • How Do I Do It?• Handle “at least” semantics on receiverto deal with duplicates– Idempotent behavior in receiver– Check message ID• Handle “at most” semantics on thesender via retries– ACK every time message is handled– Cancel repeated send@jamie_allen5
    • Durable Mailboxes?@jamie_allen6Uh,  no.
    • @jamie_allen• Doesn’t work with future-based messagesending (ask, ?)• No guarantee is there that the messageeven got to the mailbox in distributedsystems• Asking for guarantees in an uncertainworld7Durable Mailboxes
    • Event Sourcing?• Wonderful pattern for compiling a list oftime-series events• Separation of concerns from actormailboxes• Still lots of things that can go wrong– Disk writing– Replication consistency– Network partitions– Application latency@jamie_allen8
    • External Durable Message Queue• You still have to ACK• No certainty the message you neededeven got this far• Additional dependencies in yourarchitecture@jamie_allen9
    • Guaranteed Delivery Doesn’t Exist• We don’t know what we don’t know• Increased effort• Increased complexity• Increased latency• No guarantees of consistency• Doesn’t guarantee ordering@jamie_allen10
    • So What Do We Do?• This falls outside of actor supervision;nothing the actors know about hasgone wrong• Listen to Roland Kuhn:“Recovery ... should ideally beautomatic in order to restore normalservice as quickly as possible.”@jamie_allen11
    • Sentinels12@jamie_allen
    • Sentinels• Supervisors handle failure BELOW them.Sentinels handle failure ABOVE.• Responsible for querying a “source of truth” andgetting latest state• Sends data to supervisor, who resolvesdifferences in which instances of actors shouldexist versus those that do• Supervisor forwards data to instances thatshould exist for them to resolve their internalstate@jamie_allen13
    • Missed Events@jamie_allenCustomer  2Customer  1Customer  SupervisorAdd  Customer  3Delete  Customer  1Update  Customer  2
    • Sentinels@jamie_allenCustomer  2Customer  1Customer  SupervisorCustomer  SentinelCustomer  2Customer  3
    • Sentinels@jamie_allenCustomer  2Customer  1Customer  SupervisorCustomer  SentinelCustomer  2Customer  3
    • Sentinels@jamie_allenCustomer  2Customer  1Customer  SupervisorCustomer  SentinelXCustomer  2Customer  3
    • Sentinels@jamie_allenCustomer  2Customer  SupervisorCustomer  SentinelCustomer  2Customer  3Customer  2
    • Sentinels@jamie_allenCustomer  2Customer  SupervisorCustomer  SentinelCustomer  2Customer  3Customer  3
    • Sentinels• Localize them for each kind of datathat must be synchronized in yoursupervisor hierarchy• Do not create one big one and try toresolve the entire tree at once@jamie_allen20
    • Drawbacks• Doesn’t work well with localized eventsourcing - time series can be lost• Does introduce additional complexityand tunable latency over applicationswith no guarantees• Pattern only works when there is aqueryable source of truth@jamie_allen21
    • Inconsistent Views?• Using Sentinels at multiple levels of asupervisory hierarchy can lead totemporarily inconsistent views whenchild actors are resolved beforeparents on delete (no atomicity)• But is this necessarily bad?@jamie_allen22
    • Sentinels in Hierarchy@jamie_allenC2C1CSA2A1ASD2D1DSA2SSS
    • Sentinels in Hierarchy@jamie_allenC2C1CSA2A1ASD2D1DSA2SSSX
    • Sentinels in Hierarchy@jamie_allenC2C1CSA2A1ASD2D1DSA2SSSX
    • Sentinels in Hierarchy@jamie_allenC2C1CSASD2D1DS SSSX
    • Sentinels in Hierarchy@jamie_allenC2C1CSD2D1DS SSXAS S
    • Sentinels in Hierarchy@jamie_allenC1CS S
    • A Huge Win• Your system is resilient to externalfailures• You can tune sentinel updatefrequency to meet changingrequirements• Your system is considerably lesscomplex than attempting to guaranteeno message loss@jamie_allen29
    • Flow ControlBjörn Antonsson@bantonsson
    • Pure Push Applications• Often the first Actor application youwrite– Once you start telling and stop asking• Easy to implement and reason about• Fits nicely with short lived jobs thatcome at a fixed rate31@bantonsson
    • 32@bantonsson
    • Why do you need anything else?• Produce jobs faster than you can finishthem• Jobs are expensive compute/memorywise• External resources impose limits• Unpredictable job patterns33@bantonsson
    • What can you do instead?• Push with rate limiting– A fixed number of jobs per time unit arepushed• Push with acknowledgment– A fixed number of jobs can be in progress.– New jobs are pushed after old jobs finish• Pull– Jobs are pulled from the master at the ratethat they are completed34@bantonsson
    • Push with rate limiting• A timer sends the master ticks at fixedintervals• When a tick arrives, the master fills upits token count• If a job arrives and there are no tokens,it gets queued• When the master has tokens, it pullsjobs off the queue and pushes them35@bantonsson
    • 36@bantonsson
    • Push with acknowledgement• The master push a fixed number of jobsbefore waiting for an acknowledgement• If a job arrives and the master cantpush, it gets queued• To keep workers busy, push more thanone job per worker– You can use a high water mark to stop anda low water mark to start pushing37@bantonsson
    • 38@bantonsson
    • Pull• The master actor queues incoming jobs• Worker actors ask the master for a joband receives jobs when available• The workers dont need to do activepolling• Can lead to lag if jobs are smallcompared to the time it takes to get anew one– Use batching to counteract lag39@bantonsson
    • 40@bantonsson
    • References• Push with rate limiting– Kaspar Fischerhttp://letitcrash.com/post/28901663062/throttling-messages-in-akka-2• Pull– Derek Wyatthttp://letitcrash.com/post/29044669086/balancing-workload-across-nodes-with-akka-2– Michael Pollmeierhttp://www.michaelpollmeier.com/akka-work-pulling-pattern-to-throttle-work/41@bantonsson
    • DistributedWorkersPatrik Nordwall@patriknw
    • 43
    • Goal• elastic addition/removal of front endnodes• elastic addition/removal of workers• thousands of workers• jobs should not be lost44
    • 45
    • 46
    • 47
    • 48
    • 49
    • 50
    • 51DistributedPubSubMediator
    • 52
    • 53
    • 54
    • 55
    • 56