Akka - Developing SEDA Based Applications


Published on

Developing SEDA Based Applications using Akka.

  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Akka - Developing SEDA Based Applications

  1. 1. AkkaDeveloping SEDA Based Applications
  2. 2. MeBen Darfler@bdarflerhttp://bdarfler.comSenior Software Engineer at Localytics
  3. 3. LocalyticsReal time mobile analytics platform40M+ events per day and growing rapidly3x growth over the past 3 monthsHeavy users of Scala/Akka/NoSqlWe are hiring (seriously, come talk to me)
  4. 4. LocalyicsHow to keep up with our growth?
  5. 5. Actor ModelLock free approach to concurrencyNo shared state between actorsAsynchronous message passingMailboxes to buffer incoming messages
  6. 6. AkkaConfigurable Dispatchers MailboxesFault Tolerant SupervisorsGreat community @jboner @viktorklang
  7. 7. Akka Performanthttp://blog.jayway.com/2010/08/10/yet-another-akka-benchmark/
  8. 8. SEDAStaged Event Driven Architecture"Decomposes a complex, event-drivenapplication into a set of stages connectedby queues." 1"The most fundamental aspect of the SEDAarchitecture is the programming model thatsupports stage-level backpressure and loadmanagement." 11. http://www.eecs.harvard.edu/~mdw/proj/seda/
  9. 9. Backpressure Whats the big deal?
  10. 10. BackpressureManditory to prevent OutOfMemoryError Messages backup in memory faster than they can be processedCassandra was seriously bitten by this Less crappy failure mode when swamped with inserts than "run out of memory and gc-storm to death" (CASSANDRA-401) Add backpressure to StorageProxy (CASSANDRA-685)
  11. 11. BackpressureMailboxescase class UnboundedMailbox(val blocking: Boolean = false) extends MailboxTypecase class BoundedMailbox( val blocking: Boolean = false, val capacity: Int = { if (Dispatchers.MAILBOX_CAPACITY < 0) Int.MaxValue else Dispatchers.MAILBOX_CAPACITY }, val pushTimeOut: Duration = Dispatchers.MAILBOX_PUSH_TIME_OUT) extends MailboxTypeBackpressure MailboxBoundedMailbox(true, QUEUE_SIZE, Duration(-1, "millis"))
  12. 12. Stages How do we decompose the problem?
  13. 13. StagesOne actor class per stageShared dispatcherIndividually tunable I/O Bound CPU BoundEasier to reason aboutCode reuse
  14. 14. DispatchersThreadBasedDispatcher Binds one actor to its own threadExecutorBasedEventDrivenDispatcher Must be shared between actorsExecutorBasedEventDrivenWorkStealingDispatcher Must be shared between actors of the same type
  15. 15. Queues SEDA has a queue per stage model Akka actors have their own mailbox How do we evenly distribute work?
  16. 16. Work StealingExecutorBasedEventDrivenWorkStealingDispatcher"Actors of the same type can be set up to share thisdispatcher and during execution time the differentactors will steal messages from other actors if theyhave less messages to process" 11. http://doc.akka.io/dispatchers-scala
  17. 17. Work StealingReally a work "donating" dispatcher "I have implemented a work stealing dispatcher forAkka actors. Although its called "work stealing" theimplementation actually behaves more as "workdonating" because the victim actor takes the initiative.I.e. it actually donates work to its thief, ratherthan having the thief steal work from the victim." 11. http://janvanbesien.blogspot.com/2010/03/load-balancing-actors-with-work.html
  18. 18. Work Stealing Doesnt that conflict with blocking mailboxes?
  19. 19. Work StealingSending actor will block on the receiving actorsmailbox before it can "donate"Might be fixed in Akka 1.1 I owe @viktorklang a test of his latest changes
  20. 20. Load Balancing Can we distribute work on the sender side?
  21. 21. Load BalancingRouting.loadBalancerActor() Creates a new actor that forwards messages in a load balancing fashionInfiniteIterator CyclicIterator SmallestMailboxFirstIterator
  22. 22. Load BalancingDoesnt the load balancer need a blocking mailbox?
  23. 23. Load BalancingCant easily change the load balancers mailboxUse SmallestMailboxFirstIterator directlynew SmallestMailboxFirstIterator(List(actor, actor, actor))
  24. 24. Fault ToleranceSupervisors Restarts actors Stops after x times within y millisecondsRestart Strategies OneForOne AllForOne
  25. 25. Fault ToleranceGreat for transient issues Network failuresNot great for permanent issues OutOfMemoryError
  26. 26. Final Product// Actor creationval supervisor = Supervisor(SupervisorConfig( OneForOneStrategy(List(classOf[Exception]), RETRIES, WITH_IN_TIME), Supervise(myActors))defmyActors: List[Supervise] = { val mailbox = BoundedMailbox(true, QUEUE_SIZE, Duration(-1, "millis")) val dispatcher = Dispatchers.newExecutorBasedEventDrivenWorkStealingDispatcher( "my-dispatcher", mailbox).setCorePoolSize(POOL_SIZE).build (1 to POOL_SIZE toList).foldRight(List[Supervise]()) { (i, list) => Supervise(actorOf(new MyActor("my-actor-" + i, dispatcher)), Permanent) :: list }}// Sending a messageval actors = new SmallestMailboxFirstIterator(actorsFor(classOf[MyActor]).toList)def actor = actors.nextactor ! Message()
  27. 27. Thanks @bdarfler http://bdarfler.com