Successfully reported this slideshow.

Effective akka scalaio

6,936 views

Published on

Published in: Technology, Business

Effective akka scalaio

  1. 1. Effective Actors Jamie Allen Director of Consulting @jamie_allen
  2. 2. @jamie_allen 2
  3. 3. Goal Communicate  as  much  about  what  I’ve  learned   in  4+  years  of  actor  development  within  one   hour NOTE  -­‐  this  is  not  an  introductory  talk,  and  will   contain  some  advanced  concepts  about  Akka  &   asynchronous  development @jamie_allen 3
  4. 4. Extra and Cameo Patterns A  couple  of  patterns  for  how  to  handle   individual  messages  to  an  actor  without  making   it  do  all  of  the  work  before  handling  the  next   message @jamie_allen 4
  5. 5. Request Aggregation •When  an  actor  handles  a  message,  it  frequently   has  to  perform  multiple  interactions  with  other   services  to  provide  a  valid  response •We  do  not  want  the  actor  that  received  the   message  to  be  tied  up  performing  that  work •What  are  our  options? @jamie_allen 5
  6. 6. Use Case •An  actor  will  receive  a  request  to  get  all  of  the   account  balances  for  a  customer  (savings,   checkings  and  money  market) •Actor  should  not  wait  to  finish  handling  one   request  before  handling  another •Actor  should  receive  service  proxies  from  which  it   can  retrieve  each  account’s  info •Actor  should  either  get  responses  from  all  three   within  a  specified  time,  or  send  a  timeout   response  back  to  the  requestor @jamie_allen 6
  7. 7. Request Aggregation Message 4 Message 3 Message 1 Handler Message 2 Message 1 Message 2 Handler Request Receiver Message 3 Handler Message 4 Handler @jamie_allen 7
  8. 8. Transactions? •Could  be! •You  have  to  write  the  logic  of  how  to  roll  back   if  anything  fails •But  you  have  the  control  and  the  context  to  do   it,  especially  if  your  effects  are  going  to  multiple   external  places  or  data  stores @jamie_allen 8
  9. 9. All Code is on GitHub • I’m  going  to  be  showing  some  reasonably   complex  examples • Don’t  worry  about  trying  to  copy  the  code  from   the  slides http://github.com/jamie-allen/effective_akka @jamie_allen 9
  10. 10. Use Futures and Promises? •No!    Use  another  actor  responsible  for: •Capturing  the  context  (original  requestor) •Defining  how  to  handle  responses  from  other   services •Defining  the  timeout  that  will  race  against  your •Each  future  and  the  resulting  promise  have  an   additional  cost  that  we  do  not  need  to  pay @jamie_allen 10
  11. 11. Use Futures and Promises? def receive = { case GetCustomerAccountBalances(id) => val futSavings = savingsAccounts ? GetCustomerAccountBalances(id) val futChecking = checkingAccounts ? GetCustomerAccountBalances(id) val futMM = moneyMarketAccounts ? GetCustomerAccountBalances(id) Yuck! val futBalances = for { savings <- futSavings.mapTo[Option[List[(Long, BigDecimal)]]] checking <- futChecking.mapTo[Option[List[(Long, BigDecimal)]]] mm <- futMM.mapTo[Option[List[(Long, BigDecimal)]]] } yield AccountBalances(savings, checking, mm) futBalances map (sender ! _) } } @jamie_allen 11
  12. 12. Capturing the Sender •This  is  trickier  than  it  sounds •You  need  the  “sender”  value  from  the  actor   that  received  the  original  request,  not  the   sender  inside  of  the  actor  handling  it! •One  of  the  biggest  sources  of  actor  bugs @jamie_allen 12
  13. 13. Use Futures and Promises? def receive = { case GetCustomerAccountBalances(id) => val futSavings = savingsAccounts ? GetCustomerAccountBalances(id) val futChecking = checkingAccounts ? GetCustomerAccountBalances(id) val futMM = moneyMarketAccounts ? GetCustomerAccountBalances(id) val futBalances = for { savings <- futSavings.mapTo[Option[List[(Long, BigDecimal)]]] checking <- futChecking.mapTo[Option[List[(Long, BigDecimal)]]] mm <- futMM.mapTo[Option[List[(Long, BigDecimal)]]] } yield AccountBalances(savings, checking, mm) futBalances map (sender ! _) } Bug! @jamie_allen 13
  14. 14. Use an Anonymous Actor? class OuterActor extends Actor def receive = LoggingReceive { case DoWork => { val originalSender = sender context.actorOf(Props(new Actor() { def receive = LoggingReceive { case HandleResponse(value) => timeoutMessager.cancel sendResponseAndShutdown(Response(value)) case WorkTimeout => sendResponseAndShutdown(WorkTimeout) } } Anonymous  Actor def sendResponseAndShutdown(response: Any) = { originalSender ! response context.stop(self) } someService ! DoWork import context.dispatcher val timeoutMessager = context.system.scheduler.scheduleOnce(250 milliseconds) { self ! WorkTimeout } })) } } } @jamie_allen 14
  15. 15. Use an Anonymous Actor? •I  call  this  the  “Extra”  Pattern •It’s  not  bad,  but  it  has  drawbacks: •Poor  stack  traces  due  to  “name  mangled”  actor   names,  like  $a •More  difficult  to  maintain  the  cluttered  code,  and   developers  have  to  read  through  the  body  of  the   anonymous  actor  to  figure  out  what  it  is  doing •More  likely  to  “close  over”  state @jamie_allen 15
  16. 16. Anonymous Actor Example class AccountBalanceRetrieverFinal(savingsAccounts: ActorRef, checkingAccounts: ActorRef, moneyMarketAccounts: ActorRef) extends Actor with ActorLogging def receive = LoggingReceive { case GetCustomerAccountBalances(id) => { val originalSender = sender context.actorOf(Props(new Actor() { var checkingBalances, savingsBalances, mmBalances: Option[List[(Long, BigDecimal)]] = None def receive = LoggingReceive { case CheckingAccountBalances(balances) => checkingBalances = balances collectBalances case SavingsAccountBalances(balances) => savingsBalances = balances collectBalances case MoneyMarketAccountBalances(balances) => mmBalances = balances collectBalances case AccountRetrievalTimeout => sendResponseAndShutdown(AccountRetrievalTimeout) } def collectBalances = (checkingBalances, savingsBalances, mmBalances) match { case (Some(c), Some(s), Some(m)) => timeoutMessager.cancel sendResponseAndShutdown(AccountBalances(checkingBalances, savingsBalances, mmBalances)) case _ => } def sendResponseAndShutdown(response: Any) = { originalSender ! response context.stop(self) } savingsAccounts ! GetCustomerAccountBalances(id) checkingAccounts ! GetCustomerAccountBalances(id) moneyMarketAccounts ! GetCustomerAccountBalances(id) import context.dispatcher val timeoutMessager = context.system.scheduler.scheduleOnce(250 milliseconds) { self ! AccountRetrievalTimeout } })) } } @jamie_allen 16
  17. 17. Create a “Cameo” Actor •Externalize  the  behavior  of  such  an  anonymous   actor  into  a  specific  type •Anyone  maintaining  the  code  now  has  a  type   name  from  which  they  can  infer  what  the  actor   is  doing •Most  importantly,  you  can’t  close  over  state   from  an  enclosing  actor  -­‐  it  must  be  passed   explicitly @jamie_allen 17
  18. 18. Create a “Cameo” Actor class WorkerActor extends Actor { def receive = LoggingReceive { case HandleResponse(value) => timeoutMessager.cancel sendResponseAndShutdown(Response(value)) case WorkTimeout => sendResponseAndShutdown(WorkTimeout) } def sendResponseAndShutdown(response: Any) = { originalSender ! response context.stop(self) } import context.dispatcher val timeoutMessager = context.system.scheduler.scheduleOnce(250 milliseconds) { self ! WorkTimeout } } class DelegatingActor extends Actor def receive = LoggingReceive { case DoWork => { val originalSender = sender val worker = context.actorOf(WorkerActor.props(), “worker”) someService.tell(DoWork, worker) } } } @jamie_allen 18
  19. 19. Remember to Stop the Actor • When  you  are  finished  handling  a  request,   ensure  that  the  actor  used  is  shutdown • This  is  a  big  memory  leak  if  you  don’t def sendResponseAndShutdown(response: Any) = { originalSender ! response log.debug("Stopping context capturing actor") context.stop(self) } @jamie_allen 19
  20. 20. Write Tests! •Always  remember  to  write  tests  with  your   actors •Create  unit  tests  that  check  the  functionality  of   method  calls  without  actor  interations  using   TestActorRef •Create  integration  tests  that  send  messages  to   actors  and  check  the  aggregated  results @jamie_allen 20
  21. 21. Write Tests! class CameoSpec extends TestKit(ActorSystem("cameo-test-as")) with ImplicitSender with WordSpecLike with MustMatchers { val checkingAccountsProxy = system.actorOf(Props[CheckingAccountsProxyStub], "checkings") val moneyMarketAccountsProxy = system.actorOf(Props[MoneyMarketAccountsProxyStub], "money-markets") "An AccountBalanceRetriever" should { "return a list of account balances" in { val probe1 = TestProbe() val probe2 = TestProbe() val savingsAccountsProxy = system.actorOf(Props[SavingsAccountsProxyStub], "cameo-savings") val checkingAccountsProxy = system.actorOf(Props[CheckingAccountsProxyStub], "cameo-checkings") val moneyMarketAccountsProxy = system.actorOf(Props[MoneyMarketAccountsProxyStub], "cameo-mm") val accountBalanceRetriever = system.actorOf(Props(new AccountBalanceRetriever(savingsAccountsProxy, checkingAccountsProxy, moneyMarketAccountsProxy)), "cameo-retriever1") within(300 milliseconds) { probe1.send(accountBalanceRetriever, GetCustomerAccountBalances(1L)) val result = probe1.expectMsgType[AccountBalances] result must equal(AccountBalances(Some(List((3, 15000))), Some(List((1, 150000), (2, 29000))), Some(List()))) } within(300 milliseconds) { probe2.send(accountBalanceRetriever, GetCustomerAccountBalances(2L)) val result = probe2.expectMsgType[AccountBalances] result must equal(AccountBalances( Some(List((6, 640000), (7, 1125000), (8, 40000))), Some(List((5, 80000))), Some(List((9, 640000), (10, 1125000), (11, 40000))))) } } } } @jamie_allen 21
  22. 22. Write Tests! within(300 milliseconds) { probe1.send(accountBalanceRetriever, GetCustomerAccountBalances(1L)) val result = probe1.expectMsgType[AccountBalances] result must equal(AccountBalances( Some(List((3, 15000))), Some(List((1, 150000), (2, 29000))), Some(List()))) } within(300 milliseconds) { probe2.send(accountBalanceRetriever, GetCustomerAccountBalances(2L)) val result = probe2.expectMsgType[AccountBalances] result must equal(AccountBalances( Some(List((6, 640000), (7, 1125000), (8, 40000))), Some(List((5, 80000))), Some(List((9, 640000), (10, 1125000), (11, 40000))))) } @jamie_allen 22
  23. 23. Write Moar Tests! "return a TimeoutException when timeout is exceeded" in { val checkingAccountsProxy = system.actorOf(Props[CheckingAccountsProxyStub], "cameo-timing-out-checkings") within(250 milliseconds, 500 milliseconds) { probe.send(accountBalanceRetriever, GetCustomerAccountBalances(1L)) probe.expectMsg(AccountRetrievalTimeout) } } @jamie_allen 23
  24. 24. Best Practices Several  important  learnings  I’ve  gleaned  in  my   years  of  actor  development @jamie_allen 24
  25. 25. Create a Props Factory • Creating  an  actor  within  another  actor  implicitly   closes  over  “this” • Necessary  until  spores  (SIP-­‐21)  are  part  of  Scala • Create  a  Props  factory  in  a  companion  object • Anonymous  actors  can’t  have  one! object AccountBalanceResponseHandler { def props(savingsAccounts: ActorRef, checkingAccounts: ActorRef, moneyMarketAccounts: ActorRef, originalSender: ActorRef): Props = { Props(new AccountBalanceResponseHandler(savingsAccounts, checkingAccounts, moneyMarketAccounts, originalSender)) } } @jamie_allen 25
  26. 26. Keep Your Actors Simple • Do  not  conflate  responsibilities  in  actors • Becomes  hard  to  define  the  boundaries  of   responsibility • Supervision  becomes  more  difficult  as  you   handle  more  possibilities • Debugging  becomes  very  difficult @jamie_allen 26
  27. 27. Be Explicit in Supervision • Every  non-­‐leaf  node  is  technically  a   supervisor • Create  explicit  supervisors  under  each  node   for  each  type  of  child  to  be  managed @jamie_allen 27
  28. 28. Conflated Supervision Customers C1 C2 Ac1 D1 Ac2 D2 Accounts & Devices D3 Applications A1 @jamie_allen A2 A3 A4 28
  29. 29. Explicit Supervision C1 C2 Customers Ac1 Accounts & Devices DS AS D1 Ac2 A1 @jamie_allen D2 A2 D3 A3 A4 Applications 29
  30. 30. Use Failure Zones • Multiple  isolated  zones  with  their  own   resources  (thread  pools,  etc) • Prevents  starvation  of  actors • Prevents  issues  in  one  branch  from  affecting   another val responseHandler = system.actorOf( AccountBalanceResponseHandler.props(), "cameo-handler").withDispatcher( "handler-dispatcher") @jamie_allen 30
  31. 31. No Failure Zones C1 C2 Customers DS AS Accounts & Devices Ac1 D1 Ac2 A1 @jamie_allen D2 A2 D3 A3 A4 Applications 31
  32. 32. Explicit Failure Zones C1 C2 Customers DS AS Accounts & Devices Ac1 D1 Ac2 A1 @jamie_allen D2 A2 D3 A3 A4 Applications 32
  33. 33. Push, Don’t Pull • Start  with  no  guarantees  about  delivery • Add  guarantees  only  where  you  need  them • Retry  until  you  get  the  answer  you  expect • Switch  your  actor  to  a  "nominal"  state  at  that   point @jamie_allen 33
  34. 34. Beware the Thundering Herd • Actor  systems  can  be  overwhelmed  by  "storms"  of   messages  flying  about • Do  not  pass  generic  messages  that  apply  to  many  actors • Dampen  actor  messages  if  the  exact  same  message  is   being  handled  repeatedly  within  a  certain  timeframe • Tune  your  dispatchers  and  mailboxes  via  back-­‐off   policies  and  queue  sizes • Akka  has  Circuit  Breakers  to  help  you  provide  back-­‐ pressure @jamie_allen 34
  35. 35. Create Granular Messages • Non-­‐specific  messages  about  general  events   are  dangerous AccountsUpdated • Can  result  in  "event  storms"  as  all  actors  react  to   them • Use  specific  messages  forwarded  to  actors  for   handling AccountDeviceAdded(acctNum, deviceNum) @jamie_allen 35
  36. 36. Unique IDs for Messages • Allows  you  to  track  message  flow • When  you  find  a  problem,  get  the  ID  of  the   message  that  led  to  it • Use  the  ID  to  grep  your  logs  and  display   output  just  for  that  message  flow • Akka  ensures  ordering  on  a  per  actor  basis,   also  in  logging @jamie_allen 36
  37. 37. Create Specialized Exceptions • Don't  use  java.lang.Exception  to  represent   failure  in  an  actor • Specific  exceptions  can  be  handled  explicitly • State  can  be  transferred  between  actor   incarnations  in  Akka  (if  need  be) @jamie_allen 37
  38. 38. Never Reference “this” • Actors  die • Doesn't  prevent  someone  from  calling  into  an   actor  with  another  thread • Akka  solves  this  with  the  ActorRef  abstraction • Never  expose/publish  “this” • Loop  by  sending  messages  to  “self” • Register  by  sending  references  to  your  “self” @jamie_allen 38
  39. 39. Immutable Interactions •All  messages  passed  between  actors  should  be   immutable •All  mutable  data  to  be  passed  with  messages   should  be  copied  before  sending •You  don’t  want  to  accidentally  expose  the  very   state  you’re  trying  to  protect @jamie_allen 39
  40. 40. Externalize Logic • Consider  using  external  functions  to   encapsulate  complex  business  logic • Easier  to  unit  test  outside  of  actor  context • Not  a  rule  of  thumb,  but  something  to   consider  as  complexity  increases • Not  as  big  of  an  issue  with  Akka's  TestKit @jamie_allen 40
  41. 41. Semantically Useful Logging • Trace-­‐level  logs  should  have  output  that  you   can  read  easily • Use  line  breaks  and  indentation • Both  Akka  and  Erlang  support  hooking  in   multiple  listeners  to  the  event  log  stream @jamie_allen 41
  42. 42. Use the Typesafe Console in Development •Visual  representations  of  actors  at  runtime  are   invaluable  tools •Keep  an  eye  out  for  actors  whose  mailboxes  never   drain  and  keep  getting  bigger •Look  out  for  message  handling  latency  that  keeps   going  higher •These  are  signs  that  the  actors  cannot  handle  their   load •Optimize  with  routers •Rethink  usage  of  Dispatchers @jamie_allen 42
  43. 43. Use the Typesafe Console in Development @jamie_allen 43
  44. 44. Thank You! • Some  content  provided  by  members  of  the   Typesafe  team,  including: • Jonas  Bonér • Viktor  Klang • Roland  Kuhn • Havoc  Pennington @jamie_allen 44

×