Your SlideShare is downloading. ×

a million bots can't be wrong

3,164

Published on

Talk on Akka 2.0, load testing of MMO games, and performance analysis/monitoring of actor-based systems given at scaladay#2 (scaladev.ru)

Talk on Akka 2.0, load testing of MMO games, and performance analysis/monitoring of actor-based systems given at scaladay#2 (scaladev.ru)

Published in: Technology, Business
0 Comments
7 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,164
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
38
Comments
0
Likes
7
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. a million bots cant be wrong @remeniuk, Viaden Media #ScalaSBP, 18-05-2012
  • 2. In Viaden our aim is to make the best poker ever
  • 3. we know thatperformance tests should be the first-class citizens
  • 4. and kill 2 birds with one stone, using bots for testing
  • 5.  #1 we can emulate 50kplayers using just onemedium EC2 instance #2 bots are interactive,so client teams can usethem in development,and QA for testing
  • 6. everyone knows Akka, huh?
  • 7. why Scala and Akka is a perfect choice for making bots?actors are !(interactive)straightforward remotingsimple scalability/clustering~30 minutes to make a DSL and CLIwith Scala and SBT
  • 8. ..., but, I have to warn you ...
  • 9. 4 dead simple tips for keeping your sanitywhen you do asynch with Akka 2.0
  • 10. tip #1: live fast, die young
  • 11. typical approach in Akka 1.x lobby desklogin l ink bot tourney
  • 12. Akka 1.x: actors have a long lifecycle lobby desk l ink nk bot li un l inkplay game bot tourney
  • 13. Akka 1.x: actors have a long lifecycle lobby desk bot k lin bot un tourneyplay tournament link bot
  • 14. in Akka 2.0 the world has changed props paths new supervision actors
  • 15. now, youre forced to do "the right thing" (c) lobby desk tournament DeskBot desk DeskBot IdleBot
  • 16. all actors are supervised lobby logindesk
  • 17. easy come lobbydesk play game IdleBot
  • 18. easy go (when supervisor to be changed) lobby desk IdleBot dies DeskBot borns
  • 19. class Lobby extends Actor { case Login(username, password) => context.actorOf(Props(new IdlePokerBot(...))) case JoinAnyDesk(props) => findDesk ! JoinDesk(props)}class Desk extends Actor { case JoinDesk(botProps)=> context.actorOf(Props(new DeskPokerBot(botProps)))}class IdlePokerBot(props: BotProps) extends Actor { case PlayNow => context.parent ! JoinAnyDesk(props); context.stop(self)}
  • 20. Props Pattern - "the soul" of an actor IdleBot BotPropsProps remains alivebetween actor"reincarnations" DeskBot BotProps
  • 21. case class BotProperties(id: Long, login: String, script: Seq[BotEvent], aggressiveness: Int, sessionId: Protocol.SessionId)class IdlePokerBot(val botProperties: BotProperties) extends Bot[Poker]class DeskPokerBot(val botProperties: BotProperties) extends Bot[Poker]
  • 22. tip #2: think beyond
  • 23. when you know, whos supervising, lifessimple akka://gpdev/user/lobby/player1234 akka://gpdev/user/lobby/desk1/player1234 akka://gpdev/user/lobby/tournament1/desk1/ player1234
  • 24. Bad newsActorRegistry, actor UUID were removed from Akkabut what should I do, now,when I dont know, whereto look for my bot?
  • 25. you can make your own registry(using Extensions, backed with adistributed data structure)...
  • 26. or, use the full power of location transparency Projection lobby Manager Projection desk var location: ActorPath DeskBot /projection/player123 IdleBot /lobby/desk123/player123
  • 27. class Projection(var container: ActorPath) extends Actor { def receive = { case newContainer: ActorPath => container = newContainer case msg => context.actorFor(container.child(self.path.name)) !msg }}class ProjectionManager extends Actor { def receive = { case Add(actor) => context.actorOf(Props(new Projection(actor.path.parent)), actor.path.name) case Forward(msg, path) => context.actorFor(path) ! msg }}projectionManager ! Add(actorRef)projectionManager ! Forward("ping", "actor1")
  • 28. class Projection(var container: ActorPath) extends Actor { def receive = { case newContainer: ActorPath => container = newContainer case msg => context.actorFor(container.child(self.path.name)) ! msg }}class ProjectionManager extends Actor { def receive = { case Add(actor) => context.actorOf(Props(new Projection(actor.path.parent)), actor.path.name) case Forward(msg, path) => context.actorFor(path) ! msg }}projectionManager ! Add(actorRef)system.actorFor(projectionManager.path.child("actor" + i)) !"ping"
  • 29. class ProjectionManager extends Actor { def receive = { case Add(actor) => context.actorOf(Props(new Projection(actor.path.parent)), actor.path.name) case Forward(msg, path) => context.actorSelection("../*/" + path) ! msg }}val projectionManager = system.actorOf(Props[ProjectionManagerRoutee] .withRouter(RoundRobinRouter(resizer = Some (DefaultResizer(lowerBound = 10, upperBound = 20)))), "projection")projectionManager ! Add(actorRef)projectionManager ! Forward("ping", "actor1")
  • 30. case class CustomRouter(n: Int, routerDispatcher: String =DefaultDispatcherId, supervisorStrategy: SupervisorStrategy =defaultStrategy) extends RouterConfig { def createRoute(props: Props, provider: RouteeProvider) = { provider.registerRoutees((1 to n).map(i => provider.context.actorOf(Props[ProjectionManager], i.toString))) def destination(sender: ActorRef, path: String) = List(Destination(sender, provider.routees(abs(path.hashCode) %n))) { case m@(sender, Add(actor)) ⇒ destination(sender, actor.path.name) case m@(sender, Forward(_, name)) ⇒ destination(sender, name) } }}
  • 31. tip #3: dont do anything stupid
  • 32. youve tried all the options, system load is fine,only 1/10 of the heap is used, but you still canstart not more than 1k bots!?
  • 33. ulimit -n <proper value>
  • 34. your actor is lacking of throughput?       wait before adding poolsshare responsibility!one fine-grained actor is enough in 99% of the cases
  • 35. 100-300 threads are serving 300 bots!?
  • 36. spawn futures, backed with standalone[bounded] pools, for blocking operations class ThirdPartyWrapper extends Actor { case F(x) => sender ! thirdPartyService.f(x) // call to a function that takes a lot of time to // complete }class ThirdPartyWrapper extends Actor { case F(x) => val _sender = sender Future(thirdPartyService.f(x)).map(_sender ! _) // ugly, but safe, and perfectly right}
  • 37. use separate dispatcherslobby-dispatcher projection-manager-dispatcherPinnedDispatcher BalancingDispatcher lobby Projection Manager desk Projection   DeskBot projection-dispatchercontainer-dispatcher desk-bot-dispatcher DispatcherDispatcher Dispatcher
  • 38. GOTCHA: Akka successfully bootstraps, even if yourdispatcher is not configured, or the config is wrong Always check the logs to make sure that dispatchers are used! [WARN][gpdev-akka.actor.default-dispatcher-1] [Dispatchers] Dispatcher [bot-system.container-dispatcher] not configured, using default-dispatcher [WARN][gpdev-bot-system.container-dispatcher-1] [PinnedDispatcherConfigurator] PinnedDispatcher [bot- system.lobby-dispatcher] not configured to use ThreadPoolExecutor, falling back to default config. [DEBUG][gpdev-akka.actor.default-dispatcher-24] [akka: //gpdev/user/poker/lobby] logged in [DEBUG][gpdev-akka.actor.default-dispatcher-14] [akka: //gpdev/user/poker/projeciton/$g/player20013] starting projection...
  • 39. tip #4: analyze that
  • 40. how to measure? Metrics - pushes various collected metrics to GraphiteCarbon and Graphite - gather metrics, and expose them via webinterface 
  • 41. object BotMetrics {1. val loggedInCount = new Counter(Metrics.newCounter(classOf[Lobby[_]], "logged-in-count"))3. GraphiteReporter.enable(1, TimeUnit.MINUTES, "localhost", 2003)}class Lobby extends Actor {2. case Login(username, pass) => BotMetrics.loggedInCount += 1}1. add logged-in user counter 4.2. update it3. enable reporting toGraphite4. build a graph in Grtaphite
  • 42. what to measure?   - mailbox size1 - throughput - time, before the message is processed (both in actor and future)2 - time to process a message - count of threads - actor pool size - heap size1 requires implementation of a custom mailbox that can expose mailbox size2 every message should be stuffed with a timestamp
  • 43. how to tune dispatchers? VisualVM - thread timeline shows, if thread polls behind dispatchersare used effectively 
  • 44. dont neglect old good logging [ERROR][05/06/2012 12:55:43.826] [gpdev-bot-system.desk-bot-dispatcher-7][akka://gpdev/user/poker/lobby/tournament5382577/desk109129/player20121]unprocessed game event: GameEvent(CHAT,None,None)
  • 45. thanks for listening
  • 46. were hiring!viaden.com/careers/vacancies.html

×