a million bots cant be wrong               @remeniuk, Viaden Media                   #ScalaSBP, 18-05-2012
In Viaden our aim is to make the best poker ever
we know thatperformance tests should be   the first-class citizens
and kill 2 birds with one stone,     using bots for testing
 #1 we can emulate 50kplayers using just onemedium EC2 instance #2 bots are interactive,so client teams can usethem in dev...
everyone knows Akka, huh?
why Scala and Akka is a perfect   choice for making bots?actors are !(interactive)straightforward remotingsimple scalabili...
..., but, I have to warn you ...
4 dead simple tips     for keeping your sanitywhen you do asynch with Akka 2.0
tip #1: live fast, die young
typical approach in Akka 1.x                      lobby                                desklogin              l ink       ...
Akka 1.x: actors have a long lifecycle                              lobby                                        desk     ...
Akka 1.x: actors have a long lifecycle                        lobby                                         desk      bot ...
in Akka 2.0 the world has changed               props     paths                          new                       supervi...
now, youre forced to do "the right thing" (c)                     lobby                                        desk       ...
all actors are supervised       lobby                              logindesk
easy come       lobbydesk                         play game               IdleBot
easy go (when supervisor to be changed)             lobby      desk                     IdleBot   dies    DeskBot     borns
class Lobby extends Actor {    case Login(username, password) =>          context.actorOf(Props(new IdlePokerBot(...)))   ...
Props Pattern - "the soul" of an actor                         IdleBot                         BotPropsProps remains alive...
case class BotProperties(id: Long,                         login: String,                         script: Seq[BotEvent],  ...
tip #2: think beyond
when you know, whos supervising, lifessimple akka://gpdev/user/lobby/player1234 akka://gpdev/user/lobby/desk1/player1234 a...
Bad newsActorRegistry, actor UUID        were removed from Akkabut what should I do, now,when I dont know, whereto look fo...
you can make your own registry(using Extensions, backed with adistributed data structure)...
or, use the full power of location transparency                                Projection         lobby                   ...
class Projection(var container: ActorPath) extends Actor {  def receive = {    case newContainer: ActorPath => container =...
class Projection(var container: ActorPath) extends Actor {    def receive = {      case newContainer: ActorPath => contain...
class ProjectionManager extends Actor {    def receive = {      case Add(actor) => context.actorOf(Props(new              ...
case class CustomRouter(n: Int, routerDispatcher: String =DefaultDispatcherId, supervisorStrategy: SupervisorStrategy =def...
tip #3: dont do anything stupid
youve tried all the options, system load is fine,only 1/10 of the heap is used, but you still canstart not more than 1k bo...
ulimit -n <proper value>
your actor is lacking of throughput?       wait before adding poolsshare responsibility!one fine-grained actor is enough i...
100-300 threads are serving 300 bots!?
spawn futures, backed with standalone[bounded] pools, for blocking operations class ThirdPartyWrapper extends Actor {     ...
use separate dispatcherslobby-dispatcher                       projection-manager-dispatcherPinnedDispatcher              ...
GOTCHA: Akka successfully bootstraps, even if yourdispatcher is not configured, or the config is wrong Always check the lo...
tip #4: analyze that
how to measure? Metrics - pushes various collected metrics to GraphiteCarbon and Graphite - gather metrics, and expose the...
object BotMetrics {1. val   loggedInCount = new Counter(Metrics.newCounter(classOf[Lobby[_]],                             ...
what to measure?         - mailbox size1    - throughput    - time, before the message is processed (both in    actor and ...
how to tune dispatchers? VisualVM - thread timeline shows, if thread polls behind dispatchersare used effectively 
dont neglect old good logging [ERROR][05/06/2012 12:55:43.826] [gpdev-bot-system.desk-bot-dispatcher-7][akka://gpdev/user/...
thanks for listening
were hiring!viaden.com/careers/vacancies.html
a million bots can't be wrong
a million bots can't be wrong
a million bots can't be wrong
a million bots can't be wrong
Upcoming SlideShare
Loading in...5
×

a million bots can't be wrong

3,286

Published on

Talk on Akka 2.0, load testing of MMO games, and performance analysis/monitoring of actor-based systems given at scaladay#2 (scaladev.ru)

Published in: Technology, Business
0 Comments
8 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,286
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
39
Comments
0
Likes
8
Embeds 0
No embeds

No notes for slide

a million bots can't be wrong

  1. 1. a million bots cant be wrong @remeniuk, Viaden Media #ScalaSBP, 18-05-2012
  2. 2. In Viaden our aim is to make the best poker ever
  3. 3. we know thatperformance tests should be the first-class citizens
  4. 4. and kill 2 birds with one stone, using bots for testing
  5. 5.  #1 we can emulate 50kplayers using just onemedium EC2 instance #2 bots are interactive,so client teams can usethem in development,and QA for testing
  6. 6. everyone knows Akka, huh?
  7. 7. why Scala and Akka is a perfect choice for making bots?actors are !(interactive)straightforward remotingsimple scalability/clustering~30 minutes to make a DSL and CLIwith Scala and SBT
  8. 8. ..., but, I have to warn you ...
  9. 9. 4 dead simple tips for keeping your sanitywhen you do asynch with Akka 2.0
  10. 10. tip #1: live fast, die young
  11. 11. typical approach in Akka 1.x lobby desklogin l ink bot tourney
  12. 12. Akka 1.x: actors have a long lifecycle lobby desk l ink nk bot li un l inkplay game bot tourney
  13. 13. Akka 1.x: actors have a long lifecycle lobby desk bot k lin bot un tourneyplay tournament link bot
  14. 14. in Akka 2.0 the world has changed props paths new supervision actors
  15. 15. now, youre forced to do "the right thing" (c) lobby desk tournament DeskBot desk DeskBot IdleBot
  16. 16. all actors are supervised lobby logindesk
  17. 17. easy come lobbydesk play game IdleBot
  18. 18. easy go (when supervisor to be changed) lobby desk IdleBot dies DeskBot borns
  19. 19. class Lobby extends Actor { case Login(username, password) => context.actorOf(Props(new IdlePokerBot(...))) case JoinAnyDesk(props) => findDesk ! JoinDesk(props)}class Desk extends Actor { case JoinDesk(botProps)=> context.actorOf(Props(new DeskPokerBot(botProps)))}class IdlePokerBot(props: BotProps) extends Actor { case PlayNow => context.parent ! JoinAnyDesk(props); context.stop(self)}
  20. 20. Props Pattern - "the soul" of an actor IdleBot BotPropsProps remains alivebetween actor"reincarnations" DeskBot BotProps
  21. 21. case class BotProperties(id: Long, login: String, script: Seq[BotEvent], aggressiveness: Int, sessionId: Protocol.SessionId)class IdlePokerBot(val botProperties: BotProperties) extends Bot[Poker]class DeskPokerBot(val botProperties: BotProperties) extends Bot[Poker]
  22. 22. tip #2: think beyond
  23. 23. when you know, whos supervising, lifessimple akka://gpdev/user/lobby/player1234 akka://gpdev/user/lobby/desk1/player1234 akka://gpdev/user/lobby/tournament1/desk1/ player1234
  24. 24. Bad newsActorRegistry, actor UUID were removed from Akkabut what should I do, now,when I dont know, whereto look for my bot?
  25. 25. you can make your own registry(using Extensions, backed with adistributed data structure)...
  26. 26. or, use the full power of location transparency Projection lobby Manager Projection desk var location: ActorPath DeskBot /projection/player123 IdleBot /lobby/desk123/player123
  27. 27. class Projection(var container: ActorPath) extends Actor { def receive = { case newContainer: ActorPath => container = newContainer case msg => context.actorFor(container.child(self.path.name)) !msg }}class ProjectionManager extends Actor { def receive = { case Add(actor) => context.actorOf(Props(new Projection(actor.path.parent)), actor.path.name) case Forward(msg, path) => context.actorFor(path) ! msg }}projectionManager ! Add(actorRef)projectionManager ! Forward("ping", "actor1")
  28. 28. class Projection(var container: ActorPath) extends Actor { def receive = { case newContainer: ActorPath => container = newContainer case msg => context.actorFor(container.child(self.path.name)) ! msg }}class ProjectionManager extends Actor { def receive = { case Add(actor) => context.actorOf(Props(new Projection(actor.path.parent)), actor.path.name) case Forward(msg, path) => context.actorFor(path) ! msg }}projectionManager ! Add(actorRef)system.actorFor(projectionManager.path.child("actor" + i)) !"ping"
  29. 29. class ProjectionManager extends Actor { def receive = { case Add(actor) => context.actorOf(Props(new Projection(actor.path.parent)), actor.path.name) case Forward(msg, path) => context.actorSelection("../*/" + path) ! msg }}val projectionManager = system.actorOf(Props[ProjectionManagerRoutee] .withRouter(RoundRobinRouter(resizer = Some (DefaultResizer(lowerBound = 10, upperBound = 20)))), "projection")projectionManager ! Add(actorRef)projectionManager ! Forward("ping", "actor1")
  30. 30. case class CustomRouter(n: Int, routerDispatcher: String =DefaultDispatcherId, supervisorStrategy: SupervisorStrategy =defaultStrategy) extends RouterConfig { def createRoute(props: Props, provider: RouteeProvider) = { provider.registerRoutees((1 to n).map(i => provider.context.actorOf(Props[ProjectionManager], i.toString))) def destination(sender: ActorRef, path: String) = List(Destination(sender, provider.routees(abs(path.hashCode) %n))) { case m@(sender, Add(actor)) ⇒ destination(sender, actor.path.name) case m@(sender, Forward(_, name)) ⇒ destination(sender, name) } }}
  31. 31. tip #3: dont do anything stupid
  32. 32. youve tried all the options, system load is fine,only 1/10 of the heap is used, but you still canstart not more than 1k bots!?
  33. 33. ulimit -n <proper value>
  34. 34. your actor is lacking of throughput?       wait before adding poolsshare responsibility!one fine-grained actor is enough in 99% of the cases
  35. 35. 100-300 threads are serving 300 bots!?
  36. 36. spawn futures, backed with standalone[bounded] pools, for blocking operations class ThirdPartyWrapper extends Actor { case F(x) => sender ! thirdPartyService.f(x) // call to a function that takes a lot of time to // complete }class ThirdPartyWrapper extends Actor { case F(x) => val _sender = sender Future(thirdPartyService.f(x)).map(_sender ! _) // ugly, but safe, and perfectly right}
  37. 37. use separate dispatcherslobby-dispatcher projection-manager-dispatcherPinnedDispatcher BalancingDispatcher lobby Projection Manager desk Projection   DeskBot projection-dispatchercontainer-dispatcher desk-bot-dispatcher DispatcherDispatcher Dispatcher
  38. 38. GOTCHA: Akka successfully bootstraps, even if yourdispatcher is not configured, or the config is wrong Always check the logs to make sure that dispatchers are used! [WARN][gpdev-akka.actor.default-dispatcher-1] [Dispatchers] Dispatcher [bot-system.container-dispatcher] not configured, using default-dispatcher [WARN][gpdev-bot-system.container-dispatcher-1] [PinnedDispatcherConfigurator] PinnedDispatcher [bot- system.lobby-dispatcher] not configured to use ThreadPoolExecutor, falling back to default config. [DEBUG][gpdev-akka.actor.default-dispatcher-24] [akka: //gpdev/user/poker/lobby] logged in [DEBUG][gpdev-akka.actor.default-dispatcher-14] [akka: //gpdev/user/poker/projeciton/$g/player20013] starting projection...
  39. 39. tip #4: analyze that
  40. 40. how to measure? Metrics - pushes various collected metrics to GraphiteCarbon and Graphite - gather metrics, and expose them via webinterface 
  41. 41. object BotMetrics {1. val loggedInCount = new Counter(Metrics.newCounter(classOf[Lobby[_]], "logged-in-count"))3. GraphiteReporter.enable(1, TimeUnit.MINUTES, "localhost", 2003)}class Lobby extends Actor {2. case Login(username, pass) => BotMetrics.loggedInCount += 1}1. add logged-in user counter 4.2. update it3. enable reporting toGraphite4. build a graph in Grtaphite
  42. 42. what to measure?   - mailbox size1 - throughput - time, before the message is processed (both in actor and future)2 - time to process a message - count of threads - actor pool size - heap size1 requires implementation of a custom mailbox that can expose mailbox size2 every message should be stuffed with a timestamp
  43. 43. how to tune dispatchers? VisualVM - thread timeline shows, if thread polls behind dispatchersare used effectively 
  44. 44. dont neglect old good logging [ERROR][05/06/2012 12:55:43.826] [gpdev-bot-system.desk-bot-dispatcher-7][akka://gpdev/user/poker/lobby/tournament5382577/desk109129/player20121]unprocessed game event: GameEvent(CHAT,None,None)
  45. 45. thanks for listening
  46. 46. were hiring!viaden.com/careers/vacancies.html
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×