At Scale With Style (Erlang User Conference 2012)

5,032 views
4,677 views

Published on

In the world of social gaming, the classic 2-tier of web application does not cut it anymore. We need new and better solutions.

Follow along the evolution of game servers at Wooga and get an in-depth look into the next-generation backend putting the combined forces of Erlang and Ruby to work. Learn how scalability, reliability, concurrency control and beautiful code do not need to be mutually exclusive.

0 Comments
20 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
5,032
On SlideShare
0
From Embeds
0
Number of Embeds
252
Actions
Shares
0
Downloads
117
Comments
0
Likes
20
Embeds 0
No embeds

No notes for slide

At Scale With Style (Erlang User Conference 2012)

  1. AT  SCALE  WITH  STYLE Erlang  <3  <3  <3  RubyMar$n  Rehfeld,  @klickmich
  2. At  Scale  With  Style How  we  roll Server  architectures How  to  innovate Mind  the  limits
  3. Typical  game  architecture Client HTTP Backend API
  4. Typical  game  architecture Backend State  Changes ValidaHon Persistence
  5. The  scale  is  interesHng 14  billion  requests  /  month >100,000  DB  operaHons  /  second >50,000  DB  updates  /  second
  6. Wooga’s  approach  to  developmentSmall  independent  teams  for  each  gameTeam  gets  to  choose  tools  and  technologies Same  team  also  does  ops  a,er  going  liveCulture  of  sharing Look  around  what  is  there,  pick/adapt  exis;ng   solu;ons,  but  take  ownership  for  what  you   include
  7. ExisHng  backends  –  Technology  landscape
  8. At  Scale  With  Style How  we  roll Server  architectures How  to  innovate Mind  the  limits
  9. Most  games  use  stateless  applicaHon  servers Server Database
  10. And  then  there’s  sharding lb app app app app app app app app app My My SQL SQL slave slave
  11. More  app  servers,  more  sharding lb app app app app app app app app app app app app app app app app app app My My My My My My My My SQL SQL SQL SQL SQL SQL SQL SQL slave slave slave slave slave slave slave slave
  12. Wait,  seriously?! lb lbapp app app app app app app app app app app app appapp app app app app app app app app app app app appapp app app app app app app app app app app app appMy My My My My redis redis redis redis redisSQL SQL SQL SQL SQLslave slave slave slave slave slave slave slave slave slave
  13. Find  the  flaw “Stateless  applicaHon  servers   guarantee  one  thing: The  data  is  never where  you  need  it!” Paolo  Negri,  Developer  @  Wooga
  14. Strong  session  pa`ern User  starts  playing many  transformaHons of  the  same  set  of  data User  stops  playing
  15. Stateful  servers  and  DBs Server Database One  Game  Session
  16. Stateful  game  serverOne  process  per  acHve  user  gaming  session ...  holds  the  current  state  and  is  the  only  one  that  can   modify  it  (strong  encapsulaHon) ...  handles  all  API  calls  for  the  given  user  one  a,er  the  other   (concurrency  control  through  actor  model) ...  loads  the  game  state  from  storage  and  writes  it  back   periodically  and  on  process  termina;on (;meout  =  user  stopped  playing)
  17. The  DB  is  no  longer  the  bo`leneck Stateless Stateful 30.000 22.500 15.000 700 7.500 0 DB  operations  /  second
  18. Magic  Land  uses  ErlangDetails:Awesome  presentaHon  onthe  Magic  Land  game  serverby  @knuHn  &  @hungryblankh`p://www.slideshare.net/wooga/from-­‐0-­‐to-­‐1000000-­‐daily-­‐users-­‐with-­‐erlang
  19. At  Scale  With  Style How  we  roll Server  architectures How  to  innovate Mind  the  limits
  20. Erlang  &  RubyErlang  is  great Concurrent,  robust Great  for  opera;onRuby  is  great Concise,  expressive,  flexible Great  for  development
  21. Bringing  two  worlds  together Server Worker session sender Worker session Worker ... Worker receiver session Worker
  22. Do  you  know  Mongrel2? Web  Server  that  hooks up  applicaHons  via➡ We  chose  the  same  queue  setup  &   message  format Server Worker session sender Worker session Worker ... Worker receiver session Worker
  23. ConnecHng  the  dotsMongrel2  h`p://mongrel2.org/  protocol  &  ZeroMQ  setupErlang:  h`ps://github.com/hungryblank/emongrel2Ruby rack-­‐mongrel2  fork  hKps://github.com/khiltd/khi-­‐rack-­‐mongrel2 rack  protocol  hKp://rack.rubyforge.org Sinatra  hKp://www.sinatrarb.com/➡ essenHally  we  are  speaking  HTTP  over  ZeroMQ and  can  hook  up  any  Rack-­‐based  Ruby  web   framework
  24. Example  controller  in  Ruby app.game_action /:actor/fruit_tree/self/shake, :observable => true, :affects => [:fruit_trees, :user], :params => [:x, :y] do x, y = params[:x], params[:y] fruit_trees[x, y].shake endDSL-­‐like  definiHon  of  game  acHonSkinny  as  controllers  should  be
  25. Example  model  in  Ruby class FruitTree < Tree property :last_shake_time, :type => Integer, :default => 0 property :collectable_fruit_count, :type => Integer, :default => 0 def shake raise G8::Error::Validation, "no fruit!" unless carries_fruit? session.user.xp += 1 session.user.energy -= 1 self.last_shake_time = game_time self.collectable_fruit_count = config.fruit_count end endEasily  unit  testableMinimal  amount  of  code
  26. Game  stateGame  state  is  split  in  mulHple  parts user,  map,  fruit_trees  etc.Erlang  does  not  care  about  content Serialized  Ruby  objectsErlang  does  know  mapping  of  state  parts  to  URLs
  27. Looking  back  at  the  game  acHonapp.game_action /:actor/fruit_tree/self/shake, :observable => true, :affects => [:fruit_trees, :user], :params => [:x, :y] do x, y = params[:x], params[:y] fruit_trees[x, y].shakeendMapping  of  state  parts  to  game  acHons Worker  knows  mapping Worker  pushes  mapping  to  Erlang  on  startup Erlang  can  query  mapping  if  needed
  28. NICE! http://www.flickr.com/photos/aigle_dore/
  29. At  Scale  With  Style How  we  roll Server  architectures How  to  innovate Mind  the  limits
  30. Performance  impact  for  large  payloads 200  kB  payload 6000 5000Requests  /  s 4000 Pure 3000 Erlang 2000 Ruby 1000 ZMQ 400  rps 0 1 #  of  workers 8
  31. NOT  CPU  bound25% Fixed  bandwith  limit  @  300  MB/s
  32. This  has  happened  before2,000,000"1,500,000"1,000,000" 500,000" 0" Apr*10" Jul*10" Oct*10" Jan*11" Apr*11" Jul*11" Oct*11"
  33. Example  controller  in  Erlang shake(Args, State) -> Coords = g8_map:coords(Args), Map = g8_game:get_map(State), Tree = g8_map:find(Coords, Map), NewTree = g8_fruit_tree:shake(Tree), UserEffects = [incr_xp, decr_energy], NewMap = g8_map:place(Coords, NewTree, Map), [observable, {state, g8_game:set_map(NewMap, g8_user:apply(UserEffects, State))}].
  34. Example  model  in  Erlang shake(Tree) -> validate_carries_fruit(Tree), FruitCount = g8_fruit_tree_config:fruit_count(Tree), Tree#fruit_tree{last_shake_time = g8_game:gametime(), collectable_fruit_count = FruitCount}.
  35. ✓ Y ES
  36. QuesHons? MarHn  Rehfeld @klickmichslideshare.net/wooga wooga.com/jobs

×