Successfully reported this slideshow.

Erlang factory layered architecture - final

1,026 views

Published on

  • Be the first to comment

Erlang factory layered architecture - final

  1. 1. A Layered ArchitectureRobustness & Efficiencydennis.docter@spilgames.comMarch 22nd, 2013
  2. 2. 2
  3. 3. A legacy of growth
  4. 4. Social and casual gaming platform 50+ portals 190+ countries 4
  5. 5. Casual games Multi player games Social games Native games
  6. 6. 200+ million unique users per month250"200" Traffic growth150" over 1600%100" 50" 5/1/12" 1/1/12" 9/1/11" 0" 5/1/11" 1/1/11" 9/1/10" 5/1/10" 1/1/10" 9/1/09" 5/1/09" 1/1/09" 9/1/08" 5/1/08" 1/1/08" 9/1/07" 5/1/07" 1/1/07" 90" 80" 70" 60" Tech dept. 50" 40" growth 30" over 2000% 20" 10" 0" 1/1/08" 1/1/09" 1/1/10" 1/1/11" 1/1/12"
  7. 7. XML Json Interfacing Interfacing + plain GET Intefacing Intefacing Caching SQL Business Caching Authorization + Authentication Business Caching Caching logic logic Authorization Business SQL Business Business logic Caching Caching logic logic Business SQL logic Caching SQL SQL Business SQL logic Swift SQLSchema API Schema Schema Schema Mysql Write DB Mysql DB Mysql DB Mysql DBMysqlReader Memcache Mysql Mysql DB ClusterReaders 7
  8. 8. Problems• Inconsistent  design  &  interfaces• Unclear  responsibili4es• Technical  debt  • Independent  teams  developing   the  same  features.• 7  layers  of  caching  hell• Scaling  
  9. 9. Problems• Inconsistent  design  &  interfaces• Unclear  responsibili4es• Technical  debt  • Independent  teams  developing   the  same  features.• 7  layers  of  caching  hell• Scaling   not a problem :)
  10. 10. 9
  11. 11. A newapproach
  12. 12. New architecture expectations• Flexible• Predictable• Scalable TM d wor t B uzz lian p C om 11
  13. 13. Choices made• Simple• Clean  separa4on• No  horizontal  dependencies• Fault  tolerant• Distributed 12
  14. 14. Layers Responsibilities Client • Integra4on • State • Authen4ca4onInterface • Aggrega4on • Valida4on • Authoriza4onService • Composi4on • Transforma4on • CachingStorage • Persistence • Sharding 13
  15. 15. Layers Responsibilities Client • Integra4on • State • Authen4ca4onInterface • Aggrega4on Stateless • Valida4on • Authoriza4onService • Composi4on • Transforma4on • CachingStorage • Persistence • Sharding 13
  16. 16. Building blocksApplication• Small & Simple• Limited scope• Quick to deployPlatform• Abstracts complexity• Low change rate• Long lifetime 14
  17. 17. Walkthrough
  18. 18. Interfaces https post /user/get/ 3rd Party {“auth”: {“token”: “UwAA_AWeff...”} Highscore Backend } Service list 1 get([friends]) Application User 2 User 2 Interface Interface Service listget(me) 1 count /application/highscores/ 1 Friends Service 1 User Service Awards Service 16
  19. 19. Widgets
  20. 20. Widgets logo widget profile widget search widget navigation widget recommendation widget new_games widget most_played widget recent_games widget social_games widget
  21. 21. definition Userprofile logic Service template + Award html, Service [properties] 18
  22. 22. header definition User logo profile Service logic templatenavigation search + Award html, Service [properties] 18
  23. 23. ame.com /htt p://www.ag preprocess ...widget platform footer ... home_page new games header body ... definition User logo profile Service logic template navigation search + Award html, Service [properties] 18
  24. 24. Services user{“context”:{ get interface “guid”: 62623423, “siteid” : 88, “locale” : “en_US”},”guid”: 324112535} { “gid”: “324112535”, user service “username” : “-fooba-”, “avatarId” : “http://agame.com/avatars/99142.png” } check fetch filter modify { “gid”: “324112535”, “username” : “-fooba-”, profile “firstName” : “john” storage “avatarId” : 99142 }
  25. 25. Storage platform• API  instead  of  SQL• Buckets  of  data  =  Key-­‐Value  with  Schema• Abstracts  storage Bucket Bucket Bucket Bucket Bucket Bucket Storage Node 1 Storage Node 2 Cache Cache Shard 1 Shard 2 Shard .. 20
  26. 26. Job handlers Account interface-account.register AuditInterface {“guid”:7734} Handler Audit User service-user.register Log Service {“result”:”success”,”guid”:”73324”} Welcome Profile Handler bucket rabbitmq send email 21
  27. 27. Fitting ittogether
  28. 28. interface JSON Load  balancer Auth User Award Profile Game in@erface interface interface widget widget Protobuf Widget  PlaGormservice User User Game Award Award service service service service service Protobufstorage User Game Profiles Profiles Games Award Award bucket bucket bucket bucket bucket Storage  PlaGorm 23
  29. 29. How we do it• Strictly  define  EVERYTHING   • input/output  format,  urls,   • rpc  func4ons,  types,  valida4on• Strict  rules  on  changes • Everything  op4onal • Unless  you’re  really,  really  sure  (and  have  proof!)• Generate  code  and  documenta4on • Interface  derived  from  specifica4on • Documenta4on  derived  from  specifica4on 24
  30. 30. • XML:  Defini4on• XSLT:  Transforma4on• Piqi:   • Protobuf  &  Json   encoding/decoding • HTTP  RPC (h`p://piqi.org/) 25
  31. 31. What we ended up with…• Small  applica4ons  running  on  their  own  nodes• Scale  up  where  necessary• Many  of  them  on  the  same  machine• Small  cluster  already  contains  hundreds  of  nodes 26
  32. 32. Keeping it connected• Connec4ng  all  nodes  is  a  bad  idea: • Not  needed • Number  of  open  connec4ons• Solu4on:  use  hidden  nodes • Each  node  specifies  the  nodes  it  needs. • How  does  it  scale? 27
  33. 33. Hidden nodes vs. visible nodes Hidden" Visible" 160000"Number of TCP connections 145530$ 140000" 120000" 114960$ 100000" 87990$ 80000" 64620$ 60000" 44850$ 40000" 28680$ 16110$ 20000" 7140$ 1770$ 240$ 320$ 400$ 480$ 560$ 640$ 720$ 0" 2" 4" 6" 8" 10" 12" 14" 16" 18" Number of servers (~30 nodes / server) 28
  34. 34. Router• Each  applica4on  has  it’s  own  router• Scans  &  Monitors  • Balances  requests  across  nodes• Parallelizes  requests  as  much  as  possible• Applica4on  can  only  talk  to  what  it  has  defined 29
  35. 35. But what about response times?• Use  proper  transports  (Erlang  RPC  vs.  HTTP)• Yes  there  is  some  transport  overhead• Typically  <  1ms  per  layer  (+network  latency)• But  each  layer  has  predictable  response  4mes• As  long  as  enough  downward  resources  are   available 30
  36. 36. The old ... 31
  37. 37. .. and the new. 32
  38. 38. ClosingWords
  39. 39. Where are we now..• Interface  layer  has  been  defined  and  largely   implemented• ImplemenMng  the  service  &  storage  layer  step  by  step • Meanwhile  we  have  adapters  in  place• Several  of  the  new  API’s  are  in  use  in  producMon• First  portals  on  the  new  widget  plaGorm  will  be  deployed   in  the  coming  months 34
  40. 40. ... And where we’re going• Deploying  to  mul4ple  datacenters  this  year• Bring  elas4city  to  our  deployments• Decommission  our  “old  architecture”  • Erlang  game  plahorm 35
  41. 41. Lessons Learned• An  architectural  vision  is  crucial• Integra4ng  into  a  mess  is  a  mess• Layers  with  clear  responsibili4es• Strict  communica4on• Package  small• Change  is  the  only  constant• Op4mize  for  predictability  first   36
  42. 42. Questions37
  43. 43. Thank you! 38

×