Erlang, the big switch in     social games    Paolo Negri @hungryblank
About this talkErlang adoption @  how we did itand what we found
Social Games• Run in the browser• Published on social networks• Popularity measured in millions of daily  users
Social GamesFlash client (game)   HTTP API
Social GamesFlash client               • Game actions need to be                 persisted and validated               • 1...
Social Games                            HTTP API• @ 1 000 000 daily users• 5000 HTTP reqs/sec• more than 90% writes
Social Games @wooga                   HTTP API• up to 1 300 000 daily users on a single game• team of 2 people• develop fr...
The hard nuthttp://www.flickr.com/photos/mukluk/315409445/
The hard nut           60K queries/sec           Read/Write ~ 1@ 1mln daily users your SQL /NOSQL data layer is the main t...
The hard nut roots Stateless app servers     • very easy to scale (add one more)     • easy to develop     • Plenty of rea...
The hard nut roots Stateless app servers     • high pressure on data layer     • extreme query/req optimization       mean...
The case for erlang“A gaming session is a stream of closerequests altering repeatedly the samegame state”
The case for erlang“What about application servers thatunderstand and represent efficiently theconcept of a gaming session?”
The case for erlang “Which language or framework for:”• Long lived state• Safe handling of a multitude of independent,  di...
The case for erlang• gen_server, gen_fsm good approach to state  handling• Processes are very good at isolating failures• ...
Pitching the idea“This architecture let us leverage factorswe can’t currently leverage in order tosolve the scaling challe...
Pitch won! CTO says:• Develop proof of concept                             Works?• Hire 1 developer with erlang experience...
Pitch won! CTO says:• Develop proof of concept                             Works?                                      Yes...
How to hire erlang devs?http://www.flickr.com/photos/legofenris/4499417549
Hiring• mailing list• Linkedin• No agents• No paid ads
Hiring• 2 months• Handful of applications with the right skills• Hired one dev
HiringWe also got quite a few...“I’m really experienced in _somethingelse_ but I would like to learn and workwith erlang”F...
How tohttp://www.flickr.com/photos/jakeandlindsay/5524669257/                                                         With
getting startedNice books around
getting startedhttp://www.erlang.org/dockeytake(Key, N, TupleList1) -> {value, Tuple, TupleList2} | falseTypes:Key = term(...
getting startedgen_server, gen_tcp, gen_*Very well documented and awesome bydefault
getting startedThe rest of OTP was black magic andfoggy until the book was publishedStill official howtos and complete docs...
getting started                Learn you some erlang     [1]        • Hands on book        • Plenty of small examples     ...
Erlang tools                                                  we’re usinghttp://www.flickr.com/photos/loonatic/3871627355
tools we’re using                                 rebar         • Dependency management         • Eunit integration       ...
tools we’re using                                         eunit        • not tested =:= not done        • Eunit is a compl...
tools we’re using                                  erlymock         • mocking framework         • Mock all module or nothi...
tools we’re using                                        misultin       • library for building lightweight HTTP servers   ...
tools we’re using                                  gproc         • Extended process registry for Erlang         • Used to ...
tools we’re using                                    tsung         •    Multi protocol (HTTP, XMPP) benchmarking tool     ...
tools we’re using                                    tsung         •    Generates ~ 2500 reqs/sec on AWS m1.large         ...
Erlangbroughtwith it fewbenefits weweren’texplicitlylookingfor...
Freebies                 Transactionshandle_call(Request, From, State) ->    NewState = mylib:myfun(State),    {reply, ok,...
Freebies               Transactions• Immutability (erlang)• Functional approach (erlang)• Careful persistency side effects...
FreebiesDecoupling     HTTPApplication Logic Persistency/DB
Freebies                  Decoupling•   Switching from HTTP to sockets would take a day•   Changing persistency solution w...
Freebies        Test a cluster on a box•   Multiple erlang nodes on a single machine•   Continuous integration for cluster...
Freebies            New point of view•   From the erlang point of view problems look    different•   Approaches used in th...
What we          of
what we like    Low level runtime infos• erlang:statistics• erlang:system_monitor• erlang:process_info
what we like                       Performance tweaks        • Efficiency guide              [1]        • Control on initia...
what we like “erlang-questions” mailing list• Very active and responsive• “Big guns” open to help• Threads go in depth on ...
what we like                 Github• Thanks for sharing your code on github!• Lots of answers to “how is that done?”• Grow...
what we like         Github Tags AppealPlease use tags on your github projects...So we can bind rebar deps to a tag...git ...
works well with erlang...http://www.flickr.com/photos/legofenris/4563478042
works well with                 Rake!• Integration test tasks• Continuous integration setup/teardown• Other tasks specific ...
works well with     Ruby test frameworks• Flexible assertions• Rapid modeling of API• Low maintenance overhead• Easy fixtur...
works well with                  Redis  “An open source, advanced key-value store”• In memory key value store• Optional on...
works well with         Redis    Binary safe protocol            +   Binary safe datatypes
works well with            Redis  200K ops/sec on a single thread           Serialization?     term_to_binary/1     binary...
works well with               Cloud (AWS)•   Erlang understands clustering out of the box•   Nodes joining/leaving handlin...
We’re missing...
we’re missing        HashMapsuser[:inventory][:food][:apples]             => 1User#user{inventory = Inventory}            ...
we’re missingpackages à la CPAN    rubygems...
we’re missing           also...github is great but which   one of 5 forks is the   “authoritative” one?
we’re missing       will agnerA Giant Nebula of Erlang  Repositories be the        answer?   http://erlagner.org
we’re missingLive environment performance       monitoring S.A.S.       ruby/java/php  have http://newrelic.com           ...
how is it going?• On time so far• Benchmarks look good• Stay tuned... launch date approaching!
Thanks!http://www.wooga.com/jobs paolo.negri@wooga.com     @hungryblank
Upcoming SlideShare
Loading in...5
×

Erlang, the big switch in social games

10,884

Published on

Online games backend are challenging applications, a single user generates one http call every few seconds, usage volume can spike very quickly and balance between data read and write is close to 50/50 which make the use of write through cache or other common scaling approaches not so effective.
Follow how in our quest for a better architecture to serve millions of games sessions daily and reduce our resource usage we took the decision to write in Erlang our third generation game backend, see how we’re leveraging the actor model in order to change how we use and conceive our persistency layer. See also how introducing Erlang as a new tool in a company is working out, what we found hard from an organizational and technical point of view, which obstacle we hit and how as technical guys we convinced our management to take the risk of bringing in house a different technology.

Published in: Technology
0 Comments
26 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
10,884
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
198
Comments
0
Likes
26
Embeds 0
No embeds

No notes for slide

Erlang, the big switch in social games

  1. 1. Erlang, the big switch in social games Paolo Negri @hungryblank
  2. 2. About this talkErlang adoption @ how we did itand what we found
  3. 3. Social Games• Run in the browser• Published on social networks• Popularity measured in millions of daily users
  4. 4. Social GamesFlash client (game) HTTP API
  5. 5. Social GamesFlash client • Game actions need to be persisted and validated • 1 API call every 2 secs
  6. 6. Social Games HTTP API• @ 1 000 000 daily users• 5000 HTTP reqs/sec• more than 90% writes
  7. 7. Social Games @wooga HTTP API• up to 1 300 000 daily users on a single game• team of 2 people• develop from scratch• AND do deployments + live operations• Releasing on weekly schedule
  8. 8. The hard nuthttp://www.flickr.com/photos/mukluk/315409445/
  9. 9. The hard nut 60K queries/sec Read/Write ~ 1@ 1mln daily users your SQL /NOSQL data layer is the main technicalchallenge
  10. 10. The hard nut roots Stateless app servers • very easy to scale (add one more) • easy to develop • Plenty of ready frameworks
  11. 11. The hard nut roots Stateless app servers • high pressure on data layer • extreme query/req optimization means heavy refactoring • query/req of 2 best possible ratio
  12. 12. The case for erlang“A gaming session is a stream of closerequests altering repeatedly the samegame state”
  13. 13. The case for erlang“What about application servers thatunderstand and represent efficiently theconcept of a gaming session?”
  14. 14. The case for erlang “Which language or framework for:”• Long lived state• Safe handling of a multitude of independent, different states• All the above on a big cluster of machines
  15. 15. The case for erlang• gen_server, gen_fsm good approach to state handling• Processes are very good at isolating failures• clustering is a concept in the language itself
  16. 16. Pitching the idea“This architecture let us leverage factorswe can’t currently leverage in order tosolve the scaling challenge”“By the way erlang seems to be a verygood match for this architecture”
  17. 17. Pitch won! CTO says:• Develop proof of concept Works?• Hire 1 developer with erlang experience• Go on and develop the backend for our new game
  18. 18. Pitch won! CTO says:• Develop proof of concept Works? Yes!• Hire 1 developer with erlang experience• Go on and develop the backend for our new game
  19. 19. How to hire erlang devs?http://www.flickr.com/photos/legofenris/4499417549
  20. 20. Hiring• mailing list• Linkedin• No agents• No paid ads
  21. 21. Hiring• 2 months• Handful of applications with the right skills• Hired one dev
  22. 22. HiringWe also got quite a few...“I’m really experienced in _somethingelse_ but I would like to learn and workwith erlang”From people that are *good* for real
  23. 23. How tohttp://www.flickr.com/photos/jakeandlindsay/5524669257/ With
  24. 24. getting startedNice books around
  25. 25. getting startedhttp://www.erlang.org/dockeytake(Key, N, TupleList1) -> {value, Tuple, TupleList2} | falseTypes:Key = term()N = 1..tuple_size(Tuple)TupleList1 = TupleList2 = [Tuple]Tuple = tuple() Docs are complete, well written but sometime missing examples
  26. 26. getting startedgen_server, gen_tcp, gen_*Very well documented and awesome bydefault
  27. 27. getting startedThe rest of OTP was black magic andfoggy until the book was publishedStill official howtos and complete docswould be nice to have
  28. 28. getting started Learn you some erlang [1] • Hands on book • Plenty of small examples • Written with the beginners in mind[1] http://learnyousomeerlang.com
  29. 29. Erlang tools we’re usinghttp://www.flickr.com/photos/loonatic/3871627355
  30. 30. tools we’re using rebar • Dependency management • Eunit integration • Release packaging • Enforces OTP standardhttps://github.com/basho/rebar
  31. 31. tools we’re using eunit • not tested =:= not done • Eunit is a complete unit testing framework • No guidelines/tools for testing gen_servershttp://www.erlang.org/doc/apps/eunit/chapter.html
  32. 32. tools we’re using erlymock • mocking framework • Mock all module or nothing • Works, but very limitedhttps://github.com/nialscorva/erlymock
  33. 33. tools we’re using misultin • library for building lightweight HTTP servers • Tiny amount of code • Performs well • Good match for minimalist REST APIhttps://github.com/ostinelli/misultin
  34. 34. tools we’re using gproc • Extended process registry for Erlang • Used to map active users/sessions -> pids • Works fine at 100K processes/node • We use it only in local (single node) modehttps://github.com/esl/gproc
  35. 35. tools we’re using tsung • Multi protocol (HTTP, XMPP) benchmarking tool • Synthetic load only way to “scale proof” new project • Able to test non trivial call sequences • Can actually simulate game playhttp://tsung.erlang-projects.org/
  36. 36. tools we’re using tsung • Generates ~ 2500 reqs/sec on AWS m1.large • Flexible but hard to extend • Code base rather obscure, not hosted on githubhttp://tsung.erlang-projects.org/
  37. 37. Erlangbroughtwith it fewbenefits weweren’texplicitlylookingfor...
  38. 38. Freebies Transactionshandle_call(Request, From, State) -> NewState = mylib:myfun(State), {reply, ok, NewState}; Immutability -> last sane state is always known and easily available
  39. 39. Freebies Transactions• Immutability (erlang)• Functional approach (erlang)• Careful persistency side effects handling (you)
  40. 40. FreebiesDecoupling HTTPApplication Logic Persistency/DB
  41. 41. Freebies Decoupling• Switching from HTTP to sockets would take a day• Changing persistency solution would take a day (data migration excluded)• Both changes would leave application logic untouched
  42. 42. Freebies Test a cluster on a box• Multiple erlang nodes on a single machine• Continuous integration for clustering logic!• End to end test on a single box
  43. 43. Freebies New point of view• From the erlang point of view problems look different• Approaches used in this project were ported to ruby projects• Erlang makes for a good thinking exsercise
  44. 44. What we of
  45. 45. what we like Low level runtime infos• erlang:statistics• erlang:system_monitor• erlang:process_info
  46. 46. what we like Performance tweaks • Efficiency guide [1] • Control on initial process heap • Garbage collection can be measured[1] http://www.erlang.org/doc/efficiency_guide/introduction.html
  47. 47. what we like “erlang-questions” mailing list• Very active and responsive• “Big guns” open to help• Threads go in depth on interesting topics
  48. 48. what we like Github• Thanks for sharing your code on github!• Lots of answers to “how is that done?”• Growing collection of erlang libs
  49. 49. what we like Github Tags AppealPlease use tags on your github projects...So we can bind rebar deps to a tag...git  push  --tags
  50. 50. works well with erlang...http://www.flickr.com/photos/legofenris/4563478042
  51. 51. works well with Rake!• Integration test tasks• Continuous integration setup/teardown• Other tasks specific to the project
  52. 52. works well with Ruby test frameworks• Flexible assertions• Rapid modeling of API• Low maintenance overhead• Easy fixturing/mocking
  53. 53. works well with Redis “An open source, advanced key-value store”• In memory key value store• Optional on disk persistency• Optional replication
  54. 54. works well with Redis Binary safe protocol + Binary safe datatypes
  55. 55. works well with Redis 200K ops/sec on a single thread Serialization? term_to_binary/1 binary_to_term/1
  56. 56. works well with Cloud (AWS)• Erlang understands clustering out of the box• Nodes joining/leaving handling• Async I/O to compensate high latency• Cloud works better if you’re parallel
  57. 57. We’re missing...
  58. 58. we’re missing HashMapsuser[:inventory][:food][:apples] => 1User#user{inventory = Inventory} ...
  59. 59. we’re missingpackages à la CPAN rubygems...
  60. 60. we’re missing also...github is great but which one of 5 forks is the “authoritative” one?
  61. 61. we’re missing will agnerA Giant Nebula of Erlang Repositories be the answer? http://erlagner.org
  62. 62. we’re missingLive environment performance monitoring S.A.S. ruby/java/php have http://newrelic.com erlang?
  63. 63. how is it going?• On time so far• Benchmarks look good• Stay tuned... launch date approaching!
  64. 64. Thanks!http://www.wooga.com/jobs paolo.negri@wooga.com @hungryblank
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×