2012-11-30-scalable game servers

3,760 views

Published on

Published in: Technology
0 Comments
23 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,760
On SlideShare
0
From Embeds
0
Number of Embeds
126
Actions
Shares
0
Downloads
114
Comments
0
Likes
23
Embeds 0
No embeds

No notes for slide

2012-11-30-scalable game servers

  1. 1. Scalable game servers Knut Nesheim @knutintirsdag 4. desember 12
  2. 2. tirsdag 4. desember 12
  3. 3. What we do • Casual social games • Casual: simpler, easier, nicer, cuter • Social: interact with your friends • Average title: 1 000 000 players every daytirsdag 4. desember 12
  4. 4. Purpose of backend • Play anywhere, HTTP • Cheat prevention • Ensure state integrity • Highscores • Multiplayer • Paymentstirsdag 4. desember 12
  5. 5. The first backendtirsdag 4. desember 12
  6. 6. The first backend: What • Stateless: state only in DB, mutate in app • Linux, on EC2 • Apache • MySQL • Ruby on Railstirsdag 4. desember 12
  7. 7. The first backend: Challenges • MySQL becomes a bottleneck • Sharding necessary • Working data set must fit in memory • Slow restorestirsdag 4. desember 12
  8. 8. The first backend: Challenges app db 10+ db reqs per HTTP reqtirsdag 4. desember 12
  9. 9. The first backend: Challenges • Very sensitive to network latency, synchronous IO • 16 ruby processes per node * 125 nodes = 2000 concurrent reqs • 10K HTTP rps / 2000 workers = 5 reqs/worker/second • Each request must be served in 1000ms / 5 = 200 mstirsdag 4. desember 12
  10. 10. The first backend: Solutions Challenge Solution MySQL bottleneck Shard, tune, use Redis Sensitive to latency Redis, more app servers ~200 nodes Chef, Scalariumtirsdag 4. desember 12
  11. 11. The first backend: Conclusions • Pain to operate massive DB cluster • Must read & write on master • Many single points of failure • Inefficient • All data in memorytirsdag 4. desember 12
  12. 12. The second backendtirsdag 4. desember 12
  13. 13. The second backend: What • Evolution • Dedicated hardware and network • Ruby on Rails • Redis only • Fewer and faster DB reqstirsdag 4. desember 12
  14. 14. The second backend: Conclusions • Good progress • Still sensitive to latency • Single points of failure • Can we do better?tirsdag 4. desember 12
  15. 15. The third backendtirsdag 4. desember 12
  16. 16. The third backend: What • Revolution • Stateful: app stores state while user is online, flush to disk when user goes offline • Only used data in memory • Not sensitive to latency • Use Database-as-a-servicetirsdag 4. desember 12
  17. 17. The third backend: Challenges • Erlang • State • Resource locking • Load balancing • Deployment • Lack of librariestirsdag 4. desember 12
  18. 18. The third backend: Challenges State • Data structures • Functional language • Immutability • Transactionstirsdag 4. desember 12
  19. 19. The third backend: Challenges Resource locking • Only one process per user • Central serialization • Redis, CAS, SETNX • Single point of failuretirsdag 4. desember 12
  20. 20. The third backend: Challenges Load balancing • Users play for 3-5 minutes • “proxy” forwards requests • “worker” notifies “proxy” about healthtirsdag 4. desember 12
  21. 21. The third backend: Challenges Deployment • Use Erlang code reload • “git pull && make && ./upgrade.sh” • State migration on the fly • Rolling upgradestirsdag 4. desember 12
  22. 22. The third backend: Challenges Lack of libraries • No CPAN, PiP, RubyGems • Roll our own • Eredis, github.com/wooga/eredistirsdag 4. desember 12
  23. 23. The third backend: Conclusions First backend Second backend Third backend Frequent downtime Little downtime No downtime 150+ servers 10+ servers 3 servers 100k DB reqs 10k DB reqs 1k DB reqs :( :| :)tirsdag 4. desember 12
  24. 24. The fourth backendtirsdag 4. desember 12
  25. 25. The fourth backend: What • Evolution • Multiplayer! • Real-time push updates • Improve operations • Use hosted databases • Remove SPOFtirsdag 4. desember 12
  26. 26. The fourth backend: Challenges Multiplayer • User and world as Erlang processes • Serialization • Reason about concurrency • Scaletirsdag 4. desember 12
  27. 27. The fourth backend: Challenges Real-time push updates • Push updates to client • Need browser support • “Transfer-Encoding: chunked” • Elli, github.com/knutin/ellitirsdag 4. desember 12
  28. 28. The fourth backend: Challenges Improve operations • Better load balancing • Nodes broadcasts “alive” message • Local failure detector • Divergent cluster viewstirsdag 4. desember 12
  29. 29. The fourth backend: Challenges Improve operations, cont. • Collect more metrics • ETS • Histograms • github.com/knutin/statmantirsdag 4. desember 12
  30. 30. The fourth backend: Challenges Hosted databases • Amazon AWS • DynamoDB • S3 • Low operational overhead • Higher availabilitytirsdag 4. desember 12
  31. 31. The fourth backend: Challenges Remove SPOF • “locker”: distributed consistent leases • Atomic CAS • Write quorum, 2PC • Eventually consistent replication • 330 SLOCtirsdag 4. desember 12
  32. 32. The fourth backend: Conclusions • Still in development • Benchmarks shows good results • Must prove it livetirsdag 4. desember 12
  33. 33. The future backendstirsdag 4. desember 12
  34. 34. The future backends • Erlang: processes, good IO, clustering • Harder, better games • Faster, stronger backends • Stateful, also with JVM • Another revolution?tirsdag 4. desember 12
  35. 35. Conclusions • Understand the past • The future is stateful • Identify, think, dotirsdag 4. desember 12
  36. 36. http://wooga.com/jobs/students/tirsdag 4. desember 12

×