Building an MMORPG - Here be monsters

2,583 views
2,438 views

Published on

In this session, Yan will share some of his experiences of building a successful MMORPG for a social audience and insights into some of the technical challenges that his team has had to overcome along the way.

0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,583
On SlideShare
0
From Embeds
0
Number of Embeds
942
Actions
Shares
0
Downloads
12
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Building an MMORPG - Here be monsters

  1. 1. Name : Yan CuiIndustry : Social Gaming
  2. 2. Homestead
  3. 3. Homestead
  4. 4. Farming
  5. 5. CatchingMonsters!
  6. 6. In April..
  7. 7. • 200,000 unique players• 300,000 hours of game play• 100,000,000 server requests
  8. 8. • 45,000,000 items foraged• 1,000,000 butterflies caught• 4,000,000 seeds planted• 4,000,000 monsters caught
  9. 9. • 5,700,000 items bought• 24,000,000 items sold• 280,000 hours of time potion used• 526,000,000+ km travelled• around the earth 13,000 times...
  10. 10. • Bulk of the game developed by• 1 server developer• 2 flash developers• 1 producer and 2 game designers• Several artists
  11. 11. We have had to find ways to solveproblems effectively and efficiently
  12. 12. • States can be BIG• Grow with content
  13. 13. • 1000s of items on homestead• Each item has some state
  14. 14. • Storage units add 1000s more items• Players like to collect things ;-)
  15. 15. • A lot of states to track on character• 100s of items in backpack• 100s of quests/achievements• Players like to collect things ;-)
  16. 16. • Many actions requires both states at once• Operations are expensive• Best players get the worst latency
  17. 17. Stateless Server DatabaseClient
  18. 18. • Stateless server approach+ Easy to scale up and down+ Scaling can be dynamic based on load+ Every layer can scale independently
  19. 19. • Stateless server approach– Serialization/deserialization is heavy on CPU– Heavy load (1:1 read write ratio) on databases– High bandwidth usage– High latency
  20. 20. • Stateless server approach– Inefficient– Expensive
  21. 21. ServerClientSession 1Session 2
  22. 22. • Traffic is bursty• Keep state on server during bursts= Less serialization & deserialization= Less load on database= Fewer server and database nodes=
  23. 23. ServerClient Database
  24. 24. • Stateful server approach+ 500% improvement on efficiency+ 60% reduction in average latency+ Fewer game servers+ Fewer database nodes
  25. 25. But...
  26. 26. • Server affinity– Need to ensure server affinity during one burst– Need to route calls to affined server
  27. 27. • Load balancing– Some players are more active than others– Need to avoid hot spots
  28. 28. • Server hogging– Session lengths vary greatly– Need to avoid players hogging a server– Need to be able to scale down!
  29. 29. • Persist state after short inactivity• Move player to another server afterpersistence
  30. 30. • Scaling down– Need to gracefully drain player states– Need to be able to stop accepting new players
  31. 31. • Stateful server approach– Complexity+ But worthwhile!+ Others avoid scaling down to reduce complexity
  32. 32. • Progress tied to most actions• Avoid scripting• Data driven
  33. 33. Caught a Gnome
  34. 34. EXP Item GoldQuest ProgressCaught a Gnome
  35. 35. Level UpQuest ProgressEXP Item GoldCaught a GnomeQuest Complete
  36. 36. Level UpQuest ProgressEXP Item GoldCaught a GnomeQuest CompleteNew Quest
  37. 37. Level UpQuest ProgressEXP Item GoldCaught a GnomeQuest CompleteNew Quest
  38. 38. Level UpQuest ProgressEXP Item GoldCaught a GnomeQuest CompleteNew QuestAchievementProgress
  39. 39. Level UpQuest ProgressEXP Item GoldCaught a GnomeQuest CompleteNew QuestAchievementProgressAchievementComplete
  40. 40. Level UpQuest ProgressEXP Item GoldCaught a GnomeQuest CompleteNew QuestAchievementProgressAchievementComplete
  41. 41. Level UpQuest ProgressEXP Item GoldCaught a GnomeQuest CompleteNew QuestAchievementProgressAchievementComplete
  42. 42. And that’s not all...
  43. 43. • 100+ actions available in the game• Most can be tied to quests/achievements• Many yields rewards
  44. 44. • Triggered from different abstraction layers• Trapping controller• Level controller• ...
  45. 45. • Non-functional requirements• Analytics• Partner reporting• ...
  46. 46. • Message broker pattern• Something happened, it’s a FACT• Caught a Gnome• Got some EXP• ...
  47. 47. Caught Gnome TrappingQueueLevellingQuestsAchievementsAnalyticsPartnerReporting
  48. 48. Caught Gnome TrappingQueueLevellingQuestsAchievementsAnalyticsPartnerReportingIgnoreProcessProcessProcessProcessIgnore
  49. 49. Caught Gnome TrappingQueueLevellingQuestsAchievementsAnalyticsPartnerReportingEXPItemGold
  50. 50. Caught Gnome TrappingQueueLevellingQuestsAchievementsAnalyticsPartnerReportingEXPItemGold
  51. 51. Caught Gnome TrappingQueueLevellingQuestsAchievementsAnalyticsPartnerReportingEXPItemGoldProcessIgnoreIgnoreIgnoreProcessIgnore
  52. 52. Caught Gnome TrappingQueueLevellingQuestsAchievementsAnalyticsPartnerReportingEXPItemGoldLevel Up
  53. 53. Caught Gnome TrappingQueueLevellingQuestsAchievementsAnalyticsPartnerReportingEXPItemGoldLevel Up
  54. 54. • Message broker approach+ Simple+ Flexible+ Extensible
  55. 55. • Message broker approach– Requires large number of different facts+ F# to the rescue!
  56. 56. • OK for small number of DU cases• 100+ actions available in the game
  57. 57. • Players love quests
  58. 58. • Need regular content releases• Game is heavily data driven+ No developer involvement for content changes• Tech team needs to be enablers
  59. 59. • Producers need ability to• Plan changes weeks in advance• Quickly see and validate changes before release
  60. 60. • Game designers need ability to• Work on changes in isolation• Quickly iterate and test changes in isolation
  61. 61. • QA need ability to• Quickly switch between different changes to test• Smoke-test changes in live environment withoutaffecting real players
  62. 62. • Server team needs ability to• Version data changes• Rollback/hotfix data changes with no downtime
  63. 63. We version control our code,why not our data?
  64. 64. • Git + Git flow• Servers allow multiple data branches andswitching between them
  65. 65. • What do our players like to do?• Every action in the game is recorded
  66. 66. Feature Analyze
  67. 67. • Analytics is a key decision-making tool• A/B testing
  68. 68. • 150,000,000+ events recorded in April• Several GBs of analytics data a day• Fast growing!
  69. 69. • Google BigQuery+ Cheap+ Fast+ Scalable+ SQL-like query syntax
  70. 70. • Google BigQuery– OLAP– Append-only
  71. 71. • What is our code doing?
  72. 72. • Monitoring• Not the same as profiling• Should be always ON• Difficult to set up in large distributed environment
  73. 73. What to monitor?
  74. 74. • Method execution time• e.g. IO, CPU intensive• Method execution count• e.g. service entry points• Error count• ...
  75. 75. • Amazon CloudWatch+ Cheap+ Supports alarms and notifications+ Visualization tool
  76. 76. PostSharp
  77. 77. PostSharpAmazonCloudWatch+=
  78. 78. • Metrics are aggregated at instance level• Metrics gatherer has high concurrencyrequirement• F# agents to the rescue!
  79. 79. Thank You!@theburningmonk
  80. 80. Here Be Monstershttp://apps.facebook.com/herebemonstersAOP and pseudo real-time monitoring with CloudWatchhttp://bit.ly/11GL3SQhttp://bit.ly/10fe98NBuilding social games with a .Net stackhttp://bit.ly/ZKtHWbhttp://bit.ly/10CtyY9F# Agentshttp://bit.ly/ZmXuEPhttp://bit.ly/17kjbamF# Deep Diveshttp://bit.ly/XZe10E

×