Building an MMORPG - Here be monsters

2. Name : Yan Cui Industry : Social Gaming

12. Homestead

13. Homestead

14. Farming

24. Catching Monsters!

36. In April..

37. • 200,000 unique players • 300,000 hours of game play • 100,000,000 server requests

38. • 45,000,000 items foraged • 1,000,000 butterflies caught • 4,000,000 seeds planted • 4,000,000 monsters caught

39. • 5,700,000 items bought • 24,000,000 items sold • 280,000 hours of time potion used • 526,000,000+ km travelled • around the earth 13,000 times...

40. • Bulk of the game developed by • 1 server developer • 2 flash developers • 1 producer and 2 game designers • Several artists

41. We have had to find ways to solve problems effectively and efficiently

43. • States can be BIG • Grow with content

45. • 1000s of items on homestead • Each item has some state

47. • Storage units add 1000s more items • Players like to collect things ;-)

50. • A lot of states to track on character • 100s of items in backpack • 100s of quests/achievements • Players like to collect things ;-)

51. • Many actions requires both states at once • Operations are expensive • Best players get the worst latency

52. Stateless Server DatabaseClient

53. • Stateless server approach + Easy to scale up and down + Scaling can be dynamic based on load + Every layer can scale independently

54. • Stateless server approach – Serialization/deserialization is heavy on CPU – Heavy load (1:1 read write ratio) on databases – High bandwidth usage – High latency

55. • Stateless server approach – Inefficient – Expensive

56. ServerClient Session 1 Session 2

57. • Traffic is bursty • Keep state on server during bursts = Less serialization & deserialization = Less load on database = Fewer server and database nodes =

58. ServerClient Database

59. • Stateful server approach + 500% improvement on efficiency + 60% reduction in average latency + Fewer game servers + Fewer database nodes

60. But...

61. • Server affinity – Need to ensure server affinity during one burst – Need to route calls to affined server

62. • Load balancing – Some players are more active than others – Need to avoid hot spots

63. • Server hogging – Session lengths vary greatly – Need to avoid players hogging a server – Need to be able to scale down!

64. • Persist state after short inactivity • Move player to another server after persistence

66. • Scaling down – Need to gracefully drain player states – Need to be able to stop accepting new players

67. • Stateful server approach – Complexity + But worthwhile! + Others avoid scaling down to reduce complexity

68. • Progress tied to most actions • Avoid scripting • Data driven

69. Caught a Gnome

70. EXP Item Gold Quest Progress Caught a Gnome

71. Level Up Quest Progress EXP Item Gold Caught a Gnome Quest Complete

72. Level Up Quest Progress EXP Item Gold Caught a Gnome Quest Complete New Quest

73. Level Up Quest Progress EXP Item Gold Caught a Gnome Quest Complete New Quest

74. Level Up Quest Progress EXP Item Gold Caught a Gnome Quest Complete New Quest Achievement Progress

75. Level Up Quest Progress EXP Item Gold Caught a Gnome Quest Complete New Quest Achievement Progress Achievement Complete

78. And that’s not all...

79. • 100+ actions available in the game • Most can be tied to quests/achievements • Many yields rewards

80. • Triggered from different abstraction layers • Trapping controller • Level controller • ...

81. • Non-functional requirements • Analytics • Partner reporting • ...

83. • Message broker pattern • Something happened, it’s a FACT • Caught a Gnome • Got some EXP • ...

84. Caught Gnome Trapping Queue Levelling Quests Achievements Analytics Partner Reporting

85. Caught Gnome Trapping Queue Levelling Quests Achievements Analytics Partner Reporting Ignore Process Process Process Process Ignore

86. Caught Gnome Trapping Queue Levelling Quests Achievements Analytics Partner Reporting EXP Item Gold

87. Caught Gnome Trapping Queue Levelling Quests Achievements Analytics Partner Reporting EXP Item Gold

88. Caught Gnome Trapping Queue Levelling Quests Achievements Analytics Partner Reporting EXP Item Gold Process Ignore Ignore Ignore Process Ignore

89. Caught Gnome Trapping Queue Levelling Quests Achievements Analytics Partner Reporting EXP Item Gold Level Up

90. Caught Gnome Trapping Queue Levelling Quests Achievements Analytics Partner Reporting EXP Item Gold Level Up

91. • Message broker approach + Simple + Flexible + Extensible

92. • Message broker approach – Requires large number of different facts + F# to the rescue!

94. • OK for small number of DU cases • 100+ actions available in the game

96. • Players love quests

97. • Need regular content releases • Game is heavily data driven + No developer involvement for content changes • Tech team needs to be enablers

98. • Producers need ability to • Plan changes weeks in advance • Quickly see and validate changes before release

99. • Game designers need ability to • Work on changes in isolation • Quickly iterate and test changes in isolation

100. • QA need ability to • Quickly switch between different changes to test • Smoke-test changes in live environment without affecting real players

101. • Server team needs ability to • Version data changes • Rollback/hotfix data changes with no downtime

102. We version control our code, why not our data?

103. • Git + Git flow • Servers allow multiple data branches and switching between them

104. • What do our players like to do? • Every action in the game is recorded

105. Feature Analyze

106. • Analytics is a key decision-making tool • A/B testing

107. • 150,000,000+ events recorded in April • Several GBs of analytics data a day • Fast growing!

108. • Google BigQuery + Cheap + Fast + Scalable + SQL-like query syntax

109. • Google BigQuery – OLAP – Append-only

110. • What is our code doing?

111. • Monitoring • Not the same as profiling • Should be always ON • Difficult to set up in large distributed environment

112. What to monitor?

113. • Method execution time • e.g. IO, CPU intensive • Method execution count • e.g. service entry points • Error count • ...

114. • Amazon CloudWatch + Cheap + Supports alarms and notifications + Visualization tool

115. PostSharp

117. PostSharp Amazon CloudWatch + =

122. • Metrics are aggregated at instance level • Metrics gatherer has high concurrency requirement • F# agents to the rescue!

123. Thank You! @theburningmonk

124. Here Be Monsters http://apps.facebook.com/herebemonsters AOP and pseudo real-time monitoring with CloudWatch http://bit.ly/11GL3SQ http://bit.ly/10fe98N Building social games with a .Net stack http://bit.ly/ZKtHWb http://bit.ly/10CtyY9 F# Agents http://bit.ly/ZmXuEP http://bit.ly/17kjbam F# Deep Dives http://bit.ly/XZe10E

Building an MMORPG - Here be monsters

Building an MMORPG - Here be monsters

Recommended

More Related Content

What's hot

What's hot(15)

Similar to Building an MMORPG - Here be monsters

Similar to Building an MMORPG - Here be monsters(20)

More from Yan Cui

More from Yan Cui(20)

Recently uploaded

Recently uploaded(20)

Building an MMORPG - Here be monsters