Charlie Crane presented on building fast, scalable game servers using Node.js and the open source framework Pomelo. Pomelo is a distributed, scalable framework for real-time applications that abstracts servers and provides request/response and broadcast capabilities. It addresses challenges of scalability for games through techniques like area of interest and splitting processes. Performance testing showed Pomelo can support over 500 concurrent users on a single server. Pomelo has been used successfully for games and real-time messaging platforms supporting millions of users.

1. Building fast, scalable game server in
node.js
Charlie Crane
@xiecc

2. Who am I
 NASDAQ: NTES
 Senior engineer, architect(8 years) in NetEase Inc. ,
developed many web, game products
 Recently created the open source game server framework in
node.js: pomelo

3. Top cities for npm modules
https://gist.github.com/substack/7373338

4. Agenda
 State of pomelo
 Scalability
 Framework
 Performance
 Realtime application

5. The state of pomelo
Fast, scalable, distributed game server
framework for node.js
Open sourced in 2012.11.20
Newest version: V0.7
https://github.com/NetEase/pomelo

6. Pomelo position
 Game server framework
 Mobile game
 Web game
 Social game
 MMO RPG(middle scale)
Realtime Multiple User Interaction
 Realtime application server framework

7. State of pomelo
 Pomelo is not a
single project
 Almost 40 repos in total

8. Framework ---

9. https://github.com/NetEase/pomelo

10. Success cases
Card Game — ShenMu
http://shenmu.iccgame.com/index.shtml

11. Success cases
Combat Strategy Game — Ancient songs
http://www.ixuanyou.com/

12. Success cases
Realtime Communication Tools —
ZhongQingBangBang
Q&A
http://wap.youth.cn/bangbang/

13. Success cases
 The magic card hunter https://itunes.apple.com/cn/app/id6649640688
 Beauty Fried golden flower http://app.xiaomi.com/detail/41461
 Texas Hold ’ em poker http://www.yiihua.com/
 Metal Slug http://www.17erzhan.com/ww2/ww2index.php
 Coal instant messaging http://www.mtjst.com
 Yong Le golden flower http://pan.baidu.com/s/1pCmfD
 17 Bullfight http://app.91.com/Soft/Detail.aspx?Platform=iPhone
…

14. Job huntings

15. Agenda
 The state of pomelo
 Scalability
 Framework
 Performance
 Realtime appliation

16. Demo first
http://pomelo.netease.com/lordofpomelo

17. Scalability---Web and Game server
 Web server
unlimited scalability
 Game server
World record: world of tanks(bigworld), 74,536 online users
MMORPG：7k-8k maximum
Why?

18. Why does game server not scale?
 Reason 1: Long connection
 Response time
Web response time： 2000ms
Game、realtime web： <100ms
 Message direction
web: request/reponse
game: request/push/broadcast, bi-direction

19. How to solve
 Node.js to rescue
Perfect for:
Network-insentive application
Real-time application
Holding connections
Mqtt: 120,000 connections, 1 process, 10K/connection
Socket.io: 25,000 connections, 1 process, 50K/connection

20. Why does game server not scale
 Reason 2: stateful
 Web app -- stateless
 No coordinates, no adjacency
 Partition randomly, stateless
 Game server -- stateful
 Have coordinate, adjacency
 Partition by area, stateful

21. How to solve
 No good solution in technical level
 Depending on game design
 Don’t make one area too crowded
 Balance of area

22. Too crowded area

23. Disperse player

24. Why does game server not scale
Reason 3: Broadcast
MESSAGES IN
1
MESSAGES OUT
1
1
1

25. Why does game server not scale
1
2
MESSAGES IN
MESSAGES OUT
4
2
2
1

26. Why does game server not scale
MESSAGES IN
MESSAGES OUT
4
1
16
4
4
4
1
1
4
1

27. Why does game server not scale
MESSAGES IN
MESSAGES OUT
100
1
10000
100
100
100
1
1
100
1

28. Why does game server not scale
MESSAGES IN
MESSAGES OUT
1000
1
1000
1000000
1000
1000
1
1
1000
1

29. 1、How to solve --- broadcast
 AOI --- area of interested
module: pomelo-aoi

30. 2、How to solve -- broadcast
 Split process, seperate load , frontend is stateless
broadcast
Frontend
(connector)
Single Process
Game logic
Backend
(area)

31. 3、How to solve – broadcast
 Game design, don’t make one place(in area) too crowded
 Map size
 Character density
 Disperse born place

32. Disperse born place

33. 4、How to solve -- pushScheduler
app.set(‘pushSchedulerConfig’, {
scheduler: new MessageScheduler(app);
});
 Direct：send message directly
 Buffer:
buffer message in array, and flush it every 50ms
 Other Strategies: flexible, depending on client number

34. Why does game server not scale
 Reason 4: Tick (game loop)
 setInterval(tick, 100)
 What does every tick do?
 Update every entity in the scene(disappear，move, revive)
 Refresh monster
 Driving ai logic(monster, player)
Tick must be far less than 100ms

35. Problem of tick
 The entity number should be limited
 Pay attention to update algorithm: AI etc.
 GC, full gc should be limited
 V8 is good at GC when memory is under 500M, even full GC
 Memory must be limited
 Try to divide process

36. At last--- runtime architecture

37. How to solve complexity
 Too … complicated?
solution:
framework

38. Agenda
 The state of pomelo
 Scalability
 Framework
 Performance
 Realtime application

39. Pomelo Framework
The essence of pomelo:
A distributed, scalable, realtime application
framework.

40. Framework --- design goal
 Abstract of servers(processes)
 Auto extend server types
 Auto extend servers
 Abstract of request/response and broadcast/push
 Zero config request
 Simple broadcast api
 Servers communication---rpc framework

41. Framework -- some features
 Server scalability
 Network
 Plugins

42. Server scalability

43. Server Scalability-- dynamic
extensions
 pomelo add host=[host] port=[port] id=[id]
serverType=[serverType]
 node app.js env=production host=[host]
port=[port] id=[id] serverType=[serverType]

44. Network
Pomelo
component
Loader
MQTTConnector
Mobile client
Connector
SIOConnector
HybridConnector
Socket.io client
Socket, websocket
client

45. Network--Message encode, decode

46. Network--original protobuf
 Original protobuf

47. Network--pomelo protobuf
 Our protobuf

48. Network -- Proto compress
 Config:

49. Network---protobuf definition
 Handler return json, auto encoded to protobuf

50. Network--pomelo protobuf
https://github.com/pomelonode/pomelo-protobuf
As an independent project, supporting unity3d, iOS, android, C

51. Network -- Pomelo Netflow – V0.2
 Version 0.2, 400 online users
Out: 3645Kb/S in: 270Kb/s

52. Network -- Pomelo Netflow – V0.3
 Version 0.3, 400 online users
Out: 925KB/S, in: 217KB/S, 25% Netflow of 0.2

53. Plugins
 A mechanism to extend framework
 Example:
 pomelo-status-plugin , online status plugin, based on redis

54. Plugins example -- Reliability
 pomelo-masterha-plugin, based on zookeeper, master HA
 Servers auto reconnect
Zookeeper
Master
Slave
Slave

55. Agenda
 State of pomelo
 Scalability
 Framework
 Performance
 Realtime application

56. Performance --- overview
 The performance varies largely in different games,
applications
 The variation
 Application type: Game or realtime application
 Game type: round or realtime fight, room or infinite
 Game design: Map size, character density, balance of areas
 Test parameters: Think time, test action

57. Performance --- reference data
 Online users per process
 next-gen: support 20,000 concurrent players in one area
 World record – not one area
 world of tanks(bigworld), 74,536 online users
 But in real world(MMO rpg)
 The real online data: maximum 800 concurrent users per area, 7,000
concurrent users per group game servers

58. Performance--Server configuration
Openstack virtual machine

59. Performance --- stress testing
 Use Case 1: fight
 Stress on single area(area 3), increasing step by step,
login every 2 seconds
 Game logic: attack monstors or other players every
2~5 seconds

60. Performance --- stress testing

61. Performance--result
 486 onlines

62. Performance -- top
 Using toobusy, limit cpu 80%

63. Performance – stress test
 Use case 2 – move
 Stress on single area(area 3), increasing step by step,
login every 2 seconds
 Game logic: move in the map every 2~5 seconds

64. Performance – stress test
 800 online users

65. Performance --- stress testing
 Use Case 3: fight & move
 Stress on single area(area 3), increasing step by step,
login every 2 seconds
 Game logic: 50% attack monstors or other players
every 2~5 seconds, 50% move around
 Result: 558 onlines in one area

66. Agenda
 State of pomelo
 Scalability
 Framework
 Performance
 Realtime application

67. Realtime app and game
 Many success stories in realtime application
 Message push platform in NetEase
 ZhongQinBangBang
 Not that frequent messages as game
 Do do need that realtime (less than 50ms), we can use
different pushScheduler strategy
 Much more scalable than game

68. Message push platform
Supporting more than 10,000,000 online users
 Products using message push platform
 love.163.com
 music.163.com
 news.163.com
 note.youdao.com
 game.163.com
 yuehui.163.com
 yuedu.163.com

69. Message push platform
AMQP
Receiver
Receiver
publish
Receiver
Receiver
Redis,
Device
APNS publisher
APNS publisher
Channel push, socket
Connector
Connector
Connector
LVS
socket.io
Web client
mqtt
Android
client
mqtt
iOS client

70. Message push systems
 High load, More than 10,000,000 online users
 Realtime, broadcast to 5 million users in 1minute
 Reliability, qos=1 in any conditions
 Save power and network
 Support all the clients
 iOS(APNS, mqtt)
 android(mqtt)
 browser(socket.io)
 windows desktop application(mqtt)

71. Q&A
@xiecc
https://github.com/NetEase/pomelo