This is a deep-dive straight into the guts of running a low-latency multiplayer game, such as a first-person shooter, on a global scale. We dive into architectures that enable you to split apart your back-end APIs from your game servers, and Auto Scale them independently. See how to run game servers in multiple AWS regions such as China and Frankfurt, and integrate them with your central game stack. We'll even demo this in action, using AWS CloudFormation and Chef to deploy Unreal Engine game servers. In the second half, hear from PlayFab, who built the backend for the Top-10 free-to-play PC shooter Loadout. PlayFab reveals details about their architecture, including AWS Elastic Beanstalk setup, Amazon DynamoDB and Amazon RDS patterns, data sharding, and use of multiple Availability Zones. Finally, PlayFab highlights challenges they faced when deploying to AWS China, and how they solved them.
(GAM402) Deploying a Low-Latency Multiplayer Game Globally: Loadout | AWS re:Invent 2014
1. November 14, 2014 | Las Vegas, NV
Nate Wiger, Principal Gaming Solutions Architect, AWS
James Gwertzman, CEO, PlayFab
2. [Game] Online Problems Continue; Features Disabled as [Game Studio] Works on Fixes
“We are working hard to get everyone online as soon as possible,” [Game Studio] says.
Days later, [Game Studio] leaving disconnected players stranded
[Game] has a great online mode –if you can get online, that is
“This is total bull**** I paid for a preorder and I can’t even connect WTF [Game Studio]??”
8. VPC Subnet
VPC Subnet
Availability Zone
A
Availability Zone B
VPC Subnet
VPC Subnet
Auto Scaling group
WEB
VPC Subnet
WEB
JOBS
9. Region
① Login via HTTP API
② Download Game Assets
③ Matchmaking to Game Server
EC2
EC2
EC2
10. Region
① Login via HTTP API
② Download Game Assets
③ Matchmaking to Game Server
④ Connect to Server
⑤ Hack Apart Your Friends
⑥ Game Over EC2
EC2
11. Region
① Login via HTTP API
② Download Game Assets
③ Matchmaking to Game Server
④ Connect to Server
⑤ Hack Apart Your Friends
⑥ Game Over
⑦ Write via HTTP API
EC2
EC2
12. VPC Private Subnet
VPC Public Subnet
Availability Zone A Availability Zone B
VPC Public Subnet
VPC Private Subnet
GAME GAME GAME GAME GAME GAME
17. VPC Private Subnet
VPC Public Subnet
Availability Zone A Availability Zone B
VPC Public Subnet
VPC Private Subnet
GAME GAME GAME GAME GAME GAME
Auto Scaling group
18. VPC Private Subnet
VPC Public Subnet
Availability Zone A
GAME GAME GAME
Auto Scaling group
31. +
Tools for operating & monetizing live games
Complete set of cloud-basedbackend services
Eco-system of pre-integrated third-party services
.. and more
+
32.
33.
34.
35. AWS cloud: PlayFab Web Services
US
EU
Hardware Game Servers
Hardware Game Servers
Game Client
Amazon Route 53
(3.playfabapi.com)
Amazon EC2 (API handling)
Matchmaker
Instance
Instance
Game Server Monitor
DynamoDB
Amazon RDS
Amazon S3
Elastic Load Balancing
Amazon Redshift
Reports service
Instance
Logs
Matchmaker (Secondary)
Instance
US
EU
SA
AUS
Amazon EC2 Game Servers
Amazon EC2 Game Servers
Amazon EC2 Game Servers
Amazon EC2 Game Servers
38. Game Client
Amazon Route 53
(3.playfabapi.com)
API Request:
Elastic Load
Balancing
Logs
Amazon Redshift
Amazon S3 bucket
Reports service
Logs
Instance
Logs include title-specific endpoint for analytics
AWS wildcard DNS (*.playfabapi.com)
Log event:
39. API NAME
TOTAL CALLS
AVERAGE BACKEND TIME (S)
AVERAGE RESPONSE TIME (MS)
AVERAGE REQUEST TIME (MS)
TOTAL REQUEST TIME (M)
AVERAGE RECIEVED (BYTES)
AVERAGE SENT (BYTES)
TOTAL ERRORS
adminapi_userinfo
93,081,685
0.03
0.04
0.04
59.83
0
452
1979
gameserver_usercustomdatareadonly
1,582,348
0.06
0.04
0.04
1.06
42
902
290
matchmaker_userinfo
1,138,842
0.20
0.04
0.04
0.81
0
10046
319
matchmaker_playerjoined
1,138,247
0.01
0.04
0.04
0.79
64
2
93
matchmaker_playerleft
1,136,797
0.01
0.04
0.04
0.80
64
2
400
matchmaker_authuser
736,607
0.01
0.03
0.04
0.52
0
48
107
gameserver_usercustomdata
297,735
0.19
0.04
0.04
0.20
0
4080
0
gameserver_usercustomdatainternal
297,469
0.08
0.04
0.04
0.20
0
237
2
40. Instance
Health Check
Auto Scaling group
Auto Scaling service
Instance
Matchmaker (Primary)
Instance
Amazon Route 53
(matchmaker.playfabapi.com)
Elastic Load
Balancing
Matchmaker (Secondary)
1.Auto Scaling is set to keep two instances up.
2.Elastic Beanstalk Health Check terminates bad instance.
3.Amazon Route 53 fails over all traffic to secondary server.
4.Auto Scaling detects failure and starts new instance.
41. Amazon EC2 (API handling)
Elastic Load Balancing
CloudWatch
Health Check
Email Alerts
On-call Engineer Mobile Alerts
42. Amazon EC2 API Server
Server SSD
Amazon Redshift
Local Reports service
Local event logs
Instance
Events: logins and purchases
Reports service
Logs
Customer’s game management dashboard
Analytics
Server Logs
RabbitMQmessage queue
Logs
Instance
Logstashservice
Logstash
Instance
ElasticSearchInstance
Instance
43. DynamoDB
Amazon RDS
Amazon S3
Player data:
•Accounts
•Profile
•Inventory
•Save data
•Player stats
•Player currency values
•Auditing
•Log files
•Game replays
Amazon Redshift
•Analytics data
44. Amazon DynamoDB
•Player data: accounts, profile, inventory, save data
User:
UserID: Hash
TitleID: range key
-DisplayName
-Linked accounts
-…
Custom user data:
UserID: Hash
TitleID: range key
-Key/value 1
-Key/value 2
-…
Inventory:
UserID: Hash
ItemID: range key
-titleID
-Catalog Item
-…
…
45. AZ 1
AZ 2
PlayFab Web Services
PlayFab Web Services
N. California
AZ 1
AZ 2
PlayFab Web Services
PlayFab Web Services
Oregon
AZ 1
PlayFab Web Services
Beijing
•Original deployment: us-west-1
•Expanded to us-west-2 in July
•Coming soon: cn-north-1
Amazon EC2 Game Server
Singapore
Game Client
Instance
46. Player accounts are shared across all regions:
•Cross-regional wrapper for user.getsets order of databases and regions to check
•user.getthen checks for player account data
•High availability, transparent to customers and players
// Returns the first matching player record, searching DynamoDB and SimpleDB
// in both regions in the specified order.
// Updates the player record in the store and region from which it was loaded.
// Creates a player record in the default store and region.
47.
48. AWS cloud: PlayFab Web Services
US
EU
Hardware Game Servers
Hardware Game Servers
Game Client
Amazon Route 53
(3.playfabapi.com)
Amazon EC2 (API handling)
Matchmaker
Instance
Instance
Game Server Monitor
DynamoDB
Amazon RDS
Amazon S3
Elastic Load Balancing
Amazon Redshift
Reports service
Instance
Logs
Matchmaker (Secondary)
Instance
US
EU
SA
AUS
Amazon EC2 Game Servers
Amazon EC2 Game Servers
Amazon EC2 Game Servers
Amazon EC2 Game Servers