Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
1
GDC 2015 – Amazon Developer Day
Deploying a Low-Latency Multiplayer Game Globally
Nate Wiger
Principal, Gaming Solutions...
AWS Gaming Solutions
[Game] Online Problems Continue; Features
Disabled as [Game Studio] Works on Fixes
“We are working ha...
AWS Gaming Solutions
AWS Gaming Solutions http://bit.ly/verizon-latency
http://bit.ly/superdata-latency
AWS Gaming Solutions
AWS Gaming Solutions
100+ms 100+ms
100+ms
100+ms
AWS Gaming Solutions
How To Be Awesome
1.  Game servers near players
2.  Everything else as HTTP APIs
3.  Data replication...
AWS Gaming Solutions
Loosely Coupled Pods
Tokyo
Oregon
Frankfurt
Virginia
AWS Gaming Solutions
VPC Subnet
VPC Subnet
Game API Pods
Availability Zone A
 Availability Zone B
VPC Subnet
VPC Subnet
Au...
AWS Gaming Solutions
Region
①  Login via HTTP API
②  Download Game Assets
③  Matchmaking to Game Server
EC2
Game Flow
EC2
...
AWS Gaming Solutions
Region
①  Login via HTTP API
②  Download Game Assets
③  Matchmaking to Game Server
④  Connect to Serv...
AWS Gaming Solutions
Region
①  Login via HTTP API
②  Download Game Assets
③  Matchmaking to Game Server
④  Connect to Serv...
AWS Gaming Solutions
VPC Private Subnet
VPC Public Subnet
Game Server Pods
Availability Zone A
 Availability Zone B
VPC Pu...
AWS Gaming Solutions
Make. Network. Faster.
•  C3/C4 instance family (or R3 for more memory)
•  In a VPC
•  HVM AMI – not ...
AWS Gaming Solutions
Enhanced Networking (SR-IOV)
Before:
Hypervisor
After:
Hardware
AWS Gaming Solutions
VPC Private Subnet
 VPC Private Subnet
RabbitMQ + Elastic Load Balancing
Availability Zone A
 Availab...
AWS Gaming Solutions
RabbitMQ + Elastic Load Balancing
•  Clustering Tutorial
https://www.rabbitmq.com/clustering.html
•  ...
AWS Gaming Solutions
/etc/rabbitmq/rabbitmq.config
5672
rabbitmq-node1
rabbitmq-node2
AWS Gaming Solutions
VPC Private Subnet
VPC Public Subnet
Redis Pub / Sub
Availability Zone A
 Availability Zone B
VPC Pub...
AWS Gaming Solutions
VPC Private Subnet
VPC Public Subnet
CloudFormation + Chef
Availability Zone A
GAME GAME GAME
Auto Sc...
AWS Gaming Solutions
AWS Gaming Solutions
Unreal Engine with CloudFormation
UnrealServer
unreal_linux.template.json
true
my-bucket-name
Unreal-...
AWS Gaming Solutions
AWS Gaming Solutions
AWS Gaming Solutions
AWS Gaming Solutions
Where My Servers At?
Tokyo
Oregon
Frankfurt
Virginia
?
?
AWS Gaming Solutions
VPC Subnet
Server Registration
Availability Zone A
 Availability Zone B
VPC Subnet
Auto Scaling group...
AWS Gaming Solutions
Server Registration & Scaling
•  HTTPS POST /api/servers/register
•  Include an HMAC (RFC 2104)
•  Se...
AWS Gaming Solutions
VPC Private Subnet
VPC Public Subnet
Game Server Logs
Availability Zone A
S3 Bucket
Amazon CloudWatch...
AWS Gaming Solutions
VPC Private Subnet
VPC Public Subnet
Game Server Logs
Availability Zone A
Amazon

Kinesis
S3 Bucket
A...
AWS Gaming Solutions
AWS Gaming Solutions
Mini Pods Everywhere
China
Oregon
Frankfurt
Virginia
China
Leaderboard
AWS Gaming Solutions
PlayFab: Loadout Architecture
AWS Gaming Solutions
What is PlayFab?
PlayFab is a complete live game operations platform.
Focus on the fun – we’ll do the...
AWS Gaming Solutions
AWS Gaming Solutions
Some Loadout Stats
•  Launched 1/31/14
•  Over 1 billion API Calls
•  At Peak (Feb 2014):
–  300k DAU...
AWS Gaming Solutions
PlayFab Core Architecture Tenants
•  No scheduled downtime.
•  Automatic scaling for load.
•  Backend...
AWS Gaming Solutions
AWS cloud: PlayFab Web Services
US
 EU
Hardware
Game Servers
Hardware
Game Servers
Game Client
Amazon...
AWS Gaming Solutions
List of PlayFab Web APIs
Client Server
Authentication Title-Wide Data Management Shared Group Data Au...
AWS Gaming Solutions
Example of Web API Call
API Request: Response from Web Services:
AWS Gaming Solutions
ELB Logging for Title Tracking
Game Client
Amazon Route 53
(3.playfabapi.com)
API Request:
Elastic Lo...
AWS Gaming Solutions
Loadout API Usage Report (Oct 2014)
API NAME
TOTAL
CALLS
AVERAGE
BACKEND
TIME (S)
AVERAGE
RESPONSE
TI...
AWS Gaming Solutions
Instance
Making a Singleton Highly Available
Health
Check
Auto Scaling group
Auto Scaling
service
Ins...
AWS Gaming Solutions
Monitoring Service Health
Amazon EC2
(API handling)
Elastic Load
Balancing
 CloudWatch
Health
Check
E...
AWS Gaming Solutions
Handling Spikes Without Losing Data
Amazon EC2 API Server
Server SSD
Amazon Redshift
Local Reports
se...
AWS Gaming Solutions
Storage in DynamoDB, Amazon RDS, Amazon S3,
Amazon Redshift
DynamoDB
 Amazon RDS
 Amazon S3
Player da...
AWS Gaming Solutions
Managing Tables in DynamoDB
Amazon
DynamoDB
•  Player data: accounts, profile, inventory, save data
U...
AWS Gaming Solutions
Physical Deployment
AZ 1
AZ 2
PlayFab Web
Services
PlayFab Web
Services
N. California
AZ 1
AZ 2
PlayF...
AWS Gaming Solutions
Managing Cross-Region Data
Player accounts are shared
across all regions:
•  Cross-regional wrapper f...
AWS Gaming Solutions
AWS Gaming Solutions
AWS cloud: PlayFab Web Services
US
 EU
Hardware
Game Servers
Hardware
Game ServersGame Client
Amazon ...
52
Nate Wiger
Principal, Gaming Solutions
@nateware
nateware@amazon.com
Upcoming SlideShare
Loading in …5
×

GDC 2015 - Low-latency Multiplayer Gaming with AWS

12,491 views

Published on

Learn the challenges involved with deploying a low-latency multiplayer game globally, and how to overcome them with AWS. Based on real-world experiences from myself and other game developers deploying online games with tens of thousands of players.

Published in: Technology

GDC 2015 - Low-latency Multiplayer Gaming with AWS

  1. 1. 1 GDC 2015 – Amazon Developer Day Deploying a Low-Latency Multiplayer Game Globally Nate Wiger Principal, Gaming Solutions @nateware nateware@amazon.com James Gwertzman CEO, PlayFab @playfabnetwork http://playfab.com
  2. 2. AWS Gaming Solutions [Game] Online Problems Continue; Features Disabled as [Game Studio] Works on Fixes “We are working hard to get everyone online as soon as possible,” [Game Studio] says. Days later, [Game Studio] leaving disconnected players stranded [Game] has a great online mode – if you can get online, that is “This is total bull**** I paid for a preorder and I can’t even connect WTF [Game Studio]??”
  3. 3. AWS Gaming Solutions
  4. 4. AWS Gaming Solutions http://bit.ly/verizon-latency http://bit.ly/superdata-latency
  5. 5. AWS Gaming Solutions
  6. 6. AWS Gaming Solutions 100+ms 100+ms 100+ms 100+ms
  7. 7. AWS Gaming Solutions How To Be Awesome 1.  Game servers near players 2.  Everything else as HTTP APIs 3.  Data replication = bad 4.  Local caches = good 5.  Nobody really “plays everywhere”
  8. 8. AWS Gaming Solutions Loosely Coupled Pods Tokyo Oregon Frankfurt Virginia
  9. 9. AWS Gaming Solutions VPC Subnet VPC Subnet Game API Pods Availability Zone A Availability Zone B VPC Subnet VPC Subnet Auto Scaling group WEB VPC Subnet WEB JOBS Cognito SNS Mobile Push SES
  10. 10. AWS Gaming Solutions Region ①  Login via HTTP API ②  Download Game Assets ③  Matchmaking to Game Server EC2 Game Flow EC2 EC2
  11. 11. AWS Gaming Solutions Region ①  Login via HTTP API ②  Download Game Assets ③  Matchmaking to Game Server ④  Connect to Server ⑤  Hack Apart Your Friends ⑥  Game Over Game Flow EC2 EC2
  12. 12. AWS Gaming Solutions Region ①  Login via HTTP API ②  Download Game Assets ③  Matchmaking to Game Server ④  Connect to Server ⑤  Hack Apart Your Friends ⑥  Game Over ⑦  Write via HTTP API Game Flow EC2 EC2
  13. 13. AWS Gaming Solutions VPC Private Subnet VPC Public Subnet Game Server Pods Availability Zone A Availability Zone B VPC Public Subnet VPC Private Subnet GAME GAME GAME GAME GAME GAME
  14. 14. AWS Gaming Solutions Make. Network. Faster. •  C3/C4 instance family (or R3 for more memory) •  In a VPC •  HVM AMI – not PV •  Enable Enhanced Networking (SR-IOV) http://bit.ly/ec2-enhanced-net •  Linux: Tweak Networking Stack http://bit.ly/linux-tuning
  15. 15. AWS Gaming Solutions Enhanced Networking (SR-IOV) Before: Hypervisor After: Hardware
  16. 16. AWS Gaming Solutions VPC Private Subnet VPC Private Subnet RabbitMQ + Elastic Load Balancing Availability Zone A Availability Zone B 10.1.0.13 10.2.0.16 TCP 5672 rabbitmq-node1 rabbitmq-node2 TCP 4369 & 25672
  17. 17. AWS Gaming Solutions RabbitMQ + Elastic Load Balancing •  Clustering Tutorial https://www.rabbitmq.com/clustering.html •  Set Queue HA policy to "all" https://www.rabbitmq.com/ha.html •  Create Internal load balancer –  Listen Port: TCP 5672 •  VPC Security Group –  For load balancer: TCP 5672 –  Between EC2 nodes: TCP 4369 & 25672 •  Set up Client to Heartbeat
  18. 18. AWS Gaming Solutions /etc/rabbitmq/rabbitmq.config 5672 rabbitmq-node1 rabbitmq-node2
  19. 19. AWS Gaming Solutions VPC Private Subnet VPC Public Subnet Redis Pub / Sub Availability Zone A Availability Zone B VPC Public Subnet VPC Private Subnet GAME GAME GAME GAME GAME GAME Auto Scaling group
  20. 20. AWS Gaming Solutions VPC Private Subnet VPC Public Subnet CloudFormation + Chef Availability Zone A GAME GAME GAME Auto Scaling group
  21. 21. AWS Gaming Solutions
  22. 22. AWS Gaming Solutions Unreal Engine with CloudFormation UnrealServer unreal_linux.template.json true my-bucket-name Unreal-123.zip sshkeyname 1.2.3.4/24 MyPass123
  23. 23. AWS Gaming Solutions
  24. 24. AWS Gaming Solutions
  25. 25. AWS Gaming Solutions
  26. 26. AWS Gaming Solutions Where My Servers At? Tokyo Oregon Frankfurt Virginia ? ?
  27. 27. AWS Gaming Solutions VPC Subnet Server Registration Availability Zone A Availability Zone B VPC Subnet Auto Scaling group WEB WEB Oregon Tokyo VPC Subnet JOBS
  28. 28. AWS Gaming Solutions Server Registration & Scaling •  HTTPS POST /api/servers/register •  Include an HMAC (RFC 2104) •  Send Server Status –  Public IP –  # Players –  Game Modes •  Matchmaking Service –  Maintains server list –  Removes servers
  29. 29. AWS Gaming Solutions VPC Private Subnet VPC Public Subnet Game Server Logs Availability Zone A S3 Bucket Amazon CloudWatch Logs GAME GAME GAME
  30. 30. AWS Gaming Solutions VPC Private Subnet VPC Public Subnet Game Server Logs Availability Zone A Amazon
 Kinesis S3 Bucket Amazon Redshift GAME GAME GAME
  31. 31. AWS Gaming Solutions
  32. 32. AWS Gaming Solutions Mini Pods Everywhere China Oregon Frankfurt Virginia China Leaderboard
  33. 33. AWS Gaming Solutions PlayFab: Loadout Architecture
  34. 34. AWS Gaming Solutions What is PlayFab? PlayFab is a complete live game operations platform. Focus on the fun – we’ll do the boring bits. + Tools for operating & monetizing live games Complete set of cloud-based backend services Eco-system of pre-integrated third-party services .. and more +
  35. 35. AWS Gaming Solutions
  36. 36. AWS Gaming Solutions Some Loadout Stats •  Launched 1/31/14 •  Over 1 billion API Calls •  At Peak (Feb 2014): –  300k DAU –  120 fixed game servers –  110 EC2 game servers
  37. 37. AWS Gaming Solutions PlayFab Core Architecture Tenants •  No scheduled downtime. •  Automatic scaling for load. •  Backend changes should be invisible to developers. •  Use buffering to smooth over spikes or failures.
  38. 38. AWS Gaming Solutions AWS cloud: PlayFab Web Services US EU Hardware Game Servers Hardware Game Servers Game Client Amazon Route 53 (3.playfabapi.com) Amazon EC2 (API handling) Matchmaker Instance Instance Game Server Monitor DynamoDB Amazon RDS Amazon S3 Elastic Load Balancing Amazon Redshift Reports service Instance Logs ArchitectureOverview Matchmaker (Secondary) Instance US EU SA AUS Amazon EC2 Game Servers Amazon EC2 Game Servers Amazon EC2 Game Servers Amazon EC2 Game Servers
  39. 39. AWS Gaming Solutions List of PlayFab Web APIs Client Server Authentication Title-Wide Data Management Shared Group Data Authentication AddUsernamePassword GetCatalogItems AddSharedGroupMembers AuthenticateSessionTicket LoginWithAndroidDeviceID GetStoreItems CreateSharedGroup Account Management LoginWithFacebook GetTitleData GetSharedGroupData GetUserAccountInfo LoginWithGameCenter GetTitleNews RemoveSharedGroupMembers SendPushNotification LoginWithGoogleAccount AddUserVirtualCurrency UpdateSharedGroupData Player Data Management LoginWithIOSDeviceID Player Item Management Server-Side Game Logic GetLeaderboard LoginWithPlayFab ConsumeItem GetLogicServerUrl GetLeaderboardAroundUser LoginWithSteam GetUserInventory GetUserData RegisterPlayFabUser RedeemCoupon GetUserInternalData SendAccountRecoveryEmail SubtractUserVirtualCurrency GetUserReadOnlyData Account Management UnlockContainerItem GetUserStatistics GetAccountInfo StartPurchase UpdateUserData GetPlayFabIDsFromFacebookIDs PayForPurchase UpdateUserInternalData GetUserCombinedInfo ConfirmPurchase UpdateUserReadOnlyData LinkFacebookAccount PurchaseItem UpdateUserStatistics LinkGameCenterAccount Friend List Management Title-Wide Data Management LinkSteamAccount AddFriend GetCatalogItems UnlinkFacebookAccount GetFriendsList GetTitleData UnlinkGameCenterAccount RemoveFriend SetTitleData UnlinkSteamAccount SetFriendTags Player Item Management UpdateEmailAddress IOS-Specific APIs AddUserVirtualCurrency UpdatePassword RegisterForIOSPushNotification GetUserInventory UpdateUserTitleDisplayName ValidateIOSReceipt GrantItemsToUsers Player Data Management Matchmaking APIs SubtractUserVirtualCurrency GetFriendLeaderboard GetCurrentGames Matchmaking APIs GetLeaderboard GetGameServerRegions NotifyMatchmakerPlayerLeft GetLeaderboardAroundCurrentUser Matchmake RedeemMatchmakerTicket GetUserData StartGame Steam-Specific APIs GetUserReadOnlyData Android-Specific APIs AwardSteamAchievement GetUserStatistics AndroidDevicePushNotificationRegistration UpdateUserData ValidateGooglePlayPurchase UpdateUserStatistics Analytics
  40. 40. AWS Gaming Solutions Example of Web API Call API Request: Response from Web Services:
  41. 41. AWS Gaming Solutions ELB Logging for Title Tracking Game Client Amazon Route 53 (3.playfabapi.com) API Request: Elastic Load Balancing Logs Amazon Redshift Amazon S3 
 bucket Reports service Logs Instance Logs include title-specific endpoint for analytics AWS wildcard DNS (*.playfabapi.com) Log event:
  42. 42. AWS Gaming Solutions Loadout API Usage Report (Oct 2014) API NAME TOTAL CALLS AVERAGE BACKEND TIME (S) AVERAGE RESPONSE TIME (MS) AVERAGE REQUEST TIME (MS) TOTAL REQUEST TIME (M) AVERAGE RECIEVED (BYTES) AVERAGE SENT (BYTES) TOTAL ERRORS adminapi_userinfo 93,081,685 0.03 0.04 0.04 59.83 0 452 1979 gameserver_usercustomdatareadonly 1,582,348 0.06 0.04 0.04 1.06 42 902 290 matchmaker_userinfo 1,138,842 0.20 0.04 0.04 0.81 0 10046 319 matchmaker_playerjoined 1,138,247 0.01 0.04 0.04 0.79 64 2 93 matchmaker_playerleft 1,136,797 0.01 0.04 0.04 0.80 64 2 400 matchmaker_authuser 736,607 0.01 0.03 0.04 0.52 0 48 107 gameserver_usercustomdata 297,735 0.19 0.04 0.04 0.20 0 4080 0 gameserver_usercustomdatainternal 297,469 0.08 0.04 0.04 0.20 0 237 2
  43. 43. AWS Gaming Solutions Instance Making a Singleton Highly Available Health Check Auto Scaling group Auto Scaling service Instance Matchmaker (Primary) Instance Amazon Route 53) Elastic Load Balancing Matchmaker (Secondary) 1. Auto Scaling is set to keep two instances up. 2. Elastic Beanstalk Health Check terminates bad instance. 3. Amazon Route 53 fails over all traffic to secondary server. 4. Auto Scaling detects failure and starts new instance.
  44. 44. AWS Gaming Solutions Monitoring Service Health Amazon EC2 (API handling) Elastic Load Balancing CloudWatch Health Check Email Alerts On-call Engineer Mobile Alerts
  45. 45. AWS Gaming Solutions Handling Spikes Without Losing Data Amazon EC2 API Server Server SSD Amazon Redshift Local Reports service Local event logs Instance Events: logins and purchases Reports service Logs Customer’s game management dashboard Analytics Server Logs RabbitMQ message queue Logs Instance Logstash service Logstash Instance ElasticSearch Instance Instance
  46. 46. AWS Gaming Solutions Storage in DynamoDB, Amazon RDS, Amazon S3, Amazon Redshift DynamoDB Amazon RDS Amazon S3 Player data: •  Accounts •  Profile •  Inventory •  Save data •  Player stats •  Player currency values •  Auditing •  Log files •  Game replays Amazon Redshift •  Analytics data
  47. 47. AWS Gaming Solutions Managing Tables in DynamoDB Amazon DynamoDB •  Player data: accounts, profile, inventory, save data User: UserID: Hash TitleID: range key -  DisplayName -  Linked accounts -  … Custom user data: UserID: Hash TitleID: range key -  Key/value 1 -  Key/value 2 -  … Inventory: UserID: Hash ItemID: range key -  titleID -  Catalog Item -  … …
  48. 48. AWS Gaming Solutions Physical Deployment AZ 1 AZ 2 PlayFab Web Services PlayFab Web Services N. California AZ 1 AZ 2 PlayFab Web Services PlayFab Web Services Oregon AZ 1 PlayFab Web Services Beijing •  Original deployment: us-west-1 •  Expanded to us-west-2 in July •  Coming soon: cn-north-1 Amazon EC2 Game Server Singapore Game Client Instance
  49. 49. AWS Gaming Solutions Managing Cross-Region Data Player accounts are shared across all regions: •  Cross-regional wrapper for user.get sets order of databases and regions to check •  user.get then checks for player account data •  High availability, transparent to customers and players // Returns the first matching player record, searching DynamoDB and SimpleDB // in both regions in the specified order. // Updates the player record in the store and region from which it was loaded. // Creates a player record in the default store and region.
  50. 50. AWS Gaming Solutions
  51. 51. AWS Gaming Solutions AWS cloud: PlayFab Web Services US EU Hardware Game Servers Hardware Game ServersGame Client Amazon Route 53 (3.playfabapi.com) Amazon EC2 (API handling) Matchmaker Instance Instance Game Server Monitor DynamoDB Amazon RDS Amazon S3 Elastic Load Balancing Amazon Redshift Reports service Instance Logs ArchitectureOverview Matchmaker (Secondary) Instance US EU SA AUS Amazon EC2 Game Servers Amazon EC2 Game Servers Amazon EC2 Game Servers Amazon EC2 Game Servers
  52. 52. 52 Nate Wiger Principal, Gaming Solutions @nateware nateware@amazon.com

×