Behind the Scenes: Deploying a Low-Latency Multiplayer Game Globally
•Download as PPTX, PDF•
5 likes•2,244 views
A deep dive into the guts of running a low-latency multiplayer game on a global scale using Amazon Web Services. You’ll get the details on how the Top 10 F2P shooter Loadout was launched on both PC and PS4 and how PlayFab’s complete backend and live game operations platform is architected to handle the scale these kind of games demand. Delivered on July 7 at the AWS Loft in San Francisco.
![[Game] Online Problems Continue;
Features Disabled as [Game Studio]
Works on Fixes
“We are working hard
to get everyone
online as soon as
possible,” [Game
Studio] says.
Days later, [Game Studio]
leaving disconnected players
stranded
[Game] has a great online mode
– if you can get online, that is
“This is total bull**** I paid for a
preorder and I can’t even connect
WTF [Game Studio]??”](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Behind the Scenes: Deploying a Low-Latency Multiplayer Game Globally
Similar to Behind the Scenes: Deploying a Low-Latency Multiplayer Game Globally (20)
More from James Gwertzman
More from James Gwertzman (14)
Recently uploaded
Recently uploaded (20)
Behind the Scenes: Deploying a Low-Latency Multiplayer Game Globally
- 1. Behind the Scenes: Deploying a Low-Latency Multiplayer Game Globally James Gwertzman, PlayFab CEO
- 2. [Game] Online Problems Continue; Features Disabled as [Game Studio] Works on Fixes “We are working hard to get everyone online as soon as possible,” [Game Studio] says. Days later, [Game Studio] leaving disconnected players stranded [Game] has a great online mode – if you can get online, that is “This is total bull**** I paid for a preorder and I can’t even connect WTF [Game Studio]??”
- 3. “The single greatest predictor of success and of sustainable competitive advantage in this business is live game operations.” -- Owen Mahoney, CEO Nexon
- 4. Build game Launch The old days…
- 5. Build game Build backend Launch Build tools for ops team Segment & target customers Deploy servers Business intelligence Offers & Promotions Update content Host in-game events Customer service Nowadays… Business intelligence User Acquisition
- 7. + Live ops tools and dashboards (mission control for the whole team) Back-end services (cross-platform, one-stop shop) + Integration with other partners (building out the full ecosystem) All-in-one platform to build, launch, and grow and more ...
- 10. Some Loadout Stats • PC launched 1/31/14 • PS4 launched 12/17/14 • Over 1 billion API Calls • At Peak (Feb 2014): – 300k DAU, 35K CCU – 120 fixed game servers – 110 EC2 game servers
- 11. Basic tenets of PlayFab architecture • Put latency-sensitive game servers near players. • Everything else as shared services via RESTful Web APIs. • Data replication = bad. • Local caches = good. • No scheduled downtime. • Automatic scaling for load. • Backend changes should be invisible to developers. • Use buffering to smooth over spikes or failures.
- 12. Availability Zone B Oregon AWS cloud: PlayFab Web Services Amazon Route 53 (3.playfabapi.com) Amazon EC2 (API handling) Matchmaker Instance Instance Game Server Monitor DynamoDB Amazon RDS Amazon S3 Amazon RedshiftReports service Instance Logs Architecture Overview Matchmaker (Secondary) Instance Virginia Availability Zone A Amazon EC2 (API handling) Elastic Load Balancing Cross-zone storage Game Client Game Manager (Dashboards) Amazon EC2 (Virtual game servers) Tokyo Amazon EC2 (Virtual game servers) Sydney Physical game servers Physical game servers Amazon EC2 (Virtual game servers) Local disk Local disk Local disk Physical game servers
- 13. PlayFab backend services • Player Accounts – Authentication – Profile Management – Account Linking • Data Storage – Per player – Per title – Per character (under one player) – Files / CDN delivery • Commerce – Catalog Management – Virtual Currencies – Player Inventory – In-game Purchasing – Receipt validation (Apple/Google/Amazon) – Marketing and Promotion • In-game Marketing – Push Notifications – In-game Messaging • Product Management – Analytics and Reporting – Customer Segmentation – Customer Support Tools – Abuse Reporting and Banning • Social – Friends Lists – Player Chat – Leaderboards – Game forum integration – Trading / gifting • Multiplayer – Photon integration (real-time / turn-based) – Matchmaking – Custom game server hosting – Server monitoring • Game logic – Server-hosted JavaScript The last API you’ll have to integrate into your game… 13
- 14. Example of Web API Call API Request: Response from Web Services:
- 15. Oregon AWS cloud: PlayFab Web Services Matchmaker Instance Instance Game Server Monitor Basic Gameplay (1) 1. Login via Web API to PlayFab 2. Retrieve session key 3. Download content updates 4. Call backend services as needed (inventory, catalog, player state) 5. Use matchmaker to find server; retrieve game ticket Virginia API servers Game Client Game servers Storage Local diskAmazon RedshiftReports service Instance
- 16. Basic Gameplay (2) 6. Connect to game server 7. Present ticket to authenticate 8. Game server validates ticket 9. Game server loads player profile 10.Play w/ other players via server 11.Game ends 12.Server updates player profile 13.Log files get written to local disk Virginia Game Client (other players) Local disk Oregon AWS cloud: PlayFab Web Services Matchmaker Instance Instance Game Server Monitor API servers Storage Amazon RedshiftReports service Instance Game servers
- 17. Basic Gameplay (2) 14.Log files get transferred to S3 storage 15.Server monitor cleans up old games; can create new servers if needed 16.Matchmaker prepares for new game 17.Events get imported into Redshift Virginia Game Client (other players) Local disk Oregon AWS cloud: PlayFab Web Services Matchmaker Instance Instance Game Server Monitor API servers Storage Amazon RedshiftReports service Instance Game servers
- 18. ELB logging for title-specific tracking Game Client Amazon Route 53 (3.playfabapi.com) API Request: Elastic Load Balancing Logs Amazon RedshiftAmazon S3 bucket Reports service Logs Instance Logs include title-specific endpoint for analytics AWS wildcard DNS (*.playfabapi.com)
- 19. Sample Loadout API usage report API NAME TOTAL CALLS AVERAGE BACKEND TIME (S) AVERAGE RESPONSE TIME (MS) AVERAGE REQUEST TIME (MS) TOTAL REQUEST TIME (M) AVERAGE RECIEVED (BYTES) AVERAGE SENT (BYTES) TOTAL ERRORS adminapi_userinfo 93,081,685 0.03 0.04 0.04 59.83 0 452 1979 gameserver_usercustomdatareadonly 1,582,348 0.06 0.04 0.04 1.06 42 902 290 matchmaker_userinfo 1,138,842 0.20 0.04 0.04 0.81 0 10046 319 matchmaker_playerjoined 1,138,247 0.01 0.04 0.04 0.79 64 2 93 matchmaker_playerleft 1,136,797 0.01 0.04 0.04 0.80 64 2 400 matchmaker_authuser 736,607 0.01 0.03 0.04 0.52 0 48 107 gameserver_usercustomdata 297,735 0.19 0.04 0.04 0.20 0 4080 0 gameserver_usercustomdatainternal 297,469 0.08 0.04 0.04 0.20 0 237 2
- 20. Instance Making a Singleton Service Highly Available Health Check Auto Scaling group Autoscaling service Instance Matchmaker (Primary) Instance Amazon Route 53 (matchmaker.playfab api.com) Elastic Load Balancing Matchmaker (Secondary) 1. Autoscaling is set to keep one instance up in each of 2 AZ’s. 2. Client locates matchmaker through Route 53 3. All matchmaker state written through to Dynamo DB Storage Game Client AZ A AZ B
- 21. Instance Making a Singleton Service Highly Available Health Check Auto Scaling group Autoscaling service Instance Matchmaker (Primary) Instance Amazon Route 53 (matchmaker.playfab api.com) Elastic Load Balancing Matchmaker (Secondary) 4. Route53 Health Check detects bad instance and fails over all traffic to secondary. 5. Secondary loads latest state from storage 6. Autoscaling group detects bad instance and starts new one. Storage Game Client AZ A AZ B
- 22. Monitoring Service Health Amazon EC2 (API handling) Elastic Load Balancing CloudWatch Health Check Email Alerts On-call Engineer Mobile Alerts Datadog visualization
- 23. Trusted by hundreds of developers • More than 500 game studios using the PlayFab SDK • 16 titles now live (w/ more than 10 million total registered players) • Integrate at any point of your game’s life cycle 23
- 25. Easy adoption by any game developer 25 Full set of support resources makes it easy for any developer to quickly start using the PlayFab platform. Getting Started Guides SDKs for popular tools Full documentation + + + Full-time developer relations team
- 26. PlayFab benefits 26Better Faster Cheaper Developers • Best-in-class technology • Proven solution • Fully documented • Back-end engineering talent • Continual improvements • Baked-in best practices • Learn from other games • Available immediately • Less testing needed • No need to hire a new team • No up-front engineering cost • Volume of scale • Amortized costs Publishers • Share tools across games • Share players across games • Less risk • Training for new developers • Share lessons across games • All funding goes to game • No tech support costs 26PlayFab confidential
- 27. Pricing model Test Free Standard Premium Enterprise Test all PlayFab features risk-free Build, test, and launch a game risk-free Backend services for casual or mobile games Meet needs of demanding core or multiplayer games Custom agreements for multi-title publishers FREE (max 1,000 DAU) FREE (UNLIMITED DAU) $2 (per 1,000 users / day) $6 (per 1,000 users / day) negotiable All features are available with API throttling limits • Live ops game manager • Login with Device ID • Receipt validation • Catalog (RM items only) • User inventory (RM items only) • Basic reports All of free tier plus: • Player accounts • Virtual goods • Standard analytics • Friends lists • Leaderboards • In-game messaging All of standard tier plus: • Cloud scripting • Photon Realtime / Turnbased / Unity Networking / Chat / Server • Custom development • Game server hosting* All of premium tier plus: • Private server clusters • Custom payment provider • Custom authentication • Custom SLA *Actual server cost (pass through) + 10% management fee. 27PlayFab confidential
Editor's Notes
- (move to next slide as soon as we start)
- Day one challenge of games. Big problem. Players practically expect games to fail at launch. Who’s experienced? Who’s been on the developer side of this? I have… but EA not alone. I’m now CEO of PlayFab, we build game operations platform to help mitigate these problems. We help companies build, launch, operate live games. A big part of our work is specifically for multiplayer games, and in today’s talk we’ll peel back the covers and talk about how we architect to scale. We are based 100% on AWS. By end of talk, you’ll have practical steps & concepts to help leverage cloud to avoid issues like these. We’ll also give a peek into PlayFab, and show some of the other things we do to help you be successful beyond just scaling. I’m James Gwertzman, CEO, and with me today are members of my team, including Matt (CTO), Thom (head of marketing), Liz (Head of product).
- but first, I want to set the context for why this matters Part of why multiplayer games are challenging is that they exist as services – and require significant investments post-launch Asian game companies get this – they’ve been doing this for years. But it’s still relatively new here in the US Nowadays, to launch a multiplayer game, you can’t just rely on LAN parties Players want to battle other players around the world And to run a great service effectively is way harder than just building a good game.
- Consider how building a game has evolved. It used to be you’d build a game, ship it, and move on.
- Now, however, games have becomes services, and suddenly launching the game is only the beginning. There’s all this other stuff that has to happen AFTER the game goes live – the service has to be managed. (talk about the flow) Costs include: Technology cost People cost (burn out) Continuously reacting to competitors
- The amount of tech required to build and operate a live game is massive, very easy to underestimate. Most game companies don’t have much experience with this. The ones who do, who operate successful games, get it. But even with cloud platforms like AWS there’s still a ton of tech you need.
- We started PlayFab specifically to help with game operations – everything that happens post launch. That includes… Back end services Tools & dashboards Integration w/ other partners Multiplayer server hosting Education… This is all great news for most developers, because it means that once again, the field is leveling. You don’t need to spend a year building backend technology to launch a great game, anymore than you need to spend a year building animation tools. It’s possible now to build the next Candy Crush saga, or Age of War, without hiring a single server engineer.
- One of the games we host is called Loadout… FPS F2P game.
- You can see the kind of game it is --- 4v4 FPS Huge emphasis on fast-twitch action. Performance and latency really matter
- First launched on PC, integrated w/ Steam. Then launched on PS4 Overall it was a mostly smooth launch – no major glitches that prevented players from getting in and playing. In fact we hit top 10 on Steam for the original PC launch
- [CLICK] Step 1 is put our gameplay servers near our players. But the only thing our game servers are going to handle is actually running around and hacking my friends to pieces. If a game servers crashes, sure players are going to get kicked offline – not much you can do about that. But players should still be able to log right back in, and jump into another game without losing their progress. [CLICK] Step 2: Everything else will be HTTP API's: Login, friends list, leaderboards, inventory, in-app purchases – these all go into HTTP. Not only does this make developing and scaling these services easier, since you can take advantage of features such as auto-scaling and HTTP load balancing, but you also get security features like SSL that work out of the box. [CLICK] Step 3: No database-level replication. It seems like if we want a global game, so that players worldwide can to just replicate all the data everywhere. But it's a fragile architecture, since now all our regions are dependent on one another. Not to mention, it adds a ton of latency back in, since you have to wait for replication commits across regions. Instead, we're going to focus on using our HTTP API's for data access. [CLICK] Step 4: But having a local cache of hot data is a fine idea, and we'll cover that [CLICK] And the last point is that nobody really “Plays Everywhere”. Anytime I hear this, an alarm bell goes off that it's a half-designed feature. Look, if I’m sitting around… on some random Tues afternoon, having a beer, playing a few rounds of deathmatch between meetings – this is completely hypothetical by the way [LAUGH] – do I really want the game to automagically connect me from my house in San Diego to a server down in Brazil? NO! Not only is the latency going to suck, I don't even know how to swear in Portuguese! How am I going to know what the 12 year old on the other end is yelling at me? What players want is the ILLUSION of a global experience. So when we talk about having a "global game", we're really talking about a subset of features, like a global leaderboard, or global friends list. These can be easily accomplished without a giant data replication mesh, and I'll show you how.
- Three primary areas: Game Client Game services Multiplayer Game Servers (physical and AWS)
- Our goal is to cover all the core services you might need… and to integrate third party services to fill in the areas we don’t cover
- Three primary areas: Game Client Game services Multiplayer Game Servers (physical and AWS)
- Three primary areas: Game Client Game services Multiplayer Game Servers (physical and AWS)
- Three primary areas: Game Client Game services Multiplayer Game Servers (physical and AWS)
- Trick: use DNS to isolate game instances for future flexibility We assign a different endpoint to each game The API request is sent to title-specific endpoint using AWS wildcard DNS (*.playfabapi.com) Sending the wildcard CNAME for *.playfabapi.com to the ELB allows us to use 0-downtime ELB deployment with CNAME swapping ELB logging to S3: Reports service pulls logs from S3 bucket and sends them to Redshift; logs include title-specific endpoint for analytics
- This is available to us, and also to the customer via the game manager dashboard.
- There are cases when you need to have a single server available (our matchmaker needs to be a single server because it needs to know all the available games), but you can use these services to keep that single server highly available. ELB Health Checking auto-terminates EC2 instances if they become unhealthy Each instance is a single-instance Elastic Beanstalk Autoscaling service spins up a new EC2 instance if needed
- There are cases when you need to have a single server available (our matchmaker needs to be a single server because it needs to know all the available games), but you can use these services to keep that single server highly available. ELB Health Checking auto-terminates EC2 instances if they become unhealthy Each instance is a single-instance Elastic Beanstalk Autoscaling service spins up a new EC2 instance if needed
- Pingdom monitors public-facing pages for outages, sends email alerts and also sends to VictorOps. CloudWatch sends alerts to VictorOps. VictorOps sends alerts to the engineer(s) on call based on CloudWatch and Pingdom alerts. ELB health check as shown earlier.
- We’ve already got substantial traction with small and medium developers, and we’re starting to work with some of the bigger companies too, which is exciting. You can integrate at any point, not just when you’re starting out. We are working w/ hyper hippo & adventure capitalist, launching an update this week
- Here’s how we price today… but this is definitely a work in progress. We need to balance so we’re cheaper than them doing it themselves, but obviously still profitable for us. And we’re in that up front zone where we don’t yet have the economies of scale that we’ll later have as we get bigger. We have certain fixed costs just to maintain availability and scalability, and we have extra capacity not using today.
- So if you’re building a game, I hope you’re thinking about your operations strategy… And if you’re thinking about your operations strategy, I hope you’ll consider a platform like PlayFab