MIGS18 Transforming from peer-to-peer to dedicated servers on a live game
•Download as PPTX, PDF•
0 likes•41 views
Slide deck used during my talk at MIGS18 on Tuesday, November 13th.
Recommended
More Related Content
Similar to MIGS18 Transforming from peer-to-peer to dedicated servers on a live game
Similar to MIGS18 Transforming from peer-to-peer to dedicated servers on a live game (20)
Recently uploaded
Recently uploaded (20)
MIGS18 Transforming from peer-to-peer to dedicated servers on a live game
- 1. Transforming from Peer-to- Peer to Dedicated Servers on a Live game How connectivity was fixed on For Honor Laurent Chouinard Online Project Lead – For Honor Talk given at MIGS18, Tuesday, November 13th 2018 https://www.linkedin.com/in/laurentchouinard/ https://twitter.com/LaurentPointCa
- 2. / Creating a fight game Betas and launch Root cause Let’s make servers! Open test results Deploying a server infra Then what? INTRO
- 4. / Art of Battle fight mechanics use action/reaction patterns Reaction time vital to all defensive moves FIGHTING PROTOTYPE
- 6. / Most fighting games use peer-to-peer A server or host would increase latency Replication: positional updates, conflicts difficult to resolve Gameplay simulation: synchronicity guaranteed, ideal for duels, complex for melee In the end, latency was key SOME ASSUMPTIONS
- 7. / Peer-to-peer via host Peer-to-peer Full mesh Dedicated server CHOICES MADE + Deterministic Gameplay Simulation
- 8. / DETERMINISTIC GAMEPLAY SIMULATION Lockstep marching Exchanging inputs No randomness a + b = c a + b = c
- 9. / DETERMINISTIC GAMEPLAY SIMULATION 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 Player 1 frames Player 2 frames Time OK OK?!
- 10. / DETERMINISTIC GAMEPLAY SIMULATION 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 Player 1 frames Player 2 frames Time OK OK
- 11. BETAS AND LAUNCH
- 12. / DURING DEVELOPMENT Near-perfect development and testing environments Alpha and beta phases: issues were drowned by the noise Occasional glitches flagged as rare, can’t reproduce © Nan Palmero - https://www.flickr.com/photos/nanpalmero/20724139849
- 13. / LAUNCH
- 14. / LAUNCH
- 15. / #1 complaint of the community: “Disconnections!” LAUNCH https://commons.wikimedia.org/wiki/User:Jacklee https://commons.wikimedia.org/wiki/User:Stefan-Xp Dashboards are all green?
- 16. / Extreme parameters make the gameplay simulation system unable to correct and keep up. “That’s gotta be performance related.” Performance-focused update with hundreds of fixes: no change. POST-LAUNCH ANALYSIS
- 17. ROOT CAUSE
- 18. / Input received is too old or Desync cannot be recovered GAMEPLAY SIMULATION HAS TO GIVE UP
- 19. / INPUT TOO OLD 1 2 3 4 5 6 1 2 3 4 5 6 Player 1 frames Player 2 frames ! Effort to rewind and resimulate becomes too big.
- 20. / DESYNC … 10 … 20 … 30 … 10 … 20 … 30 Player 1 frames Player 2 frames ?! ?!?! Simulation buffer contains no valid snapshot
- 21. / But why? Packets are duplicated, old frames are rejected, and players that lag are removed! GAMEPLAY SIMULATION HAS TO GIVE UP
- 22. / A B C D PROBLEM #1: LOAD FEEDBACK LOOP
- 23. / PROBLEM #2: HOST MIGRATION A B C D
- 24. / PROBLEM #3: PEER ROUTING A B C D NAT NAT X
- 25. / PROBLEM #4: JOINING IN PROGRESS E B C D
- 26. / THE WEAKEST LINK
- 27. / CEILING 100s of improvements later Connectivity bugs resolved Performance problems addressed Game Stability still an issue
- 30. / SERVER DESIGN We need low latency! • Super lightweight to run as fast as possible (150-200 fps) • Server does not need to run simulation • Aware of the game state, handles session joining and leaving • Performance Monitor to isolate "the weakest link" and protect other players
- 31. / CONNECTION COMPLEXITY 1 link 3 links 6 links
- 32. / CONNECTION COMPLEXITY 28 links 8 links
- 33. / A LESSON IN LATENCY 73 ms ISP 1 ISP 2 36 ms 36 ms 6 ms AB via server 78 ms Console player: AB via P2P 73 ms (best, direct) 150 ms (worst, routed) All values are averages of the single trip (time going from player A to player B) +5 ms ?!?
- 34. / A LESSON IN LATENCY ISP 1 ISP 2 Big Cloud Providers • Goggles • Tropical Forest • IndigoVery large pipes Super optimized Well maintained Peering exchanges No strong incentive
- 35. / SERVER: MULTIPLEX AND MERGE Player A Player C Input: Attack A: Attack B: Dodge Merge Player B Input: Dodge
- 36. / SERVER: INPUT CANCEL Player A Player C Input: Attack A: Attack Cancel Player B LATE Input: Dodge
- 37. / PERFORMANCE MONITOR: KICK Goal: remove the cause of game degradation • Major framedrop: self-eject • Lag with everyone: self-eject • Contribution to advance the simulation is low: self-kick or vote-kick
- 38. /
- 39. / Connectivity errors 1 resync every 4 matches 10% pauses Weakest link impact RESULTS FOR STABILITY Reduced by 75% GONE, no resync GONE, no migrations GONE, no ripple effect
- 40. / BONUS No NAT management Players IP hidden 42% reduction in match creation errors
- 42. / AMAZON GAMELIFT MatchMaking Gamelift Queue Fleet of instances Region 1 Fleet of instances Region 2
- 43. / AMAZON GAMELIFT MatchMaking Gamelift Queue Region 1 Fleet of instances Region 1 Fleet of instances Region 2 Gamelift Queue Region 2 Gamelift Queue Region 3 Gamelift Queue Region 3 Fleet of instances Region 3 Fleet of instances Region 4
- 44. / Server-side switch that allows us to activate per game mode, per platform. • PvP modes: Move to DS • PvAI modes: Keep P2P for now Phase 1: February 2018 - PC only Phase 2: March 2018 - Consoles Phase 3: May 2018 - Move PvAI to DS as well PROGRESSIVE ROLL-OUT
- 45. THEN WHAT?
- 46. / Start with Windows to simplify development Multi-tenancy up to the limit Optimize threading and logging to bring it higher Switch to Linux More multi-tenancy Future tenancy and scaling optimizations OPTIMIZATION PATH
- 47. / 1. Don’t ignore the small things! 2. Monitoring player experience is better than servers and services. 3. Cost-driven design is a double-edged sword. Focus on the right tech for the problem, then see its optimization potential. 4. Deliver small increments often, and measure. OUR KEY LEARNINGS