Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
CGDC 2010            How to Support an Action-Heavy MMORPG             from the Angle of Server Architecture              ...
Table of Contents   Introduction    – TERA   Warming Up   Architecture    – TERA Server   Test   Summary             ...
• Introduction                                                                  • Warming UpT.E.R.A.                      ...
• Introduction                                                              • Warming UpAction-Heavy MMORPG               ...
• Introduction                                                                         • Warming UpTargeting vs. Non-Targe...
• Introduction                                                             • Warming UpTerminology                        ...
• Introduction                                                                   • Warming UpServer Architecture Requireme...
• Introduction                                                               • Warming UpSupporting Action-Heavy Combat   ...
• Introduction                                                              • Warming UpSupporting Seamless World         ...
• Introduction                                                          • Warming UpSupporting Inter-Server Contents      ...
• Introduction                                                               • Warming UpServer Architecture History (1/2)...
• Introduction                                                                   • Warming UpServer Architecture History (...
• Introduction                                                                   • Warming UpFinal TERA Server Architectur...
• Introduction                                                        • Warming UpArbiter Server                          ...
• Introduction                                                                   • Warming UpWorld Server                 ...
• Introduction                                                                    • Warming UpScalability                 ...
• Introduction                                                                   • Warming UpAdditional Issues            ...
• Introduction                                                                   • Warming UpTest with Dummy-Clients      ...
• Introduction                                                                  • Warming UpStress Test                   ...
• Introduction                                                           • Warming UpSummary                              ...
References[1] Pattern-Oriented Software Architecture: Patterns for Concurrent and Networked Object     Douglas Schmidt et ...
감사합니다   谢谢!   Thanks!   Q&A             TERA’s Server Architecture   22
Upcoming SlideShare
Loading in …5
×

TERA Server Architecture

11,015 views

Published on

How to Support an Action-Heavy MMORPG from the Angle of Server Architecture

2010 China GDC Presentation

Published in: Technology, News & Politics

TERA Server Architecture

  1. 1. CGDC 2010 How to Support an Action-Heavy MMORPG from the Angle of Server Architecture 30 JUL 2010 Seungmo Koo (具升谋) sm9@bluehole.net Copyright © 2010 Bluehole Studio, Inc.
  2. 2. Table of Contents Introduction – TERA Warming Up Architecture – TERA Server Test Summary TERA’s Server Architecture 2
  3. 3. • Introduction • Warming UpT.E.R.A. • Architecture • Test • Summary The Exiled Realm of Arborea – Open-beta test within this year TERA via Mass Media – Blockbuster MMORPG using Unreal Engine 3 – Development cost about 200,000,000 RMB during 3.5 years – Various contents on the Seamless world – Non-targeting, Non-targeting, Non-targeting, … TERA via Developers’ View – Is it possible? • (X) Converting Action-heavy console game into MMORPG • (O) Making MMORPG with strong action elements TERA’s Server Architecture 3
  4. 4. • Introduction • Warming UpAction-Heavy MMORPG • Architecture • Test • Summary Realistic Combat – Dodge or block by observing the attacks of enemies – Control the direction and distance to attack enemies – Natural in Console or MO games – High expectations in regard to high sync rate and low latency Non-targeting Combat System – A way to support action-heavy combat Server-side Support for Action-Heavy Combat – Space and collision calculation very rapidly – Send and receive packets much more  Work load proportionate to the level of action-heaviness TERA’s Server Architecture 4
  5. 5. • Introduction • Warming UpTargeting vs. Non-Targeting • Architecture • Test • Summary Targeting Non-targeting T6 T5 T4 Weapon T3 Direction T2 T1 Player Ally In Fact, 3D Volume for Collision Check Enemy  Optimized space search and collision check are needed TERA’s Server Architecture 5
  6. 6. • Introduction • Warming UpTerminology • Architecture • Test • Summary Channel – The space where players actually explore Continent – Literally, a large area of land having one or more channels World – Server contains a group of continents Planet – Group of worlds – (aka) Realm server – The unit of user clients’ view TERA’s Server Architecture 6
  7. 7. • Introduction • Warming UpServer Architecture Requirement • Architecture • Test • Summary Action-Heavy Combat – Non-targeting battle in the overall combat system • Increasing space-search and hit-judgment Seamless World – No loading and teleport within a world – No limitation to deploy contents anywhere • Creature, hunting-ground, and any type of content • Implementation should not restrict contents-deployment Inter-Server Contents – Competitive contents among planets – Community contents TERA’s Server Architecture 7
  8. 8. • Introduction • Warming UpSupporting Action-Heavy Combat • Architecture • Test • Summary Frequent Space Calculation – Object-search & Hit-judgment • CPU-intensive work – No physics engine in the server • Simplify collision volume – Using lock-free manner [4] • Lock radically increase the CPU usage Frequent Transmission – Aggregate broadcast • In inverse proportion to action-heaviness – The more frequent send, the better action-heaviness • Important to find an adequate value for aggregation TERA’s Server Architecture 8
  9. 9. • Introduction • Warming UpSupporting Seamless World • Architecture • Test • Summary Popular Thread Model for Zone-based World – A dedicated thread to a particular content or area • The decreasing complexity of source-codes – Work load could converge on a specific thread • Inappropriate for implementing a seamless world To Make Seamless World – TERA takes on (homogeneous) worker thread model [1] – No dedicated thread that handles specific contents or areas • One thread pool including symmetric worker-threads • Somewhat difficult and complex to implement – No blockings among threads via lock-free implementation [2, 3] TERA’s Server Architecture 9
  10. 10. • Introduction • Warming UpSupporting Inter-Server Contents • Architecture • Test • Summary Types – Community contents – Shared world contents • Common instance-dungeon pool • Battleground server Simple Structure – To reduce inter-server complexity – To regard the shared world as clients’ own planet – No handover of network connection Scalability – Easy to add a new world server with new contents TERA’s Server Architecture 10
  11. 11. • Introduction • Warming UpServer Architecture History (1/2) • Architecture • Test • Summary First Architecture – Project S1 prototyping version – Front Server: world container • Support action-heavy combat on a seamless world – Back Server: DB cache server – Inappropriate to inter-server contents Front Server Client Back Server Planet TERA’s Server Architecture 11
  12. 12. • Introduction • Warming UpServer Architecture History (2/2) • Architecture • Test • Summary Second Architecture – Broadcaster • To support community contents (chat-channels, party, …) • To reduce world servers’ network I/O – DB proxy • To support data-sharing contents (auction, warehouse, … ) • Serialize and validate the DB tasks – Complex: difficult and annoying to implement Broadcaster DB Proxy Client World Server Planet TERA’s Server Architecture 12
  13. 13. • Introduction • Warming UpFinal TERA Server Architecture • Architecture • Test • Summary Split by Role – World server for CPU intensive tasks – Arbiter server for I/O intensive tasks • The role of “DB proxy + Broadcaster” • In fact, DB proxy and Broadcaster have a lot in common – Shared code, similar task-style – Reduced the complexity of server architecture Client Arbiter World Server Server Planet TERA’s Server Architecture 13
  14. 14. • Introduction • Warming UpArbiter Server • Architecture • Test • Summary Single System Image [6] – The only server that game-clients access • Originated in Starcraft unit – No handover of network connection – Delegated for packet-broadcast by World servers • Dispersing network I/O load – Handling common contents among World servers • Chatting, guild, party, … Implementation Features – Using locks for common contents – Two thread pools • Blocking I/O (e.g. DB operation) • Non-blocking I/O (e.g. packet broadcast) TERA’s Server Architecture 14
  15. 15. • Introduction • Warming UpWorld Server • Architecture • Test • Summary Seamless World Container – Space where players explore • A world includes multiple continents • Specialized for space search and assessment of the combat – Sharable among multiple Arbiter servers • Battle-group, common instance dungeon pool Implementation Features – Single thread pool consists of symmetric worker-threads • No thread that exclusively handles particular contents – Robust code base for multi-threading [2, 3] • Making all tasks non-block-able through lock-free manner [4] • (c.f.) Proactor [1] & Active-object pattern [5] TERA’s Server Architecture 15
  16. 16. • Introduction • Warming UpScalability • Architecture • Test • Summary Server Topology – Can attach World servers to a single Arbiter server – Convenient to arrange continents • Different combinations of continents in multiple World servers World Server Side – Identical worker-threads as the number of CPU cores • Direct benefits from CPU core count and speed Arbiter Server Side – Possibility of bottleneck • Less critical of an issue than allocated bandwidth for service – Utilizing the surplus CPU power • Packet optimization to reduce bandwidth TERA’s Server Architecture 16
  17. 17. • Introduction • Warming UpAdditional Issues • Architecture • Test • Summary Considering Contents Deployment – Inter-server contents influence how to deploy World servers – Considering the stability • Can keep a Planet live even if a specific World server down • Quarantine unstable contents (e.g. latest patched) Split Servers Depending on Purpose – Dedicated server for users’ convenience • Chatting, searching, and so on Order of Priority – Not to put restriction on elements of fun TERA’s Server Architecture 17
  18. 18. • Introduction • Warming UpTest with Dummy-Clients • Architecture • Test • Summary Sisyphus – Client program operating dummy-bots for server-load test • Operating 1000~1500 bots per one Sisyphus • Recording real-players’ behavior patterns WAN Simulator – Give virtual latency to mimic real-network environment • Randomized latency around 200ms • Required high performance machine • Required dedicated line for load-test Sisyphus Sisyphus WAN Simulator Planet Sisyphus Sisyphus Dedicated-line for test TERA’s Server Architecture 18
  19. 19. • Introduction • Warming UpStress Test • Architecture • Test • Summary 3rd Closed Beta Test – Last Spring in Korea involving about 24000 users – Events designed to complement daily stress test • A sizable number of concurrent users were provided • Server-status goes on smoothly throughout the stress test Server Status – Very low CPU usage on World servers – Not any congestion handling network I/O on the Arbiter server – Monitoring via VTune [7] • Latest version to support Nehalem CPU [8] TERA’s Server Architecture 19
  20. 20. • Introduction • Warming UpSummary • Architecture • Test • Summary Impossible is Nothing – Not easy to make non-targeting battle in a seamless world – But, it is certainly possible Server Architecture Case by Case – Optimized for each different type of MMOG – Arbiter-World architecture for TERA Complexity – Support the “simplicity first” policy – For stability, ease of management, and … TERA’s Server Architecture 20
  21. 21. References[1] Pattern-Oriented Software Architecture: Patterns for Concurrent and Networked Object Douglas Schmidt et al. John Wiley & Sons, 2000.[2] Thread-Specific Storage for C/C++ Douglas Schmidt et al. C++ Report, 1997.[3] The Art of Multiprocessor Programming Maurice Herlihy et al. Morgan Kaufmann, 2008.[4] Simple, Fast, and Practical Non-Blocking and Blocking Concurrent Queue Algorithms Maged M. Michal et al. ACM symposium on Principles of distributed computing, 1996.[5] Active Object: An object behavioral pattern for concurrent programming R. Greg Lavender et al. Pattern languages of program design, 1996.[6] Single System Image R. Buyya et al. The international journal of high performance computing applications, 2001.[7] VTune: Performance Analyzer Intel Software Network. http://software.intel.com/en-us/intel-vtune/[8] Nehalem micro-architecture http://en.wikipedia.org/wiki/Nehalem_(microarchitecture) TERA’s Server Architecture 21
  22. 22. 감사합니다 谢谢! Thanks! Q&A TERA’s Server Architecture 22

×