An online game server's functional and non-functional features lead to non-standard challenges for both architecting as well as testing. This talk starts with an overview and then discusses one testing scenario in depth. We stress particularly on testing the asychronous nature of the application's method calls. A few general approaches of testing such applications terms are alluded to at the end.

1. Saturday, 17th March, 2012

2. Massively Multi-Player Online Game
 Nirmalya Sengupta, Independent Consultant
 Srinivas Chillara, Principal Consultant at CeeZone

3. Background
Dutch company – Online lottery giant,
decided to enter Online Poker market
Scope: Building a complete offering of
online betting Poker game environment
(first version in 12 months)
Technology choices: Java, C++, GWT,Test
Scripting, Linux, MySQL, RabbitMQ,
RedDwarf

4. Application: a bird’s eye view
Player Portal
Database
Game Server Tournament
Server
RabbitMQ
Lobby Server

5. Poker lobby – Tables list

6. Game table

7. Poker Lobby – showing all table types, tournament types etc

8. Servers must be ...
 Robust
 Network ready
 Support for ‘Statefulness’
 Easy to interface with external systems
 Highly threaded
 Minimum supportable concurrent player base: ~20000
 Logged in players don't always play but add load
 Speed and Order of message exchange between Client
and Server are of essence

9. Crucial Challenges
 Randomness must be satisfactorily built in and proven
(legal requirement)
 Arbitrarily fired Timers affect software's behaviour
 Player may log in just to watch others
 Network connection with player can drop anytime: game
must continue
 Players may decide to sit out and rejoin anytime
 Multiple simultaneous tables, Multiple simultaneous
tournaments
 Game History must be retained (legal reqmt.)

10. Tricky to test software which..
 is inherently asynchronous
 is supposed to support many users, while
maintaining performance
 is asking testers to be good/bad Poker players
 is affected indeterminately by Timers
(arbitrariness)
 requires preservation of history of every move
 is expected to display minimum latency always

11. Example of asynchronous behaviour
NEW HAND!
P1
P4 P2
ASYNCHRONOUS
MESSAGE!
step3. Hand over! step1. P2 plays
Now P3 is
P3 informed of
new Hand
allowed to re-
step2. P3 sits-out / join
gets disconnected

12. Some of bugs detected
 Disconnected player not able to login
 Negative balance for a player
 If the player's connection to the Game Server is lost
while playing his table balance is not returned to his
real balance.
 The raise rules are not correctly applied for micro-stake
tables
 The tournament ended abruptly and winners are
declared
 Application crash when player tries to join a table
 Server running out of memory

13. Steps taken to increase testability
 Separation of rules of game/business from infrastructure
– Model using a Finite State Machine
– Write separate State Transition TestDriver
 Impregnate codebase with EventWatchers
– Continuously emit info about the goings on
– JMX bean for specific data elements
– Specialized sniffer Tasks (abstraction of threads)
 Carefully structured Log messages
– Programmatically consume log message and correlate
happenings in the applications
– Simple utilities to filter logs

14. Testing infrastructure
Human Players
BOTs Player Portal
. Database
.
.
.
.
.
. `
Game Server Tournament
.
. Server
.
RabbitMQ
JMX Console
Lobby Server

15. Steps taken to increase testability
 Create controlled load on the Server
– Java-based Robots written which could play Poker
– Count, time and intelligence of the Robots,
configurable
– Homegrown binary protocol carried Roundtrip time
data
 Deal with Randomness
– Handcrafted card-sets were dealt to players
(repetition)

16. Experience reminds us that
 Testing cannot be an afterthought for planning or design
 Collaboration between testers, developers, analysts and
architect is crucial
 Extra measures taken initially, make many very difficult
problems easy to handle later on
 Competent testing starts with good thinking which
considers multiple perspectives