Testing a PC-based game poses interesting challenges. With different OS platforms and many different GPUs, the number of combinations to test grows quickly. The large number of hardware configurations used by the players of EVE Online makes it impossible to test them all. This can lead to the situation where reproducing defects in the test lab is not possible. Björgvin Reynisson shares how CCP Games got fed up with the situation and decided to make a crowd-testing app that tests the game’s graphics engine. He describes the app, the framework for deploying the app, and how they handled the test results coming in from testers. Björgvin shares how the app has become both a development tool and an automated testing tool. Discover how this approach has helped identify hard-to-catch bugs, and discover some of the challenges of crowd testing. See how this project took an unexpected turn and led the team to change how the game content is distributed.

1. CROWD TESTING A
GAME ENGINE
Björgvin Reynisson
bjorgvin@ccpgames.com
@breynisson

2. THIS TALK IS ABOUT:

3. THIS TALK IS ABOUT:
How we made a crowd-testing app

4. THIS TALK IS ABOUT:
How we made a crowd-testing app
and why...

5. THIS TALK IS ABOUT:
How we made a crowd-testing app
and why...
and how it is working for us

6. WHO AM I?
Björgvin Reynisson
QA Engineer at CCP Games
Background in mobile phones

8. COMPANY
Founded in 1997
Privately held
Approx. 350 employees worldwide
Headquarters: Reykjavik, Iceland

9. LOCATIONS
Reykjavik, Atlanta, Shanghai and Newcastle

12. EVE ONLINE
Massively multiplayer online game (MMO)
Launched May 2003
Approx. 500K players in a single world
Players in over 190 countries
Biggest markets: US, Russia, Germany, UK and Canada

16. WHY CROWD-TEST YOUR GAME
ENGINE?

17. WHY CROWD-TEST YOUR GAME
ENGINE?
Testing changes before hitting production
Catching bugs with this app vs. seeing them on live server
Increasing Test Coverage
Performance testing/comparison

18. HAS ANYONE EXPERIENCED THIS?

20. "WE CANNOT REPRODUCE THIS..."

21. WHY IS REPRODUCTION HARD?

22. EVE PLAYERS USE A LOT OF
DIFFERENT CONFIGURATIONS

23. EVE PLAYERS USE A LOT OF
DIFFERENT CONFIGURATIONS
Windows
Mac
Linux (although we don't support it)
855 different Graphics cards (GPU's)
Different GPU driver versions

24. LET'S MAKE AN APP!

25. EVE PROBE
Works on your machine!

26. EVE PROBE
Repeatable test-scenes that you play back on your
computer

27. CROWD-TESTING!
Increased coverage

32. DEMO:

34. WHAT METRICS ARE COLLECTED?

35. WHAT METRICS ARE COLLECTED?
System Info
Settings
EVE Probe version
Logs
Performance Metrics
Crash info

36. METRICS
Data is not personally identifiable
OS, GPU, Processor, RAM
Monitor Resolution
Log info, warnings, errors
Up to 150 different metrics captured per scene
Primarily monitor Framtime and Memory for performance
Collect stack trace if a crash occurs

37. DATA PIPELINE

38. DATA PIPELINE
Initially used R and IPython notebooks for
cleaning/processing/visualizing data

39. IPYTHON NOTEBOOK EXAMPLE

40. WHERE'S THE
DASHBOARD!?

41. EVEPROBE DASHBOARD

42. REPORTS

43. EXAMPLE:

44. PERFORMANCE METRICS

45. PERFORMANCE METRICS

46. DEPLOYMENT PROCESS

47. BUILDS FROM EACH BRANCH

48. DISTRIBUTING THE APP

49. DISTRIBUTING THE APP
Soft launch
Released via a forum post
Distribution method remains the same

50. FORUM CHANNEL

51. LINK:
http://bit.ly/1HkUMoY

52. DELIVERY
Resources are delivered On-Demand.
Binaries are delivered On-Demand.

53. DELIVERY

56. DOWNLOAD-ON-DEMAND NOW USED
FOR EVE-ONLINE
Lowered initial install package from 7Gb to 300Mb
Saved CDN usage
Very smooth roll-out

57. DEVELOPER TOOL
A test-framework
Repeatable scenes

58. DEMO:

60. BENCHMARKING TOOL
Setup with recommended spec hardware
Run Mainline, Staging, Live branches
Monitor performance across branches

61. HAS IT BEEN WORTH THE EFFORT?

62. SUCCESS?
Download-on-Demand
Has proven its worth for hard-to-reproduce defects
Great development tool
Big potential for benchmarking use

63. ADOPTION

64. IMPROVEMENTS?
Low adoption
Should we market the app better?
Share the app with hardware vendors
Make benchmarking part of the release process

65. QUESTIONS?