This study analyzed video game requests posted on Reddit to identify what makes finding relevant games difficult. By coding over 500 game requests, the study identified 5 main relevance aspects (content, metadata, experience, interactivity, context) and 2 information need aspects that capture what users look for in games. Game requests were found to mention an average of 4.6 relevance aspects and can reflect multiple information needs. The study aims to help improve systems for discovering games by understanding the complex factors involved in video game search.

1. TOINE BOGERS
MARIA GÄDE
MARIJN KOOLEN
VIVIEN PETRAS
METTE SKOV
AALBORG UNIVERSITY COPENHAGEN
HUMBOLDT-UNIVERSITÄT ZU BERLIN
ROYAL NETHERLANDS ACADEMY OF ARTS AND SCIENCES
HUMBOLDT-UNIVERSITÄT ZU BERLIN
AALBORG UNIVERSITY
iCONFERENCE 2019, WASHINGTON, DC, USA
“LOOKING FOR AN AMAZING GAME I CAN
RELAX AND SINK HOURS INTO...”
A STUDY OF RELEVANCE ASPECTS
IN VIDEO GAME DISCOVERY

2. MOTIVATION

3. INTRODUCTION
➤ Video games are big business
• More than 2.5 billion video gamers worldwide
• Worth 78.61 billion USD in 2018
➤ Research on
• Benefits/hazards of playing
• Gamer behavior
• Game recommendation
➤ What about game search?
https://www.wepc.com/news/video-game-statistics/
3

4. EXAMPLES OF COMPLEX GAME NEEDS 4

5. VIDEO GAME SEARCH IS COMPLEX!
➤ Not every need can be satisfied by current search engines:
• Different aspects to what makes a video game relevant
• Combination of knowledge about the current information
need and past preferences to solve them
➤ Means they are underrepresented in search/interaction logs
• Users go to forums to get help
• “Collaborative”, intellectual search process
5

6. OUR FOCUS
➤ This study: What makes these information needs hard to
solve?
• What relevance aspects do users express in complex game(s)
requests?
• Relevance aspects = components of stated information needs
with the intent of finding relevant results
➤ Future work: How can we solve these information needs?
• Content matching
• Search engine or recommender system design
6

7. METHODOLOGY

8. DATA COLLECTION
➤ Reddit is a popular discussion & social news website
• Subreddits (= forums) dedicated to any topic imaginable
• Crawled 2,226 threads in June 2018 from
– /r/gamingsuggestions
– /r/gamesuggestions
– /r/tipofmyjoystick = (re-finding video games)
8

9. CODING PROCESS
➤ Open coding
• Individual open coding on 75 game threads by 3 authors
• Information presented for each post
– Title
– Full text of first post
• Resulted in 3 different initial coding schemes with 95 different
relevance aspects
9

10. CODING PROCESS
➤ Axial coding
• Card sorting to arrive at single coding scheme
– Split, merge & label initial codes into smaller set of codes
– Axial coding to group them into top-level categories
• Calibration
– Coding scheme discussed & finalized by all five authors
• Final coding scheme
– 5 top-level relevance aspects (33 subcategories)
– 2 top-level information need aspects (8 subcategories)
10

11. CODING PROCESS
➤ Actual coding process
• Random selection of 140 Reddit threads coded by each author
• Total of 521 game requests were coded
• Overlap of 20 posts between each pair of successive coders
– Fleiss' kappa over total of 80 overlapping posts
– Top-level aspects usually show substantial agreement (κ > 0.6)
with a few exceptions
11

12. RESULTS & ANALYSIS

13. RELEVANCE ASPECTS FOR GAMES
➤ Content (What should it be about?)
• Character(s), Cutscene(s), Design, Dialogue, Gameplay mechanics,
Plot, Setting, Sound design, Time, Topic, World building
➤ Metadata (What kind of properties should it have?)
• Audience, Availability, Creator, Genre, Language, Platform,
Popularity, Price, Properties, Release date, Series, Technical
specifications, Title
➤ Experience (What kind of experience should/did it provide?)
• Mood, Perspective, Playability, (Re)play value
➤ Interactivity (How should the user interact with it?)
• Connectivity, Controls, Expandability, Game mode
➤ Context (How will it be used?)
• Context
14
11
13
4
4
1

14. RELEVANCE ASPECTS FOR GAMES
➤ Mix of 22 domain-agnostic and 11 domain-specific aspects
• Domain-agnostic aspects
– Character, Plot (Content)
– Creator (Metadata)
– Mood (Experience)
• Domain-specific aspects
– Gameplay mechanics, World building (Content)
– Playability, (Re-)play value (Experience)
– Connectivity, Controls, Expandability, Game mode
(Interactivity)
15

15. INFORMATION NEED ASPECTS FOR GAMES
➤ Information need (What type of need is it?)
• Choice, Discovery, Known-item, Similarity
➤ Search process (What could help identify relevant games?)
• Link to external resource, Not this one, Search history,
Situation of exposure
16
4
4

16. REQUEST ANALYSIS
➤ Game requests are complex!
• On average 4.6 relevance aspects
per request
➤ Example
• “I'm looking for a tactical fantasy RPG that I played on a PSX emulator
in the early- to mid-2000s. It was similar to Final Fantasy Tactics,
including the graphics and isometric style, but there were significant
story choices you could make early on that directed you into widely
different paths. I remember these choices being presented as text, with a
world map in the background -- there weren't cutscenes that I recall. The
plot concerned a war of some kind. Definitely wasn't Tactics Ogre or
Vandal Hearts.”
17
Release date
Genre
Plot
Platform
Design
Gameplay mechanics
Cutscene(s)
Not this one
Similarity
Known-item

17. REQUEST ANALYSIS
➤ Requests can reflect more than
one type of information need
➤ Example
• “Looking for a Mac/iOS/Xbox One game where I can create
buildings/bases/homes etc. I love architecture and want a game where I
can design buildings (besides Minecraft). I like the building mechanics of
Rust and Raft, but my Macbook Air isn’t powerful enough to run Rust
and Raft is only available for PC.”
18
Discovery
Similarity

18. REQUEST ANALYSIS
➤ Clear differences between information need types in which
aspects are more commonly mentioned
• Known-item relies more on content & metadata
• Context is relatively important for discovery
19
A Study of Relevance Aspects in Video Game Discovery 513

19. CONCLUSIONS
➤ Game requests can be very complicated!
• Many relevance aspects (and sometimes needs) in one game
request
• Content and metadata aspects are most frequently mentioned
(almost in every request)
• Domain-specific aspects that do not occur elsewhere
20

20. FUTURE WORK
➤ Comparison between different domains
• Books, movies (iConf 2018), games and music
➤ Which relevance aspects will be the most challenging for
search engines?
• What information is necessary to fulfill these complex search
requests?
• How and where can we extract this kind of information?
➤ What type of system could solve these complex requests?
21

21. QUESTIONS? COMMENTS?