The document discusses using sentiment and topic analysis to analyze player feedback from reviews and comments. It provides an overview of sentiment analysis and topic detection methods and challenges in implementation. Specifically, it discusses using a mix of qualitative coding, machine learning algorithms like NLTK, and AI models from OpenAI to analyze sentiment and identify topics at a granular level. The combination of these techniques helps gain insights into player sentiment, concerns, and how to improve games based on feedback.

1. Nikola Vasiljević – Head Of Insights & Analytics at Mad Head
Games
LEVERAGING SENTIMENT AND
TOPIC DETECTION FOR DECODING
PLAYER FEEDBACK

2. THE AIM IS TO GET YOU FAMILIAR WITH SENTIMENT AND
TOPIC ANALYSIS AND INSPIRE YOU TO USE THEM (for free) IN
YOUR DECISION-MAKING STACK.
YOU WILL LEARN:
1. INTRO TO SENTIMENT AND TOPIC
2. HOW WE UTILIZE THEM, WITH EXAMPLES
3. CHALLENGES WE FACED
4. WHAT METHODOLOGIES AND ALGORITHM WE
USE
5. KEY TAKEAWAYS WITH BENEFITS

3. INTRO TO SENTIMENT AND TOPIC ANALYSIS: ANALYZE YOUR PLAYERS’ REVIEW
AND COMMENTS
SENTIMENT ANALYSIS
An area of qualitative research and NLP that analyzes textual input and
assigns sentiment:
• Positive
• Neutral
• Negative
Common application: Analysis of customer satisfaction
Can be expanded to detect:
• More sentiment detail – mixed, very positive, very negative
• Specific emotions (happy, sad, thrilled, excited, angry…)
• Priority/Urgency
• Intentions (interested, not interested, purchase intent…)
QUALITATIVE DATA ANALYSIS - TOPIC DETECTION ANALYSIS
• A set of research and data analysis methods aimed
at systematically interpreting and extracting
meaningful insights from non-numerical data
• Common application: Contextual analysis of consumer
feedback
8.5

4. WE USE IT TO: ANALYZE PLAYERS’ COMMENTS AND REVIEWS ON THE INTERNET, AND
LEARN WHAT THEY ARE TALKING ABOUT AND HOW THEY FEEL ABOUT IT
This enables us to deliver insights on how we can make and improve games based on
players’ feedback
• Use main genre drivers as pillars
• Make different game design decision
• Solve bugs
• Balance out our games
• Refine our communication to manage players’ expectations
• Patch up our games
USER ATTITUDE
AND SATISFACTION
ANALYZE
COMPETITION
PLAYER
SEGMENTATION
ANALYZE GENRES’
MAIN DRIVERS

5. CHALLENGES: PROVIDE INSIGHTFUL AND TIMELY FEEDBACK DURING THE STEPS OF THE
PRODUCTION CYCLE
TRADITIONAL QUALITATIVE METHODS
• Require unbiased experts
• Take a lot of time
• Hard to scale for big data
• Cannot be fully automated
ML – NLP and LLMs
• Understanding irony, double meaning,
sarcasm
• Broken language and mistakes
• Industry-specific slang – n00b, OP, take my
money…
• Understanding context (“Andromeda anyone?“)
To get useful, actionable
insights, it requires:
• a mixture of methods
• a lot of fine tuning
• a human “bus driver“

6. • CHALLENGING, IN A POSITIVE WAY, AND FUN
• ELEMENTAL WEAPONS AND ENVIRONMENTAL
DAMAGE SYSTEM IS COOL
• BOSS FIGHTS ARE EXCITING
• LEARNING ATTACK PATTERNS IS REWARDING
• REQUIRES SKILL, WHICH IS REWARDING
• UNFAIR AND IMBALANCED COMBAT
• BOSS DIFFICULTY, ESPECIALLY THE FIRST ONE IS
UNFAIR
• NO PILLAR TO SAVE THE GAME BEFORE THE
BOSS FIGHT
• WAS THERE FOR THE STORY, BUT THE COMBAT
IS TOO MUCH
• BOSS DIFFICULTY SPIKE
• WHY IS IT SOULSIKE
WHAT
NEXT?
EXAMPLE FROM SCARS ABOVE: A TOP DOWN ANALYSIS OF PLAYERS’
SENTIMENT AND TOPIC DETECTION
Sub-topic
Sub-topic

7. GROUNDED THEORY: FORM THEORIES AND HYPOTHESES BASED ON DATA, TEST
THEM AND TAKE ACTION
What we did
• Conducted additional research to find out the nature and the differences between
story-driven players and combat-driven players, which gave us confirmation for our
hypothesis
• Lowered overall difficulty
• Adjusted the design of some boss fights
• Reworked the difficulty settings by making game design changes to the EASY mode 
removed some souslike elements, like respawning enemies
ADJUSTING COMBAT / BOSS DIFFICULTY BY MAKING IT EASIER WILL MAKE THE GAME
PLAYABLE FOR NARRATIVE-FIRST PLAYERS WHILE NOT HARMING THE EXPERIENCE OF THE
SOULSLIKE PLAYERS TARGET SUB-GROUP
Why is it important to develop theories?
• Game design elements are intertwined and work in balance – tweaking one will
inevitably impact others, so taking on them separately will get you nowhere
• We have multiple target sub-groups which sometimes are driven by totally opposite
drivers  too much catering to one may repel another

8. SENTIMENT ANALYSIS: A MIX OF APPROACHES

9. SENTIMENT ANALYSIS: A MIX OF APPROACHES
1. Devise a sample
(N=1000)
2. Go manually through
each review/comment
and assign sentiment
3. Use it as a training
model

10. SENTIMENT ANALYSIS: A MIX OF APPROACHES
1. NLTK Python library
2. Calculate sentiment for each
review/comment
3. We used a pre-trained model
with fine tuning

11. Use OpenAI API to make your
own assistant which will do
sentiment analysis for you
SENTIMENT ANALYSIS: A MIX OF APPROACHES

12. NLTK OVERVIEW: A PRE-TRAINED MODEL WITH FINE TUNING WORKS SOLIDLY
WELL
1. Text preprocessing
• Remove stop-words (words of little value
(the, and, is, in, of…)
• Word exclusion – if there are any
particular words to exclude
• Tokenization – the text is transformed
into individual tokens to facilitate
analysis at a more granular level
• Stemming – removing suffixes to get the
root of the word (run, running, runs run)
• Lemmatization - reducing words to their
base or dictionary form, known as the
"lemma." It considers the meaning of the
word and applies morphological analysis to
obtain the root form; good, better, best 
good
2. Sentiment intensity analyzer
• Apply sentiment scores – calculates
sentiment for textual input based on
tokens’ sentiment scores
3. Fine-tuning
• Manually adjusting sentiment thresholds
• Adding a custom dictionary (“sick, OP,
take my money…“)
Comment Sentiment
0 Holy hell. That was a movie-level trailer!<br>... Positive
1 Animations are so bad seeing the graphics it d... Negative
2 Pyramids and darkness? Destiny is leaking Neutral
3 <b>facepalm</b> the running animation is still... Neutral
4 This looks interesting Positive
.. ... ...
111 oh again STONG women? facepalm Negative
112 Alles sehr schön. Aber zuerst zusammen die Num... Positive
113 Let&#39;s goooo Positive
114 Awesome: I&#39;m eager for this game Positive
115 Really looking forward to this game! Positive
THE OUTPUT
ASSIGNING SENTIMENT
SCORES

13. OPEN AI: PROMPT-ENGINEER YOUR OWN ASSISTANT
1. Specify model (gpt 3.5 turbo, gpt-4…)
2. Define messages:
1. Role of the messenger: system, user, assistant
2. Content of the message – tell it what it should do
Prompt engineering step-by-step*:
a) system message
b) very specific instructions, each given in a separate prompt
c) Provide input data
d) Add contextual information
e) Set temperature (controls randomness of the output)
f) Specify desired output format (json, csv)
* Goran Milovanović: Generative AI in Action: Three case studies with OpenAI GPT(3-3.5)
THE
OUTPUT
DEFINE YOUR ASSISTANT

14. TOPIC ANALYSIS: A MIX OF APPROACHES

15. TOPIC ANALYSIS: A MIX OF APPROACHES
1. Devise a sample
(N=1000)
2. Go manually through
each review/comment
and assign tags
3. Synthesis – group
tags into subtopics
and topics

16. TOPIC ANALYSIS: A MIX OF APPROACHES
1. Take the keywords
devised with qualitative
coding
2. Setup IF THEN ELSE
classification rules for
keyword search of your
text input, and group by
topic and subtopic

17. TOPIC ANALYSIS: A MIX OF APPROACHES
BERT AND LDA on their
own didn’t get us to where
we wanted.
LDA + OPEN AI LLM:
1. Preprocess the data (as
in NLTK)
2. Extract topics and
associeated words with
LDA to a list
3. Take the extracted
topics and generate a
more user-friendly
output

18. TOPIC ANALYSIS: A MIX OF APPROACHES
Use OpenAI API to make
your own assistant which
will do sentiment analysis
for you

19. QUALITATIVE CODING AND ASPECT BASED TOPIC DETECTION: MANUAL WORK
WITH HIGH REWARD
COMBAT DIFFICULTY
HITTING WEAKSPOTS
FIRE WEAPON RELOAD
SHORT WEAKSPOT
OPPORTUNITY WINDOW
..It was sooo unfair hitting the glowing part of the chest with the fire weapon, as reload times are
too long so you miss the opporutnity window…
COMBAT DIFFICULTY
WEAPONS
COMBAT DESIGN
COMBAT DIFFICULTY
SHOOTING
RELOAD
COMBAT DESIGN
1. Take the keywords devised with qualitative
coding
2. Setup IF THEN ELSE or REGEX classification
rules for keyword search of text input,
and group by topic&subtopic
ASPECT BASED HARD-CODED TOPIC DETECTION

20. LDA + OPEN AI LLM: A WELL-ROUNDED IMPRESSIONS SUMMARY…
1. Define a function (A) to extract topics and associated
keywords from the input
2. Provide it parameters
1. Input parameters
2. Number of topics to discover
3. Number of words to include per topic
4. Output parameters – a list of topics with sublists
of keywords
3. Define a function (B) to take the previous output and
generate descriptive texts about detected topics using
detected keywords
4. Define parameters
1. LLM
2. Input parameters
3. Number of topics
4. Number of words per topic
5. Output format
OUTPUT
FUNCTION A
FUNCTION B
(a)
(b)

21. OPEN AI TOPIC DETECTION: UNDERSTANDS CONTEXT BUT LIKES TO TALK IN
SYNONYMS AND IS OCCASIONALLY STUBBORN
1. Specify model (gpt 3.5 turbo, gpt-4…)
2. Define messages:
1. Role of the messenger: system, user, assistant
2. Content of the message – tell it what it should do
Prompt engineering step-by-step*:
a) system message
b) very specific instructions, each given in a separate prompt
c) Provide input data
d) Add contextual information
e) Set temperature (controls randomness of the output)
f) Specify desired output format (json, csv)
DEFINE YOUR ASSISTANT
OUTPUT
Comment Sentiment Topic 1 Topic 2 Topic 3 Topic 4 Topic 5
0 Holy hell. That was a movie-level trailer!<br>... Positive Appreciation Trailer Comparison Quality
1 Animations are so bad seeing the graphics it d... Negative Animation Graphics Quality
2 Pyramids and darkness? Destiny is leaking Neutral Comparison Darkness
3 <b>facepalm</b> the running animation is still... Mixed Animation Appreciation
4 This looks interesting Positive Interest Appearance
.. ... ...
111 oh again STONG women? facepalm Negative Character Face
112 Alles sehr schön. Aber zuerst zusammen die Num... Mixed Potential
113 Let&#39;s goooo Positive Movement
114 Awesome: I&#39;m eager for this game Positive Interest
115 Really looking forward to this game! Positive Anticipation
[116 rows x 2 columns]

22. A HOLISTIC APPROACH TO ANALYZING PLAYERS’S FEEDBACK SENTIMENT AND
TOPIC: HOW WE DO IT
INITIAL MANUAL SENTIMENT AND TOPIC ANALYSIS USING THE QUALITATIVE CODING METHODOLOGY
• Devise a N=1000 sample, and perform a manual sentiment and topic analysis with qualitative coding
• Use it as a training dataset for ML
• Listed topics, subtopics and keywords are used as input instructions for aspect-based topic
analysis, as well as for ML fine tuning
• You get to really acquaint yourself with the data and get to know feeling, the players, the persons
behind them  you get into players’ shoes
ASPECT BASED TOPIC ANALYSIS:
• ML/AI Sentiment analysis (fine tuned using the aspects from previous step)
• Use the aspects and keywords to form keywordssubtopicstopics and fine tune your
tool/algorithm to look for them (you don’t need ML for this, a simple IF THEN ELSE to count
topics and subtopics mentioned based on keywords will do (we do this in Tableau)
• It’s all-encompassing
LLM AND OPEN AI MODELS
• Compare results
• Validate your findings from aspect-based models
• Enhance the previous results – something may pop up that you didn’t encompass with your
manually defined aspects
• Keep fine tuning
GROUNDED THEORY  CONNECTING THE DOTS BY DEVELOPING INSIGHTS AND HYPOTHESES BASED ON DATA
• It aims to develop theories or concepts grounded in the data, allowing patterns and themes to
emerge organically from the data without preconceived notions.

23. YOU CAN BUILD YOUR OWN AUTOMATED TOOLS: THIS IS OUR COMPETITION
ANALYSIS TOOL
YouTube Scrapping
script
NLTK (fine-
tuned)
Aspect-based

24. KEY TAKEAWAYS
USE SENTIMENT AND TOPIC FOR
PLAYER FEEDBACK ANALYSIS
• There’s an abundance of free data
• You will learn a lot about your players
• It WILL help you make your games better
USE THE DATA TO GET TO KNOW
YOUR PLAYERS
• Understand your players and develop
segments
• Understand that you have many target
subgroups
• Understand what are the main drivers for
each of them
• But, also understand you can’t please
everyone
DEVELOP YOUR OWN
FRAMEWORK
• Experiment with all methodologies, don’t discriminate
• Traditional (qualitative coding)
• Automated (ML algorithms, OpenAI)
• Combine them and find out what works for you
DEVELOP INSIGHTS USING
GROUNDED THEORY
MAKE IT A PART OF YOUR DECISION-
MAKING STACK
DO THE MANUAL WORK
• Take the time and do an initial manual analysis on N=1000
reviews or comments
• It will familiarize you with the data
• Use it as a training set
• Go beyond plain data and develop theories with your game
designer, based on the data and knowledge you acquired – not
just with Sentiment and Topic analysis, but with all of your
insights
• Test the hypotheses
Analyze your game life-cycle and find out how and when it can be
used in to make decisions and connect with your decision makers:
• During production
• During soft-launch / demo
• When a game is live

25. THANK
My contact info:
vasiljevic.d.nikola@gmail.com
nikola.vasiljevic@madheadgames.com
Add me on LinkedIn via QR code: