As the market is maturing and we make games for a more granular audience, user research has become a more prominent part of F2P game development. How do we make the most of user research at the very early phase of development, when we know little about our audience and the shape of the game we want to make? How to get reliable insights that can lead to critical decisions on the direction of the game, at an earlier stage than soft launch? In my talk, I will share the takeaways on the new game I am developing: how user research revealed that we were going in the wrong direction at very early stage, and which methods we used (playtestcloud, f2f live interviews, beta tester groups...) that any developer can decide to use tomorrow, at a low cost.

1. Rovio © 2018 Confidential
Testing with players in early prototyping
Sophie Vo, Executive Producer
Rovio Entertainment, Finland

2. Rovio © 2018 Confidential
Sophie

3. Rovio © 2018 Confidential
My team

4. Rovio © 2018 Confidential

5. Rovio © 2018 Confidential
*Source: Deconstructor of Fun

6. Rovio © 2018 Confidential

7. Rovio © 2018 Confidential

8. Rovio © 2018 Confidential

9. Rovio © 2018 Confidential
• We took a new and unproven approach on that genre.
• We started that game based on many assumptions
=> To a light mid-core direction
How we started...

10. Rovio © 2018 Confidential
concept
D1-D3
core
D7-30
metagame
D30+
live ops
Our incremental approach

11. Rovio © 2018 Confidential
• WHAT: A fake app store screen
• WHO: puzzle audience, 480 ppl interviewed
• HOW: Online survey
• WHY: Learn how our audience perceives our concept
Test plan 1: Concept

12. Rovio © 2018 Confidential
Method 1
Online survey

13. Rovio © 2018 Confidential
Results
Easy / Hard to understand

14. Rovio © 2018 Confidential
Results

15. Rovio © 2018 Confidential
● Easy and fast to setup
● Audience targeting
● Reasonable costs
● Informative results, to help
design decisions
Pros & Cons - Online surveys
● Doesn’t guarantee about anything :)

16. Rovio © 2018 Confidential
concept
D1-D3
core
D7-30
metagame
live ops
Next step: testing the core
D30+

17. Rovio © 2018 Confidential
• WHAT: A minimum playable
• WHO: Puzzle audience (5 testers). Also internal.
• HOW: Playtest Cloud
• WHY: See how players interact with the new core.
Test plan 2: Core game

18. Rovio © 2018 Confidential
Method 2
Unmoderated remote playtesting

19. Rovio © 2018 Confidential
Playtest Cloud results

20. Rovio © 2018 Confidential
“I really like the line feature. It gives you planning abilities.
Good for strategy players. Once I figured out that’s what they
were I was able to target in advance to hit those.” “I enjoy the lazer being on all the time -- helps
visualize my shot and prevents obviously bad shots.”
Players feedback
“It definitely removes some of the pain when you get a bad
ball and opens up some interesting opportunities that a
match may not necessarily be your best move. So far I like the
change :)”

21. Rovio © 2018 Confidential
● Easy and fast to setup
● Efficient player targeting
● Reasonable costs
● Fast and transparent results
Pros & Cons - Remote playtest
● Not 100% natural test conditions
● Small sample
● Mostly point to usability issues
● Harder to track retention...

22. Rovio © 2018 Confidential
Actionables

23. Rovio © 2018 Confidential

24. Rovio © 2018 Confidential
concept
D1-D3
core
D7-30
metagame
live ops
Next step: testing the game loop
D30+

25. Rovio © 2018 Confidential
• WHAT: A minimum engageable
• WHO: 6 PTC testers + 6 BETA testers
• HOW: Playtest Cloud, BETA testers, F2F interviews.
• WHY: See how players enjoy the core + metagame loop.
Test plan 3: core + metagame

26. Rovio © 2018 Confidential
Playtest Cloud results

27. Rovio © 2018 Confidential

28. Rovio © 2018 Confidential
Method 3
Moderated testing with f2f interviews

29. Rovio © 2018 Confidential
“I miss the puzzle challenge”
“I want to replay my level and master it”
“How do I finish the level? It looks endless. I want to
feel I complete something”
“Nothing new here”
f2f interviews feedbacks

30. Rovio © 2018 Confidential
“How do I finish the level? It looks endless. I want to
feel I complete something”
BETA testers

31. Rovio © 2018 Confidential
● Interactions based on observed
behavior, emotions and feedback
● Deeper understanding on
gameplay/UI issues
● Possibility of conversation with the tester
Pros & Cons - f2f itw
● Time and setup requirement
● Requires a trained “unbiased” moderator
● Artificial environment setup for the tester

32. Rovio © 2018 Confidential
Actionables?

33. Rovio © 2018 Confidential
Conclusion
• We took a too mid-core direction. Not fun for that audience.
• We had to refocus on the essence of a puzzle game.
=> To a casual++ direction

34. Rovio © 2018 Confidential
concept
D1-D3
core
D7-30
metagame
D30+
live ops
Back to the start again

35. Rovio © 2018 Confidential
My lessons?

36. Rovio © 2018 Confidential
Test... anything!

37. Rovio © 2018 Confidential
Rely on different methods
Ideation Design/Build Release
Ethnography
Focus groups
Interviews
Personas
Surveys
Prototype
interviews/
playtesting
Playtesting
A/B testing
In-game Surveys
Diary studies
Expert Review

38. Rovio © 2018 Confidential

39. Rovio © 2018 Confidential
Thank you!
My contact: sophievo.com