SlideShare a Scribd company logo
1 of 61
1
Data-Driven or Data-Blinded?
The Uses and Abuses of Analytics in Games
2
Data is Awesome
3
Data is a Hot Mess
4
5
So who am I to talk?
Shameless Data GeekKongregate CEO
Kongregate.com Cross-Platform Publisher Coming Summer 2018
6
How Did I Get Here?
so much data
7
Don’t Be Intimidated
You don’t need an
advanced degree in
statistics to get data
analysis right.
You can easily get it
wrong even if you do
have one.
(I’m not saying it doesn’t help, just that it’s not essential.)
8
Get Paranoid
Under an apparently solid surface data is often a steaming pile of shit
9
10
11
A Tale of Two Games
Two games on
Kongregate.com:
● Same genre
● Similar Day 1 and Day 7
retention (Game 1 slightly
higher)
● Similar lifetime buyer %
(Game 2 slightly higher)
● Similar ARPPU* (chart)
ARPU** = Buyer % x ARPPU
So they’ll have similar ARPUs,
too, right?
*ARPPU = Average Revenue per Paying User **ARPU = Average Revenue per Paying User
12
A Different Tale of Two Games
Game 1 ARPU = $2.27
Game 2 ARPU = $0.84
Game 1 has much higher:
● D30
● Transactions/Buyer
● Lifetime ARPPU (chart)
13
You’re a Detective
14
Triangulating Truth
Client Data Server Data Platform Data Benchmark Data
15
You’re also a CSI
16
Building Worlds
Map of the Universe
Limited view of enormous,
changing systems Deploy every method we can
invent
Compare observations over from
different times and angles
17
Classic Blunders
18
Audience Mix
19
Audience Mix
Large Google
Play feature
Feature cohort
US iOS-only UA
spike
UA spike cohort
20
Audience Age
21
Audience Age
22
Averages are Average
Numbers from a dominant subgroup can mask problems (or successes) in smaller subgroups.
23
Players are Not Average
Changes can have drastically different impacts on new and late-game players.
24
Funnel Subgroups
25
Small Sample Sizes
26
Nothing is Normal
IS A LIE!
27
Outliers Rule
28
Cherry Picking
29
Confirmation Bias
30
Axis of Evil
31
Correlation != Causation
32
Everything Correlates with Engagement
33
A/B Tests
34
A/B Test Traps
Sample Sizes Audience Mix
Cherry Picking
Confirmation Bias
Power Distributions
Assignment Issues
35
(Miss) Assignment
Tip: Make sure to split on what you’re actually testing
36
(Huge Miss) Assignment
Your best players ALWAYS show up first. Any
test that doesn’t take that into account is
inherently flawed.
Even if you’re testing to new players only the
people who start playing a game on Saturday
are not the same people who start on
Tuesday.
37
Downstream Impacts
38
Downstream Example
Simpler store increased conversion by 9%!
But drastically reduced ARPPU meant revenue
actually decreased by 11%. Oops.
39
Premature Analysis
Pop-Up Flash Item Sales A/B Test
40
Statistical Significance
41
Statistical Significance
True Mean
5% result
42
Abnormal Testing
A B
$1,000 $500
$100 $100
$40 $40
$20 $20
$10 $10
$5 $5
$5 $5
$5 $5
Avg $148 Avg $86 -42%
43
Medians and Distribution FTW
A B
$1,000 $500
$100 $100
$40 $40
$20 $20
$10 $10
$5 $5
$5 $5
$5 $5
Avg $148 Avg $86
Median $15 Median $15
A B
$1,000 $825
$100 $200
$40 $80
$20 $40
$10 $20
$5 $10
$5 $10
$5 $10
Avg $148 Avg $149
Median $15 Median $30
44
Testing’s Dirty Secret
Most tests don’t show a significant change.
45
Extreme Description Testing
"...a design masterpiece." 5/5 - TouchArcade
2016 Game of the Year - TouchArcade, Gamezebo
2016 Action Game of the Year - Pocket Tactics
DICE 2016 Mobile Game of the Year Nominee
Time's Top 10 Games / Top 50 Apps of 2016
"...a design masterpiece." 5/5 - TouchArcade
2016 Game of the Year - TouchArcade, Gamezebo
2016 Action Game of the Year - Pocket Tactics
DICE 2016 Mobile Game of the Year Nominee
Time's Top 10 Games / Top 50 Apps of 2016
Craft, battle, and quest your way through Crashlands, an outlandish story overflowing with sass!
Become Flux Dabes, a galactic trucker whose latest shipment gets derailed by a chin-strapped alien menace named Hewgodooko, leaving you stranded on an alien
planet. As you hustle to retrieve your packages you’ll become enmeshed in a nefarious plot of world domination, which will require all of your wits and both of your
glutes to overcome. Learn recipes from the local sentient life, make new friends, uncover ancient secrets and deadly bosses, tame everything and build yourself a
home-away-from-home as you learn to thrive on planet Woanope.
▼▼ Key Features ▼▼
● Expansive Crafting System ●
Unlock over 500 craftable items as you explore the world and learn its secrets!
● Self-managing, Infinite Inventory ●
In Crashlands, your inventory is infinite, manages itself, and retrieves your tools when you need them, so you can focus on adventuring, questing, and building. You'll
never have dig through your bag or return to your base to free up inventory space!
● RPG-Style Character Progression ●
Become more powerful through creating ever-more-amazing items! As you grow in power, you can venture to new regions of the world, meet strange characters,
discover new stories, and encounter new and interesting enemies.
● Skill-Based Combat ●
Learn the attacks of the enemies you encounter, and use your skill, agility, and wits to defeat them! You can even augment your fighting prowess with the power of
the dozens of gadgets you can craft. Set your enemies on fire, stun them, slow down time, and more!
● Intuitive Base Building ●
Building a base in Crashlands is so simple it feels like fingerpainting. You can create beautiful, sprawling bases in minutes!
● Tameable Creatures ●
Every creature in Crashlands can become a trusty combat sidekick. Find an egg, incubate it, and hatch your very own adorable or hideous bundle of joy. You can even
craft special items to grow and empower them!
● Huge World... with Huge Problems ●
Four sentient races, three continents, an epic bid for the future of the planet, and you - trapped in the middle, trying to deliver your freakin' packages. Take your time
to dive into the sidestories of the characters you meet or just rush headlong into making that special delivery. With hundreds upon hundreds of quests, there's a lot to
do and discover on planet Woanope!
● Effortless Cloud Saving ●
Just because your battery died or you accidentally dropped your device into a bottomless chasm, doesn't mean your save has to die with it. With BscotchID, you can
easily store and retrieve your save from the cloud, and move it between your devices!
● Controller Support ●
Tired of rubbing your sweaty hands all over your beautiful touchscreen? No problem! We've got support for most mobile-compatible controllers, so you can rub your
sweaty hands on some joysticks instead!
----------------------------------------
Recommended Hardware & OS:
● Android 4.1 or newer
● At least 1GB RAM
● At least 960x540px screen resolution
VS
46
Meaningful No Difference
47
A/B/C(ontext) Testing
On Google Play, Helicopter beat
Girl with Gun by 92%
...but we were using Girl with Gun
because it beat Helicopter by 47% on
Kongregate.com
48
Not Everything is Testable
game
49
Hierarchy of Testing
Advertising
Test everything, all the
time, everywhere.
Tools abound.
Conversion
As much testing of
visual assets as
possible. Tools more
limited.
Initial
Experience
Significant testing
possible but tests will
often have only minor
effects.
Late Game
Tread carefully.
Sample size, audience
expectations and
player fairness
become a challenge
First Weeks
Still possible,
especially around
store, offers, feature
unlocks.
Critical Difficult
50
Game Data Lifecycle
?
Concepting
What could we make?
?
Pre-Production
What will we make?
✕
Production
Make it!
✓
Testing/Beta
What’s working? What’s
not working? Is this
viable? What can we
make better?
✓
Launch
What’s breaking? What’s
changing? What can we
make better?
✓
Live Ops
How can we keep
players engaged? What
can we make better? Did
we break something?
Creation Optimization
All Games Games-as-a-Service
51
What Should We Make?
THE decision, always, for game companies
52
Creative Iteration
Tyrant Unleashed
Launched August 2013
$7 CPIs
Spellstone
Launched September 2015
$5 CPIs
Animation Throwdown
Launched September 2016
$2.50 CPIs
53
Pre-Production Testing
54
Live Disappointment
Source Impressions Clicks CTR Conversion
Castaway Cove Art Test Round 1 53,929 1,157 2.15% n/a
Castaway Cove Art Test Round 2 40,450 1,068 2.64% n/a
Castaway Cove Art Test Round 3 175,762 3,323 1.89% n/a
Castaway Cove Test Markets 3,912,062 43,765 1.11% 22.39% Target = 30%
CPIs for live version of Castaway Cove are okay, but much higher than we’d been targeting
55
Data Maps the Known World
56
But What Don’t We Know?
57
Known World vs Unknown World
58
Supersonic Acrobatic Rocket-
Powered Battle Cars Rocket League
Determination Driven
59
There’s No Right Answer
Familiar Innovative
Established Incumbent
Unknown Newcomer
Most games/developers
fall in the shaded area
60
Good Data, Good Luck
Keep your eyes and minds open
61
Ask Me Anything
Special thanks to Tammy Levy, Drew Levin, Zebulon
Reynolds, Heather Gainer and Butterscotch
Shenanigans for help with data examples!
More great data & talks from the whole team on our blog
https://blog.kongregate.com
Or follow us on Twitter:
@EmilyG
@KongregateDevs
And finally a good explanation of Wilcoxon Rank Sum test
can be found here:
We’re hiring for
analytics!
Open Roles
Director of Analytics
Product Manager
Data Analyst
Locations
Portland, OR
San Francisco, CA
Chicago, IL
Montreal, Canada
https://www.slideshare.net/KrysselMaeCabili/wilcoxon-ranksum-mann-whitney-u-kolmogorovsmirnov-12

More Related Content

What's hot

LiveOps as a Service | Scott Humphries
LiveOps as a Service | Scott HumphriesLiveOps as a Service | Scott Humphries
LiveOps as a Service | Scott HumphriesJessica Tams
 
Personalisation as the key to optimising your game's revenue & LTV.
Personalisation as the key to optimising your game's revenue & LTV.Personalisation as the key to optimising your game's revenue & LTV.
Personalisation as the key to optimising your game's revenue & LTV.GameCamp
 
GDC 2017: Evaluating Monetization Early
GDC 2017: Evaluating Monetization EarlyGDC 2017: Evaluating Monetization Early
GDC 2017: Evaluating Monetization EarlyAdam Telfer
 
Building Games for the Long Term: Pragmatic F2P Guild Design (GDC Europe 2013)
Building Games for the Long Term: Pragmatic F2P Guild Design (GDC Europe 2013)Building Games for the Long Term: Pragmatic F2P Guild Design (GDC Europe 2013)
Building Games for the Long Term: Pragmatic F2P Guild Design (GDC Europe 2013)Kongregate
 
What we learned: Idle Miner Tycoon and Idle Factory Tycoon - Kolibri
What we learned: Idle Miner Tycoon and Idle Factory Tycoon - KolibriWhat we learned: Idle Miner Tycoon and Idle Factory Tycoon - Kolibri
What we learned: Idle Miner Tycoon and Idle Factory Tycoon - KolibriGameAnalytics
 
Transmedia, Gamification, Advergaming
Transmedia, Gamification, AdvergamingTransmedia, Gamification, Advergaming
Transmedia, Gamification, AdvergamingGAMESbrief
 
How to Monetize your F2P Video Game
How to Monetize your F2P Video GameHow to Monetize your F2P Video Game
How to Monetize your F2P Video GameSocial Point
 
Endless frontier summary_global_2016
Endless frontier summary_global_2016Endless frontier summary_global_2016
Endless frontier summary_global_2016Ekkorr
 
Quest for Progress (GDC Europe 2016)
Quest for Progress (GDC Europe 2016)Quest for Progress (GDC Europe 2016)
Quest for Progress (GDC Europe 2016)Anthony Pecorella
 
Kongregate - Maximizing Player Retention and Monetization in Free-to-Play Gam...
Kongregate - Maximizing Player Retention and Monetization in Free-to-Play Gam...Kongregate - Maximizing Player Retention and Monetization in Free-to-Play Gam...
Kongregate - Maximizing Player Retention and Monetization in Free-to-Play Gam...David Piao Chiu
 
R2Games: Maximizing F2P Monetization Presentation from the Montreal Internati...
R2Games: Maximizing F2P Monetization Presentation from the Montreal Internati...R2Games: Maximizing F2P Monetization Presentation from the Montreal Internati...
R2Games: Maximizing F2P Monetization Presentation from the Montreal Internati...David Piao Chiu
 
Shoestring Soft Launch - Low Budget, High Value Launch Strategy for mobile games
Shoestring Soft Launch - Low Budget, High Value Launch Strategy for mobile gamesShoestring Soft Launch - Low Budget, High Value Launch Strategy for mobile games
Shoestring Soft Launch - Low Budget, High Value Launch Strategy for mobile gamesTom Kinniburgh
 
Kongregate - Maximizing Player Retention and Monetization in Free-to-Play Gam...
Kongregate - Maximizing Player Retention and Monetization in Free-to-Play Gam...Kongregate - Maximizing Player Retention and Monetization in Free-to-Play Gam...
Kongregate - Maximizing Player Retention and Monetization in Free-to-Play Gam...David Piao Chiu
 
Maximizing Monetization - Casual Connect SF 2013
Maximizing Monetization - Casual Connect SF 2013Maximizing Monetization - Casual Connect SF 2013
Maximizing Monetization - Casual Connect SF 2013emily_greer
 
R2 Games USA - Maximizing F2P Game Monetization Presentation (Respawn 2016)
R2 Games USA - Maximizing F2P Game Monetization Presentation (Respawn 2016)R2 Games USA - Maximizing F2P Game Monetization Presentation (Respawn 2016)
R2 Games USA - Maximizing F2P Game Monetization Presentation (Respawn 2016)David Piao Chiu
 
Benchmarks and metrics
Benchmarks and metricsBenchmarks and metrics
Benchmarks and metricsGAMESbrief
 
Kongregate - Maximizing Player Retention and Monetization in Free-to-Play Gam...
Kongregate - Maximizing Player Retention and Monetization in Free-to-Play Gam...Kongregate - Maximizing Player Retention and Monetization in Free-to-Play Gam...
Kongregate - Maximizing Player Retention and Monetization in Free-to-Play Gam...David Piao Chiu
 
DavidPChiu Kongregate - Maximizing Player Retention and Monetization in Free-...
DavidPChiu Kongregate - Maximizing Player Retention and Monetization in Free-...DavidPChiu Kongregate - Maximizing Player Retention and Monetization in Free-...
DavidPChiu Kongregate - Maximizing Player Retention and Monetization in Free-...David Piao Chiu
 
GDC 2018 Deconstructor of Fun: Breaking down Top Mobile Games
GDC 2018 Deconstructor of Fun: Breaking down Top Mobile GamesGDC 2018 Deconstructor of Fun: Breaking down Top Mobile Games
GDC 2018 Deconstructor of Fun: Breaking down Top Mobile GamesAdam Telfer
 

What's hot (20)

LiveOps as a Service | Scott Humphries
LiveOps as a Service | Scott HumphriesLiveOps as a Service | Scott Humphries
LiveOps as a Service | Scott Humphries
 
Idle Chatter - GDC 2016
Idle Chatter - GDC 2016Idle Chatter - GDC 2016
Idle Chatter - GDC 2016
 
Personalisation as the key to optimising your game's revenue & LTV.
Personalisation as the key to optimising your game's revenue & LTV.Personalisation as the key to optimising your game's revenue & LTV.
Personalisation as the key to optimising your game's revenue & LTV.
 
GDC 2017: Evaluating Monetization Early
GDC 2017: Evaluating Monetization EarlyGDC 2017: Evaluating Monetization Early
GDC 2017: Evaluating Monetization Early
 
Building Games for the Long Term: Pragmatic F2P Guild Design (GDC Europe 2013)
Building Games for the Long Term: Pragmatic F2P Guild Design (GDC Europe 2013)Building Games for the Long Term: Pragmatic F2P Guild Design (GDC Europe 2013)
Building Games for the Long Term: Pragmatic F2P Guild Design (GDC Europe 2013)
 
What we learned: Idle Miner Tycoon and Idle Factory Tycoon - Kolibri
What we learned: Idle Miner Tycoon and Idle Factory Tycoon - KolibriWhat we learned: Idle Miner Tycoon and Idle Factory Tycoon - Kolibri
What we learned: Idle Miner Tycoon and Idle Factory Tycoon - Kolibri
 
Transmedia, Gamification, Advergaming
Transmedia, Gamification, AdvergamingTransmedia, Gamification, Advergaming
Transmedia, Gamification, Advergaming
 
How to Monetize your F2P Video Game
How to Monetize your F2P Video GameHow to Monetize your F2P Video Game
How to Monetize your F2P Video Game
 
Endless frontier summary_global_2016
Endless frontier summary_global_2016Endless frontier summary_global_2016
Endless frontier summary_global_2016
 
Quest for Progress (GDC Europe 2016)
Quest for Progress (GDC Europe 2016)Quest for Progress (GDC Europe 2016)
Quest for Progress (GDC Europe 2016)
 
Kongregate - Maximizing Player Retention and Monetization in Free-to-Play Gam...
Kongregate - Maximizing Player Retention and Monetization in Free-to-Play Gam...Kongregate - Maximizing Player Retention and Monetization in Free-to-Play Gam...
Kongregate - Maximizing Player Retention and Monetization in Free-to-Play Gam...
 
R2Games: Maximizing F2P Monetization Presentation from the Montreal Internati...
R2Games: Maximizing F2P Monetization Presentation from the Montreal Internati...R2Games: Maximizing F2P Monetization Presentation from the Montreal Internati...
R2Games: Maximizing F2P Monetization Presentation from the Montreal Internati...
 
Shoestring Soft Launch - Low Budget, High Value Launch Strategy for mobile games
Shoestring Soft Launch - Low Budget, High Value Launch Strategy for mobile gamesShoestring Soft Launch - Low Budget, High Value Launch Strategy for mobile games
Shoestring Soft Launch - Low Budget, High Value Launch Strategy for mobile games
 
Kongregate - Maximizing Player Retention and Monetization in Free-to-Play Gam...
Kongregate - Maximizing Player Retention and Monetization in Free-to-Play Gam...Kongregate - Maximizing Player Retention and Monetization in Free-to-Play Gam...
Kongregate - Maximizing Player Retention and Monetization in Free-to-Play Gam...
 
Maximizing Monetization - Casual Connect SF 2013
Maximizing Monetization - Casual Connect SF 2013Maximizing Monetization - Casual Connect SF 2013
Maximizing Monetization - Casual Connect SF 2013
 
R2 Games USA - Maximizing F2P Game Monetization Presentation (Respawn 2016)
R2 Games USA - Maximizing F2P Game Monetization Presentation (Respawn 2016)R2 Games USA - Maximizing F2P Game Monetization Presentation (Respawn 2016)
R2 Games USA - Maximizing F2P Game Monetization Presentation (Respawn 2016)
 
Benchmarks and metrics
Benchmarks and metricsBenchmarks and metrics
Benchmarks and metrics
 
Kongregate - Maximizing Player Retention and Monetization in Free-to-Play Gam...
Kongregate - Maximizing Player Retention and Monetization in Free-to-Play Gam...Kongregate - Maximizing Player Retention and Monetization in Free-to-Play Gam...
Kongregate - Maximizing Player Retention and Monetization in Free-to-Play Gam...
 
DavidPChiu Kongregate - Maximizing Player Retention and Monetization in Free-...
DavidPChiu Kongregate - Maximizing Player Retention and Monetization in Free-...DavidPChiu Kongregate - Maximizing Player Retention and Monetization in Free-...
DavidPChiu Kongregate - Maximizing Player Retention and Monetization in Free-...
 
GDC 2018 Deconstructor of Fun: Breaking down Top Mobile Games
GDC 2018 Deconstructor of Fun: Breaking down Top Mobile GamesGDC 2018 Deconstructor of Fun: Breaking down Top Mobile Games
GDC 2018 Deconstructor of Fun: Breaking down Top Mobile Games
 

Similar to Emily Greer at GDC 2018: Data-Driven or Data-Blinded?

Rogue like-ness-ness! tgc 2018 presentation
Rogue like-ness-ness! tgc 2018 presentationRogue like-ness-ness! tgc 2018 presentation
Rogue like-ness-ness! tgc 2018 presentationAidin Zolghadr
 
GDC Taipei 2013: Creating International Hits from China
GDC Taipei 2013: Creating International Hits from ChinaGDC Taipei 2013: Creating International Hits from China
GDC Taipei 2013: Creating International Hits from ChinaCharlie Moseley
 
Playful design workshop ferrara - uxlx 2014
Playful design workshop   ferrara - uxlx 2014Playful design workshop   ferrara - uxlx 2014
Playful design workshop ferrara - uxlx 2014ferrarajc
 
Lean Live Ops - Free Your Devs (annotated edition) - Joe Raeburn
Lean Live Ops - Free Your Devs (annotated edition) - Joe RaeburnLean Live Ops - Free Your Devs (annotated edition) - Joe Raeburn
Lean Live Ops - Free Your Devs (annotated edition) - Joe RaeburnSimon Hade
 
PicoGame Presentation 2013
PicoGame Presentation 2013PicoGame Presentation 2013
PicoGame Presentation 2013picogame
 
FGS 2011: Keeping Yourself Honest in Game Design (SteamBirds)
FGS 2011: Keeping Yourself Honest in Game Design (SteamBirds)FGS 2011: Keeping Yourself Honest in Game Design (SteamBirds)
FGS 2011: Keeping Yourself Honest in Game Design (SteamBirds)mochimedia
 
F2P Monetization Secrets & Breaking Into China (or Anywhere)
F2P Monetization Secrets & Breaking Into China (or Anywhere)F2P Monetization Secrets & Breaking Into China (or Anywhere)
F2P Monetization Secrets & Breaking Into China (or Anywhere)Charlie Moseley
 
מיזם Magical Trash של תלמידי ביה"ס טשרניחובסקי בחיפה
מיזם Magical Trash של תלמידי ביה"ס טשרניחובסקי בחיפהמיזם Magical Trash של תלמידי ביה"ס טשרניחובסקי בחיפה
מיזם Magical Trash של תלמידי ביה"ס טשרניחובסקי בחיפהGalit Zamler
 
Unlock your creative potential: 7 steps to becoming a game designer
Unlock your creative potential: 7 steps to becoming a game designerUnlock your creative potential: 7 steps to becoming a game designer
Unlock your creative potential: 7 steps to becoming a game designerEthan Levy
 
Making a Game: Start with Constraints
Making a Game: Start with ConstraintsMaking a Game: Start with Constraints
Making a Game: Start with Constraintshacknjill
 
Uncharted lands, or why games are not designed but discovered
Uncharted lands, or why games are not designed but discoveredUncharted lands, or why games are not designed but discovered
Uncharted lands, or why games are not designed but discoveredJakub Stokalski
 
Understanding your game through data
Understanding your game through dataUnderstanding your game through data
Understanding your game through dataDevGAMM Conference
 
Games for Health 2014 design tutorial
Games for Health 2014 design tutorialGames for Health 2014 design tutorial
Games for Health 2014 design tutorialferrarajc
 
Ted talks
Ted talksTed talks
Ted talksFayePRF
 
There Be Dragons: Ten Potential Pitfalls of Gamification
There Be Dragons: Ten Potential Pitfalls of GamificationThere Be Dragons: Ten Potential Pitfalls of Gamification
There Be Dragons: Ten Potential Pitfalls of GamificationSebastian Deterding
 
Game Design for Game Developers by Iain Lobb
Game Design for Game Developers by Iain LobbGame Design for Game Developers by Iain Lobb
Game Design for Game Developers by Iain Lobbmochimedia
 
World Of Warcraft Intro2gd
World Of Warcraft Intro2gdWorld Of Warcraft Intro2gd
World Of Warcraft Intro2gdguest4ef2915
 

Similar to Emily Greer at GDC 2018: Data-Driven or Data-Blinded? (20)

B
BB
B
 
Rogue like-ness-ness! tgc 2018 presentation
Rogue like-ness-ness! tgc 2018 presentationRogue like-ness-ness! tgc 2018 presentation
Rogue like-ness-ness! tgc 2018 presentation
 
GDC Taipei 2013: Creating International Hits from China
GDC Taipei 2013: Creating International Hits from ChinaGDC Taipei 2013: Creating International Hits from China
GDC Taipei 2013: Creating International Hits from China
 
Playful design workshop ferrara - uxlx 2014
Playful design workshop   ferrara - uxlx 2014Playful design workshop   ferrara - uxlx 2014
Playful design workshop ferrara - uxlx 2014
 
When Random is too Random
When Random is too RandomWhen Random is too Random
When Random is too Random
 
Lean Live Ops - Free Your Devs (annotated edition) - Joe Raeburn
Lean Live Ops - Free Your Devs (annotated edition) - Joe RaeburnLean Live Ops - Free Your Devs (annotated edition) - Joe Raeburn
Lean Live Ops - Free Your Devs (annotated edition) - Joe Raeburn
 
PicoGame Presentation 2013
PicoGame Presentation 2013PicoGame Presentation 2013
PicoGame Presentation 2013
 
FGS 2011: Keeping Yourself Honest in Game Design (SteamBirds)
FGS 2011: Keeping Yourself Honest in Game Design (SteamBirds)FGS 2011: Keeping Yourself Honest in Game Design (SteamBirds)
FGS 2011: Keeping Yourself Honest in Game Design (SteamBirds)
 
F2P Monetization Secrets & Breaking Into China (or Anywhere)
F2P Monetization Secrets & Breaking Into China (or Anywhere)F2P Monetization Secrets & Breaking Into China (or Anywhere)
F2P Monetization Secrets & Breaking Into China (or Anywhere)
 
מיזם Magical Trash של תלמידי ביה"ס טשרניחובסקי בחיפה
מיזם Magical Trash של תלמידי ביה"ס טשרניחובסקי בחיפהמיזם Magical Trash של תלמידי ביה"ס טשרניחובסקי בחיפה
מיזם Magical Trash של תלמידי ביה"ס טשרניחובסקי בחיפה
 
Unlock your creative potential: 7 steps to becoming a game designer
Unlock your creative potential: 7 steps to becoming a game designerUnlock your creative potential: 7 steps to becoming a game designer
Unlock your creative potential: 7 steps to becoming a game designer
 
Making a Game: Start with Constraints
Making a Game: Start with ConstraintsMaking a Game: Start with Constraints
Making a Game: Start with Constraints
 
Uncharted lands, or why games are not designed but discovered
Uncharted lands, or why games are not designed but discoveredUncharted lands, or why games are not designed but discovered
Uncharted lands, or why games are not designed but discovered
 
Understanding your game through data
Understanding your game through dataUnderstanding your game through data
Understanding your game through data
 
Games for Health 2014 design tutorial
Games for Health 2014 design tutorialGames for Health 2014 design tutorial
Games for Health 2014 design tutorial
 
Ted talks
Ted talksTed talks
Ted talks
 
There Be Dragons: Ten Potential Pitfalls of Gamification
There Be Dragons: Ten Potential Pitfalls of GamificationThere Be Dragons: Ten Potential Pitfalls of Gamification
There Be Dragons: Ten Potential Pitfalls of Gamification
 
Game Design for Game Developers by Iain Lobb
Game Design for Game Developers by Iain LobbGame Design for Game Developers by Iain Lobb
Game Design for Game Developers by Iain Lobb
 
World Of Warcraft Intro2gd
World Of Warcraft Intro2gdWorld Of Warcraft Intro2gd
World Of Warcraft Intro2gd
 
Video games
Video gamesVideo games
Video games
 

Recently uploaded

Cyber awareness ppt on the recorded data
Cyber awareness ppt on the recorded dataCyber awareness ppt on the recorded data
Cyber awareness ppt on the recorded dataTecnoIncentive
 
Identifying Appropriate Test Statistics Involving Population Mean
Identifying Appropriate Test Statistics Involving Population MeanIdentifying Appropriate Test Statistics Involving Population Mean
Identifying Appropriate Test Statistics Involving Population MeanMYRABACSAFRA2
 
Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Conf42-LLM_Adding Generative AI to Real-Time Streaming PipelinesConf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Conf42-LLM_Adding Generative AI to Real-Time Streaming PipelinesTimothy Spann
 
Semantic Shed - Squashing and Squeezing.pptx
Semantic Shed - Squashing and Squeezing.pptxSemantic Shed - Squashing and Squeezing.pptx
Semantic Shed - Squashing and Squeezing.pptxMike Bennett
 
Data Analysis Project : Targeting the Right Customers, Presentation on Bank M...
Data Analysis Project : Targeting the Right Customers, Presentation on Bank M...Data Analysis Project : Targeting the Right Customers, Presentation on Bank M...
Data Analysis Project : Targeting the Right Customers, Presentation on Bank M...Boston Institute of Analytics
 
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...Boston Institute of Analytics
 
Real-Time AI Streaming - AI Max Princeton
Real-Time AI  Streaming - AI Max PrincetonReal-Time AI  Streaming - AI Max Princeton
Real-Time AI Streaming - AI Max PrincetonTimothy Spann
 
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...Boston Institute of Analytics
 
INTRODUCTION TO Natural language processing
INTRODUCTION TO Natural language processingINTRODUCTION TO Natural language processing
INTRODUCTION TO Natural language processingsocarem879
 
Top 5 Best Data Analytics Courses In Queens
Top 5 Best Data Analytics Courses In QueensTop 5 Best Data Analytics Courses In Queens
Top 5 Best Data Analytics Courses In Queensdataanalyticsqueen03
 
Bank Loan Approval Analysis: A Comprehensive Data Analysis Project
Bank Loan Approval Analysis: A Comprehensive Data Analysis ProjectBank Loan Approval Analysis: A Comprehensive Data Analysis Project
Bank Loan Approval Analysis: A Comprehensive Data Analysis ProjectBoston Institute of Analytics
 
NO1 Certified Black Magic Specialist Expert Amil baba in Lahore Islamabad Raw...
NO1 Certified Black Magic Specialist Expert Amil baba in Lahore Islamabad Raw...NO1 Certified Black Magic Specialist Expert Amil baba in Lahore Islamabad Raw...
NO1 Certified Black Magic Specialist Expert Amil baba in Lahore Islamabad Raw...Amil Baba Dawood bangali
 
Generative AI for Social Good at Open Data Science East 2024
Generative AI for Social Good at Open Data Science East 2024Generative AI for Social Good at Open Data Science East 2024
Generative AI for Social Good at Open Data Science East 2024Colleen Farrelly
 
The Power of Data-Driven Storytelling_ Unveiling the Layers of Insight.pptx
The Power of Data-Driven Storytelling_ Unveiling the Layers of Insight.pptxThe Power of Data-Driven Storytelling_ Unveiling the Layers of Insight.pptx
The Power of Data-Driven Storytelling_ Unveiling the Layers of Insight.pptxTasha Penwell
 
Principles and Practices of Data Visualization
Principles and Practices of Data VisualizationPrinciples and Practices of Data Visualization
Principles and Practices of Data VisualizationKianJazayeri1
 
English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdf
English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdfEnglish-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdf
English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdfblazblazml
 
Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)Cathrine Wilhelmsen
 
modul pembelajaran robotic Workshop _ by Slidesgo.pptx
modul pembelajaran robotic Workshop _ by Slidesgo.pptxmodul pembelajaran robotic Workshop _ by Slidesgo.pptx
modul pembelajaran robotic Workshop _ by Slidesgo.pptxaleedritatuxx
 

Recently uploaded (20)

Cyber awareness ppt on the recorded data
Cyber awareness ppt on the recorded dataCyber awareness ppt on the recorded data
Cyber awareness ppt on the recorded data
 
Identifying Appropriate Test Statistics Involving Population Mean
Identifying Appropriate Test Statistics Involving Population MeanIdentifying Appropriate Test Statistics Involving Population Mean
Identifying Appropriate Test Statistics Involving Population Mean
 
Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Conf42-LLM_Adding Generative AI to Real-Time Streaming PipelinesConf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
 
Semantic Shed - Squashing and Squeezing.pptx
Semantic Shed - Squashing and Squeezing.pptxSemantic Shed - Squashing and Squeezing.pptx
Semantic Shed - Squashing and Squeezing.pptx
 
Data Analysis Project: Stroke Prediction
Data Analysis Project: Stroke PredictionData Analysis Project: Stroke Prediction
Data Analysis Project: Stroke Prediction
 
Data Analysis Project : Targeting the Right Customers, Presentation on Bank M...
Data Analysis Project : Targeting the Right Customers, Presentation on Bank M...Data Analysis Project : Targeting the Right Customers, Presentation on Bank M...
Data Analysis Project : Targeting the Right Customers, Presentation on Bank M...
 
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
 
Real-Time AI Streaming - AI Max Princeton
Real-Time AI  Streaming - AI Max PrincetonReal-Time AI  Streaming - AI Max Princeton
Real-Time AI Streaming - AI Max Princeton
 
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...
 
INTRODUCTION TO Natural language processing
INTRODUCTION TO Natural language processingINTRODUCTION TO Natural language processing
INTRODUCTION TO Natural language processing
 
Top 5 Best Data Analytics Courses In Queens
Top 5 Best Data Analytics Courses In QueensTop 5 Best Data Analytics Courses In Queens
Top 5 Best Data Analytics Courses In Queens
 
Bank Loan Approval Analysis: A Comprehensive Data Analysis Project
Bank Loan Approval Analysis: A Comprehensive Data Analysis ProjectBank Loan Approval Analysis: A Comprehensive Data Analysis Project
Bank Loan Approval Analysis: A Comprehensive Data Analysis Project
 
NO1 Certified Black Magic Specialist Expert Amil baba in Lahore Islamabad Raw...
NO1 Certified Black Magic Specialist Expert Amil baba in Lahore Islamabad Raw...NO1 Certified Black Magic Specialist Expert Amil baba in Lahore Islamabad Raw...
NO1 Certified Black Magic Specialist Expert Amil baba in Lahore Islamabad Raw...
 
Insurance Churn Prediction Data Analysis Project
Insurance Churn Prediction Data Analysis ProjectInsurance Churn Prediction Data Analysis Project
Insurance Churn Prediction Data Analysis Project
 
Generative AI for Social Good at Open Data Science East 2024
Generative AI for Social Good at Open Data Science East 2024Generative AI for Social Good at Open Data Science East 2024
Generative AI for Social Good at Open Data Science East 2024
 
The Power of Data-Driven Storytelling_ Unveiling the Layers of Insight.pptx
The Power of Data-Driven Storytelling_ Unveiling the Layers of Insight.pptxThe Power of Data-Driven Storytelling_ Unveiling the Layers of Insight.pptx
The Power of Data-Driven Storytelling_ Unveiling the Layers of Insight.pptx
 
Principles and Practices of Data Visualization
Principles and Practices of Data VisualizationPrinciples and Practices of Data Visualization
Principles and Practices of Data Visualization
 
English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdf
English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdfEnglish-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdf
English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdf
 
Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)
 
modul pembelajaran robotic Workshop _ by Slidesgo.pptx
modul pembelajaran robotic Workshop _ by Slidesgo.pptxmodul pembelajaran robotic Workshop _ by Slidesgo.pptx
modul pembelajaran robotic Workshop _ by Slidesgo.pptx
 

Emily Greer at GDC 2018: Data-Driven or Data-Blinded?

  • 1. 1 Data-Driven or Data-Blinded? The Uses and Abuses of Analytics in Games
  • 3. 3 Data is a Hot Mess
  • 4. 4
  • 5. 5 So who am I to talk? Shameless Data GeekKongregate CEO Kongregate.com Cross-Platform Publisher Coming Summer 2018
  • 6. 6 How Did I Get Here? so much data
  • 7. 7 Don’t Be Intimidated You don’t need an advanced degree in statistics to get data analysis right. You can easily get it wrong even if you do have one. (I’m not saying it doesn’t help, just that it’s not essential.)
  • 8. 8 Get Paranoid Under an apparently solid surface data is often a steaming pile of shit
  • 9. 9
  • 10. 10
  • 11. 11 A Tale of Two Games Two games on Kongregate.com: ● Same genre ● Similar Day 1 and Day 7 retention (Game 1 slightly higher) ● Similar lifetime buyer % (Game 2 slightly higher) ● Similar ARPPU* (chart) ARPU** = Buyer % x ARPPU So they’ll have similar ARPUs, too, right? *ARPPU = Average Revenue per Paying User **ARPU = Average Revenue per Paying User
  • 12. 12 A Different Tale of Two Games Game 1 ARPU = $2.27 Game 2 ARPU = $0.84 Game 1 has much higher: ● D30 ● Transactions/Buyer ● Lifetime ARPPU (chart)
  • 14. 14 Triangulating Truth Client Data Server Data Platform Data Benchmark Data
  • 16. 16 Building Worlds Map of the Universe Limited view of enormous, changing systems Deploy every method we can invent Compare observations over from different times and angles
  • 19. 19 Audience Mix Large Google Play feature Feature cohort US iOS-only UA spike UA spike cohort
  • 22. 22 Averages are Average Numbers from a dominant subgroup can mask problems (or successes) in smaller subgroups.
  • 23. 23 Players are Not Average Changes can have drastically different impacts on new and late-game players.
  • 34. 34 A/B Test Traps Sample Sizes Audience Mix Cherry Picking Confirmation Bias Power Distributions Assignment Issues
  • 35. 35 (Miss) Assignment Tip: Make sure to split on what you’re actually testing
  • 36. 36 (Huge Miss) Assignment Your best players ALWAYS show up first. Any test that doesn’t take that into account is inherently flawed. Even if you’re testing to new players only the people who start playing a game on Saturday are not the same people who start on Tuesday.
  • 38. 38 Downstream Example Simpler store increased conversion by 9%! But drastically reduced ARPPU meant revenue actually decreased by 11%. Oops.
  • 39. 39 Premature Analysis Pop-Up Flash Item Sales A/B Test
  • 42. 42 Abnormal Testing A B $1,000 $500 $100 $100 $40 $40 $20 $20 $10 $10 $5 $5 $5 $5 $5 $5 Avg $148 Avg $86 -42%
  • 43. 43 Medians and Distribution FTW A B $1,000 $500 $100 $100 $40 $40 $20 $20 $10 $10 $5 $5 $5 $5 $5 $5 Avg $148 Avg $86 Median $15 Median $15 A B $1,000 $825 $100 $200 $40 $80 $20 $40 $10 $20 $5 $10 $5 $10 $5 $10 Avg $148 Avg $149 Median $15 Median $30
  • 44. 44 Testing’s Dirty Secret Most tests don’t show a significant change.
  • 45. 45 Extreme Description Testing "...a design masterpiece." 5/5 - TouchArcade 2016 Game of the Year - TouchArcade, Gamezebo 2016 Action Game of the Year - Pocket Tactics DICE 2016 Mobile Game of the Year Nominee Time's Top 10 Games / Top 50 Apps of 2016 "...a design masterpiece." 5/5 - TouchArcade 2016 Game of the Year - TouchArcade, Gamezebo 2016 Action Game of the Year - Pocket Tactics DICE 2016 Mobile Game of the Year Nominee Time's Top 10 Games / Top 50 Apps of 2016 Craft, battle, and quest your way through Crashlands, an outlandish story overflowing with sass! Become Flux Dabes, a galactic trucker whose latest shipment gets derailed by a chin-strapped alien menace named Hewgodooko, leaving you stranded on an alien planet. As you hustle to retrieve your packages you’ll become enmeshed in a nefarious plot of world domination, which will require all of your wits and both of your glutes to overcome. Learn recipes from the local sentient life, make new friends, uncover ancient secrets and deadly bosses, tame everything and build yourself a home-away-from-home as you learn to thrive on planet Woanope. ▼▼ Key Features ▼▼ ● Expansive Crafting System ● Unlock over 500 craftable items as you explore the world and learn its secrets! ● Self-managing, Infinite Inventory ● In Crashlands, your inventory is infinite, manages itself, and retrieves your tools when you need them, so you can focus on adventuring, questing, and building. You'll never have dig through your bag or return to your base to free up inventory space! ● RPG-Style Character Progression ● Become more powerful through creating ever-more-amazing items! As you grow in power, you can venture to new regions of the world, meet strange characters, discover new stories, and encounter new and interesting enemies. ● Skill-Based Combat ● Learn the attacks of the enemies you encounter, and use your skill, agility, and wits to defeat them! You can even augment your fighting prowess with the power of the dozens of gadgets you can craft. Set your enemies on fire, stun them, slow down time, and more! ● Intuitive Base Building ● Building a base in Crashlands is so simple it feels like fingerpainting. You can create beautiful, sprawling bases in minutes! ● Tameable Creatures ● Every creature in Crashlands can become a trusty combat sidekick. Find an egg, incubate it, and hatch your very own adorable or hideous bundle of joy. You can even craft special items to grow and empower them! ● Huge World... with Huge Problems ● Four sentient races, three continents, an epic bid for the future of the planet, and you - trapped in the middle, trying to deliver your freakin' packages. Take your time to dive into the sidestories of the characters you meet or just rush headlong into making that special delivery. With hundreds upon hundreds of quests, there's a lot to do and discover on planet Woanope! ● Effortless Cloud Saving ● Just because your battery died or you accidentally dropped your device into a bottomless chasm, doesn't mean your save has to die with it. With BscotchID, you can easily store and retrieve your save from the cloud, and move it between your devices! ● Controller Support ● Tired of rubbing your sweaty hands all over your beautiful touchscreen? No problem! We've got support for most mobile-compatible controllers, so you can rub your sweaty hands on some joysticks instead! ---------------------------------------- Recommended Hardware & OS: ● Android 4.1 or newer ● At least 1GB RAM ● At least 960x540px screen resolution VS
  • 47. 47 A/B/C(ontext) Testing On Google Play, Helicopter beat Girl with Gun by 92% ...but we were using Girl with Gun because it beat Helicopter by 47% on Kongregate.com
  • 48. 48 Not Everything is Testable game
  • 49. 49 Hierarchy of Testing Advertising Test everything, all the time, everywhere. Tools abound. Conversion As much testing of visual assets as possible. Tools more limited. Initial Experience Significant testing possible but tests will often have only minor effects. Late Game Tread carefully. Sample size, audience expectations and player fairness become a challenge First Weeks Still possible, especially around store, offers, feature unlocks. Critical Difficult
  • 50. 50 Game Data Lifecycle ? Concepting What could we make? ? Pre-Production What will we make? ✕ Production Make it! ✓ Testing/Beta What’s working? What’s not working? Is this viable? What can we make better? ✓ Launch What’s breaking? What’s changing? What can we make better? ✓ Live Ops How can we keep players engaged? What can we make better? Did we break something? Creation Optimization All Games Games-as-a-Service
  • 51. 51 What Should We Make? THE decision, always, for game companies
  • 52. 52 Creative Iteration Tyrant Unleashed Launched August 2013 $7 CPIs Spellstone Launched September 2015 $5 CPIs Animation Throwdown Launched September 2016 $2.50 CPIs
  • 54. 54 Live Disappointment Source Impressions Clicks CTR Conversion Castaway Cove Art Test Round 1 53,929 1,157 2.15% n/a Castaway Cove Art Test Round 2 40,450 1,068 2.64% n/a Castaway Cove Art Test Round 3 175,762 3,323 1.89% n/a Castaway Cove Test Markets 3,912,062 43,765 1.11% 22.39% Target = 30% CPIs for live version of Castaway Cove are okay, but much higher than we’d been targeting
  • 55. 55 Data Maps the Known World
  • 57. 57 Known World vs Unknown World
  • 58. 58 Supersonic Acrobatic Rocket- Powered Battle Cars Rocket League Determination Driven
  • 59. 59 There’s No Right Answer Familiar Innovative Established Incumbent Unknown Newcomer Most games/developers fall in the shaded area
  • 60. 60 Good Data, Good Luck Keep your eyes and minds open
  • 61. 61 Ask Me Anything Special thanks to Tammy Levy, Drew Levin, Zebulon Reynolds, Heather Gainer and Butterscotch Shenanigans for help with data examples! More great data & talks from the whole team on our blog https://blog.kongregate.com Or follow us on Twitter: @EmilyG @KongregateDevs And finally a good explanation of Wilcoxon Rank Sum test can be found here: We’re hiring for analytics! Open Roles Director of Analytics Product Manager Data Analyst Locations Portland, OR San Francisco, CA Chicago, IL Montreal, Canada https://www.slideshare.net/KrysselMaeCabili/wilcoxon-ranksum-mann-whitney-u-kolmogorovsmirnov-12

Editor's Notes

  1. It’s the cornerstone of many of the biggest businesses in the US, including Google & Amazon, and the backbone of most scientific undertakings.
  2. But data is just a tool, and like almost every tool it has both uses and abuses, not to mention just straight up errors. How many conflicting health studies have you seen?
  3. As a company Kongregate uses a lot of data, and some of you have probably seen talks I’ve given before where I share a lot of that data. But a lot of the time I’ve been unsure whether we’re this ship, charting a clean course to treasure, or this ship, towards disaster. Both have happened! And since I think that’s a pretty common phenomenon, I thought it would be a good talk for GDC.
  4. I love using numbers & testing to understand the world. I still probably spend at least an hour a day poking around dashboards and spreadsheets because it’s so much more fun for me than meetings.
  5. I’m mostly self-taught, majored in Eastern European Studies, not math or econ. Stumbled into direct marketing, specifically catalogs, after college, and fell in love with data. Taught myself SQL because I hated to wait for IT to pull my data, took math & econ classes to understand more theory. After 10 years in catalogs & e-commerce and a near-miss with econ grad school I co-founded Kongregate partly to do something completely different. But it hasn’t turned out to be that different after all. User acquisition in particular is fundamentally similar between catalogs & games.
  6. Part of the reason I’m telling you this is to make my first point: And for an organization to do data right you can’t toss analysis back and forth over a wall to quants. It takes intimate knowledge of a game (and the development) to do good analysis and multiple perspectives and theories are good.
  7. Sometimes it’s immediately obvious. One of the first games we launched on mobile was an endless runner. It wasn’t filtering purchases from jailbroken phones and was showing an average revenue per player of $500. That’s not very plausible and easily caught. But most issues are much more subtle – tracking pixels not firing correctly for a particular game on a particular browser, tutorial steps being completed twice by some players but not by others, clients reporting strange timestamps, etc. For this reason I recommend never relying on any analytic system where you can’t go in and inspect individual records. If you can’t check the detail there are some problems you’ll never find and fix.
  8. Even when your data is accurate it can still be deceiving. This looks like 4 separate pictures photoshopped together to create an appealing color grid, right?
  9. Wrong. So much of data is like these pictures – a set-up that appears straightforwardly to be one thing from one angle, turns out to be completely different from another.
  10. Except of course you know I’m setting you up
  11. People are playing game 1 longer than game 2, and buying repeatedly. But if you just concentrated on daily monetization stats you could miss that entirely.
  12. The witnesses may be lying or confused. The crime scene may have been tampered with. You can’t trust any one piece of evidence but by cross-checking them against each other you can figure out what’s true and false.
  13. Client data (our SDK, Adjust) vs server data App stores Benchmarking against other games Benchmarking deltas
  14. Your goal should be to create a 3-dimensional view of your players and your game. How people move through and interact with different parts. It’s a living, changing system and flat views are not enough.
  15. We tend to think of playerbases as monolithic but really they are aggregations of all sorts of subgroups created by time in game, platform, device, browser, demographics, source – and these subgroups are shifting around. Changes in key KPIs are more often the result of changes in the audience than they are of changes in the game.
  16. These examples show dramatic changes, but more subtle audience changes are happening all the time. Tracking cohorts by date of install/registration is a good way to track metrics independent of certain types of mix issues, but then it’s easy to lose track of events and changes in the game. So as ever, it’s about building a true picture across multiple sources.
  17. 75% ARPDAU decline, then a modest recover to ~50% of previous high.
  18. When you break out ARPDAU by player age you can see that the decline isn’t nearly as dramatic. There’s some decline after a big holiday sale, and then again some as we expanded UA aggressively. But most of it is from fewer el
  19. This is for a collectible card game where the player who goes first has a substantial advantage.
  20. On this chart of player win rates for Tyrant it looks like Mission 24 is very difficult (50% win rate) and mission 25 is easy (95% win rate). It’s sort of true: Mission 25 is relatively easy for those who attempt it. But by deck strength it’s harder than 22, which has a 70% win rate. Mission 25 is easy for the players who are strong enough & skilled enough to beat Mission 24, a selected subgroup of those who attempted 24.
  21. So for the last 10 minutes I’ve been ranting about how important it is too look at audience mix split
  22. The most important metrics (revenue, sessions, battles, etc) in games are all power distributions. Your business (especially in free-to-play games) is driven by outliers, and their presence or absence distorts almost any data you look at.
  23. Your outliers are your best players so it’s a good idea to do individual analysis on them to understand who they are, what drives them, and what they’re most likely to distort. Binary “yes/no” metrics like % buyer, D7 retention, tutorial completion are a lot more stable than averages involving revenue and engagement like ARPPU, $/DAU, Avg Sessions, and can be looked at in much smaller samples.
  24. Sometimes we do it consciously, but more often it’s unconscious. I’ll look at a group of cohorts and the best one is ALWAYS the most memorable. If you’re in test market and hoping to hit 50% the days you hit that number will imprint on your brain that your game has 50% D1 retention, even if the average is 45%.
  25. Cherry Picking’s great and good friend! Part of building a mental mode of your game is having theories about behavior, and if you have a theory you should test it. But it’s really easy to look for the data that supports you theory and miss the data that contradicts it, or even just muddies the picture. [Can I find an example]
  26. How you visualize data has a big impact on how you perceive it.
  27. Ice cream consumption and drowning are correlated, because they’re both more likely to happen in hot weather. But ice cream kills would be a terrible conclusion. We’ve all heard this a 1000 times but we need to keep hearing it like a mantra every day because we all make this same mistake over and over and over. We’re humans, we’re wired to search for causation. It’s our superpower and a curse.
  28. Almost every metric you look at will be positively correlated with engagement because the most engaged users do everything more. Maybe Facebook is increasing engagement. Maybe only engaged players were willing to hit the button and potentially spam their friends.
  29. This is the real way to separate correlation from causation and understand what’s really going on. But it’s not a magic bullet, because nothing is that easy. Testing has real costs in engineering time & overhead, complexity, and divisions/confusions for the players, and the more you’re running the worse that gets.
  30. There’s also a lot of ways to screw up A/B testing even though it seems so foolproof. Most A/B test traps are variations on themes I’ve mentioned but some are new, particularly issues around how people get assigned to tests
  31. For example if you’re A/B testing your store, don’t assign people to the test unless they interact with the store. It’s often easier to split people as they arrive in your game, or some other thing, but a) there’s a chance you would end up with non-equal distribution of interaction with the tested feature and b)any signal from the test group would get lost in the noise of a larger sample.
  32. Tests can have unintended consequences, you should look at additional metrics beyond the one being tested to make sure that you get the full picture. Commercial A/B products often make you choose one metric for a test to prevent you from fishing for the good result to decide the test on. I think it’s more important to understand the full effects of the change that you made (though fishing is bad, too.)
  33. Early results tend to be both volatile and fascinating – differences are exaggerated or totally change direction. People tend to remember the early, interesting results rather than the actual results. People also often want to end the test early if they see a big swing, which is a bad idea. So I recommend that you don’t look at early test results except to make sure the test isn’t totally broken. How big should your test sample be? In my opinion the bigger the better.
  34. When people talk about A/B tests you’ll often hear things like “we’ve got a statistically significant 5% lift”! And most people hear that and think that means that the lift is definitely 5%. But that’s not how statistical significance tests work.
  35. Statistical significance tests assume that there is some true difference in lift, and that if you run the same test repeatedly there will be a bell curve distribution of results, with the true lift as the average. Your 5% result could be right on the mean, or it could be an outlier on either end. If it’s statistically significant then the chance is low (usually 5% or less) that there’s no lift at all. But the true lift could be 1% or 10%. Conversely if you do a test that doesn’t show a lift, or doesn’t pass the significance test for a small lift that doesn’t mean there ISN’T a lift. This is why I like to run A/B tests with larger sample sizes. It’s like running the test again and averaging the results. It’s possible you’d get two outlier results in the same direction, but becomes less and less likely, and more likely that your test results represent the true mean.
  36. Often 70-80% of a free-to-play game’s revenue will come from a small % of buyers who spend more than $500.
  37. Large sample sizes help here, too.
  38. This can be really frustrating, even demoralizing for a team. When you’re going through the effort to make and test changes, you want them to mean something! You want to make progress. And then you get another non-result on a test. But finding out what doesn’t matter can actually be really powerful.
  39. Here’s an extreme example of this from the team at Butterscotch Shenanigans, who made the game Crashlands. They had written up an elaborate, detailed description and decided to test how much impact it had using Google’s store testing system on Android against the most extreme possible variant, no description at all. Just the accolades the game has received.
  40. They were kind enough to share the results and after 4 full months the test shows absolutely no difference, and that actually tells you a lot: specifically that the description has very little impact, and this is consistent with the testing we’ve done on our own games, as well. Time and resources are a constraint for virtually everybody, and knowing what is not important allows you to concentrate more on things that do matter. We used to argue endlessly over game names, but after doing test after test and not seeing much difference we’re all much more relaxed about it.
  41. But it’s important not to extrapolate too much. Just because you get a particular
  42. Specifically late game content is often very difficult to test, or any testing on late game players. Daniel Cook from Spryfox tweeted this recently. He was talking about YouTube and algorithms, but I think it helps frame some of the limitations of testing. As a player plays a game, the game is shaping their expectations and experience, and training them to behave in certain ways. So the same player might react very differently based on how long they had been playing the game. And when engaged players start talking to each other in chat and forums they affect each other, too. Plus you run into small sample sizes with lots of outliers and other fun problems I’ve already talked about.
  43. Tyrant successful on a small core audience, but difficult to market
  44. CPIs for live version of Castaway Cove are okay, but much higher than we’d been targeting. Lots of ways we probably went wrong
  45. So far data has helped us iterate on existing games, pointing us in the direction that helped get us from Tyrant to Animation Throwdown. But in
  46. But what we don’t know is as important as what we do know
  47. Data is alway going to tell you to make an existing successful game, but better. It’s not going to tell you to make a game unlike anything people have played before
  48. But what we don’t know is as important as what we do know
  49. Detectives, CSIs, Astronomers, Cartographers, Explorers: