Emily Greer at GDC 2018: Data-Driven or Data-Blinded?

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

It’s the cornerstone of many of the biggest businesses in the US, including
Google & Amazon, and the backbone of most scie...

But data is just a tool, and like almost every tool it has both uses and abuses,
not to mention just straight up errors. H...

As a company Kongregate uses a lot of data, and some of you have probably
seen talks I’ve given before where I share a lot...

Advertisement

I love using numbers & testing to understand the world. I still probably spend
at least an hour a day poking around dashbo...

I’m mostly self-taught, majored in Eastern European Studies, not math or
econ. Stumbled into direct marketing, specificall...

Part of the reason I’m telling you this is to make my first point:
And for an organization to do data right you can’t toss...

Sometimes it’s immediately obvious. One of the first games we launched on
mobile was an endless runner. It wasn’t filterin...

Even when your data is accurate it can still be deceiving. This looks like 4
separate pictures photoshopped together to cr...

Advertisement

Wrong.
So much of data is like these pictures – a set-up that appears
straightforwardly to be one thing from one angle, tu...

Except of course you know I’m setting you up

People are playing game 1 longer than game 2, and buying repeatedly. But if
you just concentrated on daily monetization st...

The witnesses may be lying or confused. The crime scene may have been
tampered with.
You can’t trust any one piece of evid...

Client data (our SDK, Adjust) vs server data
App stores
Benchmarking against other games
Benchmarking deltas

Advertisement

Your goal should be to create a 3-dimensional view of your players and your
game. How people move through and interact wit...

We tend to think of playerbases as monolithic but really they are
aggregations of all sorts of subgroups created by time i...

These examples show dramatic changes, but more subtle audience changes
are happening all the time. Tracking cohorts by dat...

Advertisement

75% ARPDAU decline, then a modest recover to ~50% of
previous high.

When you break out ARPDAU by player age you can see that
the decline isn’t nearly as dramatic. There’s some decline after ...

This is for a collectible card game where the player who goes first has a
substantial advantage.

On this chart of player win rates for Tyrant it looks like Mission 24 is very
difficult (50% win rate) and mission 25 is e...

Advertisement

So for the last 10 minutes I’ve been ranting about how important it is too look
at audience mix split

The most important metrics (revenue, sessions, battles, etc) in games are all
power distributions. Your business (especial...

Your outliers are your best players so it’s a good idea to do individual
analysis on them to understand who they are, what...

Sometimes we do it consciously, but more often it’s unconscious. I’ll look at a
group of cohorts and the best one is ALWAY...

Cherry Picking’s great and good friend!
Part of building a mental mode of your game is having theories about
behavior, and...

Advertisement

How you visualize data has a big impact on how you perceive it.

Ice cream consumption and drowning are correlated, because they’re both
more likely to happen in hot weather. But ice crea...

Almost every metric you look at will be positively correlated with engagement
because the most engaged users do everything...

This is the real way to separate correlation from causation and understand
what’s really going on. But it’s not a magic bu...

There’s also a lot of ways to screw up A/B testing even though it seems so
foolproof. Most A/B test traps are variations o...

Advertisement

For example if you’re A/B testing your store, don’t assign people to the test
unless they interact with the store. It’s of...

Tests can have unintended consequences, you should look at additional
metrics beyond the one being tested to make sure tha...

Early results tend to be both volatile and fascinating – differences are
exaggerated or totally change direction. People t...

Advertisement

When people talk about A/B tests you’ll often hear things like “we’ve got a
statistically significant 5% lift”! And most p...

Statistical significance tests assume that there is some true difference in lift,
and that if you run the same test repeat...

Often 70-80% of a free-to-play game’s revenue will come from a small % of
buyers who spend more than $500.

Large sample sizes help here, too.

This can be really frustrating, even demoralizing for a team. When you’re
going through the effort to make and test change...

Advertisement

Here’s an extreme example of this from the team at Butterscotch
Shenanigans, who made the game Crashlands. They had writte...

They were kind enough to share the results and after 4 full months the test
shows absolutely no difference, and that actua...

But it’s important not to extrapolate too much. Just because you get a
particular

Specifically late game content is often very difficult to test, or any testing on
late game players.
Daniel Cook from Spry...

Advertisement

Tyrant successful on a small core audience, but difficult to market

CPIs for live version of Castaway Cove are okay, but much higher than we’d
been targeting. Lots of ways we probably went w...

So far data has helped us iterate on existing games, pointing us in the
direction that helped get us from Tyrant to Animat...

But what we don’t know is as important as what we do know

Data is alway going to tell you to make an existing successful game, but
better. It’s not going to tell you to make a game...

But what we don’t know is as important as what we do know

Detectives, CSIs, Astronomers, Cartographers, Explorers:

Emily Greer at GDC 2018: Data-Driven or Data-Blinded?

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 1057327 views
Storytelling For The Web: Integrate... by Chiara Aliotta 973877 views
Artificial Intelligence, Data and C... by OECD Directorate ... 910242 views
How to Leverage AI to Boost Employe... by SocialHRCamp 385546 views
2024 State of Marketing Report – by... by Marius Sescu 218326 views
Everything You Need To Know About C... by Expeed Software 235577 views

View on Slideshare

1 of 61 Ad

View on Slideshare

1 of 61 Ad