The document discusses different types of testing that can be done prior to launching a mobile game globally. It describes internal team playtests, in-person playtests with outsiders, remote playtests on services like UserTesting, friends and family tests, closed beta tests with broader groups of players, and soft launches in select countries. The key is to use multiple testing methods in order to gather both qualitative and quantitative feedback from varied player demographics to maximize the chances of success upon global launch.
Core Games, Real Numbers: Asian & Western Gamesemily_greer
Kongregate is a distributor of more than 200 virtual-goods games developed by developers small and large, Eastern and Western, casual and hardcore. As such, it has a unique perspective on what types of mechanics and characteristics of games have the most success monetizing. This talk will focus on similarities and differences in performance between Asian & Western games, look at site-wide trends and dig into specific game metrics and mechanics to understand what makes games succeed & fail with Western audiences. This talk was given at GDC China 2012
GDC 2014 Core Games, Real Numbers: Going Cross-Platformemily_greer
Kongregate has been sharing great data on web games for years - now they're launching mobile games and are ready to start sharing iOS and Android numbers as well. Learn how monetization differs by platform, country, and device, what type of CPIs to expect for different kinds of games, and what's most important in making your free-to-play mobile game a success.
Don't Call Them Whales: Free-to-Play Spenders & Virtual Value GDC 2015emily_greer
Individual large spenders -- so called "whales"-- form the foundation of the free-to-play business model that has overtaken the game industry in the last few years. This talk from GDC 2015 examines this phenomenon by looking at demographics, play and buying patterns for big spenders on the Kongregate web platform and mobile games, how expectations of game spending and value are formed, and how attitudes toward spenders and spending shape the whole industry -- not just free-to-play.
Breaking Labels: Core, Casual, and Other Misconceptions -- Casual Connect Eur...emily_greer
As video games become more and more mainstream the industry labels of "core games" and "casual games" become less and less useful. I'll look at the forces that drove the categories, what's breaking them down, and how it changes how we should think about games.
In the last five years, data analysis, A/B testing and predictive modeling have transitioned from an afterthought to a given in the game industry. Data can be invaluable in understanding the player and making decisions, but it can just as easily lead us astray. This talk exposes common mistakes and pitfalls, both technical and emotional, as well as provide practical guidance on how to improve the rigorousness of your tests and the quality of your data.
Emily Greer at GDC 2018: Data-Driven or Data-Blinded?Kongregate
In the last decade of data analysis, A/B testing and predictive modeling have transitioned from an afterthought to a given in the game industry. Data can be invaluable in understanding the player and making decisions, but it can just as easily lead the industry astray, or worse, narrow the way the industry thinks. When should you be driven by data, and when should you let your imagination roam free? This session will expose common mistakes and pitfalls, both technical and emotional, as well as provide practical guidance on how to improve the rigorousness of your tests and the quality of your data, and how to make sure you don't lose the forest for the trees.
Core Games, Real Numbers: Asian & Western Gamesemily_greer
Kongregate is a distributor of more than 200 virtual-goods games developed by developers small and large, Eastern and Western, casual and hardcore. As such, it has a unique perspective on what types of mechanics and characteristics of games have the most success monetizing. This talk will focus on similarities and differences in performance between Asian & Western games, look at site-wide trends and dig into specific game metrics and mechanics to understand what makes games succeed & fail with Western audiences. This talk was given at GDC China 2012
GDC 2014 Core Games, Real Numbers: Going Cross-Platformemily_greer
Kongregate has been sharing great data on web games for years - now they're launching mobile games and are ready to start sharing iOS and Android numbers as well. Learn how monetization differs by platform, country, and device, what type of CPIs to expect for different kinds of games, and what's most important in making your free-to-play mobile game a success.
Don't Call Them Whales: Free-to-Play Spenders & Virtual Value GDC 2015emily_greer
Individual large spenders -- so called "whales"-- form the foundation of the free-to-play business model that has overtaken the game industry in the last few years. This talk from GDC 2015 examines this phenomenon by looking at demographics, play and buying patterns for big spenders on the Kongregate web platform and mobile games, how expectations of game spending and value are formed, and how attitudes toward spenders and spending shape the whole industry -- not just free-to-play.
Breaking Labels: Core, Casual, and Other Misconceptions -- Casual Connect Eur...emily_greer
As video games become more and more mainstream the industry labels of "core games" and "casual games" become less and less useful. I'll look at the forces that drove the categories, what's breaking them down, and how it changes how we should think about games.
In the last five years, data analysis, A/B testing and predictive modeling have transitioned from an afterthought to a given in the game industry. Data can be invaluable in understanding the player and making decisions, but it can just as easily lead us astray. This talk exposes common mistakes and pitfalls, both technical and emotional, as well as provide practical guidance on how to improve the rigorousness of your tests and the quality of your data.
Emily Greer at GDC 2018: Data-Driven or Data-Blinded?Kongregate
In the last decade of data analysis, A/B testing and predictive modeling have transitioned from an afterthought to a given in the game industry. Data can be invaluable in understanding the player and making decisions, but it can just as easily lead the industry astray, or worse, narrow the way the industry thinks. When should you be driven by data, and when should you let your imagination roam free? This session will expose common mistakes and pitfalls, both technical and emotional, as well as provide practical guidance on how to improve the rigorousness of your tests and the quality of your data, and how to make sure you don't lose the forest for the trees.
A high-speed run through some of the most important and interesting patterns and advice that we've discovered about our games on Kongregate. This talk was given by Anthony Pecorella at Casual Connect Kyiv.
GDC Talk - Nature vs Nurture: Unpacking Player Spending in F2P GamesTamara (Tammy) Levy
Have players committed to spending money in free-to-play games before they even install the game? Are you wasting time trying to get more people to spend money in your game? What monetization metrics should you be focusing on improving instead? In this talk, Tammy uses several case studies from live Kongregate games to explore how and why some player behaviors are easier and more rewarding to change, while others have a nearly unbreakable ceiling.
GDC Talk: Lifetime Value: The long tail of Mid-Core gamesTamara (Tammy) Levy
Are you maximizing the life in lifetime value? Using genre-specific predictive models developed from the Kongregate portfolio, we demonstrate how and why post-D30 retention can be key to increase your revenue. Through case studies, you'll learn strategies to boost both early and late revenue, as well as game core loop modifications to drive better player investment in the long run.
Josh Larson’s Talk at White Nights Prague '18Kongregate
During a special Valentine’s Day session at White Nights Prague ‘18, Josh presented “Mining the Web for Cross-Platform Gems." The talk explored how Kongregate has used its web platform to discover games that have potential for cross-platform success on mobile, PC and console. By looking at hits like AdVenture Capitalist, Realm Grinder and Bit Heroes, the presentation extracts the characteristics and KPIs for cross-platform potential.
DavidPChiu Kongregate - Maximizing Player Retention and Monetization in Free-...David Piao Chiu
Presentation from the Brazil Independent Games (BIG) Festival 2014 on Maximizing Player Retention and Monetization in Free-to-Play Games with a Comparative Analysis of Asian & Western F2P Games
Talk presented by Daniel James at GDC Worlds in Motion Summit 2009.3.23. Daniel is CEO of Three Rings, creators of Puzzle Pirates and Whirled. This talk is an introduction to metrics for MMOs and contains real data from Puzzle Pirates and Whirled.
Kongregate - Maximizing Player Retention and Monetization in Free-to-Play Gam...David Piao Chiu
Tokyo Game Show 2013 JETRO presentation:
Kongregate - Maximizing Player Retention and Monetization in Free-to-Play Games: Comparative Stats for Asian & Western Games
Adrian Crook's speech at GDC 2008, entitled "The Power of Free To Play". Focuses on free to play revenue models in games, design tips, growth challenges and market trends. More info at www.FreeToPlay.biz
For more information, visit: www.adriancrook.com
There are many reasons to opt for the Free-To-Play monetization model. At the forefront is that Free-to-Play games are generating more money than premium sales at nearly a 10-1 margin.
It is not just the big AAA productions that can succeed. With an average of 15,000,000 Steam Players online at any given time and 90 million console gamers in the US alone, strictly considering a mobile publishing avenue severely limits your audience reach. In addition, plenty of these platforms are doing their best to embrace the F2P evolution in gaming.
A+C has 10 years and over 130 clients worth of freemium experience. From increased monetization to retention and virality, we have delivered for clients such as LEGO, Pokemon, Electronic Arts, Zynga and many more.
This is a presentation we gave at Gameplay Space in Montreal recently, to a room full of indie developers.
Contact us at www.adriancrook.com today.
Playtika Bets on Big Data Analytics to Deliver Captivating Social Gaming Expe...Dana Gardner
Transcript of a sponsored discussion on how Playtika is using data science going and an architectural approach to conquer some of these hurdles around volume, velocity, and variety of data.
A high-speed run through some of the most important and interesting patterns and advice that we've discovered about our games on Kongregate. This talk was given by Anthony Pecorella at Casual Connect Kyiv.
GDC Talk - Nature vs Nurture: Unpacking Player Spending in F2P GamesTamara (Tammy) Levy
Have players committed to spending money in free-to-play games before they even install the game? Are you wasting time trying to get more people to spend money in your game? What monetization metrics should you be focusing on improving instead? In this talk, Tammy uses several case studies from live Kongregate games to explore how and why some player behaviors are easier and more rewarding to change, while others have a nearly unbreakable ceiling.
GDC Talk: Lifetime Value: The long tail of Mid-Core gamesTamara (Tammy) Levy
Are you maximizing the life in lifetime value? Using genre-specific predictive models developed from the Kongregate portfolio, we demonstrate how and why post-D30 retention can be key to increase your revenue. Through case studies, you'll learn strategies to boost both early and late revenue, as well as game core loop modifications to drive better player investment in the long run.
Josh Larson’s Talk at White Nights Prague '18Kongregate
During a special Valentine’s Day session at White Nights Prague ‘18, Josh presented “Mining the Web for Cross-Platform Gems." The talk explored how Kongregate has used its web platform to discover games that have potential for cross-platform success on mobile, PC and console. By looking at hits like AdVenture Capitalist, Realm Grinder and Bit Heroes, the presentation extracts the characteristics and KPIs for cross-platform potential.
DavidPChiu Kongregate - Maximizing Player Retention and Monetization in Free-...David Piao Chiu
Presentation from the Brazil Independent Games (BIG) Festival 2014 on Maximizing Player Retention and Monetization in Free-to-Play Games with a Comparative Analysis of Asian & Western F2P Games
Talk presented by Daniel James at GDC Worlds in Motion Summit 2009.3.23. Daniel is CEO of Three Rings, creators of Puzzle Pirates and Whirled. This talk is an introduction to metrics for MMOs and contains real data from Puzzle Pirates and Whirled.
Kongregate - Maximizing Player Retention and Monetization in Free-to-Play Gam...David Piao Chiu
Tokyo Game Show 2013 JETRO presentation:
Kongregate - Maximizing Player Retention and Monetization in Free-to-Play Games: Comparative Stats for Asian & Western Games
Adrian Crook's speech at GDC 2008, entitled "The Power of Free To Play". Focuses on free to play revenue models in games, design tips, growth challenges and market trends. More info at www.FreeToPlay.biz
For more information, visit: www.adriancrook.com
There are many reasons to opt for the Free-To-Play monetization model. At the forefront is that Free-to-Play games are generating more money than premium sales at nearly a 10-1 margin.
It is not just the big AAA productions that can succeed. With an average of 15,000,000 Steam Players online at any given time and 90 million console gamers in the US alone, strictly considering a mobile publishing avenue severely limits your audience reach. In addition, plenty of these platforms are doing their best to embrace the F2P evolution in gaming.
A+C has 10 years and over 130 clients worth of freemium experience. From increased monetization to retention and virality, we have delivered for clients such as LEGO, Pokemon, Electronic Arts, Zynga and many more.
This is a presentation we gave at Gameplay Space in Montreal recently, to a room full of indie developers.
Contact us at www.adriancrook.com today.
Playtika Bets on Big Data Analytics to Deliver Captivating Social Gaming Expe...Dana Gardner
Transcript of a sponsored discussion on how Playtika is using data science going and an architectural approach to conquer some of these hurdles around volume, velocity, and variety of data.
This research is conducted with the companies Blend, Sioux, ESPM and with the partnership of ACIGAMES (Industrial and Commercial Association for games in Brazil) on the gaming market in Mobile and Tablets for 2013 in Brazil
This presentation covers tips and tricks for game developers and covers a little bit of everything for both work for hire developers, as well as those creating their own IP.
Massively multiplayer data challenges in mobile game analyticsJak Marshall
I gave this talk at the Institute of Mathematics and its Applications Conference for Early Career Mathematicians at Warwick University on November 2016.
The talk details my experiences of working in the mobile games industry as a data scientist, drawing on my experience working at both Sega Hardlight and Exient Malta studios.
It was the final talk of the day and it was well received.
Massively multiplayer data challenges in mobile game analyticsJak Marshall
I gave this talk at the IMA Early Career Mathematicians' Conference at Warwick University on the 5th November 2016.
It was the final talk of the day and it was well received by the professional mathematicians and students that were in attendance.
This presentation is a summary of ten themes that made an impact on me at SXSW Interactive in 2011.
The contents are broken into two sections:
1). Five themes relevant to marketers
2). Five themes that aren't so relevant but rock.
Feel free to contact me.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Let's dive deeper into the world of ODC! Ricardo Alves (OutSystems) will join us to tell all about the new Data Fabric. After that, Sezen de Bruijn (OutSystems) will get into the details on how to best design a sturdy architecture within ODC.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
"Impact of front-end architecture on development cost", Viktor TurskyiFwdays
I have heard many times that architecture is not important for the front-end. Also, many times I have seen how developers implement features on the front-end just following the standard rules for a framework and think that this is enough to successfully launch the project, and then the project fails. How to prevent this and what approach to choose? I have launched dozens of complex projects and during the talk we will analyze which approaches have worked for me and which have not.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
2. Back a few years ago when social games were exploding on facebook the
conventional wisdom was that you wanted to release your minimum viable product as
quickly as you could, and iterate on it in the wild with real data from players. But that
only made sense in a world where if you “wasted” your early traffic on a poor game it
was relatively easy to get more. Mobile is the opposite: traffic is always at a premium
and a strong global launch has become crucial to success. It’s the best chance to get
substantial features from the platforms, to get noticed in the new charts, and
potentially get picked up by recommendation algorithms.
Ideally we’d all be like Blizzard & Supercell and be able to polish and test games
internally for years, but there are all sorts of pressures and costs pushing games out
the door. For companies with <$1B in annual profit it’s important to be realistic about
what, why, and how to test to maximize our chance at success.
3. We started Kongregate back in 2006 as an open platform for browser games, a little
like YouTube for games: anyone can upload, we then add chat, comments, forums,
achievements, and a lot of other social features that make the site a whole game
itself. In 10 years more than 100,000 games have been put on our site, covering
pretty much every genre imaginable. A fairly wide array of games are popular, from
casual puzzles and launch games to MMOs and collectible card games. Overall our
audience trends male, with heavy overlap with console and Steam.
4. Four years ago we started publishing third party games on mobile, and have
launched more than 20 games in the last 3 years. Like on Kongregate itself we
publish a fairly broad range of games, from more niche, high monetizing RPGs and
CCGs to single-player games with mostly ad monetization. To give you a feel of the
range here are the games we published in 2015.
5. And like everything in life they have different strengths and weaknesses. To use them
properly it’s important to understand what those are, so I’m going to spend a bit of
time going through the different types.
One note: I’m not going to be talking much about defect and bug-oriented QA testing,
which is not to say it’s not important. It’s very important, and you should do it, ideally
with dedicated in-house resources supplemented by 3rd party resources. But I only
have an hour so I have to skimp on some topics.
6. By Team Playtests I mean both the testing you do naturally as you add on features to
the game, and also scheduled team sessions to look at the game more broadly.
Game is available, everyone’s getting paid to do this
You don’t play like a player, you know how things are supposed to work. And you
know games too well – every convention is obvious to you. And since you’re testing to
make sure features are working you play the game in totally unnatural ways “I’m
jumping right to x” “I’m pushing all the buttons” “I’m going to play with an OP account
to breeze through”
7. By in-person playtests I mean getting a 3rd party unrelated to game development to
play the game. It could be as informal as handing somebody a device out in the wild,
or could be an organized, in-office thing where you hire people to come in and play
the game.
• They’re a pain-in-the-ass to arrange whether you’re bringing strangers from
Craigslist into your office or harassing them at a coffee shop. And they take a lot
of skill to run & analyze well – not to prompt the person, to jump to solutions,
conflate problems, hear beyond what they say
• They are psychologically difficult – exposing your work is hard, It’s pretty
equivalent to the feeling most of us have about getting up to sing in a public
Karaoke bar, only without it being appropriate to get a little drunk first. when
combined with it being a pita, get pushed off indefinitely “it’s not ready” “they’ll
only tell us what we already know”.
• Depending on how you’re recruiting
8. Companies like Usertesting.com (whom we use) and others
• Generally a directed task testers are supposed to complete, so less natural
experience than just picking up a game and exploring, higher willingness to
“figure it out”
• Limited view of body language, narration expresses conscious thoughts, not
unconscious, no chance to follow-up
• Still limited to first session experience, now with a time limit
• Still small sample, luck of the draw on testers, some selection bias in terms of
who takes these kind of gigs
9. By this I don’t mean in person, but sending out a mobile build or a link to a web
version to a group of people you know and seeing what happens
• More realistic experience, they’re testing the whole game across as many
sessions as they want to play
• Depending on who it goes to you can get good qualitative feedback,
• Audience tends to be biased in your support, professional game developers, or
both. a lot of people won’t want to hurt your feelings. You’re unlikely to hear “this
sucks” even if it’s true.
• Low sample size & unrepresentative/biased audience = mostly garbage metrics
10.
11. The way to think about this: if we assume that tutorial completion is around 80%
globally, if I get 400 installs representative of the global audience 95% of the time
their tutorial completion rate will be between 72% and 88%.
12. Now just because you have a smaller sample than this doesn’t mean metrics are
useless. In a normal distribution you are more likely to be closer to the mean than
farther away. My unscientific rule of thumb is that you start getting directionally useful
if not accurate metrics when an event has occurred 75 or more times. So for
directionally useful tutorial results you need about 100 people installing a game, and
for directionally useful buyer conversion rate more like 5,000.
13. By this I mean inviting a broader group of players to play a game not yet released,
either on web or Android, usually volunteers from a fanbase
• All the benefits of friends & family tests with larger sample sizes!
• REALLY engaged audience excited to give feedback, not constrained by
politeness. They’ll tell you your game sucks.
• Chance to build a community for your game pre-release
14. On Kongregate.com beta access to games is a benefit of Kong+, our ad-free version
players pay $29.99/year. We also gift it to our volunteer moderators and big spenders.
There are about 30,000 MAU, and the average beta game gets ~3k beta users.
We consistently have 5-10 games in closed betas, which we use for our publishing
portfolio, but is also open to other developers
15. Metrics are the product of the audience and the underlying game. You can get
average metrics by either putting poorly qualified traffic into an amazing game, or by
putting amazing traffic into a mediocre game. Now this is somewhat obvious, but if
you’ve been working on a game for a year it’s easy to forget about audience, and
think metrics are entirely about the game. It is especially easy to underestimate how
BIG the audience swing can be.
This was the most extreme split we’ve ever seen, with a 9X difference in % buyers,
which is more commonly 3-4x higher in beta than in global release. But even though
it’s inflated it’s still useful: we could tell this was going to be a high ARPU game with
mediocre initial retention but good long-term retention. (Note, web d1 tends to be
much lower than mobile, but they are comparable by D30.
16. This is the now classic method for mobile, releasing fully but in just a few countries,
often Canada and/or Australia
The real thing is hard, and a lot of the weakness are just aspects of the mobile game
market
• It’s hard work releasing anything on mobile – builds, screenshots, but particularly
getting games working on such a broad range of devices
• Long Apple approval times makes iteration slow even when you can quickly fix a
problem
• Traffic doesn’t magically show up in games and buying it is expensive – average
is ~$3 per install in Canada & Australia, a bit less on Android
18. Australia & Canada may be good proxies for the US, but that’s likely to be <1/3 of
installs
They used to be closer, but the gap widened after the release of the iPhone 6 when a
lot of high end device users switched back to iOS
Especially for more niche, high LTV genres like CCGs, RPGs & Strategy games
whether it’s CPIs going up or retention & LTV going down after your first “golden
cohort”.
In a small market a few big buyers can blow out a market. And test market spend
tends to be less ROI focused, so you see weird patterns.
19. Spellstone, a polished collectible card game from Synapse games, shows some of
the dynamics. You can see that the performance of iOS is generally much stronger
than Android, and that paid generally is quite a bit stronger than organics. But the
really dramatic number is the huge drop in performance on organic traffic coming from
the substantial features that the game got, with around a 70% drop in the ARPU on
both iOS and Android. This is most dramatic with more niche, high LTV genres like
CCGs, RPGs & Strategy games. For a more casual, broad audience game like
AdVenture Capitalist we don’t typically see a big delta.
20. Your game is driven by outliers and their presence or absence distort almost anything
you look at.
Binary “yes/no” metrics like D1 retention or tutorial completion are more reliable than
averages involving engagement or revenue. And the deeper your game, the less
spending is capped, the more unpredictable those averages get. So as much as
possible look at binary metrics that are proxies for the averages, or answer the
questions. % repeat buyers, for example, rather than ARPPU.
21. Don’t get me wrong, I love A/B tests. They divorce correlation from causation, and
manage the audience mix problems well, too. But don’t expect to be able to run a lot
of A/B tests in test markets unless you’re willing to spend major $$s. The problem is
sample size. Take the numbers I was giving earlier for cohort sizes then double them
for an A/B test. And with an A/B test you need to measure the results fairly precisely,
directional numbers are not good enough, or you can make bad choices based on the
results.
22. More of a strategy than a testing method, but one I think is underused. I’m biased of
course! I have a web portal. But this strategy has worked for a lot of big successes,
from King with Candy Crush to Blizzard recently with Hearthstone.
– Steam, Kongregate, Facebook, Miniclip, Newgrounds, Addicting Games and
hundreds more. But it’s important to find the right audience fit – Facebook is a much
better choice for a very casual game than Steam or Kongregate
Better social feature support (forums, videos, streaming, etc) to build community
Comparable LTVs (at least on Kongregate)
Chrome no longer supports the Unity plug-in, and Firefox will likely kill it by the end of
the year. Flash is still going for now, but will likely be phased out in a few years. But
the webGL export from Unity is improving rapidly, and there are a lot of other good
cross-platform frameworks to work with, such as Haxe.
23. Over the next 6 months they worked on polishing the UI, adding monetization, all
while releasing it on dozens of additional platforms across the web, and were able to
extend the content and improve the balance while building a bigger and bigger
audience.
24. Mobile test markets lasted about 2 months, focused on mobile device stability, FTUE
and the new rewarded video integration, a huge addition to monetization
25.
26.
27.
28. And since each type of test adds something, the ideal plan is to use all of them, in
approximately this order, with company playtests throughout a given.
29. How much money and time you have are intimately related: time is generally the
biggest cost because each month of a studio’s burn rate adds up. But there are
situations where that’s not true: in a big company you may have intense pressure to
launch by a certain date but a lavish budget for test market marketing. Or an indie
doing work-for-hire or with a full-time jobs may have little time pressure but no money
to buy traffic. The indie should focus on in-person play tests and closed betas, while
with enough money the big studio can blast their way through geo-locked test
markets.
A simple puzzle game or endless runner that are easy to pick up and play are going
to get more value out of the experiential testing that help them nail the fun in the core
experience, but isolated, single session tests are much less useful to multiplayer
games with deep metagames and economies, where long-term metric-based testing
is crucial to getting things right.
Games with lots of graphics, or are otherwise technically demanding, are going to
need extra time in mobile geo-locked test markets to deal with all the problems that
crop up with low memory and low GPU devices on both Android & iOS
And finally: what are your goals? Do you expect to get significant features from the
platforms? Are you look for top 10 grossing? Top 1000? The bigger the launch you’re
expecting, the higher the stakes and the more crucial it is to have the game in the
best state possible at launch.
30. Assuming both time and money are at least somewhat constrained. Say a mid-sized
studio with most of their burn covered by existing game income, but not a big cushion.
This still assumes 6 months from friends/family to global mobile release. If you cut it
much shorter on a multiplayer game you will almost certainly regret it.
31. Single-player game made by a small cash-strapped team, mobile-specific controls
and ad-based monetization. In this case most of the real game testing and iteration
should be on the back of rigorous, frequent, in-person testing. Then mobile test
market can concentrate on just a few metrics.
32. During pre-production & production the key question you’re asking is “Is this fun? Are
we on the right track?” The sooner you figure out something isn’t working as you
expected, the easier it is to fix. The more you are departing from convention and
comparables the more you need to validate as you go along.
33.
34. As you approach release you’re asking “Is This Ready”? It’s a great time to do remote
playtests focused on the first time user experience, and make sure that analytics are
hooked up and firing correctly – that last is not a given, analytics are very easy to
screw up. This is also a great time to send your game to a 3rd party QA service to test
on a broad range of devices if you’re going straight to mobile.
35. It’s the next Clash of Clans! Or Crossy Road! Everything is broken! Who would even
play this piece of shit? Total failure.
Depending on the person, they may cherry-pick the good, or focus only on the bad.
36. This is where the rubber really meets the road. To know if the game is working, you
have to know what you’re looking for.
37. We have a very successful game with 20% D1 retention. We have very successful
games with $0.03 ARPDAUs.
38. So here are some sample metrics ranges, from low to high, for the genres I’ve been
using for example test plans, then for all genres. This is loosely based on the metrics
we’ve seen from games in our portfolio and more generally on Kongregate.com. As
you can see the “all genres” low/medium/high can be pretty different from the ranges
by genres. Good retention for a multiplayer RPG game is drastically lower than an
idle game. Good monetization for a casual runner would be terrible for that same
RPG. What’s missing from this is expectations around traffic: that casual runner has
probably 10x the potential traffic of the multiplayer RPG.
39. Here’s a couple of outcome models based on the average metrics for each of these
genres, then broken out by different levels of traffic. You’ll notice these profitability
scenarios aren’t great: games with average metrics need exceptional traffic to
become a sustainable business. In general for a success you need at least one or two
metrics to fall in one of the “high scenarios”.
Set your goals matching realistic expectations of the genre and acceptable (not ideal)
business outcomes.
If you need to hit top 20 grossing to justify huge budget/company expectations, your
goals should be much higher than if you mostly want to learn.
40. It’s not that you don’t look at other metrics, but you want to set the gates based on the
most important one for that stage.
41. One of the benefits of breaking your testing into stages is that on mobile it allows you
to use a wider variety of test markets. Canada & Australia are not only expensive
places to test stability, they’re a bad choice because they have a much lower % of the
low-end devices most likely to trigger issues. That’s better tested in emerging
markets. Overall testing in a range of countries from tier 1 to tier 3 will give you a
much more representative view of your global performance than limiting to a few
English-speaking markets. We’ve used more than 20 different countries in the last
year, all shown in this map.
42. Note that the sample sizes are cumulative. 12k isn’t enough for statistical significance
on buyer %, but 25k is.
43.
44. This is not optional, but a surprising number of developers don’t. Crashes are
annoying to players, effecting both retention and your ratings and reviews in the app
stores. We recommend a 3rd party service like Fabric/Crashlytics or Hockey App,
though there’s a quite bit of info in Google Play console as well.
45. Stage 2 is where you’re really optimizing your game, and you should look at in the
same stream that players are moving through the game, as that’s the order in which
sample sizes will get large enough, and because problems in one will likely flow into
the next. You start with the progression through the first time user experience,
checking drop-off pre-tutorial, and then at each tutorial step. Then you’ll watch PVE
progression, what’s the progression through missions, what are the win rates. After
players have been in the game a little longer you want to look at PVP participation
and win rates, and then finally at the economy, where you should look at the full sinks
and sources flow, but keep a particularly close eye on resource balances and how
they grow.
46. Retention is the KPI reflection of progression. Without longer term retention, which
reflects commitment and engagement with the game, few people will pay. Conversion
reflects both engagement and balance: do they care enough to buy something, and is
there a good reason to. If retention is good but conversion is bad, then either what’s
being sold is not compelling, the balance is not challenging enough for it to be useful,
or the economy is imbalanced in a way where there’s no reason to purchase,
because you can get it for free. Note that there is some tension between retention
and conversion, though, because tight economies may make players more likely to
churn.
New buyer packs can be great at boosting conversion, while masking underlying
problems. One of the most important stats to look at is repeat conversion – how many
people buy a 2nd time? A 3rd? Repeat conversion shows both how players feel about
their first purchase and whether there is depth of spend. If you have a high
conversion rate but low repeat purchasing, your game will just pop and drop.
Note that I haven’t mentioned ARPPU. That’s because while it’s an important metric,
it’s not one you can really look at with any reliability because revenue is an
exponential distribution, and very erratic in small samples. However ARPPUs are
capped fairly low unless there are repeat purchases, so looking at that statistic, which
is a normal distribution, answers the same question with better stability.
47. It’s human nature to project causation, so as you make changes to your game you’re
likely to look at daily numbers and think it’s the result of what you’ve done. Resist.
Even with a game doing a reasonably big test market you get tons and tons of
random variation in the daily numbers. These numbers are from Spellstone’s test
markets, which had between 1000-1500 DAU through most of this but the daily
numbers are all over the place.
We track rolling averages, which helps, but if you want to look at the impact of
changes roll up cohorts from before and after the change to get a statistically
significant sample to look at. And still take that with a grain of salt because of
audience mix and confounding effects from other changes you’re making.
48. An important part of mobile test markets is optimizing the assets you’re using in the
app store – we regularly see substantial gains testing icons, video, screenshots, and
copy, though we’ve never seen a significant difference from game name testing. But
context is important – results are often inconsistent between app stores and geos. For
that reason we don’t start our ASO until we’ve expanded beyond T3 markets.
Google’s tools for this are great, but for iOS you need to use a paid service like Store
Maven.
49. Test market marketing isn’t just about driving installs, it’s about testing the marketing
itself, optimizing creatives & targeting, generally figuring out: will this work? Will I be
able to drive audience into my game? Can I do it profitably? Test a lot of creatives
across a lot of networks. Keep refreshing. You never know what’s going to work
because again context matters a lot.
50. There’s nothing worse than having a big feature and then have the servers crumble
under the load. It feels like flushing success down the toilet, and you are. Now “how to
load test a game” is a subject that deserves it’s own GDC talk and I’m not the person
to give it. But if you have a server based game and any hope of success you need to
do this.
51. Now I want to go through a case study of what this looks like with a game that that
was neither a triumph or a failure.
Raid Brigade is a one-handed party-based Action RPG with base-building elements
and an unusual one-handed control scheme. It’s the first game made by San Diego
studio Ultrabit made up of mostly Zynga veterans. After about 12 months of
development we released it to mobile test markets last June, skipping our normal PC
stage because of questions on how the controls might work. We were all excited for
the game, but the initial results were way below our goals and expectations: the first
few weeks saw dreadful performance with only 40% of people getting through the
tutorial and D7 retention 75% below the goal numbers.
52. The good news is that there were lots of obvious things to fix. There were long
loading times for assets being streamed in, essential to keep the initial download size
of a game with 3D art under 100MB. Improving those and the tutorial got tutorial
completion up to 70% and doubled D7 retention, though still well below our goal of
18%.
53. To help people get in to the various branching systems we then switched from a linear
tutorial to one based on a series of quests, which again helped increase D7 retention
significantly, this time up to about 12%. Again it was great to see that much
improvement, now 3x what we started at…but still below goals. And gains in retention
after that became harder to get. Though we kept working on that, along with many
other things.
After 3 months in test markets we we had to face the dilemma: the game was meeting
some of our goals (crash rate, tutorial, D1 retention, conversion) but not all of them
(D7, D30 retention, Repeat Buyer %) and the developer’s runway was starting to be a
concern. We could go ahead and launch in October as planned, or keep working on
it, cut into the developer’s runway, and go up against the glut of games coming out in
the holiday season.
54. When your metrics are good the decision path on when to launch is easy. When you
have plenty of time and money, you can usually keep working on the game, though
even there it’s important to be realistic about whether the game is fixable. If you’re
pretty sure you understand what’s wrong and have a good idea to fix it that’s great,
but there may be diminishing returns or unfixable flaws. When you get to the point
where you either can’t keep working on the game, or it doesn’t seem worthwhile then
the question becomes: do you launch?
At that point it’s time to think of the money you’ve spent on development as a sunk
cost, and think just about the effort needed to launch and support the game after
launch. For a single-player game without servers, this is fairly simple, and in most
cases you should go ahead and launch and see what happens. But with multiplayer
games this becomes more complicated as there are ongoing server costs and the
necessity of releasing additional content to drive revenue and engagement, as well as
critical player mass and opportunity cost issues. Supercell and some other companies
would only support a game if it’s a huge success, but exactly what that level should
be is going to have different answers by studio. But personally I don’t think you should
launch a game unless you can support it for a fairly extended time frame, a year plus
at least. Players are investing in your game, and that should be respected.
In 4 years of publishing we’ve never cancelled a single-player game, though we did
stop one from bothering to do an Android version. But this year alone we canceled
three multiplayer games, one after a Kong+ beta and two during mobile test markets.
55. Making a game these days can feel like walking into a dark forest. But remember: you
have many tools at your disposal. Be prepared, like the boy scouts say, have a plan,
be realistic, and hopefully you can make it through the forest and find the treasure
you were looking for.