With games being designed that seek changes in intention, attitudes and behavior it’s more important than ever that we evaluate the games we design and understand their true effects. A team from the University of Miami designed a game-based intervention with the goal to reduce intentions to indoor tan. We found that games can be a powerful persuasion tactic but not always in the ways one would expect! Join us as we share our design process and findings, which indicate the importance of knowing your target audience and how there isn’t a one-size fits all approach to designing such interventions.
Boost Fertility New Invention Ups Success Rates.pdf
Hypothesis Tested: Designing Games with Theoretical Frameworks
1. HYPOTHESIS TESTED
Designing Social Impact Games with Theoretical Frameworks
Clay Ewing
@claytical
c.ewing@miami.edu
Associate Professor
University of Miami
4. TANZANIAN SOCIAL
ACTION FUND (2012)
Working with the Tanzanian government
and the World Bank, we developed a
facilitated tabletop game intended to teach
rural farmers about a new conditional cash
transfer program.
5. “
”
IF YOU CAN MEASURE IT, THEN IT’S NOT THE CHANGE
I WANT TO SEE… IF YOUR GAME OR TECHNOLOGY
REALLY ‘WORKS’ IT FREAKS ME OUT
Paolo Pedercini, Making Games in a F****d Up World , Games for Change 2014
Pedercini, P. (2014) Making Games in a F****d Up World, Games for Change 2014. Retrieved from
https://www.youtube.com/watch?v=MflkwKt7tl4
6. VANITY
A tabletop game where players take on the
role of aspiring actors competing at
auditions. The game aims to teach
teenagers about the dangers of using
tanning beds.
Preliminary results are inconclusive but
promising. The game may be an acceptable
preventative tool to keep people from
engaging in tanning behavior that could
lead to skin cancer.
Tran, Carcioppolo, Colantonio, Ewing, Lang, Beyea, Beecker. “The Empty Comfort of Vanity: Assessing the
Effectiveness of an Interactive Game to Increase Skin Cancer Prevention Outcomes” (GLS10)
7. UNSAVORY (2015)
You play as a minimum wage worker in a
fast casual restaurant preparing burritos.
Your goal is to survive a month on the job.
Eventually, your work is interrupted by an
oncoming sneeze. If you sneeze, your boss
knows you’re sick and sends you home. If
you miss too much work, you’re fired and
won’t be able to pay your bills.
8. “
”
AFTER I ANALYZED THE RESULTS FROM THIS STUDY, I WAS
DISMAYED TO FIND THAT PLAYING THE GAME HAD NO
EFFECT ON POSITIVE FEELINGS TOWARD THE POOR. IN
FACT, THE GAME HAD A NEGATIVE EFFECT ON ATTITUDES
AMONG CERTAIN PARTICIPANTS—INCLUDING SOME
PEOPLE WHO WERE SYMPATHETIC TO THE POOR TO BEGIN
WITH.
Gina Roussos, When Good Intentions Go Awry, Psychology Today
Roussos, G., & Dovidio, J. F. (2016). Playing below the poverty line: Investigating an online game as a way to reduce
prejudice toward the poor. Cyberpsychology: Journal of Psychosocial Research on Cyberspace, 10(2).
Roussos, G. (2015). When Good Intentions Go Awry: The Counterintuitive effects of a prosocial online game.
Psychology Today. Retrieved from https://www.psychologytoday.com/us/blog/sound-science-sound-
policy/201512/when-good-intentions-go-awry
9. “
”
WHEN IMPACT IS DEFINED TOO NARROWLY, SOME
GAMES ARE DISMISSED FOR THE WRONG REASONS
AND THEIR IMPACT IS OVERLOOKED.
Impact with Games: A Fragmented Field
Stokes, B., O’Shea, G., Walden, N., Nasso, F., Mariutto, G., Hill, A., & Burak, A. (2016). IMPACT WITH GAMES: A
FRAGMENTED FIELD. Pittsburgh, PA: ETC Press. Retrieved from http://gameimpact.net/reports/fragmented-
field/
10. Integrated Model of Behavioral Prediction: Fishbein, 2000, 2007; Fishbein & Yzer, 2003
Health Belief Model: Rosenstock, 1974; Rosenstock, Strecher & Becker, 1988
11. POTENTIAL SUITORS
Anti Tanning Neutral Pro Tanning
For each tanning belief, we created 3 personality types that would cover different combinations of tanning beliefs and life values.
14. THE EPILOGUE
The final piece of the game has an anti-
tanning health message implicitly
presented.
It gets shown when the character went to a
tanning salon or they chose to date
someone with a history of using tanning
beds.
15. STUDY RESULTS
The game was the most effective intervention at reducing intent to tan with women
and people who haven’t indoor tanned previously.
However, they did not enjoy the game as much as the other interventions.
Kim, S., Zhou, C., Carcioppolo, N., Ewing, C. (2018). Rethinking the Role of Engagement and Enjoyment: The Effect of a
Digital Health Game on Indoor Tanning Attitude and Intention. Kentucky Conference on Health Communication.
Retrieved from https://comm.uky.edu/kchc/abstract/768.
16. STUDY RESULTS
There was a boomerang effect!
Men and people who have indoor tanned previously enjoyed the game and are more
likely to tan after playing the game.
Kim, S., Zhou, C., Carcioppolo, N., Ewing, C. (2018). Rethinking the Role of Engagement and Enjoyment: The Effect of a
Digital Health Game on Indoor Tanning Attitude and Intention. Kentucky Conference on Health Communication.
Retrieved from https://comm.uky.edu/kchc/abstract/768.
17. STUDY RESULTS
We assumed increasing engagement would have a positive effect but the increase in
variables can decrease the persuasive effects of the game.
Kim, S., Zhou, C., Carcioppolo, N., Ewing, C. (2018). Rethinking the Role of Engagement and Enjoyment: The Effect of a
Digital Health Game on Indoor Tanning Attitude and Intention. Kentucky Conference on Health Communication.
Retrieved from https://comm.uky.edu/kchc/abstract/768.
18. STUDY RESULTS
In traditional interventions, increased transportation leads to better outcomes. In the
game condition, the players were transported but it had a negative effect. We can’t
make a conclusion on this yet.
Kim, S., Zhou, C., Carcioppolo, N., Ewing, C. (2018). Rethinking the Role of Engagement and Enjoyment: The Effect of a
Digital Health Game on Indoor Tanning Attitude and Intention. Kentucky Conference on Health Communication.
Retrieved from https://comm.uky.edu/kchc/abstract/768.
19. It would be irresponsible to publish the game knowing what we found during the first study
20. When we figure out how to eliminate the boomerang effect, the game will be a solid health intervention tool.
21. THE ROAD AHEAD
• Limited character versions that focus on one category (tanning vs. anti-tanning)
• Deep dive into specific character messages and player behaviors
22. DREAMY – NON TANNING EDITION
The idea of interacting with a series of characters with the pretext of getting a date is
a pathway for other public health interventions.
23. THANK YOU!
Clay Ewing (@claytical)
c.ewing@miami.edu
http://www.claytical.com
Interactive Media Program
http://interactive.miami.edu
NERDLab
http://nerdlab.miami
Editor's Notes
CONTACTS:
Clay Ewing
c.ewing@miami.edu
Dr. Soyoon Kim
soyoon.kim@miami.edu
This project was funded by the School of Communication at the University of Miami (http://com.Miami.edu) and the Center for Communication, Culture, and Change (http://com.miami.edu/center-for-ccc)
Integrated Belief Model: https://www.sibson.com/media/1536/using-theory-to-design-effective-health-behavior-interventions.pdf
Health Belief Model: http://foresttwpseniorcenter.com/5695732-the-health-belief-model-and-personal-health-behavior.pdf
Hello!
I’m Clay Ewing and I’m an associate professor at the University of Miami. I’m the Co-Director of NERDLab, where students and faculty work together on social impact projects that are primarily game based. Most of the student members of NERDLab come from our MFA program in Interactive Media.
At the School of Communication, we also offer a PhD in Communication. Our PhD program has a strong health communication influence, so it was only a matter of time before we started working together on evidence based games for public health.
Today I’m going to talk about a game I designed and developed with my colleague Dr. Soyoon Kim. Dr. Kim is a health communication researcher and together we worked to integrate theoretical frameworks into gameplay in order to change tanning intentions.
But before I get to that, I want to step back in time to provide a bit of context for my relationship with measuring impact and how my experience evaluating games has evolved.
In 2010, I was here in New York at Parsons finishing up my MFA in Design and Technology. My thesis project was Hustling Health Care, a board game that spoke to the debate about health insurance before the Affordable Care Act had been passed.
My feeling at the time was fairly simple. If you presented players with an argument through a system, they would understand it better. If you used real world data, as I was doing, a player would come away with hard facts as well. And if the game was engaging enough, it could perhaps spark a deeper conversation.
The game was playtested thoroughly. It was definitely engaging. I can remember Nick Fortugno coming by after this picture was taken and saying something to the effect of “If you can keep 4 high school students playing your game for 2 hours in the middle of Playtech, you have succeeded.”
If you’re not familiar with Playtech, it’s an annual event held at Parsons where a bunch of kids are brought in to playtest a wide variety of games and experiences in development. It’s basically a big free arcade.
Hustling Health Care pushed me towards social impact game design as a career. When my partner Lien Tran and I were asked to design a game for the World Bank and the Tanzanian government, it felt like we were moving towards something.
The Tanzanian government had created the Tanzanian Social Action Fund with the objective of enabling poor households to increase incomes and opportunities while improving consumption. The game we designed set out to teach the TASAF staff about the program. It was well received and the staff suggested that the game could be modified to teach the community about the program as well.
When it was piloted, the game had a debrief that reinforced ideas and concepts as well as asking about engagement and enjoyment. In certain contexts, this would be a sufficient evaluation, but the for study I’m focusing on today, the scope is much more specific.
The thing that drives me to make social impact games is that they have the possibility of making a difference. When Paolo Pedercini commented in 2014 at G4C “If you can measure it, then it’s not the change I want to see… if your game or technology really works, it freaks me out”, I felt a bit of “Yeah, but…”
Weaponizing games as a propaganda is scary, but just as film and radio continue to be used effectively, it’s inevitable that games will be used as well. Understanding the way in which they can persuade us is important otherwise we’re bound to be manipulated.
Also, we use film and TV to deliver persuasive arguments for prosocial purposes. A good example would be East Los High on Hulu. Embedded in East Los are educational narratives that promote sexual and reproductive health. It makes sense to do something like this with games as well.
At the same time we were developing the game for TASAF, we started working on a tabletop game funded by the Canadian Dermatology Association called Vanity. The game sought to prevent use of indoor tanning beds, particularly at an early age. We designed the game with a dermatologist as the content expert.
There was no evaluation plan but as designers we felt that working with content experts would ensure the game was accurate and provide good information.
When we started working with researchers at UM, Vanity was selected to evaluate from a health communication perspective. Our preliminary results were promising, but collecting data has been incredibly slow as the game requires a minimum of 3 players and more than an hour of play.
Thinking about positive change and previous projects, I wanted to make something with a much greater reach. I had been working an app with ROC United on a Diners guide for ethical eating and one of the things that struck me was that most restaurant workers don’t receive paid sick days.
Following my feeling with Hustling Health Care, I sought to create a game that advocated for restaurant workers by allowing the player to walk in their shoes for a month, working a restaurant making minimum wage and getting sick.
But then…
I was here at Games for Change and during her presentation, Colleen Macklin mentioned this article in Psychology Today.
[READ QUOTE]
The article is about a game called Spent. Is anyone familiar with Spent? I’m a big fan of Spent, I love it using it as an example in my design classes. So hearing that a well designed game had no effect on positive feelings toward the poor was interesting. But the fact that some participants who were sympathetic to the poor beforehand had a negative effect on attitude was troublesome. It freaked me out a bit.
Since that Psychology Today article was published, the full study has been published in the journal Cyberpsychology.
As an FYI, one of the key takeaways was that playing the game wasn’t effective at positive change, but watching someone else play it was.
I think that we often have trouble with evaluating games because we debate what is considered impact and what it means to properly evaluate a game.
Educational games are probably the most straight forward as it’s pretty easy for the designers and researchers understand the test results before and after. Increases in test scores means that the game is effective at teaching a subject.
With games for public health and communication, you’re dealing with intentions and attitudes. Through work with our colleagues on the research side, we were finding that games with a narrative focus seemed to align well with theoretical frameworks in communication as where games focused more on play were a different beast.
The games research group in our school came to the conclusion that a tanning intervention game was a good idea but as a tabletop requiring multiple players, it was requiring too much time to collect data.
So now that you have a bit of context from my previous work, let’s dive into Dreamy.
Dreamy is a single player dating simulator. It’s also digital. As the first game at the University of Miami’s School of Communication to be developed with a researcher from day one, we thought it was a great opportunity to match existing theoretical frameworks to the game mechanics.
We chose two behavior prediction models: Fishbein’s integrated model of behavioral prediction and Rosenstock’s health belief model.
The game mechanics target salient beliefs and attitudes to determine behavioral intention to engage in tanning behavior and discourage indoor tanning.
The game has 9 conversation trees based on a 3x3 matrix of character combinations: outgoing, health conscious, and average paired with their attitude on tanning: anti, neutral and pro.
In case you’re wondering, the way the each potential suitor looks is randomly generated. At the beginning of the game, a friendly guide asks about their gender preference to find matches. They can choose men, women or say that love has no boundaries. Based on this preference, the game generates the suitor avatars.
Once the player starts a conversation with one of these characters, they won’t be able to talk to any other characters with tanning beliefs until they fail with the current conversation tree.
You might also be wondering how all the characters end up talking about tanning. As the different characters have specific reasons for tanning or not tanning, we attempted to build conversations that would naturally end up mentioning tanning. There are dead end characters that never talk about tanning but the player can never get a date with them. In early playtests, we allowed the player to talk to multiple tanning suitors and while players didn’t think “This must be a tanning intervention” they did think the designers had some hangups about tanning.
In this example, the player is talking to a character that is pro-tanning. You can see with the potential replies that the player is able to challenge the character’s beliefs with “I don’t think tanning beds are supposed to be used like that. Or at all.”
Or they can agree with the character choosing “I’ve never thought of a tanning bed as being so corrective! That’s smart!”
In the game, the player is invited to visit a tanning salon to improve their physical attractiveness. The player’s avatar can burn and even get moles.
If they choose to go to the tanning salon, a dermatologist sends them a message offering a free check up. The dermatologist gives accurate information regarding tanning beds and skin cancer.
To win the game, the player has to navigate successfully through a character’s conversation tree to get a date. If they date someone with a tanning history or engage in tanning behavior, their epilogue changes.
There are a series of outcomes for both the player and their love interest that include having a brush with death due to skin cancer and in some cases, dying from skin cancer.
So what did we find out?
Our target demographic was young women, so in that aspect, the game can be said to have a positive impact because it reduced their intent to tan. They really didn’t enjoy the game though.
Men and people who have indoor tanned enjoyed the game. However, they were more likely to tan after playing the game. That’s the exact opposite of what we wanted.
We went really deep on data collection and intervention design. The results I’m sharing today are from one study but we will have quite a bit more as the research team analyzes all of the data.
It’s also a problem figuring out what specifically caused the boomerang effect because there are many variables: how many characters they interacted with and which characters they interacted with, along with marketing offers, the tanning salon and the dermatologist. There are 117 health specific messages alone from just the suitors in the game.
Transportation refers the feeling of being so absorbed in a story that the connection to the real world is lost for some time. One assumption that we’ve had for games is that increased transportation is a good thing. We’re not sure about this and we need more data, specifically limiting the amount of health messages the player receives.
During the development process, Chun Zhou, one of our doctoral students, decided to use the game for her dissertation. She created a modified version to study interactivity. In the game, you’re able to construct your avatar, so she created a version that removed the avatar creation process as well as streamlined the conversations so that players choices were predetermined. Her initial findings have shown that story based interactivity is more effective than character based interactivity.
After working on Dreamy, I’ve been thinking a lot about impact and how we talk about games for change.
Dreamy was definitely impactful. By playing the game, it will change a person’s intent to tan. But not always in the right direction.
The most important takeaway for me is thinking deeply about failure. We tend to celebrate and talk about games that either feel like they’re creating impact or are proven to have impact, but we shy away from telling people about the ones that didn’t come out the way we expected and might have unintended consequences and effects. We need to discuss our failures more.
The exciting part is that we can iterate and figure out what works. We can also analyze the things that didn’t work as expected and try to figure out what happened so we don’t do it again.
Our first step in figuring out the boomerang effect is to simplify the game and to do this we’re going to create multiple versions that focus on specific categories. For example, would our results change if we removed all pro-tanning characters? What if the game pushed every player to tan but then had a dermatologist intervene?
For the designers in the room, you probably know what I’m talking about when I say that working with researchers can setup some hurdles. One of the bigger ones is finding common language. I’ve found that sometimes, envisioning a game through a design brief isn’t enough for a researcher to wrap their head around an idea. But prototyping a game just to explain an idea is a big ask of the developers.
It’s much easier now to talk about using the skeleton of Dreamy where players talk to suitors with embedded health messages to think about other possible interventions.
I’m a designer and I’m always happy to talk shop. Although if you are interested in the research side, I’m happy to connect you to my colleagues for further information regarding the nitty gritty of the studies. And just in case you’re interested in more information on our MFA program and our lab, here are the URLs.
If there are any questions/comments and we have time, fire away. Thank you!