Designing Learning Games for Return on Investment

Game Design for Return on Investment
The review, planning, and production of a digital training tool for long-term care certified nursing assistants
Brock Dubbels
Center for Cognitive Sciences
The University of Minnesota Brock@vgAlt.com

What I’m Playing...

Why games for learning?
Games, by their very nature, assess, measure, and evaluate.
But, you still need to know what you are assessing, measuring, and evaluating.
Assessing inside and out of the game.
The creation of construct validity and method.

Topics
Yoga – perceptual / conceptual -- activity
Background
The game
Development talking points
Take home

The state of long term care
55% white, 35% Black, 10% Hispanic
Most workers are economically disadvantaged
Low levels of educational attainment.
Physically and emotionally demanding work, but often among the lowest paid in the service industry.
Viewed as an unpleasant occupation: primarily a maid service taking care of incontinent, cognitively unaware old people.
45% attrition rate in first 90 days. Some reports show 100% turn over a year.
Great shortage—with shortages, come reductions in quality of care.
Expected growth rate of 85% with Baby Boomers retiring.
Regulation tends to emphasize entry training, with limited attention to continued career growth or development.
Supervisors with “good people skills,” promotion of worker autonomy are the most important predictors of higher job satisfaction and lower turnover rates.
From “Who Will Care for Us?” US Dept. Health & Human Services

Hypothesis
Will improved people skills and increased worker autonomy reduce attrition through improving the perceived quality of life.
Will perceived quality of life improve relations and autonomy in the facility?
Increase well-being in residents and nursing assistants?
Reduce pain management?
Reduce catastrophic care?
Confidence and accuracy in information gathering and reporting?

The User Story
Functionally capable, but not skilled.
Soft skills
Documentation skills
High school education +/-1
Like popular culture.
A bit irreverent about job, but this is coffee break coping mechanism
Limited care load while training.
Enjoy popular culture – soaps, drama, etc.

Pre-Design Methods
Elicitation – do you really need a game?
Cognitive Ethnography
Identification of quality of life variables, theoretic processes (presence, CC, AL), & documentation variables for kiosk.
Fidelity
Assessment, measurement, and evaluation methods.

Design Decisions
FPH –the first person healer
Flash / with database
Web-host as well as installed with data upload.
Time serves as game element
Functional task selection vs. Interpersonal Communication
Dialog driven
Dialog supported by video cut scenes with voice narration
Mini games such as room clean up
Reward system (blue stars)
Preceptor / Optimal Path / Debrief
Interface with task log, resident information, clock, pause.
Use real people’s faces as avatars
Increase engagement through subtle but tasteful irreverence.

Task List

Profile: Al

Profile Eunice

Branching Dialog
Blue star for Constructive Conversation—
you constructively redirected.

Interaction

Debrief

Four questions
Can I take any credit for any changes that have happened in an individuals learning?
Does this have a connection to my instructional activities?
Does these instructional activities equate to a return on investment?
How do I know this?

Tension in workflow
Software Design
Typically based upon an economic consideration.
How will this solve a problem?
What are the first steps in production?
The focus is on stages of production:
Business Partner Relations, Function, Behavior, Structure, & Non-Function (qualities).
Research Design
Typically based upon answering a testable question.
How will this solve a problem?
How do I know this?
The focus is on method and hypothesis testing:
Construct validity, reliability, reliability, and probability.

Theoretical perspective
Improve soft skills and documentation.
Quality of Life Measures
How these are affected through: Presence, Constructive Conversation, Active Listening
These are used for game mechanics and coding dialog.
Reducing attrition, improving Perceived Quality of Life (PQoL), and improving documentation will reduce costs and allow for more hires, better wages
Reduce: pain meds, attendant care, catastrophic care.
Documentation
Reduce elicitation from medical staff
Improve medical staff objective knowledge on daily living skills

Analysis Tools
In order to measure whether the game does what it was designed to do:
Analysis criteria must exist inside and outside of the game for evaluation.
Same underlying measures from game
Inside:
Scoring system weighted dialog
Story content equated to kiosk input
Outside:
Observational scoring tool for preceptors
Survey for self-report
Resident survey

Construct Validity—How do we know what we measure means anything?
MTMM to measure convergent/discriminatory validity.
You need 3 measures for construct validity measures.
In-game measures
Observational workplace rubric
Resident survey
Without construct validity, you cannot conduct ROI analysis with certainty.

Cognitive Ethnography Design/ Usability Perspectives for MTMM validity measure

Mental Representation

Outcome Analysis
General Linear Model
Quality of Life Variables
Operationalized in soft skills and PQoL construct
Presence, Active Listening, and Constructive Conversation
Longitudinal study
Pre, Game Play, Post
Compare performance in: surveys, objective observer data, game play, non-game play controls, self-report.
Game play – construct sub-level scoring, i.e. number of residents, rewards, optimal path decisions.
Institutional data pre / post
Compare catastrophic care, pain meds, independence, attrition
Use game play, survey and observational tools as co-variates.

Take home
Can you pose a testable question– hypothesis?
Tension between design process and measurement
Needs – behavior, function, non-function, structure.
Construct validity – are you measuring what you think you are measuring?
Assessments, measures, & evaluations
Spend time understanding the sample population
Beliefs, likes, skills, & abilities.
irreverence increases engagement, but reduces happiness of business partner.
Usability testing should align with construct
Again, emphasis on validity
Without it, there is no capability for ROI analysis

http://vgalt.com	180
url_unknown	4
http://www.slideshare.net	1

Designing Learning Games for Return on Investment

by Brock Dubbels on May 22, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

3 Embeds 185

Statistics

Designing Learning Games for Return on Investment — Presentation Transcript