Your SlideShare is downloading. ×
  • Like
  • Save
 Designing Learning Games for Return on Investment
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Designing Learning Games for Return on Investment


This presentation was offered as a talk at Ludica Medica for the Games for Health Conference 2011. The talk emphasizes assessment, measurement, and evaluation utilizing the MTMM model for statistical …

This presentation was offered as a talk at Ludica Medica for the Games for Health Conference 2011. The talk emphasizes assessment, measurement, and evaluation utilizing the MTMM model for statistical analysis of construct validity and roi measures.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads


Total Views
On SlideShare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide
  • cognitive ethnography assumes that cognition is distributed through rules, roles, language, relationships and coordinated activities, and can be embodied in artifacts and objects (Dubbels, 2008).Conceptual Space Analysis, Physical Space Analysis, Social Space AnalysisCognitive ethnography assumes that human cognition adapts to its natural surroundings. Therefore, the role of cognitive ethnographer is to transform observational data and interpretation into meaningful representations so that cognitive properties of the system become visible Ethnography often involves the researcher living in the community of study, learning the language, doing what members of the community do—learning to see the world as it is seen by the natives in their cultural context, Fetterman (1998).Cognitive ethnography follows the same protocol, but its purpose is to understand cognitive process and context—examining them together, thus, eliminating the false dichotomy between psychology and anthropology.
  • We gathered from the ethnographies that many watched reality television, soapy kind of programming, liked comedies, popular culture, gossip, irreverent, and suggested that for the game we have wheelchair jousting.We chose to offer the game as a soap opera, with intrigues at the residency.


  • 1. Game Design for Return on Investment
    The review, planning, and production of a digital training tool for long-term care certified nursing assistants
    Brock Dubbels
    Center for Cognitive Sciences
    The University of Minnesota
  • 2. What I’m Playing...
  • 3. Why games for learning?
    Games, by their very nature, assess, measure, and evaluate.
    But, you still need to know what you are assessing, measuring, and evaluating.
    Assessing inside and out of the game.
    The creation of construct validity and method.
  • 4. Topics
    Yoga – perceptual / conceptual -- activity
    The game
    Development talking points
    Take home
  • 5. The state of long term care
    55% white, 35% Black, 10% Hispanic
    Most workers are economically disadvantaged
    Low levels of educational attainment.
    Physically and emotionally demanding work, but often among the lowest paid in the service industry.
    Viewed as an unpleasant occupation: primarily a maid service taking care of incontinent, cognitively unaware old people.
    45% attrition rate in first 90 days. Some reports show 100% turn over a year.
    Great shortage—with shortages, come reductions in quality of care.
    Expected growth rate of 85% with Baby Boomers retiring.
    Regulation tends to emphasize entry training, with limited attention to continued career growth or development.
    Supervisors with “good people skills,” promotion of worker autonomy are the most important predictors of higher job satisfaction and lower turnover rates.
    From “Who Will Care for Us?” US Dept. Health & Human Services
  • 6. Hypothesis
    Will improved people skills and increased worker autonomy reduce attrition through improving the perceived quality of life.
    Will perceived quality of life improve relations and autonomy in the facility?
    Increase well-being in residents and nursing assistants?
    Reduce pain management?
    Reduce catastrophic care?
    Confidence and accuracy in information gathering and reporting?
  • 7. The User Story
    Functionally capable, but not skilled.
    Soft skills
    Documentation skills
    High school education +/-1
    Like popular culture.
    A bit irreverent about job, but this is coffee break coping mechanism
    Limited care load while training.
    Enjoy popular culture – soaps, drama, etc.
  • 8. Pre-Design Methods
    Elicitation – do you really need a game?
    Cognitive Ethnography
    Identification of quality of life variables, theoretic processes (presence, CC, AL), & documentation variables for kiosk.
    Assessment, measurement, and evaluation methods.
  • 9. Design Decisions
    FPH –the first person healer
    Flash / with database
    Web-host as well as installed with data upload.
    Time serves as game element
    Functional task selection vs. Interpersonal Communication
    Dialog driven
    Dialog supported by video cut scenes with voice narration
    Mini games such as room clean up
    Reward system (blue stars)
    Preceptor / Optimal Path / Debrief
    Interface with task log, resident information, clock, pause.
    Use real people’s faces as avatars
    Increase engagement through subtle but tasteful irreverence.
  • 10.
  • 11. Task List
  • 12. Profile: Al
  • 13. Profile Eunice
  • 14. Kiosk
  • 15. Branching Dialog
    Blue star for Constructive Conversation—
    you constructively redirected.
  • 16. Interaction
  • 17. Rooms
  • 18. Debrief
  • 19. Four questions
    Can I take any credit for any changes that have happened in an individuals learning?
    Does this have a connection to my instructional activities?
    Does these instructional activities equate to a return on investment?
    How do I know this?
  • 20. Tension in workflow
    Software Design
    Typically based upon an economic consideration.
    How will this solve a problem?
    What are the first steps in production?
    The focus is on stages of production:
    Business Partner Relations, Function, Behavior, Structure, & Non-Function (qualities).
    Research Design
    Typically based upon answering a testable question.
    How will this solve a problem?
    How do I know this?
    The focus is on method and hypothesis testing:
    Construct validity, reliability, reliability, and probability.
  • 21. Theoretical perspective
    Improve soft skills and documentation.
    Quality of Life Measures
    How these are affected through: Presence, Constructive Conversation, Active Listening
    These are used for game mechanics and coding dialog.
    Reducing attrition, improving Perceived Quality of Life (PQoL), and improving documentation will reduce costs and allow for more hires, better wages
    Reduce: pain meds, attendant care, catastrophic care.
    Reduce elicitation from medical staff
    Improve medical staff objective knowledge on daily living skills
  • 22. Analysis Tools
    In order to measure whether the game does what it was designed to do:
    Analysis criteria must exist inside and outside of the game for evaluation.
    Same underlying measures from game
    Scoring system weighted dialog
    Story content equated to kiosk input
    Observational scoring tool for preceptors
    Survey for self-report
    Resident survey
  • 23. Construct Validity—How do we know what we measure means anything?
    MTMM to measure convergent/discriminatory validity.
    You need 3 measures for construct validity measures.
    In-game measures
    Observational workplace rubric
    Resident survey
    Without construct validity, you cannot conduct ROI analysis with certainty.
  • 24. Cognitive Ethnography Design/ Usability Perspectives for MTMM validity measure
  • 25. Mental Representation
  • 26. Outcome Analysis
    General Linear Model
    Quality of Life Variables
    Operationalized in soft skills and PQoL construct
    Presence, Active Listening, and Constructive Conversation
    Longitudinal study
    Pre, Game Play, Post
    Compare performance in: surveys, objective observer data, game play, non-game play controls, self-report.
    Game play – construct sub-level scoring, i.e. number of residents, rewards, optimal path decisions.
    Institutional data pre / post
    Compare catastrophic care, pain meds, independence, attrition
    Use game play, survey and observational tools as co-variates.
  • 27. Take home
    Can you pose a testable question– hypothesis?
    Tension between design process and measurement
    Needs – behavior, function, non-function, structure.
    Construct validity – are you measuring what you think you are measuring?
    Assessments, measures, & evaluations
    Spend time understanding the sample population
    Beliefs, likes, skills, & abilities.
    irreverence increases engagement, but reduces happiness of business partner.
    Usability testing should align with construct
    Again, emphasis on validity
    Without it, there is no capability for ROI analysis