G-ScalE
Brock R. Dubbels PhD.
G-ScalE Game Development and Testing Lab
McMaster University
Dubbels@McMaster.ca
Games Unit
Inquiry
Reading comprehension
Composition
Sustained engagement
Behavioral management
Planning
Cooperative learning
Classroom as game
Outcomes




     Dubbels, B.R. (in press) Video games, reading, and
     transmedial comprehension. In R. E. Ferdig
     (Ed.), Handbook of research on effective electronic
     gaming in education. Information Science Reference.
How about Math and
Science?
Scientific Habits of mind
Applied curriculum
Modeling
Simulation
STEM
Dance Dance Education
Because kids won’t let an education get in the way of their learning
Games are a structured form of play.
What are the elements of this game?
What makes the play emergent?
Is it non-linear?
Games as a metaphor for instructional design
Invoking play

                                               Probability
                                   Branching

                           Rules

                Roles &
                Identity




Imagery &
visualization
Towards top sight
Built like a game
What if we made a game about going
            to Las Vegas
It sounds kind of fun
And it is . . .
What do you end up with?
The Vegas Effect
Should everything that happens in games, stay in games?
It is not enough to invoke games and play.
Serious games should provide evidence that they delivered.
This should be quantifiable in performance metrics
UI Diagram
                                                   1. Are you sure .NET isn’t better?

                            I need a new Java      2. Who will the users be?
                              based portal for
                            Emergency rooms.
                                                   3. What information will it provide?




                                                                                 Direct the
                                                                                 conversation




                                                        Press when ready to move to design

         Client   Analyst




                                                  Start
Values    Functions         Qualities            Design
Functional Requirements
• Playable in < 5 minutes
• Engaging enough that target audience chooses
  to play multiple times
• Easy scripting for additional scenarios
• Results in demonstrable improvement in
  requirements elicitation
VFQ for this Project
• Values
  – Better understanding of how to apply RSA level 2
  – Learners able to “think” RSA
• Functions
  – Graphical
• Qualities
  – Engaging enough that people will choose to play it
    for 10-15 minutes
Values, Functions, and Qualities
• Values
   – Why a project is needed
   – Should have a success measure
   – e.g. Decrease claims processing time
• Functions
   – What is needed
   – Typically yes/no
   – e.g. must accept Medicare claims
• Qualities
   – How Well
   – Should have a success measure
   – e.g. don’t add to manual adjudication queue
UI Diagram
                            Press when
                            ready to see
                            what
                            happens
                                                                    Solution/Design
                                                             1. Create a new portal
Client   Analyst
                                           Implement         2. Implement using Java

                                                             3. Redesign business process
              Requirements                   Project
                                                             4. …

                                              Drag best
                                              solutions to
                                              SSD below


                                                                 Possible Solutions



                                                                Solution Summary
Values             Functions Qualities                              Document
UI Diagram
This isn’t what I         When will it be
    wanted.                  over?




                                              Replace with
                                              a simplified
                                              visual of
                                              UDP




                                                             Show key events during
                                                             implementation, that
                                                             result from choices the
                                                             user made
               Client     Analyst




                                                                Solution Summary
 Values                 Functions Qualities                         Document
BP: We need an online method to train people for the United Compliance Project




What would                     Do you have a                       Have you had
you like the                   timeline in mind?                   projects similar
project to                                                         to this in the
accomplish?                                                        past
Ratings
• Cohen’s Kappa
• Fleiss Kappa
• Agreement at .8
  – Kappa of <0.2 is
    considered poor
    agreement, 0.21-0.4 fair,
    0.41-0.6 moderate, 0.61-
    0.8 strong, and more
    than 0.8 near complete
    agreement.
Learning Games
• “It is worth noting that none of the featured
  companies started in instructional design…they come
  from business strategy, marketing, and the games
  industry”.
• “Most game-based learning approaches do not
  employ that particular category of expert
  whatsoever”
   – (Squire, 2005, p. 35).
Surface (face) & Content Validity
• Games are often built
  on these.
   – It looks like it measures what
     it is supposed to measure.
   – Have checked how the game
     represents the content
     against the relevant content
     domain.
   – Approach assumes that you
     have a good detailed
     description of the content
     domain, something that's not
     always true.
Criterion Validity
• Criteria-related validity,
  you check the performance
  of your operationalization
  against some criterion.
   – Predictive validity: Improve
     performance.
   – Concurrent Validity: correlates to
     other validated tools.
   – Convergent Validity: correlate
     the scores on our test with other
     constructs
   – Discriminant Validity: the
     constructt is not similar to other
      constructs
HumanFIRST Lab



           ng
Virtual Clinic
Gigaheart
Design
            Deliver
            content




            Student
            Learning   Motivate
Content
              Play        &
Frame
                       Engage




             Apply
               &
             Reflect
roles
image
beliefs
language
writing
public perception
                    Are you on your way up?
producing
performing
making the band
How about the cover art? Creating a web presence? Press kits? A
band itinerary? A band identity and branding? Video?

MAKING THE BAND
Rhythm & Flow
       •   High interest
       •   Role Playing
       •   Performance
       •   Technology
       •   Writing
       •   Video
       •   Music
Making the grades
         • Character/
           Characterization
         • diction
         • Plot/ story grammar
         • Setting
         • Point of View
         • Theme
         • Tone
         • Voice
         • Word choice
Modeling
Kiosk
ADL
• The term “activities of daily living” refers to a
  set of common, everyday tasks, performance
  of which is required for personal self-care and
  independent living. The most often used
  measure of functional ability is the Katz
  Activities of Daily Living Scale (Katz et al.,
  1963; Katz, 1983).
     • Wiener, Hanley, Clark, Van Nostrand (1990, pg.1 )
Complex Relationship Building
•   Establish therapeutic relationship with patient
    to promote behavioral change.
1. Identify own attitude toward patient and
    situation.
2. Determine ethical boundaries of the
    relationship.
3. Deal with personal feeling evoked by the patient
    that my interfere with effectiveness
4. Provide for physical comfort before interaction
5. Discuss confidentiality of information shared
6. Create climate of warmth and acceptance
7. Reassure patient of your interest in them as a
    person
8. Return at established time to demonstrate trust
9. Maintain open body posture
10. Monitor, seek clarification & respond to non-
    verbal messages.
      –   10 of 31, NIC revised 5th edition 2008
Facial Action Coding
• The Facial Action       1.
                          2.
                                Anger
                                Disgust
  Coding Systems (FACS)   3.
                          4.
                                Fear
                                Happiness
  – (Ekman and Frisen     5.    sadness
                          6.    surprise
    1978).                7.    Amusement
  – 2 or more raters      8.    Contempt
                          9.    Contentment
                          10.   Embarrassment
                          11.   Excitement
                          12.   Guilt
                          13.   Pride in achievement
                          14.   Relief
                          15.   satisfaction
                          16.   sensory Pleasure
                          17.   shame
Tension in workflow
• Software Design                    • Research Design
  – Typically based upon an            – Typically based upon
    economic consideration.              answering a testable
     • How will this solve a             question.
       problem?                           • How will this solve a problem?
     • What are the first steps in        • How do I know this?
       production?
                                       – The focus is on method and
  – The focus is on stages of            hypothesis testing:
    production:                           • Construct validity, reliability,
     • Business Partner                     reliability, and probability.
       Relations, Function,
       Behavior, Structure, &
       Non-Function (qualities).
Training Development Process
                                                                               <<Design Review>>


              Overall Design
                                                                                          Maintenance
                                                          Beta         Deploy
                                                                                              CPI
            Prototype                  Build
                                                          Stakeholders Signed Off
                                                          Edit OK
                                                          Standards Followed
 Inputs
                                  Review
                     [Changes]




                                 {At least one must be
                                  classroom delivery}



  Charter                                                                    Required Outputs

                                     Assess
Templates
                                      Risk                                 Project Plan
                                                                                          Dashboard



Style Guide                                                                               Build
                                                                             Design
MATRICS
Take home
• Can you pose a testable question– hypothesis?
   – Tension between design process and measurement
      • How will you assure game mechanics are measuring what
        you think you are measuring? Theoretically? Conceptually?
          – Assessments, measures, & evaluations

• Usability testing should align with construct
   – Testing should be happen in development.
• Again, emphasis on validity
   – Without it, there is no capability for ROI analysis
• The G-ScalE Laboratory




                                                   G-ScalE
  – Game Development and Testing
  – Scaling across screen size,
    resolution, and geometry
     • Critical Print Size, Visual Acuity, Prior
       Knowledge, and HCI/UX

Building Serious Games for Medical Intervention and Training

  • 1.
    G-ScalE Brock R. DubbelsPhD. G-ScalE Game Development and Testing Lab McMaster University Dubbels@McMaster.ca
  • 2.
    Games Unit Inquiry Reading comprehension Composition Sustainedengagement Behavioral management Planning Cooperative learning Classroom as game Outcomes Dubbels, B.R. (in press) Video games, reading, and transmedial comprehension. In R. E. Ferdig (Ed.), Handbook of research on effective electronic gaming in education. Information Science Reference.
  • 3.
    How about Mathand Science? Scientific Habits of mind Applied curriculum Modeling Simulation STEM
  • 4.
    Dance Dance Education Becausekids won’t let an education get in the way of their learning
  • 5.
    Games are astructured form of play. What are the elements of this game? What makes the play emergent? Is it non-linear? Games as a metaphor for instructional design
  • 6.
    Invoking play Probability Branching Rules Roles & Identity Imagery & visualization
  • 7.
  • 8.
  • 9.
    What if wemade a game about going to Las Vegas
  • 10.
  • 11.
    And it is. . .
  • 12.
    What do youend up with?
  • 13.
    The Vegas Effect Shouldeverything that happens in games, stay in games? It is not enough to invoke games and play. Serious games should provide evidence that they delivered. This should be quantifiable in performance metrics
  • 14.
    UI Diagram 1. Are you sure .NET isn’t better? I need a new Java 2. Who will the users be? based portal for Emergency rooms. 3. What information will it provide? Direct the conversation Press when ready to move to design Client Analyst Start Values Functions Qualities Design
  • 15.
    Functional Requirements • Playablein < 5 minutes • Engaging enough that target audience chooses to play multiple times • Easy scripting for additional scenarios • Results in demonstrable improvement in requirements elicitation
  • 16.
    VFQ for thisProject • Values – Better understanding of how to apply RSA level 2 – Learners able to “think” RSA • Functions – Graphical • Qualities – Engaging enough that people will choose to play it for 10-15 minutes
  • 17.
    Values, Functions, andQualities • Values – Why a project is needed – Should have a success measure – e.g. Decrease claims processing time • Functions – What is needed – Typically yes/no – e.g. must accept Medicare claims • Qualities – How Well – Should have a success measure – e.g. don’t add to manual adjudication queue
  • 18.
    UI Diagram Press when ready to see what happens Solution/Design 1. Create a new portal Client Analyst Implement 2. Implement using Java 3. Redesign business process Requirements Project 4. … Drag best solutions to SSD below Possible Solutions Solution Summary Values Functions Qualities Document
  • 19.
    UI Diagram This isn’twhat I When will it be wanted. over? Replace with a simplified visual of UDP Show key events during implementation, that result from choices the user made Client Analyst Solution Summary Values Functions Qualities Document
  • 20.
    BP: We needan online method to train people for the United Compliance Project What would Do you have a Have you had you like the timeline in mind? projects similar project to to this in the accomplish? past
  • 21.
    Ratings • Cohen’s Kappa •Fleiss Kappa • Agreement at .8 – Kappa of <0.2 is considered poor agreement, 0.21-0.4 fair, 0.41-0.6 moderate, 0.61- 0.8 strong, and more than 0.8 near complete agreement.
  • 22.
    Learning Games • “Itis worth noting that none of the featured companies started in instructional design…they come from business strategy, marketing, and the games industry”. • “Most game-based learning approaches do not employ that particular category of expert whatsoever” – (Squire, 2005, p. 35).
  • 23.
    Surface (face) &Content Validity • Games are often built on these. – It looks like it measures what it is supposed to measure. – Have checked how the game represents the content against the relevant content domain. – Approach assumes that you have a good detailed description of the content domain, something that's not always true.
  • 24.
    Criterion Validity • Criteria-relatedvalidity, you check the performance of your operationalization against some criterion. – Predictive validity: Improve performance. – Concurrent Validity: correlates to other validated tools. – Convergent Validity: correlate the scores on our test with other constructs – Discriminant Validity: the constructt is not similar to other constructs
  • 25.
  • 26.
  • 30.
  • 31.
    Design Deliver content Student Learning Motivate Content Play & Frame Engage Apply & Reflect
  • 32.
    roles image beliefs language writing public perception Are you on your way up? producing performing making the band
  • 33.
    How about thecover art? Creating a web presence? Press kits? A band itinerary? A band identity and branding? Video? MAKING THE BAND
  • 34.
    Rhythm & Flow • High interest • Role Playing • Performance • Technology • Writing • Video • Music
  • 35.
    Making the grades • Character/ Characterization • diction • Plot/ story grammar • Setting • Point of View • Theme • Tone • Voice • Word choice
  • 37.
  • 38.
  • 39.
    ADL • The term“activities of daily living” refers to a set of common, everyday tasks, performance of which is required for personal self-care and independent living. The most often used measure of functional ability is the Katz Activities of Daily Living Scale (Katz et al., 1963; Katz, 1983). • Wiener, Hanley, Clark, Van Nostrand (1990, pg.1 )
  • 40.
    Complex Relationship Building • Establish therapeutic relationship with patient to promote behavioral change. 1. Identify own attitude toward patient and situation. 2. Determine ethical boundaries of the relationship. 3. Deal with personal feeling evoked by the patient that my interfere with effectiveness 4. Provide for physical comfort before interaction 5. Discuss confidentiality of information shared 6. Create climate of warmth and acceptance 7. Reassure patient of your interest in them as a person 8. Return at established time to demonstrate trust 9. Maintain open body posture 10. Monitor, seek clarification & respond to non- verbal messages. – 10 of 31, NIC revised 5th edition 2008
  • 41.
    Facial Action Coding •The Facial Action 1. 2. Anger Disgust Coding Systems (FACS) 3. 4. Fear Happiness – (Ekman and Frisen 5. sadness 6. surprise 1978). 7. Amusement – 2 or more raters 8. Contempt 9. Contentment 10. Embarrassment 11. Excitement 12. Guilt 13. Pride in achievement 14. Relief 15. satisfaction 16. sensory Pleasure 17. shame
  • 42.
    Tension in workflow •Software Design • Research Design – Typically based upon an – Typically based upon economic consideration. answering a testable • How will this solve a question. problem? • How will this solve a problem? • What are the first steps in • How do I know this? production? – The focus is on method and – The focus is on stages of hypothesis testing: production: • Construct validity, reliability, • Business Partner reliability, and probability. Relations, Function, Behavior, Structure, & Non-Function (qualities).
  • 43.
    Training Development Process <<Design Review>> Overall Design Maintenance Beta Deploy CPI Prototype Build  Stakeholders Signed Off  Edit OK  Standards Followed Inputs Review [Changes] {At least one must be classroom delivery} Charter Required Outputs Assess Templates Risk Project Plan Dashboard Style Guide Build Design
  • 44.
  • 45.
    Take home • Canyou pose a testable question– hypothesis? – Tension between design process and measurement • How will you assure game mechanics are measuring what you think you are measuring? Theoretically? Conceptually? – Assessments, measures, & evaluations • Usability testing should align with construct – Testing should be happen in development. • Again, emphasis on validity – Without it, there is no capability for ROI analysis
  • 46.
    • The G-ScalELaboratory G-ScalE – Game Development and Testing – Scaling across screen size, resolution, and geometry • Critical Print Size, Visual Acuity, Prior Knowledge, and HCI/UX

Editor's Notes

  • #23 Squire 20005 conducted case studies of three companies that develop game-based learning products.
  • #24 Face -- you might observe a teenage pregnancy prevention program and conclude that, &quot;Yep, this is indeed a teenage pregnancy prevention program.&quot; Of course, if this is all you do to assess face validity, it would clearly be weak evidence because it is essentially a subjective judgment call.Content – the domain is drawn from observations—is this good enough for scientific results? ROI?
  • #25 How is this different from content validity? In content validity, the criteria are the construct definition itself -- it is a direct comparison. In criterion-related validity, we usually make a prediction about how the operationalization will perform based on our theory of the construct. The differences among the different criterion-related validity types is in the criteria they use as the standard for judgment.
  • #37 In our own work we developed for a number of criterion tools. This talk will examine Activities of Daily Living Explicitly