Serious Gaming Research
© Dr. Igor Mayer i.s.mayer@tudelft.nl
Faculty Technology, Policy and Management (TPM) & Signature ...
Challenge the future

2
References (1)
1. Mayer, I. S., Bekebrede, G., Harteveld, C., Warmelink, H. J. G., Zhou, Q., van Ruijven, T., … Wenzler, I...
References (2)
1. Mayer, I. S., van Dierendonck, D., van Ruijven, T., & Wenzler, I. (2013). Stealth Assessment of Teams in...
Review articles
1. Papastergiou, M. (2009). Exploring the potential of computer and video games for health and physical
ed...
Towards a Science of SG
(SoSG)?

Frames and
discourses

Methodology

Research designs
and data-gathering

Validated resear...
Game theory versus gaming
Game theory

Gaming

Rigid rule-based, closed

Free form, seminar, open

Formal, mathematical, q...
Framing game related
research
Frame

Example

Research theory

Game theory as in economics, political science

Research co...
Strong/weak aspects of gaming as a
research method
Strong aspects

Weak aspects

Flexible, adaptable, multi-purpose,

Time...
Requirements for SG Meth.
Multipurpose
d

Broad in
scope
Compar
ative

Fast
and
nontime
consumi
ng

Standar
dized

Require...
Underlying research questions
What is the relation between
(virtual) gaming and (professional)
learning in socio-technical...
Group
intervie
ws focus
groups

Scenario
analysis

Q
methodo
logy

(particip
atory)
Observa
tion

Docume
nt
analysis

Inte...
Shortcomings
Few indications how to use the models, for what purpose, with what scope and under what conditions.
Few proce...
Research framework
1. Domain application:
infrastructures

1.1 Water, Rail,
Ports, Energy,
Tunnels, etc...

Framing

Resea...
Research design

MSP Challenge 2011, TU-Delft, ministry I&M, ICES, OSPAR, HELCOM, VASPAP

When?

Pre-game

Observation
num...
Quasi experimental design
(Mayer, et al. under review EJEE, BJET, 2012)

O1 =
O1 =
X1 =
O3 =
Ox1...n =
O4 =

Intervention ...
Measuring indicators

MSP Challenge 2011, TU-Delft, ministry I&M, ICES, OSPAR, HELCOM, VASPAP

Quantitative

Qualitative

...
•

References

•

Mayer, I. S., Warmelink, H., & Bekebrede, G.
(n.d.).

Learning

in

a

Game-based

Virtual

Environment:...
Structural equation modelling
(Mayer, et al. under review EJEE, 2012)
.32

MOTIVATION

Would like to
play other SGs
in edu...
Upcoming SlideShare
Loading in …5
×

Serious Gaming: Research

1,526 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,526
On SlideShare
0
From Embeds
0
Number of Embeds
12
Actions
Shares
0
Downloads
41
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Serious Gaming: Research

  1. 1. Serious Gaming Research © Dr. Igor Mayer i.s.mayer@tudelft.nl Faculty Technology, Policy and Management (TPM) & Signature Games Delft University of technology, The Netherlands Challenge the future 1
  2. 2. Challenge the future 2
  3. 3. References (1) 1. Mayer, I. S., Bekebrede, G., Harteveld, C., Warmelink, H. J. G., Zhou, Q., van Ruijven, T., … Wenzler, I. (2013). The research and evaluation of serious games: Toward a comprehensive methodology. British Journal of Educational Technology, n/a–n/a. doi:10.1111/bjet.12067 2. Mayer, I. S., Bekebrede, G., Warmelink, H. J. G., & Zhou, Q. (2013). A Brief Methodology for Researching and Evaluating Serious Games and Game-Based Learning. In T. M. Connolly, L. Boyle, T. Hainey, G. Baxter, & P. Moreno-Ger (Eds.), Psychology, Pedagogy and Assessment in Serious Games (in press) (pp. 357–393). IGI Global. doi:10.4018/978-1-46664773-2.ch017 3. Mayer, I. S., Kortmann, R., Wenzler, I., Wetters, Á., & Johan, S. (2014). Game-based Entrepreneurship Education: Identifying Enterprising Personality, Motivation and Intentions amongst Engineering Students. International Journal of Entrepreneurship Education (in press). Challenge the future 3
  4. 4. References (2) 1. Mayer, I. S., van Dierendonck, D., van Ruijven, T., & Wenzler, I. (2013). Stealth Assessment of Teams in a Digital Game Environment. In GALA 2013 Conference, Paris (pp. 1–13). Paris, France: Springer. 2. Mayer, I. S., Warmelink, H. J. G., & Bekebrede, G. (2013). Learning in a game-based virtual environment: a comparative evaluation in higher education. European Journal of Engineering Education, 38(1), 85–106. doi:10.1080/03043797.2012.742872 3. Mayer, I. S., Warmelink, H. J. G., & Zhou, Q. (2014). The Utility of Games for Society, Business and Politics: A Frame Reflective Analysis. In Nick Rushby & D. Surry (Eds.), Wiley Handbook of Learning Technology (in press). Wiley. 4. Mayer, I. S., Wolff, A., & Wenzler, I. (2013). Learning Efficacy of the “Hazard Recognition” Serious Game: A Quasi Experimental Study. In M. Ma, M. F. Oliveira, S. Petersen, & J. Baalsrud Hauge (Eds.), 4th International Conference, SGDA 2013, Trondheim, Norway, September 25-27, 2013. Proceedings (pp. 118–129). Berlin Heidelberg: Springer Verlag. doi:10.1007/978-3-642-40790-1_12 Challenge the future 4
  5. 5. Review articles 1. Papastergiou, M. (2009). Exploring the potential of computer and video games for health and physical education: A literature review. Computers & Education, 53(3), 603–622. doi:10.1016/j.compedu.2009.04.001 2. Connolly, T. M., Boyle, E. A., MacArthur, E., Hainey, T., & Boyle, J. M. (2012). A systematic literature review of empirical evidence on computer games and serious games. Computers & Education, 59(2), 661–686. doi:10.1016/j.compedu.2012.03.004 3. Vogel, J. J., Vogel, D. S., Cannon-Bowers, J. A., Bowers, C., Muse, K., & Wright, M. (2006). Computer Gaming and Interactive Simulations for Learning: a Meta-Analysis. Journal of Educational Computing Research, 34(3), 229–243. doi:10.2190/FLHV-K4WA-WPVQ-H0YM 4. Gosen, J., & Washbush, J. (2004). A Review of Scholarship on Assessing Experiential Learning Effectiveness. Simulation & Gaming, 35(2), 270–293. doi:10.1177/1046878104263544 Challenge the future 5
  6. 6. Towards a Science of SG (SoSG)? Frames and discourses Methodology Research designs and data-gathering Validated research instruments and tools A dynamic body of knowledge identifying the state of the art and knowledge gaps. Professional ethics of the SG designer, the SG advocate, the SG seller, the SG interventionist, etc. Challenge the future 6
  7. 7. Game theory versus gaming Game theory Gaming Rigid rule-based, closed Free form, seminar, open Formal, mathematical, quantifiable, economics, Informal, social, interpretative and qualitative, psychology social, political intervention sciences Experimental control, objective, separation Semi- or non experimental, subjective, interaction researcher and subjects, large „ n‟ , player cannot between researcher and subjects, small „ n‟ ; change the rules or setting players should change the rules or setting Theory-based, hypothesis testing, generalization, Theory construction, exploratory, hypothesis prediction and forecasting formulation, constructing the future „ Players‟ are research objects; „Players‟ are learning subjects; they bring with them tacit knowledge, social relations etc. Challenge the future 7
  8. 8. Framing game related research Frame Example Research theory Game theory as in economics, political science Research concept Political decision-making as a strategic game Research object Studying game cultures, game economics, game politics Design artifact Socio-technical design etc. Research method Quasi-experiment Intervention method Therapy, learning, change or decision-method Data-gathering method Observation, group interview, data-modeling Challenge the future 8
  9. 9. Strong/weak aspects of gaming as a research method Strong aspects Weak aspects Flexible, adaptable, multi-purpose, Time and resource consuming. Difficult to complementary manage. No guarantee for success. Multi-, interdisciplinary, innovative, appealing Problematic legitimization as „science‟ Research in action, interaction with real Dependency on others (financers, players, stakeholder, problem owners, users, etc. stakeholders, designers). Deliverable by or end product. Game takes over the research Concrete, practical, tangible, fun A lot of fuzz that is not research … … Challenge the future 9
  10. 10. Requirements for SG Meth. Multipurpose d Broad in scope Compar ative Fast and nontime consumi ng Standar dized Requirements Meth. SG Research Unobtru sive Specific Expand able Flexible Validate d Triangul ated Multileveled Challenge the future 10
  11. 11. Underlying research questions What is the relation between (virtual) gaming and (professional) learning in socio-technical systems / multi-actor contexts? Design-oriented research (artifact): ‘making it (better)’ Policy, management oriented research (intervention): ‘making it work’ Domain-oriented research (energy, rail, water, ports, tunnels etc.): ‘making it matter’ Scientific game research: ‘making it understandable’ • (How) do (virtual) gaming experiences (a)(e)ffect learning of professionals in sociotechnical systems / multi-actor contexts? • (How) does (virtual) game-based learning (a)(e)ffect Real World (RW) policy-making in socio-technical systems / multi-actor contexts? • Development and testing of design and validation theories, methods & tools • Validation studies of specific and generic game-based artifacts and events. • Development and testing of game evaluation, measurement and feedback theories, methods & tools • Studies into the learning effectiveness of game-based interventions • Studies into the transfer game-based interventions to the RW • Studies into the design and use of SG as a research instrument, e.g. e.g. quasi experimental, questionnaires, video observation, Q-method, etc. • Studies using game based research in infrastructures. • Studies on SG in a cultural, organizational, political, economic context • Theory construction on serious gaming „as‟ / „in‟ socio-technical / multi-actor systems Challenge the future 11
  12. 12. Group intervie ws focus groups Scenario analysis Q methodo logy (particip atory) Observa tion Docume nt analysis Inter views Case study Participatory research methods Serious gaming Path tracking Crowd sourcing Computer simulation Lage schale data analysis Video observat ion Experimental research Bio & psychoMetric tests Panel research Surveys Quasi experim ental research Eye tracking Mayer, I. S., Warmelink, H., & Bekebrede, G. (n.d.). Learning in a Game-based Virtual Environment: a Comparative Evaluation in Higher Education. European Journal of Engineering Education. Challenge the future 12
  13. 13. Shortcomings Few indications how to use the models, for what purpose, with what scope and under what conditions. Few procedures how to validate the conceptual research / evaluation model. Few research hypothesis and research designs. Few definitions, relations and interrelations between the concepts in the model. Few operationalization and validation of constructs. Furthermore in the application of the models we see: A dominance of single case studies, one game, one context of application. Lacking information on the questionnaires used. A focus on GBL of children in formal education; little attention to advanced-professional learning, outside education; A focus on learning of individuals in formal training or educational context; little attention to learning of teams, groups, organizations, networks or systems in policy or organizational context; Challenge the future 13
  14. 14. Research framework 1. Domain application: infrastructures 1.1 Water, Rail, Ports, Energy, Tunnels, etc... Framing Research question 1 Sub question 4 Sub question 5 m Fra 2.1 Complex adaptive systems, resilience, integrated planning, self-organization, sense-making, etc... 2. Theory: Complex, multi actor systems and policy making. ing sub question 6 CaseExperiments, e.g. Levee Patroller, SimPort, Water Game, Research question 3 Research question 2 Fra m ing 3.1 Design and evaluation of (Serious) Gaming-Simulation 3. Method: Modeling, simulation & gaming (MSG) Challenge the future 14
  15. 15. Research design MSP Challenge 2011, TU-Delft, ministry I&M, ICES, OSPAR, HELCOM, VASPAP When? Pre-game Observation number O1 How? Online survey What? Soc. Dem. Involvement in MSP In-game O2 O3 Paper quest. Paper quest. O4 Paper quest. Post-game O5 O6 End of game debriefing Online survey Analysis of maps MSP process MSP process Influence After action review 41 41 Knowledge in MSP Influence in MSP MSP in country Valid response Additional data gathering Game play Emotions 63 50 40 38 Video registration – Observation – Data logging Challenge the future 15
  16. 16. Quasi experimental design (Mayer, et al. under review EJEE, BJET, 2012) O1 = O1 = X1 = O3 = Ox1...n = O4 = Intervention process Case study O4 Learning proces In game data logging O3 X O2 Transfer process Observation O1 Serious Gaming session Ox1 Ox2 Ox3 Oxn Observation 1 Learning objectives, context Observation 2 pre-game measurement Intervention = gaming session Observation 3 post game measurement In game observations / measurements Observation 4 Learning objectives, context Time Challenge the future 16
  17. 17. Measuring indicators MSP Challenge 2011, TU-Delft, ministry I&M, ICES, OSPAR, HELCOM, VASPAP Quantitative Qualitative The level of engagement of the players in the game. The influence attributed to each Self-reported by the participants stakeholder, by other stakeholders. The quality of each of the four MSPs as The lessons and insights on the process and outcome reported during and after the game. assessed by each stakeholder on ten criteria. The observations on how the MSP process goes, by the game The square nautical miles assigned to Observed by the facilitators facilitators. the different spatial functions in the The quality of each of the four MSPs. four MSPs. The overlap, conflicts, internal contradictions etc. between different spatial functions in or between MSPs. Challenge the future 17
  18. 18. • References • Mayer, I. S., Warmelink, H., & Bekebrede, G. (n.d.). Learning in a Game-based Virtual Environment: a Comparative Evaluation in Higher Education. European Journal of Engineering Education. • Mayer, I. S. (2012). Towards a Comprehensive Methodology for the Research and Evaluation of Serious Games. Vs games (pp. 1–15). Genoa: Procedia Computer Science 00 (2012) 000–000. Retrieved from www.sciencedirect.com Challenge the future 18
  19. 19. Structural equation modelling (Mayer, et al. under review EJEE, 2012) .32 MOTIVATION Would like to play other SGs in education. Attitude GBL .62 EXPSERGAME_PRE .22 .18 Attitude GBL .2* EXPSERGAME_POST .78 .42 Engagement .28 .34 FUN .36 Learning expectations ACHIEVEMENT Quality VLE SOCSKILLS_PRE .2 .12* .34 .17* .18* Attitude VLE VALUEDL .13 PROFSKILLS_PRE .22 .2 (t-test) .24 .25 .14 SEX .18 .1 -.17 FREQ. DIGITAL .16 .12* GAMES Learning style preferences .17* CYBIMPRESS; .17 EXPKNOW_PRE .24 .26 .28* .40 RELGAME .39 .48 PROFSKILLS_POST .27 .27 .39 .38 .46 .39 EXPKNOW_POST .58 LRNPREF 0.11 .3 FREQ NON-DIGITAL GAMES .1 FREQ SG .53 .43 Quality facilitator QUAFACIL. EFFORTPLAY .44 .46 .41 Quality game design QUAGAME; Game play QUAPLAY; SOCSKILLS_POST .26 CYBCONTENT .51 .19 .24 .27 1.1 (t-test) .34 CYBACCESS; .38 .42 AGE CYBFUNCTION; CYNCLEARNESS; .45 .16* -.12 Learning satisfaction Q game Challenge the future 19

×