On Evaluating The 80 Days Geography Game For School Students Mahmoud Abdulwahed

Pre Evaluation Understand the utilized pedagogical theories. Understand the utilized cognitive theories, i.e. the CbKST. Understand the game architecture, in particular: The adaptive engine The Narration engine

Understand the utilized pedagogical theories.

Understand the utilized cognitive theories, i.e. the CbKST.

Understand the game architecture, in particular:

The adaptive engine

The Narration engine

Pedagogical Usability Cognitive Affective Economy Others: such as Neuroscience, the gaming factor, etc. The DEGs’ Evaluation Dimensions

Pedagogical

Usability

Cognitive

Affective

Economy

Others: such as Neuroscience, the gaming factor, etc.

The DEGs Learning Outcomes Intended learning outcomes, such as: Countries capitals Demographic information Climate states and any such intended geography learning outcomes by the school curricula Occasional, or un intended learning outcomes (nevertheless its still learning !), such as: Organizational skills Decision making skills Strategic skills and any such generic learning outcomes as a result of playing digital games Two Types of Learning Outcomes are Associated with DEGs

Intended learning outcomes, such as:

Countries capitals

Demographic information

Climate states

and any such intended geography learning outcomes by the school curricula

Occasional, or un intended learning outcomes (nevertheless its still learning !), such as:

Organizational skills

Decision making skills

Strategic skills

and any such generic learning outcomes as a result of playing digital games

Pedagogical Variables of DGs Generic Learning Types Associated with Digital Games (Prensky 2001)

Some Important Pedagogical Measures Depth of the gained knowledge, i.e. deep vs. surface learning. Sustainability of learning (Direct impact on life-long learning), i.e. do students seek to learn more about geography outside the game environment as a result of DEG based learning ? Breadth of learning outcome for DEGs vs. Classical Pedagogies. Time spent of the learning objectives. Learning styles change? i.e. does it increase the experiential learning style? Social learning: Proximal, i.e. is the DEG an optimal platform for implementing effective social learning experience for students in the classroom or the lab? Cyber, i.e. online collaborative learning with DEG through the web

Depth of the gained knowledge, i.e. deep vs. surface learning.

Sustainability of learning (Direct impact on life-long learning), i.e. do students seek to learn more about geography outside the game environment as a result of DEG based learning ?

Breadth of learning outcome for DEGs vs. Classical Pedagogies.

Time spent of the learning objectives.

Learning styles change? i.e. does it increase the experiential learning style?

Social learning:

Proximal, i.e. is the DEG an optimal platform for implementing effective social learning experience for students in the classroom or the lab?

Cyber, i.e. online collaborative learning with DEG through the web

Some Proposed Methods and Tools Analysing the game logged data Questionnaire Pre and Post tests Comparative evaluation, i.e. classical teaching vs. DEG based teaching. Learning styles inventories. Interviews Focus groups And many others

Analysing the game logged data

Questionnaire

Pre and Post tests

Comparative evaluation, i.e. classical teaching vs. DEG based teaching.

Learning styles inventories.

Interviews

Focus groups

And many others

Usability Problems of Digital Games Unpredictable / inconsistent response to user’s actions Does not allow enough customization Artificial intelligence problems Mismatch between camera/view and action Does not let user skip non-playable content Clumsy input scheme Difficult to control actions in the game Does not provide enough information on game status Does not provide adequate training and help Command sequences are too complex Visual representations are difficult to interpret Response to user’s action not timely enough Generic List of DGs Usability Problems (Pinelle et al 2008):

Unpredictable / inconsistent response to user’s actions

Does not allow enough customization

Artificial intelligence problems

Mismatch between camera/view and action

Does not let user skip non-playable content

Clumsy input scheme

Difficult to control actions in the game

Does not provide enough information on game status

Does not provide adequate training and help

Command sequences are too complex

Visual representations are difficult to interpret

Response to user’s action not timely enough

Setting up The Usability Problems of DEGs Many usability problems of the generic DGs usability problems may apply for DEGs. Build upon the educational usability findings of educational software research. Extension towards the notion of DEGs Educational Usability will be necessary when evaluating DEGs usability, i.e. the combination of the learning elements make the game boring?, knowledge is difficult to navigate or retrieve?, schools does not have enough IT equipments for running the 3D games (high quality graphic cards)?, too much time needed to learn basic knowledge? etc.

Many usability problems of the generic DGs usability problems may apply for DEGs.

Build upon the educational usability findings of educational software research.

Extension towards the notion of DEGs Educational Usability will be necessary when evaluating DEGs usability, i.e. the combination of the learning elements make the game boring?, knowledge is difficult to navigate or retrieve?, schools does not have enough IT equipments for running the 3D games (high quality graphic cards)?, too much time needed to learn basic knowledge? etc.

Usability Evaluation Methods Coaching Method Co-discovery Learning Performance Measurement Question-asking Protocol Remote Testing Retrospective Testing Shadowing Method Teaching Method Thinking Aloud Protocol Cognitive Walkthroughs Feature Inspection Heuristic Evaluation Pluralistic Walkthrough Perspective-based Inspection Focus Groups Interviews Logging Actual Use Questionnaires (QUIS, PUEU , NAU NHE, CSUQ, ASQ, PHUE, PUTQ, USE ) From http://www.usabilityhome.com/

Coaching Method

Co-discovery Learning

Performance Measurement

Question-asking Protocol

Remote Testing

Retrospective Testing

Shadowing Method

Teaching Method

Thinking Aloud Protocol

Cognitive Walkthroughs

Feature Inspection

Heuristic Evaluation

Pluralistic Walkthrough

Perspective-based Inspection

Focus Groups

Interviews

Logging Actual Use

Questionnaires (QUIS, PUEU , NAU

NHE, CSUQ, ASQ, PHUE, PUTQ, USE )

Usability Evaluation of DEGs Methodology: The Adoption, Adaptation, and Extension Principle

Cognitive and Affective Variables Gender issues of the game perception? Cognitive overload? The content dimension The spent time dimension Information retaining: Short term. Long term. Like or dislike Engagement Addiction? Self esteem Preference in comparison to commercial educational games during leisure times, i.e. is it alternative? Supplement? or no preference at all? Confidence in geography Motivation towards schooling

Gender issues of the game perception?

Cognitive overload?

The content dimension

The spent time dimension

Information retaining:

Short term.

Long term.

Like or dislike

Engagement

Addiction?

Self esteem

Preference in comparison to commercial educational games during leisure times, i.e. is it alternative? Supplement? or no preference at all?

Confidence in geography

Motivation towards schooling

Summary: Evaluating, How To? Hybrid Evaluation Methods Data Analysis of the Game. Pedagogical measurements methods and instruments, i.e. the learning style inventory LSI. Cognitive measurements instruments, i.e. memory tests, mind maps, IQ tests, etc. Quantitative methods. Qualitative methods. Usability instruments. Suggested Tools: Comparative Evaluations X X Treatment Pre Post Experimental Group Equivalent groups Different Outcome? Control Group Y Yt

Data Analysis of the Game.

Pedagogical measurements methods and instruments, i.e. the learning style inventory LSI.

Cognitive measurements instruments, i.e. memory tests, mind maps, IQ tests, etc.

Quantitative methods.

Qualitative methods.

Usability instruments.

Conclusions Objectives based evaluation. i.e. identifying the objectives of the game on all perspectives and evaluating whether they met or not and to which extent. In-Game and Out Game evaluation. Surveying the potential current tools and models of evaluation (pedagogical, usability, cognitive, etc)  Adaptation  Extension. Hence, developing Novel Models. Inventing new models from scratch when needed. Comparative evaluations when possible. Evaluation during the design process phase, and final product evaluation. Hybrid evaluation approach, i.e. quantitative and qualitative. Embedding more science in explaining the evaluation findings, i.e. relating the findings to the pedagogical theories and cognitive theories, explaining from systems dynamics and game theory perspectives. There is a lot of objectives and variables to be evaluated; load of evaluation experiments, methodologies and models that can be designed or used  comprehensive work, BUT , a lot of high quality publications!. Robust and Holistic evaluation models would be developed after having more insight on the project, contacting with the partners, and getting closer view of the associated multidisciplinary domains.

Objectives based evaluation. i.e. identifying the objectives of the game on all perspectives and evaluating whether they met or not and to which extent.

In-Game and Out Game evaluation.

Surveying the potential current tools and models of evaluation (pedagogical, usability, cognitive, etc)  Adaptation  Extension. Hence, developing Novel Models.

Inventing new models from scratch when needed.

Comparative evaluations when possible.

Evaluation during the design process phase, and final product evaluation.

Hybrid evaluation approach, i.e. quantitative and qualitative.

Embedding more science in explaining the evaluation findings, i.e. relating the findings to the pedagogical theories and cognitive theories, explaining from systems dynamics and game theory perspectives.

There is a lot of objectives and variables to be evaluated; load of evaluation experiments, methodologies and models that can be designed or used  comprehensive work, BUT , a lot of high quality publications!.

Robust and Holistic evaluation models would be developed after having more insight on the project, contacting with the partners, and getting closer view of the associated multidisciplinary domains.