Presentation of my paper that examines the impact of an pedagogical agent (PA), in a 3D collaborative educational game in the virtual environment of OpenSim platform. The main goal of this paper is to evaluate the impact of a pedagogical agent to 3D collaborative game based learning by developing an agent that interacts with the students both at team and personal level. In order to enhance the already implemented 3D game functions, a pedagogical agent that supports students during the game learning process was implemented. The evaluation approach attempts to investigate the impact of PAs in 3D collaborative educational games focusing on two dimensions: a) on PAs usefulness in collaborative 3D games and b) on students' teams performance in the game learning process. This research concludes that the intervention of pedagogical agent's assistance in the game has a strong positive impact on teams performance and can be considered as useful in collaborative serious games.

1. Aristotle University of Thessaloniki – Department of Informatics
Greece
EDUCON 2014 - Istanbul, Turkey
The Impact of Pedagogical Agents in 3D
Collaborative Serious Games
Theodouli Terzidou & Thrasyvoulos Tsiatsos

2. Overview
 Research goal
 Theoretical Background
 The Agent’s environment - OSgame
 Design & Implementation of Aristotle Pedagogical
Game Agent (PGA)
 Evaluation
 Results and Conclusions
EDUCON 2014 - Istanbul, Turkey Terzidou T. and Tsiatsos T.2

3. Pedagogical Agents: Definitions [1/3]
 Pedagogical Agent (PA)
A Pedagogical Agent is an intelligent agent who
supports variously educational procedures, taking
place in synchronous learning environments aiming to
achieve better learning; educational procedures
comprise students' or/and teacher's role's relevancy.
 Collaborative Pedagogical Agent (CPA)
A Pedagogical agent that aims to foster by any means
collaboration among students.
[Terzidou & Tsiatsos 2014]
EDUCON 2014 - Istanbul, Turkey Terzidou T. and Tsiatsos T.3

4. Pedagogical agents, according to their tasks can be
classified as:
 Tutors
 Mentors
 Coaches
 Motivators
 Assistance
 Learning companions
 Co-learners
 Collaborators
 Competitors
 Trouble Makers
Pedagogical Agents: Classification [2/3]
EDUCON 2014 - Istanbul, Turkey Terzidou T. and Tsiatsos T.4

5. Pedagogical Agents: Roles in CVEs [3/3]
 Monitoring collaborative learning process
 Feedback and guidance to activate interaction and
collaboration among participants
 Information on the current state of a learner’s
interaction in the collaborative learning process
 Advice on the learning process according to the
process and strategy of collaborative learning by
comparing the current and ideal states.
[Hmelo-Silver, 2002; Soller et al, 2004]
EDUCON 2014 - Istanbul, Turkey Terzidou T. and Tsiatsos T.5

6. Research Questions
RQ1
• Do students perceive pedagogical agents
as useful in collaborative 3D games?
RQ2
• Does the agent's support have an impact
in the performance of a team?
EDUCON 2014 - Istanbul, Turkey Terzidou T. and Tsiatsos T.6

7. The Aristotle Pedagogical Game Agent
IS
a Model-based reflex agent
DESIGNED
for the 3D Collaborative
OSgame
ACTS
in an OpenSim World
INTERACTS
with the students both at
team and personal level
Aristotle
PGA
EDUCON 2014 - Istanbul, Turkey Terzidou T. and Tsiatsos T.7

8. The Aristotle Pedagogical Game Agent
 The PGA senses the game environment.
 The PGA has an internal model related to the OSgame
rules.
 The PGA acts-reacts according to:
 Current game state
 Students’ Teams game progress
 Students’ actions
 Keeps track of students’ teams performance (score).
EDUCON 2014 - Istanbul, Turkey Terzidou T. and Tsiatsos T.8

9. OSgame Description
 Osgame is a 3D collaborative knowledge game.
 Students play in teams of three or four.
 Team members need to collaborate and develop strategy.
 Aim: find the hidden questions and answer them correctly.
 Questions are in form of multiple choice based on the
course syllabus, randomly hidden in virtual objects.
 Teams have to discover the virtual objects containing the
questions.
 There is a specific time limit.
 Correct answer gets 1 point.
 The team with the highest score wins!
Players can check teams' progress and score through a user interface HUD.
EDUCON 2014 - Istanbul, Turkey Terzidou T. and Tsiatsos T.9

10. A look to the agent’s world…
The OSgame
EDUCON 2014 - Istanbul, Turkey Terzidou T. and Tsiatsos T.10

11. Agent’s Design [1/2]
OSgame
Characteristics
Agent's
Requirements
Decision/ functionality
Agent has to...
Players act in time
pressure
discreet, accurate and not
wordy
 interact with text messages
 provide short and precise messages
Limited resources
regarding the 3D
environment
non-system-resource-
intensive
 2D implementation
 non animated
 use local and IM chat
Players interact
with 3D objects
observe players interactions  Sense 3D environment through virtual objects.
 Identify problematic situations
It is a
collaborative
game
observe players collaboration  motivate students to collaborate
 encourage students
 monitor game status
Lack of online
help for the game.
The agent has to provide
information about game rules
 provide feedback and awareness
 offer a complementary interface to visualize
game status (Progress Map).
EDUCON 2014 - Istanbul, Turkey Terzidou T. and Tsiatsos T.11

12. Design: Agent’s Goals [2/2]
 Motivate students to collaborate
 Reinforce competition
 Encourage involvement
 Trigger interaction between players
 Reward players
 Remind of aim and game rules
 Provide help and awareness about game
 Sense 3D environment through virtual objects
EDUCON 2014 - Istanbul, Turkey Terzidou T. and Tsiatsos T.12

13. Implementation [1/2]
The agent is implemented in the OpenSim platform using the LSL Scripting
language.
 Stores real time data in a MySQL database.
 Senses his environment through virtual objects.
EDUCON 2014 - Istanbul, Turkey Terzidou T. and Tsiatsos T.13
OpenSim Platform
Game
Script
(LSL)
Game
Agent
(LSL)
Students’ teams
with
OpenSim Viewer

14. Implementation [2/2]
The agent is implemented in the OpenSim platform using the LSL
Scripting language and...
 Notifies the players of any change in teams' ranking
 Informs the top two ranked teams for any change in their rating
classification
 Checks if player is inactive (motionless) and sends him/her a message.
 Encourage players to chat more with their team members (if they were
inactive for a specific time)
 Rewards players that achieved specific percentages of correct answers.
 Draws attention to continuous incorrect responses and assistance
 Alerts for impending expiry of the game
 Checks repeated clicks and sends message.
 Shows a Progress-Map and Highlights the positions of the answered
questions.
EDUCON 2014 - Istanbul, Turkey Terzidou T. and Tsiatsos T.14

15. Agent
Intelligent OSgame Interface
Progress Map
EDUCON 2014 - Istanbul, Turkey Terzidou T. and Tsiatsos T.15

16. The Case Study:
Evaluating Collaborative Pedagogical Game Agent in OSgame
 37 undergraduate
Students [13 females and 24
males]
 attending the “Educational
Environments” course, of
the Informatics Department
in our University, were
 randomly divided into
Control and Experimental
groups
EDUCON 2014 - Istanbul, Turkey Terzidou T. and Tsiatsos T.16

17. Phases Actions
in/for OSgame
Evaluation
(data collection
method)
Phase 1 Before OSgame
Create OSgame teams, familiarization
with the virtual environment. Pre-evaluation
questionnaire
Form control and experimental groups
Phase 2 During OSgame
Collaboration among the members of each
game team to achieve game goal.
Database records
Observation
Text chat log files
Without PA
intervention
Control Group
With PA
intervention
Experimental
Group
Phase 3 After OSgame
Post-evaluation
questionnaire
Evaluation: Methodology [1/2]
EDUCON 2014 - Istanbul, Turkey Terzidou T. and Tsiatsos T.17

18. Evaluation: Data collection methods [2/2]
 Questionnaires (were in most based on the
literature investigation)
 Pre-evaluation questionnaire [102 questions divided into 9 question
groups]
 Post-evaluation questionnaire [+27 new questions and +6 new
question groups for the experimental group].
 Log files
 Agent’s database [MySQL Database]
 Chat logs
 In-world observation
[Sisson, 2011; Fraser; Compeau and Higgins, 1995; Wua, et al., 2010, Davis, 1989; Chin et
al., 1988]
EDUCON 2014 - Istanbul, Turkey Terzidou T. and Tsiatsos T.18

19. Results: Pre Evaluation [1/3]
Mean
Std.
Deviation
Technology familiarization 3,98 0,56
Attitude towards course 3,84 0,39
Attitude towards games 4,12 0,47
Self-efficacy 3D 3,40 0,45
Self-efficacy VLE 3,26 0,45
Self-efficacy Internet 3,56 0,52
Perceived agent's usefulness 3,97 0,49
What is happening in class 3,15 0,54
PRE-EVALUATION DESCRIPTIVE STATISTICS
 A quite high
technologically
familiarized sample (as
expected by Informatics
students).
 Highly motivated
towards games and with
a positive attitude
towards the specific
course.
 Consider the assumed
existence of a
pedagogical agent as
useful. EDUCON 2014 - Istanbul, Turkey Terzidou T. and Tsiatsos T.19

20. Results: Post Evaluation [2/3]
Subgroups Mean Std. Deviation
Attitude towards Course 3,87 0,49
Value Course 3,91 0,54
Enjoyment Course 3,84 0,48
Motivation Course 3,89 0,58
Attitude towards Games 4,12 0,51
Value Game 4,07 0,54
Enjoyment Game 4,21 0,53
Motivation Game 4,03 0,58
What Is Happening In Class 3,26 0,56
Coherence 3,43 0,66
Collaboration 3,43 0,72
Involvement 2,92 0,72
Self Efficacy 3D 3,48 0,45
Self Efficacy VLE 3,34 0,40
Self Efficacy Internet 3,53 0,43
Perceived Usefulness of Agent 3,86 0,58
Agent's Pedagogical Value 3,43 0,78
Agent's Message Perception 3,51 0,58
Agent's Support Perception 3,34 0,97
EDUCON 2014 - Istanbul, Turkey Terzidou T. and Tsiatsos T.20

21. Results: Agent’s technical attributes [Experimental group]
[3/3]
 The technical attributes of the agent were quite high
evaluated (all mean values > 3,5).
Mean
Std.
Deviation
User Interface Satisfaction Agent 3,41 0,65
User Interface Satisfaction Progress Map 3,46 0,76
Screens Agent 3,75 0,81
Terminology Agent 3,78 0,65
Learning Agent 3,58 0,91
Capabilities Agent 3,56 0,77
Descriptive Statistics for Agent’s Variables - Post-evaluation
phase
EDUCON 2014 - Istanbul, Turkey Terzidou T. and Tsiatsos T.21

22. What is Happening During the Osgame?
[Experimental group]
Agent's messages and
game score for Orange Team
Agent's messages and
game score for Blue Team
EDUCON 2014 - Istanbul, Turkey Terzidou T. and Tsiatsos T.22

23. Results synopsis
 Independent-samples t-test indicated a statistically significant
difference between control and experimental group for the Perceived
Usefulness of Agent.
 This outcome implies the effect on pedagogical agent's usefulness in
the OSgame.
 Database analysis revealed a strong linear relationship (R2 = 0,9764)
between the number of messages delivered by the agent and the
teams' score.
 Agent’s messages were positively perceived from the students.
Team Score versus Agent's
messages for Blue Team
EDUCON 2014 - Istanbul, Turkey Terzidou T. and Tsiatsos T.23

24. RQ1: Do students perceive pedagogical agents as useful in
collaborative 3D games?
 Students belonging to the experimental group did not seem to consider
additional support necessary.
The statistically significant difference for the perceived pedagogical agent's
usefulness between control and experimental group is due to the decline in
experimental group's mean and may be interpreted in two different ways:
 Students did not consider the presence of the agent to be very helpful. This
is contradicted by the quantitative results that clearly indicate the
usefulness of the agent. Both are correct! The first corresponds to a
personal level and the second one to a team level.
 Individual level: Students were most likely not satisfied by the agent's
assistance on an individual level, declaring that they would prefer shorter and less
frequent messages (they operate under stress while playing the game).
 Team level: the agent seems to have fulfilled its purpose through its' messages
and hints.
 Students could have perceived agent as an additional feature of the
OSgame (they were intentionally not informed about the presence of the
agent). Based on this interpretation, the hypothesis about the usefulness of
the agent is validated. EDUCON 2014 - Istanbul, Turkey Terzidou T. and Tsiatsos T.24

25. RQ2: Does the agent's support have an impact in the
performance of a team?
 As stated from the results the answer is “Yes".
 The support provided by the agent in the OSgame
had a positive impact on teams' performance.
 The agent seems to have a beneficial impact to the
motivation, team collaboration and self-efficacy
of the participated students.
The intervention of pedagogical agent's assistance in
collaborative serious games has a strong positive impact on
teams performance and can be considered as useful.
EDUCON 2014 - Istanbul, Turkey Terzidou T. and Tsiatsos T.25

26. Thank you ! Questions?
Contact:
E-mail:
Lterz@csd.auth.gr
LinkedIn Profile:
http://gr.linkedin.com/in/theodouli
Twitter: LiaTerzidou
Google: +LiaTerzidou
Department of Informatics
Aristotle University of Thessaloniki:
http://www.csd.auth.gr
Multimedia lab:
http://mlab.csd.auth.gr/
EDUCON 2014 - Istanbul, Turkey Terzidou T. and Tsiatsos T.26