Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Selection of Theses


Published on

  • Be the first to comment

  • Be the first to like this

Selection of Theses

  1. 1. A synopsis of Educational Technology R&D: Lessons learned from a selection of theses submitted to Open University of Cyprus Dr. Panagiotis Zaharias Open University of Catalonia - 4/6/2015
  2. 2. Experience from master theses supervision at Open University of Cyprus  Mentored/supervised more than 20 students in their master thesis along topics such as:  - Usability and UX methods,  - Serious games / Game-based learning,  - Online and video games,  - Virtual worlds,  - Gamification,  - Social media in education etc.
  3. 3. Assessing long-term UX in the context of MMOGs
  4. 4. Time spans of User Experience (UX) Source:
  5. 5. UX in the long term: an important but neglected issue  While the importance of temporality has been repeatedly highlighted in user experience research, it has rarely been systematically addressed.  …Due to the effort involved in conducting longitudinal studies.  or a lack of sufficient interest, induced by a belief that motivating prolonged use does not necessarily lead to increased commercial revenues.
  6. 6. Context of the study  Based on:  Thesis submitted by Moschou, Eirini - thesis title: “Development of a UX evaluation method for Virtual Learning Environments” - January 2014  Why long term UX in MMOGs?  Shortage of relevant studies so far  Players usually have a long lasting relationship with the game they play  The main objective of the study:  To investigate, understand and interpret the changes of players’ experience over time
  7. 7. The method: A new version of UX Curve  UX Curve: A retrospective method for assessing long term UX proposed by Kujala et al. (2011). Based on drawings made by users Cost efficient as a long term usage can be covered in a single session Results give an overview of the most relevant experiences Memories are as important or more than actual experiences
  8. 8. UX Curve templates
  9. 9. Context of empirical study  The game: League of Legends  The players: 9 students aged between 18-20 years old who had been playing LoL for periods ranging from 6 to 16 months (mean: 12.5 months SD: 3.1 months)  The new version of UX Curve: UX Curve • General UX • Attractiveness • Ease of use • Utility • Usage Volume A new UX Curve (MMOG version) • General UX • Degree of usage • Ease of use • Immersion • Social interaction • Engagement
  10. 10. Indicative results (1)  A total of 54 curves were collected. Most of the curves were improved revealing thus an increasing user experience General UX Curve for each user ID The immersion UX Curve for each user ID
  11. 11. Indicative results (2)  In order to draw the curves, players described the factors that improved their experience over time or caused it to deteriorate Reason Categories Positive Negative Usability 6 4 Utility 3 2 Aesthetics 7 0 Gameplay 5 3 Challenge 6 0 Social interaction 7 5 Interest 6 0 Miscellaneous 3 0 Sum 43 14 Table2 -The categories of the reasons for general UX curve Reason Categories Positive Negative General UX 43 14 Ease of use 41 26 Immersion 43 10 Social interaction 38 12 Engagement 32 10 Total 197 72 Table 1- Number of reasons for general and specific UX dimensions
  12. 12. Indicative results (3)  All except for two “Ease of use” curves were improving or stable, with pragmatic-related reasons to be the most frequent.  most negative perceptions related to usability reasons  Overall, most of the issues influencing the long-term user experience in the game were related to non-pragmatic issues such as fun, immersion, challenge, interest and control.
  13. 13. Key Takeaways  It is crucial to measure UX in the long term  Retrospective methods can greatly help in understanding how UX changes over time  Memories are as or more important than actual experiences  UX Curve is a cost efficient and effective method to use for analyzing and understanding long term UX  The new proposed version of UX Curve (customized to gaming environments) seems to be a valid and effective method for assessing long term UX of MMOG players  It can be applied to many other contexts for evaluating products/services/systems
  14. 14. Development of an E-learning UX Measurement System
  15. 15. Context of the study  Based on:  Thesis submitted by Evangelos Loutas - thesis title: “Development of an E-learning UX Measurement System” - May 2013  Usability and UX measurement in e-learning applications  Studies show that usability/UX is a crucial success factor in e- learning and facilitates learners to achieve learning objectives and gain knowledge effectively and efficiently  The main objective of the study:  To develop a web-based system that will measure UX, according to Mo2L usability evaluation method.
  16. 16. Mo2L questionnaire  The Mo2L instrument can be used to measure different dimensions of the e-learning user experience,  i.e. from typical usability attributes such as content, navigation and learnability etc. to affective learning issues such as motivation to learn.  A validated questionnaire with 49 items along 8 scales:  Content, Learning design and support, Visual design, Navigation, Accessibility, Interactivity, Self-assessment & Learnability and Motivation to Learn
  17. 17. UX e-learning measurement  Towards quantification of e-learning usability perceptions  The method provides calculation of a total “e-learning usability value” which takes into consideration all items on the questionnaire and reflects a global usability score of the e-learning application.  In addition calculations are made along all the different scales of the questionnaire presenting thus quantifiable information for all the aspects of an e-learning application.
  18. 18. Main workflow of the system as developed 1. Interested researcher or practitioner has to register 2. A new instance of the system is created and a link to the researcher is sent. Researcher can use this link to initiate his/her usability study by calling users/participants. 3. The enquiry participants will be provided with a 49-item questionnaire that contains also some demographics questions. 4. All questions are mandatory. Failure to reply to a question will result to the invalidity of the participant's response. 5. After a certain period of time and a fair number of enquiry participants the questionnaire link will close and further information will be given to registered users on how to access the reports containing the usability measures. 6. The reports are extracted by using an automated procedure. Custom report generation is possible upon request to the administrator.
  19. 19. The main features of the system  Reports for the mean Global E-learning Usability Score (GEUS)  Reports for all the respective Mo2L scales: mean Motivation to Learn score, mean navigation score, etc.  Reports for individual scores and  Reports for open questions
  20. 20. Impact and future research  From May 2013 – today more than 70 researchers and practitioners around the world have expressed their interest to use the system in their usability and UX studies.  Quite recently five instances of the system were open and being used by researchers from US, England and Portugal.  Shortly we are planning a redesign of the system in order to make it more robust and usable  Check out the site: 
  21. 21. Development of a serious 3D game as a tool for organizational learning
  22. 22. Context of the study  Based on:  Thesis submitted by Demetrios Mouzouros, thesis title: “Development of a serious 3D game as a tool for organizational learning” - August 2012  Organizational learning processes and human development  New methods, techniques and tools are needed to effectively support process such as onboarding, etc.  The main objective of the study:  To develop a serious game and use it as an organizational learning tool  To empirically test it for onboarding employees in a real company environment
  23. 23. The development of the “Knowledge Donor”  The game takes place in a 3D world that simulates a business environment and gives the user a realistic feeling of an actual corporate training.  The game scenario is based on two main pillars:  a) the existing procedure that takes place in an insurance company for the orientation of new employees, and  b) the fact that a percentage of the profit of this company goes to charity (strong relations of the company with a scholarship foundation)
  24. 24. The main game scenario  The player is a new employee. Other employees of the company will take the player through an orientation and initial training session  Content about the company creation and history, the products and marketing routines and the personnel of the company.  The game consists of three stages.  At the end of each stage, the player is tested for the “knowledge” that has acquired within specific time limits  Successful completion of the tests will reward the player with money, all of which is collected as a donation to the scholarship foundation.
  25. 25. Empirical study  The “Knowledge Donor” game was evaluated within a period of 20 days where the employees of the insurance company had the chance to play the game.  After the play sessions, interviews were conducted (by the student) to get feedback from the players.  Along with the interviews, the users were asked to fill-in a user- experience questionnaire which was based on the AttrakDiff Lite Questionnaire.  The questionnaire measures hedonic qualities and pragmatic qualities through 10 items that are presented in a 7-point semantic differential scale
  26. 26. Some key findings  Reactions and comments of the users revealed a positive approach towards this new intervention and a great interest in playing the game.  Almost 80% said that this could be a new and effective method to support onboarding processes in the company.  The same users mentioned that the game helped them refresh their memory regarding certain things about business processes.  However only 50% would choose this method over the traditional face-to- face training sessions.  Users also reported some problems with their interaction with the 3D world.  They faced mainly navigation problems and they complained regarding the content
  27. 27. Main limitations and future prospect  Sample of participants  Duration of the study  Future implementations should be based on detailed UX driven process of capturing requirements  Future implementation must involve the real target users: new employees that join the company
  29. 29. Context of the study  Based on:  Thesis submitted by Ioanna Chatzeparaskeuaidou, thesis title: “Development of an educational game as a tool for learning geography in elementary schools” - August 2012  Design of 2-dimensional and 3-dimensional serious games in education  Limited knowledge and relevant empirical research in formal educational settings that compares their effectiveness  The main objective of the study:  To develop an educational game for learning geography in elementary schools – develop two versions a 2D and 3D  To empirically compare their effects on learning effectiveness, motivation to learn, and UX.
  30. 30. Empirical study  Research question: What effects do 2D and 3Dversions of the game have on learning effectiveness, motivation to learn and user experience?  The game:  Design of two versions of the game with Scratch platform and Kodu.  The game scenario: quite simple and structured around an interactive map of Greece.  The game follows a question and answer method: it prompts the player to choose one geographical region and continues with several related questions that are region specific. This is happening repeatedly for other regions as well and the total number is twenty questions.
  31. 31. Snapshots of the game
  32. 32. Snapshots of the game
  33. 33. Experimental process  Participants were students from eight elementary schools located in areas of Northern Greece.  159 students participated in the experiments, 94 girls (59.1%) and 65 boys (40.9%), attending the 5th and 6th grade, from 16 urban and suburban parts.  The age ranged between 10 and 12 years old.  Data were collected through questionnaires and mini interviews:  a demographic survey was employed along with a multiple choice test for assessing learning effectiveness on Geography and two scales measuring motivation to learn and user experience
  34. 34. Experimental process  Set up: A control group with students playing the 2D version of the game and the experimental group with students playing the 3D version (Students were randomly selected based on the "lottery" method)  A mixed between-within subjects ANOVA model:  A session (2: Pretest, Posttest) x design modality (2: 2D, 3D) ANOVA was conducted with session as the within-subjects factor and design modality as the between subjects factor  The experiments took place at the computer labs in 8 elementary schools. Participation of the students and their teachers was on voluntary basis during the whole process
  35. 35. Experimental process and results  Before running the mixed ANOVA an independent samples t-test for differences between the mean scores of male and female subjects was performed. - No statistical significant differences were found.  Learning effectiveness:  It was found that there is a significant interaction between design modality and session, Wilks Lambda = .84, F (1, 149) = 27.08, p < .0005, partial eta squared = .15.  There was a substantial main effect for session (pretest and posttest), Wilks Lambda = .72, F (1, 149) = 57.05, p < .0005, partial eta squared = .27, with both groups showing an increase in learning effectiveness scores across the two time periods.
  36. 36. Results  Learning effectiveness: players in 2D group performed better in geography knowledge tests than players in 3D group.  Authors observed that:  students in 2D group were more focused on the successful completion of the game while  students playing the 3D game were almost distracted from the visual design of the game trying to learn to navigate and willing to explore the elements of the environment beyond the learning focus of the game
  37. 37. Results  Motivation to learn:  Results revealed that there is no significant interaction between design modality and session, Wilks Lambda = .97, F (1, 149) = 3.19, p = .076, partial eta squared = .021.  There was a substantial main effect for session (pretest and posttest), Wilks Lambda = .81, F (1, 149) = 33.69, p < .0005, partial eta squared = .184, with both groups showing a decrease in motivation to learn scores across the two time periods
  38. 38. Results  Motivation to learn:  We observed that students entered the experimental process with a great enthusiasm  while they were engaged with game activities they oriented themselves more towards fun than learning.  This may explain the fact that motivation to learn decreased after the end of the experiment. Confirmed in other studies as well (Yang 2012, Annetta et al. 2009), as the novelty effect fades motivation gets lower.
  39. 39. Results  Motivation to learn:  The main effect comparing the two versions of game was also significant, F (1, 149) =7.71, p = .006, partial eta squared = .049, suggesting a difference between the two design modalities.  We see that players in 3D group showed greater motivation to learn than those who played the 2D version.
  40. 40. Results  UX:  There is no significant interaction between design modality and session, Wilks Lambda = .99, F (1, 149) = .28, p = .595, partial eta squared = .002.  There was a substantial main effect for session (pretest and posttest), Wilks Lambda = .77, F (1, 149) = 42.46, p < .0005, partial eta squared = .222, with both groups showing a decrease in user experience scores across the two sessions  Observation: Students had great expectations in terms of anticipated UX in both groups
  41. 41. Results  UX:  The main effect comparing the two versions of game was also significant, F (1, 149) =4.75, p = .031, partial eta squared = .031, suggesting a difference between the two design modalities.  players in 3D group expressed a higher degree of user experience than the players in the 2D group.  We argue that 3D learning environments are supposed to enhance the user experience in terms of flow, presence, etc.  We attribute this effect on the perceived hedonic qualities of 3D environments such as novelty and greater fidelity.
  42. 42. Future research  Longer experiments are needed  Different types of knowledge tests  Design modalities (2d and 3d) can be combined in a single game  The games could be designed in a multiplayer mode with a different pedagogical focus where activities will require collaboration for successful completion of the game.
  43. 43. Some papers associated with the theses  [C21] Chatzeparaskeuaidou, I. and Zaharias, P. Hedonic and pragmatic qualities as predictors for motivation to learn in serious educational games. Accepted at the Foundations of Digital Games 2013, Workshop on Games for Learning.  [J17] Chatzeparaskeuaidou, I. and Zaharias, P. The effects of a 2D and 3D game on learning effectiveness and motivation. Submitted to Journal of Educational Technology & Society  [C24] Moshou, E. and Zaharias, P. (2013). The UX Curve revisited: Assessing long term UX for games. Accepted at the Workshop on Designing Gamification: Creating Gameful and Playful Experiences at CHI 2013 - Conference on Human Factors in Computing Systems.  [J19] Zaharias, P. and Moschou, E. Drawing Curves for assessing long term UX of Massive Multiplayer Online Games. Submitted to Interacting with Computers  [J16] Mouzouros, D. and Zaharias, P. Development of a 3D serious game for knowledge management: an empirical investigation. Submitted to Journal of Virtual Worlds Research
  44. 44. Thank you very much ! Any Questions? Contact info:  |  