Downloaded 21 times
![Thank you! Questions? Dr. Niki Lambropoulos [email_address] Prof. Fintan Culwin [email_address] EuroCAT Pedagogical Usability >>](https://image.slidesharecdn.com/lambropoulos-culwin-avatar-100206025759-phpapp01/85/Tools-and-Evaluation-Techniques-to-Support-Social-Awareness-in-CSCeL-The-AVATAR-Lambropoulos-Culwin-22-320.jpg){kind=avatar}
Uploaded byNiki Lambropoulos PhD
Tools and Evaluation Techniques to Support Social Awareness in CSCeL: The AVATAR - Lambropoulos Culwin
1. The document describes a 7-step process for designing participation avatars to increase engagement in an online teacher training program in Greece. 2. Initial research found low participation. The design process included evaluating prototypes with users and experts to determine the best way to calculate and represent participation levels visually. 3. The final participation avatars displayed on the forum interface represented participation levels based on number of posts by the most active student, which usability testing found to be most effective. Evaluation of the tool found it increased satisfaction and frequency of use.
More Related Content
Similar to Tools and Evaluation Techniques to Support Social Awareness in CSCeL: The AVATAR - Lambropoulos Culwin
Tools and Evaluation Techniques to Support Social Awareness in CSCeL: The AVATAR - Lambropoulos Culwin
- 1. Design Example: ParticipationAvatars Dr. Niki Lambropoulos & Prof. Fintan Culwin Department of Informatics, London South Bank University
- 2. Agenda HCI-Ed, theLSBU’s Role in Euro-CAT HCI-Ed 7 steps Design Example: Participation Avatars Conclusions & Insights
- 3. Goals for EducationalComputing Goals for Computing (Shneiderman, 2008): Useful Usable Universal Goals for Educational Computing ( Tricot, 2007 ; Lambropoulos & Culwin, 2009) Pedagogical Utility : degree of reaching learning goal Pedagogical Usability : degree of ease of use Pedagogical Acceptability : degree of compatibility with organisational context & practices, learners’ motivation, affects, culture and values
- 4. HCI-Ed @ LSBUHCI-Ed emphasises understanding of the learning context and values, before the design, evaluation and implementation of systems and tools, from a user/learner-centred perspective. This is complemented by the study of the major phenomena surrounding the impact of these processes.
- 5. HCI-Ed in 7Steps Context & Learning Values - Hypothesis (Iterative) Design – Requirements Evaluation with user groups/experts Development Evaluation with user groups Re-Design & Development Study & Research - Tool Release 1CH 2DR 3E 4D 5E 6D 7RTR
- 6. Research Problem GreekTeachers’ inactivity between 2001-2004 on the online training hosted on Moodle by the Greek School Network (GSN) Note: The Greek Ministry of Education & GSN targeted active participation in order to to form a Community of Practice (CoP)
- 7. Ethno-Technological Analysis of behaviour Greek Teachers’ CoP Moodle@GSN Native’s point of view, background and characteristics Learners’ Intentions (learning goals): Mirroring capabilities for Presence & Co-Presence Self -Presentation, -Locus, -Assessment, -Management Co -Presentation, -Locus, -Assessment, -Management Users’ operations (actions/tasks): Pedagogical Usability 1C H 2DR 3E 4D 5E 6D 7RTR Step1a: Context & Learning Values (Social Awareness)
- 8. Social Awareness: Presence& Co-Presence Degree of salience Other people being real Feeling of being together tangibility proximity accessibility connectedness Ability to emotionally and socially project oneself Being Human Awareness of relations affiliation intimacy immediacy sociability Co-presence Attentional allocation Perceived message understanding Perceived affective understanding Perceived affective interdependence Perceived behavioural interdependence engagement participation Psychological involvement Mutual awareness Feeling accessibility Co-location empathy 1CH 2DR 3E 4D 5E 6D 7RTR
- 9. If these e-learnerswere able to observe their behaviour in relation to themselves and their peers they may be able to alter it. 1CH 2DR 3E 4D 5E 6D 7RTR Step1b: Hypothesis
- 10. Step2: (Iterative) Design - Requirements 1CH 2DR 3E 4D 5E 6D 7RTR Participation Levels (Other: Social Network Analysis, CSCeL Argumentation)
- 11. Evaluation Indicators Presence:individual’s level of activity Co-presence: pattern of activity in the group 1 2 3 4 1CH 2DR 3E 4D 5E 6D 7RTR
- 12. Issues What isthe best calculation basis? Total number of messages? Quartile? Highest poster? E-tutor? E-student? What about the colours? 1CH 2DR 3E 4D 5E 6D 7RTR
- 13. Step3: User/ExpertEvaluation @LSBU 1CH 2DR 3E 4D 5E 6D 7RTR Best calculation basis Total number of messages Quartile Highest poster Colours
- 14. Prototype: Avatars onthe Forum Interface Step4: Development 1CH 2DR 3E 4D 5E 6D 7RTR
- 15. Step5: Evaluationwith User Groups 1CH 2DR 3E 4D 5E 6D 7RTR Teachers’ case study: comments & PU (Likert 1-5) Teachers/Developers’ focus group: comments & PU (Overall)
- 16. Issues Whatis the best calculation basis? 1CH 2DR 3E 4D 5E 6D 7RTR Highest poster E-tutor? E-student? What about the colours?
- 17. Step6: Re-Design& Development 1CH 2DR 3E 4D 5E 6D 7RTR Colours Best calculation basis: Highest E-student
- 18. 1CH 2DR 3E4D 5E 6D 7RTR Results on both logs and Avatars (control/experimental group)- Recommendations Step7: Study & Research in context – Tool Release
- 19. Step7: Study& Research – Tool Release 1CH 2DR 3E 4D 5E 6D 7RTR Greek Teachers on Pedagogical Usability Overall satisfaction = 3.2/5 Low Originality (2.4) Learnability (2.7) > Frequency of use (2.4)
- 20. Avatars Today 1CH2DR 3E 4D 5E 6D 7RTR In our courses GAPMET Comenius current project University of Athens GAPMET Da Vinci project Moodle module?
- 21. Conclusions & InsightsDirect fit between educational task and the method chosen to pursue it Tools like Avatars can support the Exploration and verification of participation levels Differentiation between social communication & learning tasks Organisational convergence Diversity in teaching & learning styles Adaptation to individuals’ changing needs, learning and interaction styles Social Intelligence: act wisely in human relations (Thorndike, 1920)
- 22. Thank you! Questions?Dr. Niki Lambropoulos [email_address] Prof. Fintan Culwin [email_address] EuroCAT Pedagogical Usability >>
- 23. EuroCAT Pedagogical UsabilityURL: http://www.survey.bris.ac.uk/lsbu/euro-cat
- 24. Participation startswith enrolment into the Learning Management System, in this case Moodle, and is divided into passive and active participation. Null passive participation is defined as absence of activity; that is the online participant registers without any further actions whatsoever. Passive participation is defined as visiting and reading posts, without posting. It is categorised into three levels: low, medium and high. These levels are obtained from the number of days during the course when at least one visit is made. Low passive participation is where the average number of days is one quarter or less of the duration of the course. High passive participation is more than three quarters of the duration and medium passive participation is between these two categories. Online active participation is defined as the presence of activity; that is, the online participant registers, reads and posts. Active participation is categorised into three levels: low, medium and high. These levels are defined by the number of posts sent by the most active e-learner. Low active participation is where the average number of posts is one quarter or less of the total posts from the most active e-learner. High active participation is more than three quarters of the posts and medium active participation falls between these two categories.
- 25. Ergonomic Analysis PedagogicalUsability should question whether the tools, contents, interfaces, and tasks provided within the e-learning environments can support e-learners (Silius et al., 2003; Nokelainen, 2006; Zaharias, 2006; System Usability Scale, Lewis & Sauro, 2009). Pedagogical Utility is the degree to which the functionality of the system allows the learner to reach his/her learning goal (Silius et al., 2003). Lastly, Pedagogical Acceptability refers to the previous compatibility as well as the degree the system is compatible with organisational context & learners’ motivation, affects, culture and values (Tricot, 2007).
- 26. Ethnotechnology Context: Moodle@GreekSchoolNetworkLegislation, Archives Discussions (Greek teachers associations, e-mint online community managers) Opinions (developers, Greek teachers, graphic designer) Observation notes from active participation in the Community of Practice (online courses) Surveys
- 27.
- 28. Examples (2004-2009)2005-9: Visualisation Interaction Tools (Social Network Analysis ) 2004-8: HySynTag (Argumentation Tags) 2007-8: Graphs & Avatars Participation Levels – Participation Eyeball (McDonald, 2003 ) Participation Avatars 1CH 2DR 3E 4D 5E 6D 7RTR
- 29. Design The designcan be defined as an individual and collective activity, finalized by a project to develop a physical and symbolic artefact. The artefact is rarely well defined in this step.
- 30. What is agood external representation for learning ? It is useful: – it allows the learner to reach his/her learning goal It is usable: – it is easy to learn to use it, efficient, it is easy to memorize how it works, it doesn’t cause errors, it is satisfying (Nielsen, 1994) It is acceptable: – it is compatible with the goals and the constraints of the context (time, space, organization) – it is compatible with learner’s motivation, affects, culture, values
- 31.
- 32.
- 33. Goals for TheNew Computing 1) Usable: Reliable & comprehensible 2) Universal: Diverse users & varied equipment 3) Useful: In harmony with human needs Leonardo's Laptop: Human Needs and the New Computing Ben Shneiderman
Editor's Notes
- #4 Tricot: external representation of learning
- #8 The first HCI-Ed stage is the understanding of the context, key concepts and tasks prior to their translation into design. The understanding of the context is attained by identifying issues within it and the key factors that are at play with the associated learning values.
- #9 e.g. feeling of community the learner experiences (Tu & McIsaac, 2002), the degree to which a person is perceived as a ‘real person’ in a mediated communication (Gunawardena, 1995), Social presence theory is defined as the ‘degree of salience of the other person in the interaction and the consequent salience of the interpersonal relationships…’ (Short et al., 1976:65), Rafaeli (1990) argues that social presence is a subjective measure of the presence of others, while ‘interactivity’ is the actual quality of a communication sequence or context. salience-=projection, salient=outstanding (important, striking)
- #11 The second HCI-Ed stage is iterative design based on reflections upon processes and outputs from the previous stage. The design can be defined as an individual and collective activity, finalized by a project to develop a physical and symbolic artefact. The problem is rarely well defined.
- #13 In descriptive statistics , a quartile is any of the three values which divide the sorted data set into four equal parts, so that each part represents one fourth of the sampled population.
- #15 The fourth HCI-Ed stage involves building prototypes in order to acquire feedback from design. Exploratory prototypes are useful to allow different groups of users/learners to comment upon design. Such prototypes might be thrown away or might evolve towards a final version. For example, an initial version is useful for an early evaluation; and early access can be provided to different groups of users/learners as well as developers and e-tutors for community previews. Pre-release prototypes shown at community reviews will facilitate the final tweaking of the system.
- #16 The fifth HCI-Ed stage is the evaluation of system’s quality by measuring the degree to which the system meets its purpose as defined in the first two stages. Live evaluation of the system in use is of extreme importance. The avatars are divided into three regions: head, arms and legs, with a colour scheme indicating the levels of participation this was in an attempt to allow other users to see very quickly what kind of forum member they were communicating with. The baseline study was conducted on a Moodle server at the Greek School Network and this was used to determine the level of participation. There were 8 topics, 28 new discussions and 77 messages sent by 45 participants, 32 of whom were passive and 13 active, making an average of about 6 messages per poster. In the case study the prototypes of the various tools were tested. There were 56 participants and 5 e-tutors including the first author (Niki Lambropoulos). The majority (43) were Greek teachers and the rest (13) were from different parts of the world. In the tools evaluation discussion, there were 33 sent messages and 10 active users with an average of about 3 messages per poster. The comments were analysed and on the basis of these discussions, the tools were redesigned and tested in a focus group of 3 male Greek teachers who were also Moodle developers. None of them had participated in any of the previous studies. The case study and focus group active participants completed an online questionnaire to evaluate the tools. It used a 1-5 Likert scale and focused on efficiency, effectiveness and satisfaction (ISO DIS 9241-11, 1994), as well as enjoyability, learnability and imagination (Zaharias, 2004; Silius et al., 2003). Overall, the tools scored 4.2 out of 5 indicating that the participants were satisfied with the tools. The most important changes that came out of this fieldwork are described in the next section. Six participants were passive in the first study, 1 was a null passive participant, 4 were low passive participants and 1 was medium. Of these 6, 2 became active participants in the second study. However, 5 who were active became passive. There were 9 passive participants in the second study: 5 low, 2 medium and 2 high. Figure 7 shows an overview of the participation levels that appeared in the second study. The calculation for the second case study was based on P50, who posted 12 messages. There are 1 low, 5 medium and 4 high participation learners. The active participation data in the first case study were obtained from the system logs. The calculation for the first case study was based on the most active e-learner, P37, who posted 36 messages. Overall, there were 29 active participants in the first case study: 22 low active participants, 5 medium and 2 high. There were 26 active participants in the second case study: 16 low, 5 medium and 5 high. Figure 8 illustrates these results.
- #19 The final HCI-Ed stage is the study and research of all phenomena surrounding human-computer interactions. These can be supported by observing the context from different perspectives. Such multi-disciplinary approach is essential to effective HCI-Ed. Interface and contextual information are needed to facilitate the acquisition of structured and unstructured data in numeric, textual and visual formats to reveal converged patterns. Statistical analysis can work on numeric data from questionnaires and logs as well as quantified qualitative analysis. Qualitative analysis can work on textual data, open questions, focus groups and interviews; visual data can provide pictorial perspective of the e-learning phenomena.
- #20 New PU Q was developed.
- #26 Tricot: external representation of learning
- #27 The right methodology gives us the tools to investigate the research context and in fact, to suggest the criteria and metrics for internal and external evaluation and feedback. Ethnotechnology is a perspective rather than a methodology. This means that all approaches are legitimate to acquire relevant information from the situated context, in this case the Greek teachers. No presence indicated stillness and thus, change was needed. The techniques were associated with the aims, for example, an e-learning environment can provide data for human-human and human-computer interactions by quantitative analysis and grounded theory can be used for message analysis. The collaborative e-learning analytical framework has been already suggested for evaluation.
- #31 http://pagesperso-orange.fr/andre.tricot/Earli_Tricot_Slides.pdf Tricot, A., (2007). Utility, usability and acceptability: an ergonomic approach to the evaluation of external representations for learning. EARLI Symposium « Understanding the role of external representations in supporting learning » , Budapest, August 28 – September 1.