This document discusses groupware evaluation methods. It provides an overview of groupware applications and evaluation objectives. Both traditional single-user evaluation methods and groupware-specific methods are described, including mechanics of collaboration analysis and collaboration usability analysis. Analytical methods like groupware task analysis are also covered. The document examines examples of evaluation studies that combine traditional and groupware methods and analyzes awareness mechanisms. It concludes that evaluating groupware is complex due to many impacting factors and that both tool interfaces and collaboration processes must be assessed.
"From Making to Learning" : Dev Camps as a Blueprint for Re-inventing Project...Irene-Angelica Chounta
Dev Camps are events that enable participants to tackle challenges using software tools and different kinds of hardware devices in collaborative project style activities. The participants conceptualize and develop their solutions in a self-directed way, involving technical, organizational and social skills. In this sense, they are autonomous producers or " makers ". The Dev Camp activity format resonates with skills such as communication, critical thinking, creativity, decision making and planning and can be considered as a bridge between education and industry. In this paper we present and analyze experience from a series of such events that were co-organized between an industrial partner acting as a host and several university partners. We take this as an indication to envision new opportunities for project-based learning in more formal educational scenarios.
Multilevel analysis of collaborative activities based on a Mobile Learning Sc...Irene-Angelica Chounta
CollabTech2014 (http://www.collabtech.org/) - This paper describes the analysis of collaborative mobile learning activities. We explore the use of learning analytics for the evaluation of the performance of stu-dents as individuals and the performance of teams. We argue that traditional met-rics used for learning analytics can provide insight with respect to the quality of the activity and the learning outcome. We propose a way to integrate innovative mobile learning scenarios into traditional classrooms and to analyze collaborative learning activities on both the group and the individual level.
Two make a network: using network graphs to assess the quality of collaborati...Irene-Angelica Chounta
CRIWG2014 - In this paper we explore the application of network analysis techniques in order to analyze synchronous collaborative activities of dyads. The collaborative activi-ties are represented and visualized as networks. We argue that the characteristics and properties of the networks reflect the quality of collaboration and therefore can support the analysis of collaborative activities in an automated way. To sup-port this argument we studied the collaborative practice of 228 dyads based on network graphs. The properties of each graph were evaluated in comparison to ratings of collaboration quality as assessed by human experts. The activities were also examined with respect to the solution quality. The paper presents the method and the findings of the study.
"From Making to Learning" : Dev Camps as a Blueprint for Re-inventing Project...Irene-Angelica Chounta
Dev Camps are events that enable participants to tackle challenges using software tools and different kinds of hardware devices in collaborative project style activities. The participants conceptualize and develop their solutions in a self-directed way, involving technical, organizational and social skills. In this sense, they are autonomous producers or " makers ". The Dev Camp activity format resonates with skills such as communication, critical thinking, creativity, decision making and planning and can be considered as a bridge between education and industry. In this paper we present and analyze experience from a series of such events that were co-organized between an industrial partner acting as a host and several university partners. We take this as an indication to envision new opportunities for project-based learning in more formal educational scenarios.
Multilevel analysis of collaborative activities based on a Mobile Learning Sc...Irene-Angelica Chounta
CollabTech2014 (http://www.collabtech.org/) - This paper describes the analysis of collaborative mobile learning activities. We explore the use of learning analytics for the evaluation of the performance of stu-dents as individuals and the performance of teams. We argue that traditional met-rics used for learning analytics can provide insight with respect to the quality of the activity and the learning outcome. We propose a way to integrate innovative mobile learning scenarios into traditional classrooms and to analyze collaborative learning activities on both the group and the individual level.
Two make a network: using network graphs to assess the quality of collaborati...Irene-Angelica Chounta
CRIWG2014 - In this paper we explore the application of network analysis techniques in order to analyze synchronous collaborative activities of dyads. The collaborative activi-ties are represented and visualized as networks. We argue that the characteristics and properties of the networks reflect the quality of collaboration and therefore can support the analysis of collaborative activities in an automated way. To sup-port this argument we studied the collaborative practice of 228 dyads based on network graphs. The properties of each graph were evaluated in comparison to ratings of collaboration quality as assessed by human experts. The activities were also examined with respect to the solution quality. The paper presents the method and the findings of the study.
The presentation slides for workshop 4, “Structuring online collaboration though 3 Ts : task time & teams” in 2nd STELLARnet Alpine Rendez-Vous in the French Alps 27th to 31st March 2011
A preliminary study where a multiplayer location-based game’s logfiles were used for the assessment of the overall practice of teams. We explore the use of activity metrics previously introduced and applied to CSCL settings. We argue that these metrics, if adapted in a meaningful way, will provide insight of the progress of a location-based gaming activity and its quality regarding the score. Moreover, we assert that this can be achieved in an automated way. A small set of activity metrics, related to game characteristics and player activity, is applied to a set of gaming activities. The results are analyzed regarding team performance and score. The paper proposes
a way to analyze group activity in the context of location-based games while taking into account the characteristics of successful collaborative activities. Future work is proposed towards the development of automated metrics for the analysis of location-based gaming activities with emphasis on collaboration and group dynamics.
Chapter 19: Groupware
from
Dix, Finlay, Abowd and Beale (2004).
Human-Computer Interaction, third edition.
Prentice Hall. ISBN 0-13-239864-8.
http://www.hcibook.com/e3/
HCI 3e - Ch 14: Communication and collaboration modelsAlan Dix
Chapter 14: Communication and collaboration models
from
Dix, Finlay, Abowd and Beale (2004).
Human-Computer Interaction, third edition.
Prentice Hall. ISBN 0-13-239864-8.
http://www.hcibook.com/e3/
It's all about time: towards the real time evaluation of collaborative activi...Irene-Angelica Chounta
In this paper, we propose a method for the real time evaluation of collaborative activities. The method aims to classify collaborative activities using a memory-based learning model and to evaluate them using a reference set of pre-evaluated activities. The proposed approach uses time series to represent activities and classifies them regarding their similarity. In order to evaluate the results, the method is used as an automatic rater of collaboration quality and the ratings are compared to ratings of human experts.
Chapter 9: Evaluation techniques
from
Dix, Finlay, Abowd and Beale (2004).
Human-Computer Interaction, third edition.
Prentice Hall. ISBN 0-13-239864-8.
http://www.hcibook.com/e3/
The Pragmatic Evaluation of Tool System InteroperabilityCommunitySense
A. de Moor (2007). The Pragmatic Evaluation of Tool System Interoperability (invited paper). In Proc. of the 2nd ICCS Conceptual Structures Tool Interoperability Workshop (CS-TIW 2007), Sheffield, UK, July 22, 2007. Research Press International, Bristol, UK, pp.1-19.
Examples of evaluation studies in educational technolgycreativeperson
This presenation show summaries of two evaluation studies in educational technolgy. The first summary focus on the methodolgy of the study.The second summary focus on specific educational technolgy features.
An institutional perspective on analytics that focusses on a particular tool developed using an agile methodology to visualise learner behaviours in MOOCs via Sankey diagrams.
ICT are transforming Cuban higher education towards the adoption of blended-learning and distance learning. This dissertation focuses on investigating the effectiveness of using social software to support collaborative learning in a Cuban university. Five studies were conducted within three phases that included diagnostic, integration and validation of the social software that was used to support collaborative learning. A didactic model was created to integrate social software within Cuban teaching and learning in higher education. Social Network Analysis and content analysis were used to evaluate the effectiveness of social software to support students' learning through their collaborative learning relationships and through their posts in wiki pages and online discussions. Statistical analysis was used to evaluate students' self-efficacy as a measure of their achievements in social software-supported collaborative learning. The findings confirmed social software’s suitability to support collaborative learning, as it increased collaborative learning's effectiveness, compared to face-to-face collaborative learning. Specific findings were revealed for the use of wikis and online discussions within teaching and learning, which are extendable to other social software tools. A didactic model to integrate social software in Cuban teaching and learning, as well as a framework to analyse students' interactions, were used for first time and validated to extend its use among Cuban university stakeholders.
AFEL-REC: A Recommender System for Providing Learning Resource Recommendation...Dominik Kowald
In this paper, we present preliminary results of AFEL-REC, a recommender system for social learning environments. AFEL-REC is build upon a scalable software architecture to provide recommendations of learning resources in near real-time. Furthermore, AFEL-REC can cope with any kind of data that is present in social learning environments such as resource metadata, user interactions or social tags. We provide a preliminary evaluation of three recommendation use cases implemented in AFEL-REC and we find that utilizing social data in form of tags is helpful for not only improving recommendation accuracy but also coverage. This paper should be valuable for both researchers and practitioners interested in providing resource recommendations in social learning environments.
The presentation slides for workshop 4, “Structuring online collaboration though 3 Ts : task time & teams” in 2nd STELLARnet Alpine Rendez-Vous in the French Alps 27th to 31st March 2011
A preliminary study where a multiplayer location-based game’s logfiles were used for the assessment of the overall practice of teams. We explore the use of activity metrics previously introduced and applied to CSCL settings. We argue that these metrics, if adapted in a meaningful way, will provide insight of the progress of a location-based gaming activity and its quality regarding the score. Moreover, we assert that this can be achieved in an automated way. A small set of activity metrics, related to game characteristics and player activity, is applied to a set of gaming activities. The results are analyzed regarding team performance and score. The paper proposes
a way to analyze group activity in the context of location-based games while taking into account the characteristics of successful collaborative activities. Future work is proposed towards the development of automated metrics for the analysis of location-based gaming activities with emphasis on collaboration and group dynamics.
Chapter 19: Groupware
from
Dix, Finlay, Abowd and Beale (2004).
Human-Computer Interaction, third edition.
Prentice Hall. ISBN 0-13-239864-8.
http://www.hcibook.com/e3/
HCI 3e - Ch 14: Communication and collaboration modelsAlan Dix
Chapter 14: Communication and collaboration models
from
Dix, Finlay, Abowd and Beale (2004).
Human-Computer Interaction, third edition.
Prentice Hall. ISBN 0-13-239864-8.
http://www.hcibook.com/e3/
It's all about time: towards the real time evaluation of collaborative activi...Irene-Angelica Chounta
In this paper, we propose a method for the real time evaluation of collaborative activities. The method aims to classify collaborative activities using a memory-based learning model and to evaluate them using a reference set of pre-evaluated activities. The proposed approach uses time series to represent activities and classifies them regarding their similarity. In order to evaluate the results, the method is used as an automatic rater of collaboration quality and the ratings are compared to ratings of human experts.
Chapter 9: Evaluation techniques
from
Dix, Finlay, Abowd and Beale (2004).
Human-Computer Interaction, third edition.
Prentice Hall. ISBN 0-13-239864-8.
http://www.hcibook.com/e3/
The Pragmatic Evaluation of Tool System InteroperabilityCommunitySense
A. de Moor (2007). The Pragmatic Evaluation of Tool System Interoperability (invited paper). In Proc. of the 2nd ICCS Conceptual Structures Tool Interoperability Workshop (CS-TIW 2007), Sheffield, UK, July 22, 2007. Research Press International, Bristol, UK, pp.1-19.
Examples of evaluation studies in educational technolgycreativeperson
This presenation show summaries of two evaluation studies in educational technolgy. The first summary focus on the methodolgy of the study.The second summary focus on specific educational technolgy features.
An institutional perspective on analytics that focusses on a particular tool developed using an agile methodology to visualise learner behaviours in MOOCs via Sankey diagrams.
ICT are transforming Cuban higher education towards the adoption of blended-learning and distance learning. This dissertation focuses on investigating the effectiveness of using social software to support collaborative learning in a Cuban university. Five studies were conducted within three phases that included diagnostic, integration and validation of the social software that was used to support collaborative learning. A didactic model was created to integrate social software within Cuban teaching and learning in higher education. Social Network Analysis and content analysis were used to evaluate the effectiveness of social software to support students' learning through their collaborative learning relationships and through their posts in wiki pages and online discussions. Statistical analysis was used to evaluate students' self-efficacy as a measure of their achievements in social software-supported collaborative learning. The findings confirmed social software’s suitability to support collaborative learning, as it increased collaborative learning's effectiveness, compared to face-to-face collaborative learning. Specific findings were revealed for the use of wikis and online discussions within teaching and learning, which are extendable to other social software tools. A didactic model to integrate social software in Cuban teaching and learning, as well as a framework to analyse students' interactions, were used for first time and validated to extend its use among Cuban university stakeholders.
AFEL-REC: A Recommender System for Providing Learning Resource Recommendation...Dominik Kowald
In this paper, we present preliminary results of AFEL-REC, a recommender system for social learning environments. AFEL-REC is build upon a scalable software architecture to provide recommendations of learning resources in near real-time. Furthermore, AFEL-REC can cope with any kind of data that is present in social learning environments such as resource metadata, user interactions or social tags. We provide a preliminary evaluation of three recommendation use cases implemented in AFEL-REC and we find that utilizing social data in form of tags is helpful for not only improving recommendation accuracy but also coverage. This paper should be valuable for both researchers and practitioners interested in providing resource recommendations in social learning environments.
Enhancing School Community through Technology Professional Development for Te...Kendra Minor
This presentation provides an overview of the roles and responsibilities of each collaborative partner; narrative about the process used to analyze, design, develop, implement and evaluate the professional development workshop; and the tools and community generated by the collaborative.
LTS Lunch 27 Jan 10 - Tools for Learning Designaewp2
__What is Learning Design?
The focus of course designers is often first on covering the requisite ground in a logical structure, second on developing the best possible explications of difficult areas, and only third on lighting a fire of inquiry in their students. This fire is nourished when students are involved and challenged, stifled when they are passive recipients of knowledge. Planning how learning will happen needs as much consideration and care as what will be learned.
__Design Tools
Modern technology can be integrated in teaching to give new flexibility in teaching styles, whether at the level of activities, courses or whole programmes. It also provides tools to help us extract the benefits of that flexibility:
- by making the pedagogical structure more visible and explicit to students and planners, thereby promoting understanding and reflection
- by serving as a description or template to be collaborated on, adapted and reused
- by sharing best practices and understanding between those involved in design and teaching
__Course Tools
CARET's Course Tools project is developing easy-to-use web tools for fast and flexible access to course information and planning at Cambridge. I will briefly introduce the Lecture Explorer and Oxford's Phoebe.
__Learning Design Initiative
CARET is partnering with the OU to develop “a learning design methodology and suite of practical tools and resources that bridge between good pedagogic practice and effective use of new technologies”; broadly structured around understanding the curriculum design process, supporting collaborative design, offering support such as case studies, course templates and ‘best practice’ checklists, visualising designs in various ways, and sharing the results. I will introduce CompendiumLD and Cloudworks.
Will time tell? Exploring the relationship between step duration and student ...Irene-Angelica Chounta
==ICLS 2018 talk==
Abstract: In this paper, we explore how the time students take to solve a problem may relate to their success. Even though prior research indicates that students’ response times can provide some indication regarding correctness, time is not consistently and broadly used when modeling students’ performance. We aim to clarify the relationship between the step duration
– that is, the time a student takes to carry out a step of a learning task – and the outcome of
this step with respect to correctness for STEM-related courses. Then, we discuss our early
findings, how they can be used to enhance student modeling and to provide meaningful and
timely feedback to students.
Modeling the Zone of Proximal Development with a Computational ApproachIrene-Angelica Chounta
In this paper, we propose a computational approach to modeling the Zone of Proximal Development of students who learn using a natural-language tutoring system for physics. We employ a student model to predict students’ performance based on their prior knowledge and activity when using a dialogue tutor to practice with conceptual, reflection questions about high-school level physics. Furthermore, we introduce the concept of the “Grey Area”, the area in which the student model cannot predict with acceptable accuracy whether a student has mastered the knowledge components or skills present in a particular step.
---
To appear in Educational Data Mining 2017
Building Arguments Together or Alone?Using Learning Analytics to Study the Co...Irene-Angelica Chounta
Research has shown that the construction of visual representations may have a positive effect on cognitive skills, including argumentation. In this paper we present a study on learning argumentation through computer-supported argument diagramming. We specifically focus on whether students, when provided with an argument-diagramming tool, create better diagrams, are more motivated, and learn more when working with other students or on their own. We use learning analytics to evaluate a variety of student activities: pre and post questionnaires to explore motivational changes; the argument diagrams created by students to evaluate richness, complexity and completion; and pre and post knowledge tests to evaluate learning gains.
To a large extent, current tutorial dialogue systems lack the ability to gauge students’ level of mastery over the curriculum. Human tutors do gauge the level of knowledge and understanding of their tutees to some degree, although they are not very adept at diagnosing the causes of student errors.
We propose integrating a student model that evaluates the student’s understanding of curriculum elements into tutorial dialogue and that doing so can address these differences between human and simulated tutors.
LASAD Grammar App: A collaborative application to support students who study ...Irene-Angelica Chounta
Our goal is to support students who learn form and function of sentences in English Grammar. We designed and implemented an app that helps students to identify the critical parts of a sentence and to map a sentence to its constituent parts. The app can be used individually (i.e. a student working alone) and collaboratively (a group of students working on the same sentence). Additionally the app can support teachers who want to monitor students’ progress and it allows them to intervene when necessary.
Chounta Talk @HCII: Linking Dialogue with Student Modeling to Create an Enhan...Irene-Angelica Chounta
The learning process and its outcomes depend greatly on the social interaction between teachers and students and, in particular, on the proficient and focused use of language through written text or discussions. Our overarching goal in this project is to better understand how to make automated tutorial dialogues effective and adaptive to student characteristics, such as prior knowledge. The specific goal of our current project is to develop an adaptive, natural-language tutoring system, driven by a student model, which can effectively carry out reflective conversations with students after they solve physics problems. Towards this end, we continue our work in identifying linguistic features of tutoring that predict learning gains, and extend it by characterizing the “level of support” to provide to students based on their current level of understanding particular physics concepts and principles, as dynamically captured by the student model. In this talk, I will describe the features of dialogic discourse underlying “level of support” that we have identified through the analysis of human-to-human tutorial dialogues as well as the construction and application of a coding scheme for the characterization of the “level of support”. I will present initial teacher feedback on dialogues that apply these features to coach students at different levels. I will also discuss how this line of research affects the authoring of tutorial dialogues used by an intelligent tutoring system for students who exhibit different levels of understanding.
Acetabularia Information For Class 9 .docxvaibhavrinwa19
Acetabularia acetabulum is a single-celled green alga that in its vegetative state is morphologically differentiated into a basal rhizoid and an axially elongated stalk, which bears whorls of branching hairs. The single diploid nucleus resides in the rhizoid.
Read| The latest issue of The Challenger is here! We are thrilled to announce that our school paper has qualified for the NATIONAL SCHOOLS PRESS CONFERENCE (NSPC) 2024. Thank you for your unwavering support and trust. Dive into the stories that made us stand out!
Unit 8 - Information and Communication Technology (Paper I).pdfThiyagu K
This slides describes the basic concepts of ICT, basics of Email, Emerging Technology and Digital Initiatives in Education. This presentations aligns with the UGC Paper I syllabus.
How to Make a Field invisible in Odoo 17Celine George
It is possible to hide or invisible some fields in odoo. Commonly using “invisible” attribute in the field definition to invisible the fields. This slide will show how to make a field invisible in odoo 17.
2024.06.01 Introducing a competency framework for languag learning materials ...Sandy Millin
http://sandymillin.wordpress.com/iateflwebinar2024
Published classroom materials form the basis of syllabuses, drive teacher professional development, and have a potentially huge influence on learners, teachers and education systems. All teachers also create their own materials, whether a few sentences on a blackboard, a highly-structured fully-realised online course, or anything in between. Despite this, the knowledge and skills needed to create effective language learning materials are rarely part of teacher training, and are mostly learnt by trial and error.
Knowledge and skills frameworks, generally called competency frameworks, for ELT teachers, trainers and managers have existed for a few years now. However, until I created one for my MA dissertation, there wasn’t one drawing together what we need to know and do to be able to effectively produce language learning materials.
This webinar will introduce you to my framework, highlighting the key competencies I identified from my research. It will also show how anybody involved in language teaching (any language, not just English!), teacher training, managing schools or developing language learning materials can benefit from using the framework.
Operation “Blue Star” is the only event in the history of Independent India where the state went into war with its own people. Even after about 40 years it is not clear if it was culmination of states anger over people of the region, a political game of power or start of dictatorial chapter in the democratic setup.
The people of Punjab felt alienated from main stream due to denial of their just demands during a long democratic struggle since independence. As it happen all over the word, it led to militant struggle with great loss of lives of military, police and civilian personnel. Killing of Indira Gandhi and massacre of innocent Sikhs in Delhi and other India cities was also associated with this movement.
Embracing GenAI - A Strategic ImperativePeter Windle
Artificial Intelligence (AI) technologies such as Generative AI, Image Generators and Large Language Models have had a dramatic impact on teaching, learning and assessment over the past 18 months. The most immediate threat AI posed was to Academic Integrity with Higher Education Institutes (HEIs) focusing their efforts on combating the use of GenAI in assessment. Guidelines were developed for staff and students, policies put in place too. Innovative educators have forged paths in the use of Generative AI for teaching, learning and assessments leading to pockets of transformation springing up across HEIs, often with little or no top-down guidance, support or direction.
This Gasta posits a strategic approach to integrating AI into HEIs to prepare staff, students and the curriculum for an evolving world and workplace. We will highlight the advantages of working with these technologies beyond the realm of teaching, learning and assessment by considering prompt engineering skills, industry impact, curriculum changes, and the need for staff upskilling. In contrast, not engaging strategically with Generative AI poses risks, including falling behind peers, missed opportunities and failing to ensure our graduates remain employable. The rapid evolution of AI technologies necessitates a proactive and strategic approach if we are to remain relevant.
Introduction to AI for Nonprofits with Tapp NetworkTechSoup
Dive into the world of AI! Experts Jon Hill and Tareq Monaur will guide you through AI's role in enhancing nonprofit websites and basic marketing strategies, making it easy to understand and apply.
1. Groupware Evaluation:
An Overview
Irene-Angelica Chounta, Nikolaos Avouris
University of Patras, Greece
1
2. Overview
• A brief review on Groupware
Evaluation
• Lessons learned from evaluation
studies
• Discussion
2
http://hci.ece.upatras.gr
3. Groupware Applications
software systems that support people
involved in a common task to achieve
their goals
3
http://hci.ece.upatras.gr
4. Groupware Evaluation
the assessment of the strengths and weaknesses of
groupware applications and systems
Groupware systems are expected to support:
– communication between partners
– support for awareness of others' actions
– the establishment of shared understanding
and goals
4
http://hci.ece.upatras.gr
5. Objectives of groupware evaluation
• the groupware application
used as an interactive
system
• the mechanics of group
interactions or
communication means used
• the collaborative experience
• the group activity outcome 5
http://hci.ece.upatras.gr
6. Methods based on single user
evaluation practice
•use of heuristics
•user testing
•interviews & questionnaires
•focus groups
•ethnographic methods
Question:
Are they enough to capture completely the groupware case?
How easily are applied in groupware (e.g. groupware
evaluation in real-case scenario) 6
http://hci.ece.upatras.gr
7. Groupware-specific methods
• Mechanics of collaboration: Collaboration,
communication and awareness aspects, are analyzed
and mapped through small-scale actions (Gutwin, C.,
Greenberg, S., 2000)
• Collaboration Usability Analysis (CUA): high and low-
level representation of collaborative activity and the
interactions through field studies and task analysis
techniques (Pinelle,D. et. al., 2003)
Gutwin, C., Greenberg, S. “The Mechanics of Collaboration: Developing Low Cost Usability Evaluation Methods for Shared
Workspaces”, (2000)
Pinelle,D., Gutwin, C., Greenberg. S. Task analysis for groupware usability evaluation: Modeling shared-workspace tasks with the
mechanics of collaboration”, (2003)
7
http://hci.ece.upatras.gr
8. Analytical methods
• Groupware Task Analysis (GTA): Hierarchical
task analysis in combination with human
information processing models that focuses
on the triplet people, work, situation (Van der
Veer et. al., 1996)
• Distributed GOMS (DGOMS): representation
of group activity to predict execution time,
distribution of workload and other
performance variables. (Min, D. et. al., 1999)
Van der Veer, G.C, Lenting, B.,F., Bergevoet, B.A.J. “GTA: Groupware task analysis -- Modeling complexity”, (1996)
Min, D., Koo, S., Chung, Y.H., Kim, B. “Distributed GOMS: an extension of GOMS to group task”, (1999) 8
http://hci.ece.upatras.gr
9. Groupware-specific frameworks
• Participatory evaluation (PETRA): A framework
designed to address both theoretical concerns and practical
design issues of groupware evaluation (Ross, S., et. al., 1995)
• Modeling and mapping awareness within a
collaborative setting focusing on the central relationships
underlying the processes of distributed group work (Neale, D.,
et. al., 2004)
• Breakdown analysis classifying different breakdown and
repair scenarios to highlight implications (Hartswood, M. et.
al. 2000)
Ross, S., Ramage, M., Rogers, Y. “PETRA: participatory evaluation through redesign and analysis”, (1995)
Neale, D.C., Carroll, J.M, Rosson, M.B. “Evaluating computer-supported cooperative work: models and frameworks”, (2004)
Hartswood, M., Procter, R. “Design guidelines for dealing with breakdowns and repairs in collaborative work settings”, (2000)
9
http://hci.ece.upatras.gr
10. Examples of Evaluation Studies
•traditional HCI methods were combined with
groupware evaluation methodologies
•special emphasis to the effectiveness of
alternative awareness mechanisms.
•qualitative analysis of video to assess the
collaboration activity
10
http://hci.ece.upatras.gr
11. Study A
•Evaluation of a web-based
argumentation tool used by
communities of practice [1]
• heuristic evaluation for single-user interface was
combined with Groupware Heuristic Evaluation
• both synchronous and asynchronous collaboration
was studied
• Overall fifty (50) participants took part in the study
[1] Chounta, I.A., Avouris, N. “Heuristic Evaluation of an Argumentation Tool used by Communities of Practice”, (2009)
11
http://hci.ece.upatras.gr
12. Results of Study A
• Various aspects need to be studied and evaluated
separately for an overall assessment
• Expert based inspection methods need to be combined
with user observation
• We need not only focus on the collaborative
functionality but also on the user interface design
issues.
• Most of the issues observed in a collaborative session
were due to flows of the interface design rather than
communication and awareness problems.
12
http://hci.ece.upatras.gr
13. Study B
•Qualitative study of synchronous
collaboration for problem-solving
on allocation of attention resources
during different collaborative
sessions [1]
•Three dyads' practice was monitored by an eyetracker and
analyzed.
•The dyads were formed in order to study different group
dynamics.
•The logfiles of the collaborative activity were combined with
the logfiles of the eyetracker to analyze the interplay between
task, awareness mechanisms and collaborative practice.
[1] Chounta I.A., Avouris N. “Study of the effect of awareness on synchronous collaborative problem-solving”, (2010)
13
http://hci.ece.upatras.gr
14. Results of Study B
•awareness and communication failures are often
interpreted as unwillingness towards collaboration or
gradual lost of interest in a collaborative activity
•the lack of adaptive awareness mechanisms that help
users to set priorities leads users to withdraw from the
joint activity
•partners remain visible in the common workspaces,
are aware of the actions of their partners but take no
actual role in the collaborative activity
14
http://hci.ece.upatras.gr
15. Discussion
• The complexity of group activity setting makes it difficult
to identify the source of observed problems
• The outcome of collaborative activities rely on many
factors such as: the quality of collaboration, the context
of the activity and the tools that mediate the activity.
• Indications to use combination of single user evaluation
methodologies and groupware specific methods
• Still a long way to go in establishing an evaluation
framework for a wide range of collaborative applications
• Open issue: how to include the context of use and the
quality of the outcome in the evaluation process
15
http://hci.ece.upatras.gr
16. thank you
hci.edu.gr
16
http://hci.ece.upatras.gr