Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Chounta avouris limassol2011


Published on

Presentation slides from the 1st European Workshop on HCI Design and Evaluation, April 8th, Cyprus

Published in: Education
  • Be the first to comment

  • Be the first to like this

Chounta avouris limassol2011

  1. 1. Groupware Evaluation: An OverviewIrene-Angelica Chounta, Nikolaos Avouris University of Patras, Greece 1
  2. 2. Overview • A brief review on Groupware Evaluation • Lessons learned from evaluation studies • Discussion 2
  3. 3. Groupware Applications software systems that support people involved in a common task to achieve their goals 3
  4. 4. Groupware Evaluation the assessment of the strengths and weaknesses of groupware applications and systems Groupware systems are expected to support: – communication between partners – support for awareness of others actions – the establishment of shared understanding and goals 4
  5. 5. Objectives of groupware evaluation • the groupware application used as an interactive system • the mechanics of group interactions or communication means used • the collaborative experience • the group activity outcome 5
  6. 6. Methods based on single user evaluation practice •use of heuristics •user testing •interviews & questionnaires •focus groups •ethnographic methods Question: Are they enough to capture completely the groupware case? How easily are applied in groupware (e.g. groupware evaluation in real-case scenario) 6
  7. 7. Groupware-specific methods • Mechanics of collaboration: Collaboration, communication and awareness aspects, are analyzed and mapped through small-scale actions (Gutwin, C., Greenberg, S., 2000) • Collaboration Usability Analysis (CUA): high and low- level representation of collaborative activity and the interactions through field studies and task analysis techniques (Pinelle,D. et. al., 2003)Gutwin, C., Greenberg, S. “The Mechanics of Collaboration: Developing Low Cost Usability Evaluation Methods for SharedWorkspaces”, (2000)Pinelle,D., Gutwin, C., Greenberg. S. Task analysis for groupware usability evaluation: Modeling shared-workspace tasks with themechanics of collaboration”, (2003) 7
  8. 8. Analytical methods • Groupware Task Analysis (GTA): Hierarchical task analysis in combination with human information processing models that focuses on the triplet people, work, situation (Van der Veer et. al., 1996) • Distributed GOMS (DGOMS): representation of group activity to predict execution time, distribution of workload and other performance variables. (Min, D. et. al., 1999)Van der Veer, G.C, Lenting, B.,F., Bergevoet, B.A.J. “GTA: Groupware task analysis -- Modeling complexity”, (1996)Min, D., Koo, S., Chung, Y.H., Kim, B. “Distributed GOMS: an extension of GOMS to group task”, (1999) 8
  9. 9. Groupware-specific frameworks • Participatory evaluation (PETRA): A framework designed to address both theoretical concerns and practical design issues of groupware evaluation (Ross, S., et. al., 1995) • Modeling and mapping awareness within a collaborative setting focusing on the central relationships underlying the processes of distributed group work (Neale, D., et. al., 2004) • Breakdown analysis classifying different breakdown and repair scenarios to highlight implications (Hartswood, M. et. al. 2000) Ross, S., Ramage, M., Rogers, Y. “PETRA: participatory evaluation through redesign and analysis”, (1995) Neale, D.C., Carroll, J.M, Rosson, M.B. “Evaluating computer-supported cooperative work: models and frameworks”, (2004) Hartswood, M., Procter, R. “Design guidelines for dealing with breakdowns and repairs in collaborative work settings”, (2000) 9
  10. 10. Examples of Evaluation Studies •traditional HCI methods were combined with groupware evaluation methodologies •special emphasis to the effectiveness of alternative awareness mechanisms. •qualitative analysis of video to assess the collaboration activity 10
  11. 11. Study A •Evaluation of a web-based argumentation tool used by communities of practice [1] • heuristic evaluation for single-user interface was combined with Groupware Heuristic Evaluation • both synchronous and asynchronous collaboration was studied • Overall fifty (50) participants took part in the study [1] Chounta, I.A., Avouris, N. “Heuristic Evaluation of an Argumentation Tool used by Communities of Practice”, (2009) 11
  12. 12. Results of Study A • Various aspects need to be studied and evaluated separately for an overall assessment • Expert based inspection methods need to be combined with user observation • We need not only focus on the collaborative functionality but also on the user interface design issues. • Most of the issues observed in a collaborative session were due to flows of the interface design rather than communication and awareness problems. 12
  13. 13. Study B •Qualitative study of synchronous collaboration for problem-solving on allocation of attention resources during different collaborative sessions [1] •Three dyads practice was monitored by an eyetracker and analyzed. •The dyads were formed in order to study different group dynamics. •The logfiles of the collaborative activity were combined with the logfiles of the eyetracker to analyze the interplay between task, awareness mechanisms and collaborative practice.[1] Chounta I.A., Avouris N. “Study of the effect of awareness on synchronous collaborative problem-solving”, (2010) 13
  14. 14. Results of Study B •awareness and communication failures are often interpreted as unwillingness towards collaboration or gradual lost of interest in a collaborative activity •the lack of adaptive awareness mechanisms that help users to set priorities leads users to withdraw from the joint activity •partners remain visible in the common workspaces, are aware of the actions of their partners but take no actual role in the collaborative activity 14
  15. 15. Discussion • The complexity of group activity setting makes it difficult to identify the source of observed problems • The outcome of collaborative activities rely on many factors such as: the quality of collaboration, the context of the activity and the tools that mediate the activity. • Indications to use combination of single user evaluation methodologies and groupware specific methods • Still a long way to go in establishing an evaluation framework for a wide range of collaborative applications • Open issue: how to include the context of use and the quality of the outcome in the evaluation process 15
  16. 16. thank you 16