Academic Paper Writing Service http://StudyHub.vip/A-Laboratory-Method-For-Studying-Activi 👈

1. A Laboratory Method for Studying Activity Awareness
Gregorio Convertino (1)
, Dennis C. Neale (2)
, Laurian Hobby (2)
,
John M. Carroll (1)
and Mary Beth Rosson (1)
(1)
School of Information Science and Technology,
The Pennsylvania State University
University Park, PA 16802
{gconvertino, jcarroll, mrosson}@ist.psu.edu
+1 814 863 2476
(2)
Center for Human-Computer Interaction,
Department of Computer Science
Virginia Tech, Blacksburg, VA 24061
{dneale, lhobby}@vt.edu
+1 540 231 7542
ABSTRACT
Many failures in long-term collaboration occur because of a
lack of activity awareness. Activity awareness is a broad
concept that involves awareness of synchronous and
asynchronous interactions over extended time periods. We
describe a procedure to evaluate activity awareness and
collaborative activities in a controlled setting. The activities
used are modeled on real-world collaborations documented
earlier in a field study. We developed an experimental
method to study these activity awareness problems in the
laboratory. Participants worked on a simulated long-term
project in the laboratory over multiple experimental
sessions with a confederate, who partially scripted activities
and probes. We present evidence showing that this method
represents a valid model of real collaboration, based on
participants' active engagement, lively negotiation, and
awareness difficulties. We found that having the ability to
define, reproduce, and systematically manipulate
collaborative situations allowed us to assess the effect of
realistic conditions on activity awareness in remote
collaboration.
Author Keywords
Awareness, activity awareness, coordination, CSCW
ACM Classification Keywords
H5.m. Information interfaces and presentation (e.g., HCI):
Miscellaneous.
INTRODUCTION
Evaluation is crucial to both designers and researchers: it
allows the former to verify that a design proposal meets
users’ requirements, and gives the latter an opportunity to
formulate and refine their theories and models of human-
computer interaction (HCI). For user-centered design in
general, design and evaluation methods have evolved in
parallel. However, in computer-supported cooperative work
(CSCW), considerable effort has been put into the design of
new systems, with much less attention to the systematic
evaluation of CSCW systems [26]. This deficiency in
methodology may be due to the intrinsic complexity that
characterizes processes, products, and contexts of
collaboration [13]. Additionally, a fundamental limitation
of existing approaches to evaluate distributed CSCW
systems is that they tend to be method-driven rather than
theory-driven. Consequently, they do not sufficiently
inform researchers about how specific methods map to
specific constructs in CSCW, and what critical questions
should be answered [21].
Researchers in HCI have developed precise methods for
evaluating interactions between an individual and a
computer — for example: controlled experiments, time and
error studies of human performance, and mathematical
models of human behavior (e.g. Fitts’ Law). But for
CSCW researchers, there remain many open questions
concerning how if at all collaboration can be studied with
cost-effective methods and in controlled settings without
compromising the ecological validity of the studied
phenomena.
In this paper, we describe a method developed to evaluate
collaborative activities in a controlled setting. The activities
invoked are modeled on real world collaborations
documented earlier in a field study. More specifically, we
developed the laboratory methods to study a pervasive
collaborative phenomenon that impacts the success of
CSCW systems - activity awareness [4]. Activity awareness
subsumes the prior notions of awareness in collaboration
(e.g., workspace awareness or awareness of asynchronous
collaboration) and refers to the understanding that
collaborators have of prior, current, and future activity.
Although we designed and implemented a method to study
activity awareness, this approach can be used to study other
aspects of collaboration that are heavily dependent on
context and that arise during extended collaboration
activities.

2. Typically, the rich interactions of collaborative work are
examined through field studies and ethnographic methods.
On one hand, these observational methods are essential for
understanding authentic practices, but on the other they
demand large investments of resources and time.
Additionally, field methods do not lend themselves to direct
manipulation or control of the studied phenomena (a field
study may allow for process or system changes to be
studied, but it is often indirect, uncontrolled, and slow to
produce results). To investigate the relationships between
specific contextual events, such as collaborative
breakdowns and their effect on collaborative phenomena
(awareness, coordination, common ground, and planning),
we need research methods that allow systematic
investigation of such relationships within semi-naturalistic
controlled settings.
Researchers emphasize that CSCW evaluation should occur
in the context of actual use [31, 25]. Unfortunately,
fieldwork evaluation does not mesh well with fast paced
system development lifecycles. The procedures adopted in
the field are often opportunistic and based on informal
evaluation sessions [31]. As a result, CSCW systems have
failed to a much greater degree than single-user systems
due to inadequate usability feedback, and more importantly,
feedback concerning the proposed functionality.
One approach to more systematic CSCW evaluation is to
simulate real usage conditions: evaluate functionality and
usability in semi-controlled conditions designed to mirror
real-world contexts. Given the difficulty of reproducing
such a realistic setting, researchers in CSCW have recently
developed discount evaluation methods extending the
methods developed for individual-based applications to
collaborative systems (basic inspection [29]; cognitive
walkthrough [23]; heuristic evaluation [2]). Specifically,
some authors [14, 2] have advocated the use of heuristic
evaluation methods customized for groupware systems.
Instead of studying real work situations, groupware systems
can be evaluated by experts with reference to a set of
CSCW usability principles – usage issues influencing the
mechanics of collaboration [2]. Others have proposed new
modeling techniques, such as Collaboration Usability
Analysis [24] (a task analysis technique for studying
teamwork), to provide evaluators with new analysis
schemes that are appropriate for groupware usability
evaluation.
Discount evaluation methods are an important
methodological contribution because they promote savings
in costs and allow evaluation to take place during early
development, when there is no operational prototype for
users to test in real work settings [24]. However, they are
unlikely to be sufficient to support the full evaluation
“lifecycle” of CSCW systems. For example, Cockton and
Woolrych have discussed several methodological problems
with discount usability evaluation methods [6]. They
suggest that evaluation methods should be judged not
simply for their benefits (e.g., rapid feedback), but also for
their cost and risk of error (e.g., poor ecological validity).
Continuing this general argument, we propose that
controlled laboratory methods should be judged with
respect to both their benefits (control and precision) and
risks (poor ecological validity). More specifically, we argue
that by modeling laboratory tasks on field observations and
simulating them in controlled settings, we can obtain the
benefits of a laboratory study, while at the same time
reduce our risk of missing important properties relevant to
real system usage.
To address CSCW evaluation in all stages of development,
we present a laboratory method that was conceived as part
of a larger research paradigm to study collaborative
activities through both field and laboratory studies.
Although the laboratory methods are grounded in the results
of field research, ultimately we propose a research
paradigm that is bidirectional: laboratory studies should be
able to inform field studies and vice versa. We believe that
doing research on collaboration by interleaving lab and
field studies will increase the heuristic potential of both
methodologies, offset the drawbacks of individual methods
[19, 3], and reveal new important aspects of the studied
phenomena – in this case activity awareness. In this paper,
we describe our first instantiation of the laboratory method
and show that it is a valid CSCW experimental approach.
ACTIVITY AWARENESS
The CSCW problem domain we used to instantiate and
validate our laboratory methods was the study of the high-
level and complex collaborative phenomenon of activity
awareness [4].
Adopting the notion of activity to study awareness in
collaborative work requires a conceptual shift from a
knowledge-centered to an activity-centered perspective.
Several studies that have investigated awareness in
individuals and teams have focused on knowledge
representations that underlie awareness (e.g., theorizations
on situation awareness by Endsley [9] and Salas [27]).
These studies have focuses on individual and shared mental
models as precursor products of team’s awareness. We
argue that the notion of shared mental models attributes
more emphasis to convergence than divergence among
team members [20] and characterizes teamwork through its
products (knowledge) rather than group process. This
reduces the explanatory power that this theoretic approach
has on the complex and dynamic phenomenon of
awareness. In fact, awareness is not only dependent on
shared knowledge and common ground, but is also directly
related to members’ coping with divergent views, resolving
conflicts and uncertainty, adapting to changes over time,
and understanding how current activities fit with prior
activities, future plans, members’ roles, norms in the group,
and in the larger socio-organizational context.
Understanding, defining, and operationalizing the many
roles of awareness in collaboration is a key problem for the
success of CSCW systems [16]. Much of the research in

3. this area has focused on social (who is present) and action
(what is s/he doing) awareness. However, field observations
have shown that even when social and action awareness are
supported, many disruptive awareness breakdowns still
occur. Most of these problems can be attributed to the lack
of activity awareness [4]. Activity awareness refers to
people’s ability to get and maintain “the big picture” about
the ongoing collaboration while they are working together
on long-term projects. Activity awareness is a precondition
for effective communication, planning, coordination,
decision-making, and actions during long-term
collaboration.
With respect to the other concerns about awareness in
CSCW research (e.g., workspace and social awareness),
activity awareness is articulated at a higher level of
analysis. Thus, it has a larger scope since it pertains to the
understanding of the overall activity being performed in the
collaboration. Drawing from the conceptualization of
activity in activity theory [18], we consider an activity as a
sequence of actions, directed towards a goal or object,
mediated by tools, and situated in many embedded contexts
(e.g., work practices, culture, organizational structures,
interpersonal relations). Activity awareness pertains to
group activity that takes place over an extended period of
time. This implies that in order to maintain awareness of the
entire activity, the group members need to develop and
maintain common understanding of shared goals, plans,
norms and roles; monitor the resources over time; and
remain aware of the actual status of the execution of the
group activity and its relationship with the prior aspects.
Little empirical research has been conducted on activity
awareness. Several authors have proposed different
categories of awareness: peripheral, situational, informal,
group, social, workspace, etc. However, in order to
efficiently support synchronous and asynchronous
collaboration within long-term planned activities, we need a
broader view of awareness that interprets the multiple
categories of awareness and their evolution in the larger
context of activities. Because activity awareness implies an
extended collaboration process, it was an ideal candidate
for exploring our new laboratory methods. In this initial
investigation, we began by modeling and studying activity
awareness breakdowns observed from a field setting [4].
MODELING REAL-WORLD CONTEXTS
The field study that grounded our laboratory work involved
inter-classroom collaboration in a long-term school project.
Small groups of middle school students, typically 3 students
using one computer, were paired across two different
classrooms (6th
and 8th
grades) and collaborated remotely on
a science project. The remote collaboration was supported
through the BRIDGE software, a Java-based collaborative
system [11], which provided the users with planning tools
(calendar and timeline) and a collaborative multimedia
notebook [4]. A multifaceted evaluation framework was
adopted to identify factors that disrupted or contributed to
activity awareness, and data was collected about the process
and outcomes of collaboration [22].
We analyzed our field study records to categorize the
circumstances under which activity awareness breakdowns
occurred. On this basis, we developed a set of collaborative
scenarios to be used in our laboratory study. For example,
we found that collaborative breakdowns are distributed
across collaborators and contexts and often involve multiple
people and actions to identify and repair. They inherently
involve events with interdependencies and multiple
consequences for different collaborators. Situation, group,
task, and tool factors were identified as four fundamental
categories classifying activity awareness problems.
Leveraging this framework and its ability to identify
activity awareness problems occurring in similar contexts,
we modeled the scenarios used in this research on these
four factors.
A total of seven scenarios were used in the lab study (Table
1). Each involved the participation of a confederate [7, 33]
who was a trained research assistant playing the part of a
remote collaborator. In the first scenario, Tool Use, the
confederate simply encouraged the participant to use a
planning tool. Each of the remaining scenarios then
“implemented” a typical awareness breakdown observed in
the field [4], and corresponded to one of the factors
identified in the fieldwork analysis: situation, group, task,
and tool (Table 1, second column). The scenario introduced
the breakdowns through a combination of instructions to the
confederate and changes made to the collaborative
environment (i.e., before beginning a given session).
Additional Work and Task Data Changes were determined
by alterations in the length and the content of the task
respectively (task factors); Schedule Changes and
Completion Failure were motivated by changes in the class
schedule and unavailability of internet connection
(situational factors); Role Changes was implemented as the
partner’s tendency to work alone combined with
uncertainty about the partner’s abilities (group dynamics);
Tool Changes was determined by the individual decision of
putting data in a different tool (tool factors). These
scenarios describe unexpected changes occurring within
specific collaborative contexts.

4. THE LABORATORY METHOD
Based on the data collected from the field study, we
developed a method that allows simulating and
manipulating authentic collaborative situations in a
laboratory setting. This method is characterized by three
major properties:
1. The use of authentic tasks and collaborative situations
2. The use of a confederate
3. The use of multiple collaborative sessions over time
The first characteristic refers to collaborative scenarios that
were developed from field observations. This aspect of the
lab method directly supports ecologic validity of the
collaborative context and the realism of the tasks performed
during the collaboration. The second characteristic was
introduced to reduce the sources of variability in the
laboratory setting and to control and manipulate the setting
systematically. The third characteristic was motivated by
the necessity to study realistic long-term activities and the
factors that influence the way these activities unfold. By
using realistic long-term activities, we can investigate the
evolution of complex phenomena (i.e. planning and activity
awareness). In fact, long-term collaboration requires users
to be aware of state information about the workspace and
the shared plans that constantly change within and between
the collaborative sessions.
Participants and laboratory setting
The participants were six 7th and 8th graders (P1… P6):
three females and three males. They participated in four
weekly laboratory sessions in which they had to collaborate
through a CSCW system (described below, see Figure 1)
with another student (the confederate) on a group project.
During the four sessions they had to complete an
environmental project (Environmental Quality by NeoSCI
Corporation). Each session lasted approximately one hour.
The laboratory setting is an important component of our
method. The participant and the confederate were located in
separate rooms. Both were able to talk to the experimenter
through a microphone and a video camera, and they were
both able to hear the experimenter through computer
speakers. The experimenter had visual and auditory access
to both participant and confederate, and was able to switch
the audio communication between the two. Through the use
of four screens (two displaying the participant’s and
confederate’s collaborative workspace, and two showing
their interaction captured from the video cameras), the
experimenter could give personalized instructions, take
time-stamped notes, and guide the sessions. The
experimenter also had his own computer station running a
Groove client (Figure 1). This allowed the experimenter to
access the collaborative workspace and intervene when
necessary. The confederate and participant could not see or
hear each other, and both were physically separated from
the experimenter. In the context of a realistic collaborative
situation, this setting allows the experimenter to monitor
and influence the participant or confederate individually
without impacting the other.
Figure 1. Groove workspace [12]. The major components of
the user interface: buddy list (A), planning tool (B), the actual
set of tools hierarchically organized within a large tabbed
panel (C), and the chat tool (D). Several awareness features
are supported (e.g., small pop-ups and notifications inform the
user about partners' movements through the workspace or
when they are typing a new message (E)).
Scenario Breakdown
factor
What the confederate (C)
does in the scenario
Tool Use
Tool factors:
the planning
tool is used
The confederate (C) encourages
the use of a planning tool
Additional
Work
Task factors:
the task is
extended
C completes additional work
because of new teachers’
instructions: 3 additional
vocabulary terms were added
Schedule
Changes
Situational
factors: the
class schedule
changes
C changes the dates in the
planning tool: two dates were
changed in the planning tool
because the class schedule
changes
Completion
Failure
Situational
factors:
unavailability
of the internet
connection
C fails to complete a task
because of local contingencies:
additional information was not
gathered from the Web
Role
Changes
Group
dynamics: a
task is executed
ahead of
schedule
C executes a task ahead of
schedule because of his habit to
work alone and uncertainty
with the partner’s abilities
Task Data
Changes
Task factors:
the content of
the task
changes
C executes a task because of
new teachers’ instructions: the
levels of pollutant considered
were different from what is
listed in the activity guide
Tool Change
Tool factors:
the task is
completed in a
different tool
C completes a task in a tool that
is different from the one they
had previously agreed on
Table 1. Scenarios summarized by breakdown factor and
confederate’s activities.
A
D
E
C
B

5. Tool and tasks
The participants collaborated on a long-term project
through Groove, a computer-supported collaborative tool.
This is a groupware system that supports asynchronous and
synchronous collaboration through a shared workspace [12]
(Figure 1).
The group project included the following collaborative
activities that were conducted in the Groove workspace:
Getting Acquainted (responding to questions about personal
experiences with environmental problems); Identifying the
Problem (identifying the overall goal of the project);
Developing a Plan (planning the activities for the entire
project); Vocabulary (defining a small set of scientific
terms); Research Questions (identifying the questions to be
investigated); Web Research (collecting relevant
information from the Web); Develop and Conduct two lab
studies (collecting lab data about acid rain and water);
Organize, Graph, and Analyze Data (presenting the data
collected in the laboratories studies and from the Web);
Draw Conclusions and Final Report (reporting about the
whole project).
Based on the categorization of activity awareness problems
that emerged from the fieldwork [4], we developed a set of
scenarios that modeled the circumstances in which
awareness breakdowns occurred (see the section “Modeling
real-world context”). The scenarios were simulated through
the confederate, who followed loosely scripted activities
during the four collaborative sessions. The confederate
played the role of a middle school student of the same age
and gender. The use of pairs of the same gender allowed
mitigating any confounding dynamics that could occur
among males and females students of this age group.
Here we provide an example of a script used by the
confederate to simulate the scenario ‘Schedule Changes’.
Scenario: Schedule Changes. Because of a change in the teacher’s
planning of class activities, the dates of two activities in the project
manager have been changed.
The day after your meeting with your partner, your teacher has decided
to swap the order of two class activities. Both of these activities are
related to the work you are doing in the project. Since she requires you
to perform the project task in parallel with the related class activity, she
has asked you to adjust the plan about the lab activities. Following her
suggestion, you have changed the schedule regarding the two activities
in the Project Manager.
Experimental procedure
This section describes the organization of the four sessions
(Table 2). In the first session, after signing the informed
consent, the participant was informed that s/he was going to
work with another middle school student who was located
in a neighboring school. The participant then read a
description of the experimental procedure and a brief
outline of the project activities. After s/he was trained on
how to use the workspace, s/he was given a demonstration
on how to think-aloud during the session.
The tasks to be accomplished during the first collaborative
session included three activities: Getting Acquainted,
Identifying the Problem, and Developing a Plan.
During the other three collaborative sessions, the
participants and the confederate had to plan their work for
the week and perform the scheduled activities. In the time
interval between the collaborative sessions, the participants
did not have to actually do the work. Instead, they received
all their work for each session when they arrived. This
simulated the work and allowed for a greater level of
control. The schedule followed during the four sessions is
summarized in Table 2.
Sessions Activities within and
between sessions
Scenarios
Session 1:
1. the participant received:
a) detailed information
about the project,
b) basic training on how to
use Groove workspace and
the think-aloud technique.
2. collaborative session
All participants were
exposed to the scenario Tool
Use
between
sessions
The participant received the
tasks already accomplished
by email and read it
The workspace was modified
according to the scenarios
run in the second session.
Session 2:
1. the participant was asked
to insert her/his work in the
workspace and then explore
whole the content
2. collaborative session
All participants but P1 were
exposed to the scenarios:
Task Expands a n d
Schedule Changes
between
sessions
The participant received the
tasks already accomplished
by email and read it
The workspace was modified
according to the scenarios
run in the third session.
Session 3:
1. the participant was asked
to insert her/his work in the
workspace and then explore
the whole content.
2. collaborative session
All participants were
exposed to one or more of
the scenarios Completion
Failure, Role Changes, and
Task Data Changes
between
sessions
The participant received the
tasks already accomplished
by email and read it
The workspace was modified
according to the scenarios
run in the fourth session.
Session 4:
1. the participant was asked
to insert her/his work in the
workspace and then explore
the whole content.
2. collaborative session
3. questionnaire, interview
All participants were
exposed to one or both the
scenarios C o m p l e t i o n
Failure and Role Changes.
after the
fourth
session
The participant was paid
and was informed about the
simulation
Table 2. Collaborative sessions and scenarios schedule. The
table summarizes the schedule of activities performed (second
column) and scenarios run (third column) by session.
At the end of each session the participants were asked for
informal feedback regarding the recent session. At the end
of the experiment, the participants filled out a questionnaire
and participated in an interview. Finally, participants were
compensated $20 for their participation and were debriefed.

6. Before the experiment, the confederate was trained to
simulate the scripted scenarios using Groove, (i.e. the
experimenter acted as a subject and the confederate worked
with the system to complete the script). Moreover, before
each collaborative session the confederate reviewed the
scripts for each scenario (Table 1) scheduled and then
discussed with the experimenter how to flexibly adapt the
scripts to the specific participant. Except for the first
scenario, each scenario was scheduled in accordance with
the plan of the activities defined by the participant and
confederate dyad (see third column on Table 2).
Data collection and analysis
Multiple data collection techniques were used. During each
session, the interaction between participant and confederate
was synchronously monitored and recorded using video
cameras and a screen-capture tool. Both participant and
confederate used the think-aloud method to inform the
experimenter about what was occurring during the session.
A session-by-session logging captured changes to the
workspace and the tasks assigned. The confederate also
made notes during the experiment. Finally, a small
questionnaire (Table 3) was given and a semi-structured
interview was administered at the end of the last session.
We also used contextual inquiry to interview the
participants during the collaborative sessions.
We conducted three different types of analysis on the data
collected: analysis by scenarios, questionnaire and
interview, and breakdown analysis.
In the analysis by scenario, the participant’s activity
awareness was assessed with respect to the changes (related
to situation, people, task, and tool) occurring in each
collaborative scenario. Two judges, both graduate HCI
researchers, conducted the assessment for the level of
awareness in participants by adhering to the following
coding scheme:
1. Participants were evaluated ‘fully aware’ when they
had spontaneously noticed the inconsistencies.
2. They were evaluated ‘partially aware’ if they noticed
the inconsistencies after being prompted by the
confederate or the experimenter.
3. They were considered ‘unaware’ in all remaining
cases.
In the case of ‘fully aware’, the participant directly (e.g.
through explicit statements) or indirectly (e.g. through
related comments or actions) showed that s/he was
conscious of the changes that occurred in a specific
scenario. In the second and third cases, the participant did
not become spontaneously aware after being exposed to the
change. It was only after s/he moved on to another activity
that was unrelated to the scenario, that s/he was given one
prompt (through a comment or a question) from either the
experimenter or the confederate (directed by the
experimenter). If the participant then provided any direct or
indirect signs of being aware then s/he was evaluated as
‘partially aware’. Otherwise, the participant was considered
‘unaware’.
The questionnaire (Table 3) contains thirteen statements
about activity awareness using a 7-point Likert-type scale
modeled on the rating scales proposed by Watts et al. [31].
A follow-up semi-structured interview was also used to
collect qualitative data from the questionnaire
(interpretation and reasons for the answers to each item)
and to gather some additional issues that had emerged
during the experiment.
The breakdown analysis was conducted with explicit
definitions of breakdown and critical incident. A
collaborative breakdown occurs in an interaction when the
expectations of one participant do not match with the action
of another [34, 8]. Partially overlapping with this concept,
we consider critical incidents as behaviors and experiences
leading to surprisingly bad or good results [10].
Consistently with the evaluation framework used for the
analysis of the breakdown in fieldwork [4], we considered
that these breakdowns might be determined by different
classes of factors: situational (environment), group (users
and their roles), task (plans), and tool (tools and
workspace). Using this evaluation framework, the two
judges analyzed the communication transcripts to identify
instances of collaborative breakdowns and critical incidents
and to classify them according to the framework. Then the
judges compared, discussed, and agreed upon each case.
RESULTS: A VALID LABORATORY METHOD OF
ACTIVITY AWARENESS
In this section we present a collection of results that support
the validity of this laboratory method for studying activity
awareness. The validity of the method is assessed by
comparing the results of our study (collected through the
three types of analyses) to what we observed in the field. If
the results of this method are representative of what occurs
in the field, then the initial stages of validating the method
will have been achieved. From the scenario analysis, we
observed several attributes of real collaboration occurring
across the four sessions. The participants appeared visibly
interested in the activities and generally motivated to work
with their partner (only one participant was noticeably less
interested in the topic, but this appeared to be due to
personal factors of the participant). For example, in
response to the Tool Use scenario, all the participants
agreed to use one of the planning tools, and continued to
use it during the remaining sessions. The participants did
not need to be prompted to keep using the planning tools
after being exposed to the Tool Use scenario, demonstrating
that they were highly engaged over the four sessions.
In all the remaining scenarios, several activity awareness
problems were observed. In more than half of the situations
(18/31), participants did not become aware of the changes
introduced by the scenarios. Even among the situations in
which participants were aware (13/31), only in 30% of the

7. situations (4/13) were changes noticed after being prompted
by the experimenter or the confederate. In several
situations, the participants were not fully aware of the
changes made to the content, workspace, and the tasks that
they had agreed to perform.
Other awareness problems appeared to be caused by the
lack of a clear overview of the shared plan. In fact, in
several cases participants appeared to not fully understand
the duration of tasks, and tended to underestimate the time
needed for the whole project. They tended to refine their
plans more during multiple sessions, negotiating decisions
with their partner as the work unfolded. For example, P1,
P3, and P6 kept readjusting their plan until the third or
fourth session. Defining a clear shared plan during the
collaboration and maintaining a constant awareness of the
plan and time needed for each activity are difficult in real
collaborations, and the results from our laboratory method
confirm this. These activity awareness difficulties assessed
during the experiment appear representative of ones
observed in real collaborative contexts and with theoretic
accounts for the importance of opportunistic aspects of the
planning activity [17, 30].
Examples A. Negotiation and collaborative problem solving
• P1 discusses with the partner how to split the work and how to
realize each activity. Later, he will propose to assign a priority
value to each one of them.
• P3, after noticing that they did not have enough time to complete
the project, invites the partner to re-examine the plan.
• P5 actively debates with her partner about the organization of the
final report until they reach an agreement.
• P6 evaluates with the partner about the time to be assigned to the
final report.
Examples B. Creativity
• P1, when adding the tasks to the planning tool, proposes to
distinguish and assign different roles so that one can add the tasks
to the tool and the other evaluates the accuracy and suggests
changes.
• P3 and P5 decided to use both Calendar and Project Manager,
and, in order to transfer the data from one tool to another, they
strategically coordinated the work of the pair so that one of them
would read and type in the chat the tasks and the other would
insert the tasks into the planning tool.
Another aspect that confirms the validity of our method is
that during the collaborative sessions, the participants
engaged in lively negotiation and collaborative problem
solving with their partner (see examples above). We also
observed several cases in which the pairs of collaborators
accomplished some tasks creatively by defining their own
strategy (see examples above). This shows that the
experimental procedure was flexible enough to allow the
pair to organize the work creatively, as students are often
encouraged to do in school contexts.
Finally, the analysis of the different data collected during
the interaction (videos, the chat history, and think-aloud
verbalizations) showed that the level of engagement of the
participants clearly increased when they had non-task
related communication with the partner. For example, with
P6 the confederate engaged in a conversation about Harry
Potter and with P5 about a recent movie. When the
confederate promoted more interpersonal and informal
communication with three participants, they appeared to
enjoy the opportunity to communicate informally and were
more engaged in the activities. Being more familiar with
their partner seemed to help.
Questionnaire Av. s.d.
1. I found it difficult to tell what work my partner had
done after being absent from the workspace for a week.
2.8 1.7
2. It was easy to find what my partner had worked on in
the collaborative space.
5.0 1.5
3. I could tell what my partner was doing while we
were collaborating online.
6.2 0.8
4. I always knew what my partner was going to work on
over the week.
6.3 0.8
5. It was always clear what my partner was going to do. 5.7 0.5
6. I became more aware of my partner’s plans over
time.
6.2 1.2
7. My partner and I planned adequately. 5.8 0.8
8. My partner and I communicated well with each
other.
6.5 0.5
9. My partner collaborated with me to complete the
project.
6.5 0.5
10. My partner contributed equally to this project. 7.0 0.0
11. I enjoyed collaborating with a partner online. 6.5 0.5
12. I would enjoy interacting with others in the
community (outside of the school system with interest
or knowledge in science) on my group science project.
4.8 1.8
13. I would prefer to work on group projects over other
types of school learning activities.
4.2 1.5
Table 3. Questionnaire results with average rankings and
standard deviation. The 7-points Likert-type scales range
from: 1 = Strongly Disagree to 7 = Strongly Agree.
These results are also confirmed by data from the
questionnaire (see Table 3) and the interview. Specifically,
the results from the questionnaire show that participants felt
they were collaborating during the experiment, enjoyed the
experience, and were satisfied about the work and their
partner. Moreover, the same participants who had engaged
in non-task related communications appeared to enjoy the
collaboration with their partner more. During the interview
they explicitly stated that communicating informally with
their partner had made the other person more familiar and
intimate (a student like them), and created a more realistic
and informal context of collaboration. This supports the
ecologic validity of the method since in the real world
social behavior and work are always interleaved.

8. Using the evaluation framework defined in the analysis of
the breakdowns within the prior fieldwork (see sub-section
“Experimental procedure”), we conducted both a qualitative
and a quantitative analysis of the data about breakdowns
extracted by the two judges. We found that breakdowns and
critical incidents occurred during all four sessions and did
not appear to be directly related to the manipulation
introduced through the scenarios. For example, in the first
session, although the scenario Tool Use did not expose the
participants to any inconsistencies, at least one third of the
total breakdowns occurred within this session.
The analysis of the breakdowns by category has shown that
in more than one third of the cases (37%) the breakdowns
occurred because of problems related to communication,
roles, and the relationship between partners. About another
third (32%) of the breakdowns were determined by task
factors. Less that one fourth (23%) were related to tool
factors and a small portion (7%) were caused by situational
factors. We observed that the number of cases of
breakdowns tended to decrease along the four sessions
(44%, 22%, 19%, 15%). This trend is easily explained by
the fact that the participants gradually became more
familiar with the tool, the task, and the partner.
The problems related to communication, roles and
relationship between partners have been well acknowledged
by studies of computer mediated communication in real
settings, and have been considered to be a consequence of
the restrictions imposed by the medium of communication
(e.g., limited support for non-verbal communication,
reduced number of auditory cues, deictic and spatial co-
references that are difficult to resolve). Collaborators need
to constantly repair or remediate miscommunications and
undertake explicit actions to maintain common ground [5]
and reciprocal awareness among the collaborators.
The category of breakdowns related to tasks and plans
appeared strictly associated with participants’ problems in
activity awareness. Several breakdowns that we identified
as part of this category occurred when participants noticed
that their plan was inappropriate because various
constraints (e.g., duration or order of the activities) had not
been considered (P1, P3). Moreover, in some conditions the
participants were not able to predict what they needed to do
next, and they appeared confused (P4, P6). These types of
breakdowns reveal that in these situations the participants
were unable to maintain a clear overview of the plan and of
the time available, and were not fully aware of the current
status of the work. This essentially reveals that they were
lacking activity awareness.
The results obtained through the three different types of
analysis show that this laboratory method was a valid
model of real collaboration. In fact, this method was able to
promote engagement, lively discussion, autonomous
initiatives, collaborative problem-solving, and activity
awareness difficulties, which are representative attributes of
real collaboration, and that were also observed in field
studies conducted in similar conditions.
DISCUSSION AND CONCLUSION
In the previous section we presented data and arguments in
support of the validity of our laboratory method. Our
purpose was to provide evidence of ecological validity; that
is, the extent to which our results were comparable to what
happens in the “real world”. Integrating the data from the
three different types of analyses shows that several
attributes of real collaboration were exhibited during the
experimental simulations: engagement, lively discussion,
autonomous initiatives, collaborative problem solving, and
activity awareness difficulties.
This method also has provided us with relevant information
about activity awareness. It allowed us to assess the
participant’s level of activity awareness within the
collaborative scenarios simulated through the confederate.
Moreover, by systematically manipulating the collaborative
conditions, we were able to identify possible factors that
affect participant’s activity awareness. Our analysis by
scenario has shown that the participants’ level of awareness
varied for the different scenarios and across the four
sessions. The difference between scenarios is particularly
evident if we compare the scenario Completion Failure and
Additional Work. While all the participants were aware of
the inconsistencies that occurred in the Completion Failure,
none of them were aware of the breakdowns introduced
during the Additional Work scenario. This difference
suggests that participants were generally more aware of
macroscopic changes occurring to objects in the workspace
(Completion Failure, Role Changes, and Tool Changes)
rather that of changes of “smaller granularity” occurring to
symbols (Additional Work, Task Data Changes) within the
workspace. Another possible factor affecting participants’
level of awareness is the growing familiarity with the
workspace, partner, and tasks during the experiment. We
also observed that as participants advanced through the four
sessions there was an increase in the number of cases of
participants’ awareness of said inconsistencies. This finding
was confirmed by all three type of data analysis. For
example, the breakdown analysis had shown that the
number of cases of breakdowns tended to decrease along
the four sessions.
Based on our systematic manipulation of the experimental
conditions, we observed an increased trend in the subjects’
level of activity awareness over the four sessions. This
trend can be explained by general explanatory concepts
used in HCI and CSCW such as expertise, familiarity [15],
and increasing common ground [5]. However, future
investigations are needed to evaluate the specific role that
these factors play in activity awareness. Further
investigation of these issues will require settings where
factors such as familiarity and common ground can be
manipulated. Since these background factors are difficult to
control in field studies, a valid and cost-effective method

9. for research in this area would be the laboratory method we
have presented.
Having the ability to design, reproduce, and systematically
manipulate collaborative situations in semi-naturalistic
settings means researchers can develop ad hoc
implementations of this method suited to study complex
collaborative phenomena. This can be accomplished by
using a limited investment of resources and time. The
method of manipulating the experimental setting and using
collaborative scenarios makes this method categorically
different from field studies, yet complementary to them. For
example, contextual factors cannot be controlled within the
field studies, but they may have very complex influences on
the variable of measurement in the field. At the same time
manipulation is possible in a lab simulation, but the
contextual factors would be modeled using criteria and
results derived by empirical studies in the field. In
agreement with other authors [1, 32], our research paradigm
to study collaboration combines methods from different
research traditions in order to provide richer and broader
contributions to our understanding of groups as complex
systems. In fact, in our line of research we investigate
activity awareness by interleaving lab and field studies.
The use of a multiplicity of data collection techniques in an
integrated fashion allows maximization of the amount of
knowledge extracted from the phenomena investigated by
triangulating or mediating [3, 28] multiple types of data and
interpretations. In our study, on several occasions, we were
able to integrate and triangulate data from the different data
collection techniques to increase our understanding of the
data and test the validity of the method. For example, the
fact that the participants’ behavior evolved along the four
sessions was confirmed by different types of data: the
increasing trend in the number cases of participants’
awareness of inconsistencies and the decreasing trend in the
number of breakdowns occurred.
Our results show the possibility for more articulated
investigations of the contrasting impacts that contextual
factors have on people’s overt and covert behaviors.
Specifically, we observed that in some cases the data
collected through direct observation of overt behaviors
were inconsistent with the participants’ individual
experiences. Through alternative active investigation
techniques (through contextual inquiry, think-aloud
protocol and interviews), we were able to investigate
underlying experiences of the participants (e.g., personal
reaction to the partner’s behavior) that would not be
observable from their overt behavior during the session. An
example of this phenomenon was P2’s reaction to the
scenario Role Changes, where the confederate had
completed additional work without any prior agreement
with her. During the collaborative session with her partner,
she appeared pleased that her partner had accomplished
some extra work ahead of time. However, after the session,
when she was asked to comment on this specific situation,
she expressed that she had felt disappointed. These types of
inconsistencies would have been difficult to study in the
real world. People, in fact, tend to conform to the norms
and constraints of the context (e.g., social rules); for
example, the participants avoided to express personal
concern about their collaborators and their work (e.g. see
results for the 10th
item of the questionnaire, Table 3).
Although we designed and implemented the method to
study activity awareness, this laboratory method might also
be used to study other aspects of collaboration arising in
real contexts and during extended collaboration activities.
For example, from our data analysis we were able to
observe phenomena related with social and action
awareness, planning, common ground, familiarity, learning,
etc. We believe that this method may offer a new
opportunity to study these phenomena in the laboratory
using valid methods.
FUTURE WORK
Future work should focus on the research method, the data
collection, the data analysis, and the domain of study. To
improve our research method in the future, specifically the
collaborative setting, we plan to increase engagement and
ecological validity: increase realism by making the
participants actually complete the assigned work, which has
a cost-benefit tradeoffs; construct multiple session
scenarios, which would assent to observing how scenarios
affect collaboration in the long term; and, encourage
informal communication between participant and
confederate before and after each session, which can
become a motivational factor that induces engagement and
intimacy. Another area that has potential for improvement
is the think-aloud method, specifically as a formative data
collection technique for activity awareness. By asking the
participants to think-aloud about what they think their
partner knows, we could increase our ability to monitor
their awareness knowledge level. Our study also allows for
future work in the area of awareness-sensible designs for
CSCW systems within education. Particularly, the issues of
human-computer interaction associated with middle and
high school students’ use of technology and their
developing meta-cognitive and meta-communication
abilities could be further developed.
REFERENCES
1. Arrow H., McGrath, J. E. Berdahl, J. L., Small Groups
as Complex Systems: Formation, Coordination,
Development, and Adaptation. Sage Publications, 2000.
2. Baker, K. Heuristic Evaluation of Shared Workspace
Groupware based on the Mechanics of Collaboration.
MSc Thesis, Department of Computer Science,
University of Calgary, Alberta, Canada, 2002.
3. Campbell, D., & Fiske, D. W. Convergent and
discriminate validation by the multitraitmultimethod
matrix. Psychological Bulletin 4, (1959), 297-312.
4. Carroll, J. M., Neale, D. C., Isenhour, P. L., Rosson, M.
B., and McCrickard, D. S. Notification and awareness:

10. synchronizing task-oriented collaborative activity. Int..
Journ.. of Human-Computer Studies 58, (2003) 605-32.
5. Clark, H. H. Using Language. New York: Cambridge
University Press, 1996.
6. Cockton, G., & Woolrych, A. Sale must end: Should
discount methods be cleared off HCI's shelves?,
interactions 9(5) (2002), 13-18.
7. Connolly, T., Jessup, L., & Valacich, J.S., "Effects of
anonymity and evaluative tone on idea generation in
computer-mediated groups," Management Science 36,
(1990), 689-703.
8. Easterbrook, S. (ed.), CSCW: Cooperation or Conflict,
Springer-Verlag, New York, 1993.
9. Endsley, M. R., Toward a theory of situation awareness
in dynamic systems. Human Factors 37, (1995), 32-64.
10.Flanagan, J. C. 1954. The critical incident technique.
Psychological bulletin 51(4), (1954), 327-358.
11.Ganoe, C. H., Somervell, J. P., Neale, D. C., Isenhour,
P. L., Carroll, J. M., Rosson, M. B., and McCrickard, D.
S. Classroom BRIDGE: Using Collaborative Public and
Desktop Timelines to Support Activity Awareness.
Proc. of the ACM '03 Symposium on User Interface
Software and Technology, ACM Press, (2003) 21-30.
12.Groove Networks Inc., http://www.Groove.com, 2003.
13.Grudin, J., Why CSCW Applications Fail: Problems in
the Design and Evaluation of Organizational Interfaces,
Proc. CSCW'88, (1988), 85-93.
14.Gutwin, C. & Greenberg, S. (2000). The Mechanics of
Collaboration: Developing Low Cost Usability
Evaluation Methods for Shared Workspaces. IEEE 9th
Int’l Workshop WET-ICE'00 (2000).
15.Gutwin, C. & Greenberg, S., Workspace Awareness
Position paper for the ACM CHI'97 Workshop on
Awareness in Collaborative Systems, organized by
McDaniel S. E., & Brinck T., Atlanta, Georgia (1997).
16.Gutwin, C. & Greenberg, S., Workspace awareness for
groupware. Proc. CHI’96, ACM Press, (1996),
208–209.
17.Hayes-Roth, B. & Hayes-Roth, F. A cognitive model of
planning. Cognitive Science, 3, (1979), 275-310.
18.Kuutti, K. & Arvonen, T. Identifying potential CSCW
applications by means of Activity Theory concepts: A
case example. Proc. CSCW '92, ACM Press, (1992),
233-240.
19.McGrath, J. E., Methodology matters: Doing research in
the behavioral and social sciences. In R. M. Baecker, J.
Grudin, W. A. S. Buxton, & S. Greenberg (Eds.),
Readings in human-computer interaction: Toward the
year 2000, Morgan Kaufmann, (1995), 152-169.
20.Mohammed, S., & Dumville, B., Team Mental Models:
Expanding Theory and Measurement Through Cross-
Disciplinary Boundaries. Journal of Organizational
Behavior, 22 (2001), 89-106.
21.Neale, D. C., Carroll, J. M., Rosson, M. B., Evaluating
Computer-Supported Cooperative Work: Models and
Frameworks, Proc. CSCW ’04, ACM Press, (2004) (to
appear).
22.Neale, D.C., Carroll, J.M., 1999. Multi-faceted
evaluation for complex, distributed activities. Proc. of
the CSCL’99 Computer Supported Cooperative
Learning. Lawrence Erlbaum, 425–433.
23.Pinelle, D., & Gutwin, C., Groupware walkthrough:
Adding context to groupware usability evaluation. Proc.
2002, ACM Press, (2002) 455–462.
24.Pinelle, D., Gutwin, C., & Greenberg, S., Task Analysis
for Groupware Usability Evaluation: Modeling Shared-
Workspace Tasks with the Mechanics of Collaboration,
ACM Transactions on Computer-Human Interaction,
10, (4) (2003), 281-311.
25.Prinz W., Mark G., Pankoke-Babatz P. (1998),
Designing Groupware for Congruency in Use, Proc.
CSCW'98, ACM Press, 373-382.
26.Ramage M., The Learning Way: Evaluating Co-
operative Systems, PhD thesis, Lancaster Univ., 1999.
27.Salas, E., Prince, C., Baker, D.P., & Shrestha, L.,
Situation awareness in team performance: Implications
for measurements and training, Human Factors, 37
(1995), 123-136.
28.Scriven, M. The methodology of evaluation. In R. Tyler,
R. Gagne & M. Scriven (Eds.), Perspectives of
curriculum evaluation. Chicago: Rand McNally, p.39-
83, 1967.
29.Steves, M., Morse, E., Gutwin, C., & Greengerg, S., A
comparison of usage evaluation and inspection methods
for assessing groupware usability. Proc. 2001
International ACM SIGGROUP Conference on
Supporting Group Work. ACM Press (2001), 125–134.
30.Suchman, L.A. Plans and situated actions. New York:
Cambridge University Press, 1986.
31.Twidale, M. Randal, D., Bentley, R. (1994), Situated
Evaluation for Cooperative Systems. Proc. CSCW'94,
ACM Press (1994), 22-26
32.Watts, L., Monk, A., & Daly-Jones, O. Inter-Personal
Awareness and Synchronization: Assessing the Value of
Communication Technologies. International Journal of
Human-Computer Studies, 44 (6), (1996), 849-873.
33.Weisband, S., Overcoming social awareness in
computer-supported groups: Does anonymity really
help?, Proc. CSCW’94 (2) (1994), 285-297.
34.Winograd, T., & Flores, F., Understanding computers
and cognition: A new foundation for design. Norwood,
NJ: Ablex, 1986.