Professional Writing Service http://StudyHub.vip/A-Case-Study-For-Evaluating-Interface-D 👈

1. 308
A Case Study for Evaluating Interface Design
through Communicability
Raquel O. Prates
2,1
Simone D.J. Barbosa
1
Clarisse S. de Souza
1
1
Informatics Department, PUC-Rio
R. Marquês de São Vicente, 225 – Gávea
Rio de Janeiro – RJ – Brazil – 22453-900
2
Computer Science and Informatics Department, IME/UERJ
R. São Francisco Xavier, 524 – 6o. andar – Maracanã
Rio de Janeiro – RJ – Brazil – 20550-013
sim@les.inf.puc-rio.br, raquel@les.inf.puc-rio.br, clarisse@inf.puc-rio.br
ABSTRACT
Communicability evaluation is a method based on semiotic
engineering that aims at assessing how designers communicate to
users their design intents and chosen interactive principles, and
thus complements traditional usability evaluation methods.
In this paper, we present a case study in which we evaluate how
communicablity tagging of an application changes along users’
learning curves. Our main goal was to have indications of how
communicability evaluation along a learning period helps provide
valuable information about interface designs, and identify
communicative and interactive problems, as users become more
proficient in the application.
Keywords
Communicability, interface design evaluation, users’ learning
curves, semiotic engineering.
1. INTRODUCTION
Most of the research in HCI aims at providing users with usable
computer systems [5]. Evaluation plays a crucial role here, in that
it reveals actual and potential design problems. The majority of
well-known evaluation methods focus on usability evaluation.
They have been developed to measure if the system is easy to use
and learn, and if it is useful and pleasant to use [9]. However,
usability evaluation does not focus on how designers
communicate to users their design intents and chosen interactive
principles, and how well users make sense of the application.
Our research is motivated by a semiotic engineering perspective
[2], in which user interfaces are perceived as one-shot, higher-
order messages sent from designers to users. We claim that the
degree to which a user will be able to successfully interact with an
application and carry out his tasks is closely related to whether he
understands the designers’ intentions and the interactive
principles that guided the application's design.
Motivated by the need to evaluate how well designers convey
their intentions to users through an application’s interface, we
have developed the communicability evaluation method [8],
which can be perceived as complementary to usability evaluation.
It aims at measuring whether the software successfully conveys to
users the designers’ intentions and interactive principles. By
evaluating the communicability of their software, designers can
appreciate how well users are getting the intended messages
across the interface and identify communication breakdowns that
may take place during interaction. This information is valuable to
designers, either during formative or summative evaluation, since
it allows them to identify persistent communicative problems in
the interface, as well as unexpected use or declination of interface
elements.
2. COMMUNICABILITY EVALUATION:
THE METHOD
In communicability evaluation, evaluators must also define a set
of tasks for users to perform and record their interaction using
software that is able to capture mouse-pointer movements and
other screen events (e.g., Lotus® ScreenCam™). The evaluation
method itself comprises three steps [8]:
1. Tagging
2. Interpretation
3. Semiotic Profiling
1. Tagging
Tagging is the process of relating a sequence of interactive steps
to an utterance (from a predefined set), in an attempt to express
the user’s reaction when a communication breakdown occurs. The
tags used in communicability evaluation were identified based on
literature about explanatory design [3]. Table 1 presents each one
of these tags with a brief description of the context in which they
occur.
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and
that copies bear this notice and the full citation on the first page. To
copy otherwise, or republish, to post on servers or to redistribute to
lists, requires prior specific permission and/or a fee.
DIS ’00, Brooklyn, New York.
Copyright 2000 ACM 1-58113-219-0/00/0008…$5.00.
308

2. 309
Table 1 – Tags used in communicability evaluation
Tag Description
• Where is?
• What now?
The user seems to be searching for a specific function but demonstrates difficulty in
locating it. So, he sequentially (worse case) or thematically (better case) browses
menus and/or toolbars for that function, without triggering any action. This category
includes a special case we have called What now? which applies when a user is
clearly searching for a clue of what to do next and not searching for a specific function
that he hopes will achieve what he wants to do.
• What’s this?
• Object or action?
The user seems to be exploring the possibilities of interaction to gain more (or some)
understanding of what a specific function achieves. He lingers on some symbol waiting
for a tool tip and/or explicitly calls for help about that symbol, or he hesitates between
what he thinks are equivalent options. This category also includes cases in which users
are confused about widgets being associated with objects instead of actions and vice
versa (Object or action?).
• Oops!
• I can’t do it this way.
• Where am I?
This category accounts for cases in which a user performs some action to achieve a
specific state of affairs, but the outcome is not what he expected. The user then either
immediately corrects his decision (typically via Undo or by attempting to restore some
previous state) or completes the task with an additional sequence of actions.
Sometimes the user follows some path of action and then realizes that it’s not leading
him where he expected. He then cancels the sequence of actions and chooses a
different path. In this case the associated utterance is I can’t do it this way. This
category includes another one, Where am I?, in which the user performs some action
that is appropriate in another context but not in the current one.
• Why doesn’t it?
• What happened?
This category involves cases in which the user expects some sort of outcome but does
not achieve it. The subsequent scenario is that he then insists on the same path, as if he
were so sure that some function should do what he expects that he simply cannot
accept the fact that it doesn't. Movies show that users carefully step through the path
again and again to check that they are not doing something wrong. The alternative
scenario (What happened?) is when they do not get feedback from the system and are
apparently unable to assign meaning to the function’s outcome (halt for a moment).
• Looks fine to me… The user achieves some result he believes is the expected one. At times he
misinterprets feedback from the application and does not realize that the result is not
the expected one.
• I can’t do it. The user is unable to achieve the proposed goal, either because he does not know how
to or because he does not have enough resources (time, will, patience, etc.) to do it.
• Thanks, but no, thanks.
• I can do otherwise.
The user ignores some preferential intended affordance present in the application’s
interface and finds another way around task execution to achieve his goal. If the user
has successfully used the afforded strategy before and still decides to switch to a
different path of action, then it is a case of Thanks, but no, thanks. If the user is not
aware of the intended affordance or has not been able to use it effectively then it is a
case of I can do otherwise. Whereas Thanks, but no, thanks is an explicit declination
of some affordance, I can do otherwise is a case of missing some intended affordance.
• Help! The user accesses the help system.
Tagging can be perceived as “putting words in the user's mouth”,
inasmuch as the evaluator selects the “words”, one of the
utterances from the set shown in Table 1, when he identifies a
system-user communication breakdown. Thus, this step can be
thought of as an attempt to recreate a verbal protocol [4].
2. Interpretation
In the interpretation step, the evaluator tabulates the data collected
during tagging and maps the utterances (communicability tags)
onto HCI ontologies of problems or design guidelines. The
general classes of problems identified are navigation, meaning
assignment, task accomplishment, missing of affordance, and
declination of affordance (Table 2). HCI experts may use
alternative taxonomies for a more precise user-system interaction
diagnosis. For instance, utterances may be mapped to such distinct
taxonomies as Nielsen’s discount evaluation guidelines [6],
Shneiderman’s eight golden rules [11], Norman’s gulfs of
execution and evaluation [7], or even Sellen and Nicol’s
taxonomy of contents for building online help [10].
309

3. 308
Table 2 – Mapping communicability tags to typical HCI problems
Navigation Meaning
Assignment
Task
Accomplishment
Declination /
Missing of
Affordance
Where is?
What now?
What’s this?
Object or action?
Why doesn’t it?
What happened?
Oops!
I can't do it this way.
Where am I?
I can’t do it.
Looks fine to me…
Thanks, but no, thanks.
I can do otherwise.
.
The first three general classes of problems are well known and
dealt with by other evaluation techniques. Declination of
affordance occurs when a user knows how to use certain interface
elements to carry out a certain task, but takes another course of
action he prefers. Missing of affordance is a more serious
problem, because the user cannot follow a path predicted by the
designer for a given task, possibly because the interface elements
are not evident enough, or are inadequately mapped to the
corresponding functionality, or are inconsistent with the
application as a whole.
In other words, declination of affordance means that the user
understood how the designer wanted him to carry out a task, but
decided otherwise, whereas missing of affordance indicates that
the user did not understand the designer’s intentions at all for the
corresponding task.
3. Semiotic profiling
In the semiotic profiling step, the evaluators proceed to interpret
the tabulation in semiotic terms, in an attempt to retrieve the
original designer’s meta-communication, that is, the meaning of
the overall designer-to-user message. This step should be
performed by a semiotic engineering expert, due to the nature of
the analysis. For instance, according to the tabulated results, we
may find an application to be easy to use for novices, but
cumbersome for expert users, indicated by an increase of
declination of affordance tags along the evaluated period. This
could be due to a design approach that is tutorial but does not
provide easy access or shortcuts.
3. APPLICATIONS USED IN OUR CASE
STUDY
We have arbitrarily selected two HTML tag-based editors –
Arachnophilia [1] and SpiderPad [1212]. Figures 1 and 2 show
their main application windows. Notice that, when a user creates a
new document, SpiderPad presents a blank document, whereas
Arachnophilia presents the basic structure of an HTML document
as a starting point.
310

4. 309
Figure 1 – SpiderPad main application window.
Figure 2 – Arachnophilia main application window.
In a previous case study [3], we have evaluated the
communicability of both editors for a set of predefined tasks.
Participants who had no previous contact with either editor were
split up into two groups: one that worked first with
Arachnophilia, and another that worked first with SpiderPad.
(The order in which participants interacted with editors did not
play a relevant role in the experiment.) Users were asked to
perform the same tasks in both editors.
In this study all three steps of the methods were performed for
both editors. It is interesting to notice the different semiotic
profiles obtained for each one of them. In Arachnophilia the
designer adopts a tutorial approach, and his discourse to the user
is more conversational and verbose. On the other hand,
SpiderPad designer’s discourse is more terse and descriptive,
with a functional approach, conveying SpiderPad as a toolbox
for HTML.
311

5. 310
4. CASE STUDY
By observing how communicability tags (utterances) change as
users learn to use an application, we wanted to assess how its
interface design supports users along their learning curve, i.e.,
whether users are able to grasp the intended message, and in
which ways it succeeds or fails. Although our focus is on
communication, the study was also meant to reveal some
interaction problems, typically disclosed by traditional
evaluation methods. Analyzing communicative and interactive
problems together and in the context where they occur, we
wanted to provide designers with some indication of how they
could improve their interface design, and thus their
communication to users.
The case study was done using SpiderPad [12] and
Arachnophilia [1], and it included 4 participants and 3 different
tests throughout a semester. The participants were students in an
HCI grad course, who had never used any of the editors, and had
different knowledge of HTML. In the first test users were asked
to do the same tasks in both editors. The participants were then
“assigned” to using one of the editors (2 participants to each
editor). From one test to the next participants were asked to
perform specific tasks using their assigned editor. They were
also asked to use it during the semester to do any other HTML
editing they needed.
1. Description of tests
Each test was composed by some tasks. Users only had access to
one task at the time, in a given order. Users had a maximum
amount of time allowed for each one of the tasks. In the first
test, users performed tasks in both editors, whereas on the other
two they performed tasks in their assigned editor. Table 3 shows
the tasks users had to perform in each test. Notice that at each
test at least one new task was included.
2. Results
One of the goals of the case study was to observe how taggings
changed as users learned the software. The expected result was
that the occurrence of all taggings (except for Thanks, but no
thanks) would decrease. After all, as users interact with the
software they learn the functions that are available to them, their
corresponding interface elements, and the overall organization
and communication code presented by the designer. Thus, one
can expect that the communication breakdowns will decrease.
However, this was not what happened in all cases.
As for Thanks, but no thanks the expected result was that it
would increase. Often in a software application one function is
afforded in different ways to users. Once users become
experienced in it, they develop their own personal way and
preferences for interacting with it. These preferences don't
necessarily include the primary1
affordance provided by the
designer, and can be perceived as a declination of the afforded
strategy.
1
We consider the primary affordance to be the ones that are
more salient in the interface and typically taught in “How to”
section of the documentation.
Figure 3 and Figure 4 show the results obtained in the three
tests for Arachnnophilia and SpiderPad, respectively. We next
discuss the obtained results for each one of the utterances in
both editors.
Table 3 – Tasks performed in tests.
Test Date Tasks
Test 1 Apr. 5, 99 In both editors, users were asked to:
1. Create nested lists, with different
bullet types.
2. Change the page’s background
color
3. Create a 2x2 table
4. Merge cells of first line of table
Test 2 Apr. 20, 99 In their assigned editor, users were
asked to:
1. Create a table, containing lists,
cells with different background
colors, and merged cells.
2. Create a nested list, with different
bullet types
3. Create a page with links to
previous pages (containing table
and list)
Test 3 May 11, 99 In their assigned editor, users were
asked to:
1. Create a title page, containing a
logo and a text with a different
background color
2. Create a page containing nested
lists with different bullet types, a
table, and links to the same page
and text.
3. Create a page containing links to
previous page
4. Create a page using frames and
previous pages.
312

6. 311
Arachnophilia
0% 5% 10% 15% 20% 25% 30% 35% 40%
Whereis?
What'sthis?
I can't doit thisway.
What happened?
I can't doit.
I candootherwise
Utterances
%tagged
Test 1
Test 2
Test 3
Figure 3 - Tagging results for Arachnophilia
SpiderPad
0% 5% 10% 15% 20% 25% 30% 35% 40%
Whereis?
What now?
What'sthis?
Oops!
I can't doit thisway.
Whydoesn't it?
What happened?
Looksfinetome…
I can't doit.
Thanks, but nothanks
I candootherwise
Help
Utterances
%tagged
Test 1
Test 2
Test 3
Figure 4 - Tagging results for SpiderPad
5. DISCUSSION
Some of the taggings changed as expected in both editors,
namely Where is?, What’s this?, Looks fine to me…, and
Thanks, but no, thanks. The other tagging changes in time were
more erratic, varied from one editor to the other and not always
followed the expected pattern. We will discuss the result
obtained for each tagging, as well as some factors that
contributed to this outcome.
Where is?
In both editors the number of Where is? utterances decreased, as
expected. Although in the long term we would expect it to tend
to zero, this was not expected during these tests, since every test
presented a new task to the user.
If we compare the occurrences of this tag in both editors, we
will note that initially SpiderPad presents a higher number of
313

7. 312
Where is? occurrences than Arachnophilia. However, in the last
test this number is much smaller in SpiderPad than in
Arachnophilia. This is an indication that interface elements
relative to more basic functions were easier to find in
Arachnophilia than in SpiderPad. However, as tasks became
more complex, functions were harder to find in Arachnophilia.
This indication is in line with their semiotic profile (see the
previous section). Being more tutorial, Arachnophilia is meant
for people who will use only basic HTML, whereas SpiderPad's
toolbox approach is for people who are more proficient.
What's this?
In both editors the number of occurrences of What's this?
decreased, as expected. Notice that in SpiderPad the number of
utterances of this tag is significantly smaller than in
Arachnophilia. This indicates that the signs used in SpiderPad
were more easily remembered by users than those presented in
Arachnophilia. Besides, Arachnophilia supplied users with
Command Bars that could be easily turned on and off by the
users, depending on their needs. When users turned on a
Command Bar they didn't regularly use, they would usually
check the meaning of one or more buttons of the Command Bar,
or check the difference between two of the buttons, thus
increasing the number of What's this? uttered by the users.
Looks fine to me…
The percentage of Looks fine to me… decreased in both editors.
This was an expected result, for users learned to interpret and
better understand the feedback presented by the editors, as they
became more experienced in them.
Thanks, but no, thanks.
As expected, the percentage of Thanks, but no, thanks increased
in both editors. Both editors presented distinct paths to perform
a task, and we considered dialogs and wizards to be the primary
afforded interaction form. In the tests we noticed that users
would often choose not to use this primary affordance, or would
only use it partially. For instance, dialogs for editing tables
allowed users to define a great number of attributes relative to
the table or its rows or cells. However, users would frequently
use it just to create the template (code structure) of the table and
then would edit attributes and values directly on code or by
modifying the tag afterwards. Another common strategy
observed during the tests occurred when users created more than
one instance of an object (e.g. table or list). In this case, they
would create the first one using the dialog and then for the
others they would just copy and paste directly the code and then
edit it. In the first example users would be declining the
opportunity to define (most) attributes during the creation of the
table, whereas in the second they would decline the wizard or
dialog altogether after creating the first object.
What now?
Although we expected the What now? utterance to decrease as
users learned how to interact, it did not, in either editor. In each
editor it behaved differently, but in both it decreased in the 2nd
test, and then increased again. We relate this result to the fact
that a What now? breakdown depends not only on the interface,
but also on the task and the user’s knowledge of the domain.
Frequently users did not know how to achieve a result in
HTML, and thus, did not know what to look for in the interface.
Therefore, the What now? breakdowns could be qualified in two
categories:
• Interactive breakdown: the user knows what he wants to
do, but does not know how to achieve that next step
through the interface.
• Task breakdown: the user didn’t know what to do to
achieve a result in the domain (in this case, in HTML).
In our experiment, the fact that the 3rd
test presented more
complex tasks to the user could explain why this utterance
increased from the 2nd
to the 3rd
test in both editors.
Oops!
If we observe the occurrences of Oops! in Figures 3 and 4, we
notice that not only did it not decrease, but also it behaved very
differently in each editor. In order to understand this result, we
analyzed the movies and taggings more thoroughly trying to
identify the possible causes. Our conclusion was that Oops!
breakdown happened for different reasons and could be further
qualified. The possible categories we identified are:
• Expectation breakdown: the user expects a result but the
outcome is not what he expected.
• Similarity breakdown: two “similar” options are displayed
next to each other and the user ends up clicking on the
wrong one.
• Exploratory breakdown: user does not know what a specific
option does, he then clicks on it in order to find out.
• Status breakdown: the user opens the correct dialog, and
only then realizes the software is not in the necessary state
to achieve the desired result.
It seems that the occurring subcategories of Oops! change along
users’ learning curve. We could expect Expectation breakdowns
to happen when the user does not know the software well and is
still building his usability model of the application. However, in
order to have a Status breakdown the user must know enough to
identify that the software's current state is not the necessary state
to achieve the desired result.
I can’t do it this way.
In both editors, in the 1st
test the number of occurrences was
very small, whereas in the 2nd
they disappeared. Up to this point
the results are in line with what we had expected. However, in
the 3rd
test in SpiderPad this tagging appeared once again, but
the number of occurrences decreased in comparison to the 1st
test. This probably resulted from the 3rd
test being relatively
more complex than the 2nd
.
In Arachnophilia this tagging behaved differently in the 3rd
test,
having a significant increase when compared to the 1st
test. By
analyzing the movies we noticed that this was caused by an
interaction strategy adopted by the users. Arachnophilia does
not provide users with any functions to change HTML tags’
attributes (it could only be done via code). Furthermore, most of
the HMTL attributes are not available in the help system. Thus,
Arachnophilia only presents users the more “basic” attributes
and only in the creation dialog. Consequently, when users knew
they had to change an attribute, but did not know what the
attribute was or its possible values, they would open the tag’s
creation dialog (observed for lists and tables) to try and find out.
Since only a limited number of the possible attributes was
presented, often they could not get the desired information,
characterizing an I can’t do it this way breakdown.
314

8. 313
This result in Arachnophilia indicates users’ need to have access
to HTML tags’ attributes and their possible values, which was
not fulfilled by the designer’s solution.
I can do otherwise.
In both editors this utterance has the highest number of
occurrences in the 2nd
test. We relate this result to the fact that
when users couldn’t remember where a desired interface element
was, or couldn’t find it immediately, they would quickly give up
searching and would type in the code. This tendency did not
continue in Test 3, since it was a little more complex than Test
2. Thus, users frequently didn’t know the exact syntax to type
in, so they had to find the intended interface element.
This result does not necessarily indicate a problem in the
interface, but rather one of the users' strategy. If the users knew
the complete syntax for an HTML object and couldn’t easily
find the corresponding interface element, they would perceive
the cost of looking for it higher than its benefit and would
choose to type it in.
Help!
In SpiderPad this utterance occurred in the 1st
test, and then
disappeared in the other 2. That is, it behaved as expected. In
Arachnophilia, although it behaved as expected in Tests 1 and 2,
it increased in Test 3. This result was due to the way the task –
to create an HTML document containing frames – is achieved in
Arachnophilia.
In Arachnophilia a new page always contains the basic structure
(body, head, title and html tags) of an HTML document.
However, part of this structure must be removed when the user
creates a frame. Although in the Help an example was shown,
this point was never emphasized, and was overlooked by users.
This result points out an inconsistent message sent from
designer to users. When the designer creates a new HTML
document containing the basic structure he conveys to the user
the idea that these tags should always be part of an HTML
document. Nonetheless, this is not the case when creating
frames.
What happened?
The What happened? tag did not behave as expected. This result
indicates a problem with the feedback provided by both editors.
Frequently, the user would try to achieve a result using a
specific function, but “nothing” would happen. There were two
main causes that led to this situation:
• the feedback was too subtle and the user was unable to
perceive it
• the application was not in the necessary state to perform an
action, but the action was available to the user; once
activated it would not be executed, but it wouldn’t provide
the user with an error message.
In both cases there was a communication breakdown between
user and system, and they both indicate the need for a review of
the feedbacks provided to users.
The higher number of occurrences in Test 3 is probably due to it
being more complex than Test 2.
Why doesn’t it?
In both editors this utterance behaved very similarly to the What
happened? tag. This was not a coincidence, but resulted from the
users not having perceived or received a feedback. Users would
often try to activate the same function one more time, or even a
few more times. This also corroborates the indication that the
feedback provided to users should be reviewed.
In this test, we did not notice any cases of Why doesn’t it? in
which users insisted on a path because they thought it had to be
the right path, even though they realized they had not achieved
the desired result. Rather, we noticed that in some cases users
would not find the function they were looking for, and decided
to go back to what they seemed to have considered the most
probable location for it and check if they hadn’t missed it by any
chance.
6. FINAL REMARKS
Although the results presented in this paper were not statistically
significant, they provide us with relevant indications of how
designers and users can benefit from observing how
communicative breakdowns change along users’ learning
curves. If these changes do not behave as expected, they point
out to the designer a problem in his meta-message to the user,
and should be looked into, as was indicated by the What
happened? and Why doesn’t it? tags. Also, the tags that behave
accordingly to the expected pattern point to a consistent
communicative act from designer to user. This happened with
the utterances: Where is?, What’s this? and Looks fine to me….
Even then, the rate of change of the tags may provide designers
with valuable indications. For instance, if the Where is?
utterance took a long period of time to decrease, or did not
decrease significantly, it would indicate to the designer, that
although users were able to understand and learn how the
interface was organized, it was not an easy task for them.
The communicability method not only identifies problems, but
also gives the designer hints to a solution. The utterance
category narrows down the possible causes of the problem, and
the pieces of tagged movies frequently provide the designer with
evidence of the users intuitions about a solution. Furthermore,
this method provides the designer with some information that
other evaluation methods do not consider, such as the
declination or missing of affordance and also some of the
strategies developed or chosen by the users.
By identifying interface elements whose meaning users were
unable to grasp (missing of affordance) the designer may choose
to represent those elements differently. That is, "rephrase" his
message to the user. As for declination of affordance, the ability
to identify this breakdown allows the designer to evaluate the
cost/benefit ratio of offering such an element (especially if
identified during formative evaluation) or to decide in favor of
other interactive elements, preferred by users. This study has
also provided valuable information to our research in the
communicability evaluation method and has set the need for
further research. It has pointed to the need to look into how and
when utterances should be qualified. The qualification of
utterances would refine further the evaluation and would
provide designers with more specific indications. For instance, if
Similarity Oops! breakdowns often occurred when interacting
with a specific menu, it could indicate that 2 or more options of
that menu were not distinctive enough and should probably be
changed. If instead regular Oops! utterances were identified, a
further investigation of their causes would be necessary to
precisely pinpoint the interactive problem.
315

9. 314
The co-occurrence between the What happened? and Why
doesn’t it? tags indicates that we should investigate not only the
tags, but also sequences of tags. That is, examine which
sequences have an associated meaning to them, and if this
meaning adds to the meaning of individual tags.
Furthermore, as we have pointed out, the ability to identify
declination of affordance would be particularly interesting if
done during formative evaluation. Thus, it would be interesting
to investigate how effective communicability evaluation method
would be when applied to prototypes, storyboards and the like
during the design process. Clearly some of the utterances may
emerge in tests done without a running prototype.
This case study also justifies the carrying out of a more
extensive study, which would include more participants and a
longer period of time. Such a study would provide us with more
statistically significant results and more than providing us with
indications, it would allow us to make claims as to the meaning
of communicative breakdowns changing patterns along users
learning curves.
Moreover, we could ask users to tag movies of their own
interaction. In this case, they would provide evaluators with
their own report of the conversational breakdowns experienced,
as opposed to the evaluator's interpretation. Users' tagging could
either be done based on the interaction movie, or during the
interaction. Whereas the former could be considered a case of
controlled2
post-event protocol, the latter could be considered a
case of controlled think aloud protocol [9]. The advantage of
doing the tagging based on the interaction movie is that users'
attention would not be divided between performing the task and
doing the tagging.
It would also be interesting to have non-HCI experts tag movies.
The comparison of their taggings with those of the users (for a
same interaction movie) could give us more insights about
communicability and user-developer communication [8].
7. ACKNOWLEDGMENTS
The authors would like to thank the participants of the tests for
their time and interest. They would also like to thank CNPq,
FAPERJ, TeCGraf and LES for their support.
2
Controlled in the sense that users would have to express
themselves using the set of utterances of the communicability
evaluation method.
8. REFERENCES
]1] Arachnophilia 3.9 Paul Lutus. 1996-1998.
[2] de Souza, C. S. (1993). “The Semiotic Engineering of User
Interface Languages”. International Journal of Man-
Machine Studies, 39, 753-773.
[3] de Souza, C.S., Prates, R.O., Barbosa, S.D.J. "A Method
for Evaluating Software Communicability". Proceedings of
the Second Brazilian Workshop in Human–Computer
Interaction (IHC '99), in CD-ROM, Campinas, Brazil,
October 1999.
[4] Ericsson, K. A. and Simon, H. A. (1985). Protocol
Analysis: Verbal Reports as Data. MIT Press, Cambridge,
MA.
[5] Hartson, H. R. (1998). “Human-computer interaction:
Interdisciplinary roots and trends.” The Journal of System
and Software, 43, 103-118
[6] Nielsen, J. Usability Engineering. Academic Press, Boston,
MA, 1994.
[7] Norman, D. Cognitive Engineering. In Norman, D. A. and
Draper, S. W. (eds.), User-Centered System Design.
Lawrence Erlbaum Associates, Hillsdale, NJ, 1986, pp.
411–432.
[8] Prates, R.O., de Souza, C.S., Barbosa, S.D.J. (2000) “A
Method for Evaluating the Communicability of User
Interfaces”. Interactions. Jan-Feb 2000.
[9] Preece, J., Rogers, Y., Sharp. H., Benyon, D., Holland, S.
and Carey, T. (1994). Human-Computer Interaction.
Addison-Wesley.
[10] Sellen, A. and Nicol, A. Building User-Entered Online
Help. In B. Laurel (ed.), The Art of Human-Computer
Interface Design. Addison-Wesley, Reading, MA, 1990.
[11] Shneiderman, B. Designing the User Interface. Addison-
Wesley, Reading, MA, 1998.
[12] SpiderPad 1.5.3. Six-Legged Software. 1996-1997.
316