The document discusses Clippy, the animated character in Microsoft Office that was intended to provide assistance to users. It describes how Clippy was based on a Bayesian network that used evidence from user actions and background to infer their intentions and needs. However, Clippy received widespread criticism for being annoying and prompting help excessively. The document analyzes reasons for Clippy's failure, including that it did not account for different user expertise levels or learn from past interactions. It proposes updates to the Bayesian network, like incorporating a "historical acknowledgement" node, to better determine when assistance is required.

1. 1
Bayesian Network backbone of
Microsoft Office Assistant – Clippy:
A CASE STUDY
Naveen Bharathi Pitchandi
North Carolina State University
Abstract
User interface agents are increasingly employed to enhance software products. Websites
(e.g. buy.com, extempo.com, ananova.com, mysimon.com) now use characters to guide users
through processes or present information. Microsoft Office Assistant - Clippy is a well-known
animated character that was part of Microsoft’s graphical user interface for its elements of the
Microsoft Office Suite. Clippy was first included in the 1997 release of the Office suite and
continued to be part of the product line until 2007 when it was permanently removed. due to
widespread hatred towards the animated character. Clippy was one of the many cartoon
characters that were available as the engine’s front end interacting with the user.
The Office Assistant was an Intelligent User Interface for Microsoft Office that assisted
users by way of an interactive animated character, which interfaced with the Office help content.
The back end of the Office Assistant was based on Bayesian Networks, to provide the
corresponding assistance to the user by inferring from user background, user actions, and
program state. The Bayesian method of analysis computes the likelihood of alternative concepts
based on the user’s inputs/actions and models the user’s intention to provide assistance.
Within few years of its existence, the office assistant received wide spread hatred and was
deemed “annoying” and one of the Top 50 worst inventions. The purpose of this paper to
investigate the working of the assistant with a help of a sample network, reasons behind Clippy’s
massive failure. And propose a solution of including the “historical acknowledgement” to
elicited assistance as part of a node in the network to update the probabilities that would capture
the User’s expertise level and need for assistance better.

2. 2
1. ORIGIN OF CLIPPY
Microsoft Office assistant was a partial implementation of the Lumiere project at
Microsoft Research that was initiated in 1993 with the goal of developing methods and an
architecture for reasoning about the goals and needs of software users as they work with
software. At the heart of Lumiere are Bayesian models that capture the uncertain relationships
between the goals and needs of a user and observations about program state, sequences of actions
over time, and words in a user's query (when such a query has been made).
Early on in the Lumiere project, studies were performed in the Microsoft usability labs to
investigate key issues in determining how best to assist a user as they worked. The studies were
aimed at exploring how experts in specific software applications worked to understand problems
that users might be having with software from the user's behaviors. The Lumiere prototypes have
explored the combination of a Bayesian perspective on integrating information from user
background, user actions, and program state, along with a Bayesian analysis of the words in a
user's query (although the commercially available Office products had a separation between user
interface events and word-based queries). Research on new application and extensions continued
in parallel with efforts to integrate portions of the prototype into commercially available software
applications. In January 1997, a derivative of Lumiere research shipped as the Office Assistant in
Microsoft Office '97 applications.
2. DOWNFALL OF CLIPPY
“It looks like you’re writing a letter” – one of the most irked phrases used by Clippy,
proposing assistance to write a letter when a user types “Dear”. Accurate as it was in estimating
the intensions of the user based on actions/word queries, its repeated prompting for assistance
irked many users. Clippy offered to help incessantly and popped up with suggestions related to
the actions made despite the user denying for assistance repeatedly. Eventually, Clippy became
one of the most hated features of the Office Suite. And it was eventually deemed as one of the
Top 50 worst inventions.

3. 3
3. REASONS BEHIND CLIPPY’S FAILURE
Clippy was permanently removed from the Office Suite from 2007, and there has been no
attempt yet to revive the assistant back (except for using the designed character for a game –
Ribbon Hero2 – in 2011). But on thorough research in looking at areas where Clippy fell behind,
a couple of research papers throw some light on the issue.
The first paper, [1] by Luke Swartz, 2003 analyses from a psychological perspective of
what lacked in Clippy. The Office Assistant’s letter-writing proactive help feature, thus, breaks
every relevant etiquette rule: it ignores social conventions of when to disturb someone, it does not
learn from its mistakes, it does not develop a long-term relationship, and (one might argue) it does
not even provide a helpful service. Since this feature is the most cited annoyance in the popular press,
one cannot help but wonder how much better the Office Assistant would be perceived if this one
feature had been fixed or eliminated before its release.
While the above conveyed in terms of a human perception, the following article elicits the
technical drawbacks of the implementation of the Office assistant from its parent project – Lumiere.
[2] The paper by E.Horvitz published in 1998 at the Conference on Uncertainty in Artificial
Intelligence (UAI) by the Lumiere team described the inner workings of the Assistant’s inference
engine and also how much of it was included in the released version of Office 97. 1) The system
does not take into account of any persistent user profile, wherein the assistant does not take into
account of all actions taken by user and acclimatize to User’s competency level to offer the
required assistance. Instead, the assistant worked on a single profile and hence was a more rigid
structure than what we would’ve liked it to be. 2) The system does not use rich combination of
events over time; rather system only considers a small set of relatively atomic user actions. And
only those actions which are present in the small event queue present are considered. 2) The
system separates the analysis of words. 3) The system separates the analysis of words and of
events. When words are available, the system does not exploit information about context and
recent actions. Finally 4) the automated facility of providing assistance based on the likelihood
that a user may need assistance or on the expected utility of such autonomous action was not
employed.

4. 4
Based on inferences from the two papers [1] ,[2] the feature of automated facility of
providing assistance based on likelihood that user may need assistance, seemed to have a higher
impact on the failure of the office assistant. The paper proposes a possible solution that could be
incorporated onto the office assistant, which would update probabilities for the likelihood such
that the assistant would be smarter as it gains more knowledge on whether the user would need
assistance for a particular action given what was the user response for previous assistance for the
same action. This would solve the problem of unnecessary pop-ups irking the user providing
assistance when it is not warranted.
Another possible way to improve the performance of the office assistant would be to
integrate the search of words and actions, which would again reduce the most common error of
prompting to assist with a letter. To better understand how the Bayesian network infers from
User’s actions and elicits user intentions, the following sections would explain a segment of how
it works.
4. BAYESIAN NETWORK : Teaching notes
A Bayesian Network is a graphical model that applies probability theory and graph
theory to construct probabilistic inference and reasoning models. The nodes represent variables,
events or evidence, whilst the arc between two nodes represents conditional dependences
between the nodes.
A Bayesian Network is defined as a Directed Acyclic Graph (DAG) in which the arcs are
Unidirectional and feedback loops are not allowed in the graph. Because of this feature, it is easy
to identify the parent-child relationship or the probability dependency between two nodes.
A small network based on user’s recent actions – usage of Chart option on Excel – has
been mapped. Though typically the actual network could be bigger and more complex, for
understanding purposes a simple network to explain the basic theory behind eliciting appropriate
assistance based on user actions is used (Fig 1.). This network does not exactly replicate the
actual network behind Office Assistant. This is a hypothesized network built based on the
working of the Assistant. Based on evidence of any of the user actions (Nodes in the bottom
row), the likelihood of which type of and if User is looking to create/edit a chart can be obtained.

5. 5
Fig. 1
To further simplify the computation involved, and for better understanding considering a
sub network from the above given by Fig 2
Fig. 2

6. 6
To explain the network, probabilities were assigned to each node [3] (though they aren’t
the actual probability distribution used in the real network behind the Assistant). There are
different states for each node that have a probability assigned to it. A causal relationship can be
derived from the network by using the Bayes rule.
A B P (B|A) = P (A, B)/P (B)
Hence, the inference from the network can be obtained by:
• Conditional Probability: P(Q = q|E) = P(q,E)/P(E)
Where Q is the query variable and E is the evidence.
X1, ...,Xn be unknown network variables that q depends on
• Joint Probability :
P(Q=q, E=e) = 𝑃 𝑞 𝑒, 𝑥1, … 𝑥𝑛 𝑃(𝑒, 𝑥1, … . 𝑥𝑛),-,…,,.
• General Product rule for Bayesian Networks:
P(X1, ...,Xn) = 𝑃(𝑋𝑖|𝑝𝑎𝑟𝑒𝑛𝑡𝑠 𝑋𝑖 ).
78-
Table 1
Node Insert Chart
State Yes No
Prior
Probability
0.500 0.500

7. 7
A prior probability of being neutral to any decision (50-50) is assigned to the Insert chart
node such that with further evidence, the posterior probability would indicate if User’s intention
is to insert a chart. The conditional probabilities of the remaining nodes are established below.
Table 2
Table 3 Table 4
Table 5
The Assistant would be able to infer if the user wishes to create a line chart if the user
Highlights cells or mouse over insert chart or mouse over line chart. The prior conditional
probabilities from the above tables are used to derive inferences. The first step towards that is to
find the joint probability density function.
Node Name Line
State Yes No
Insert Chart = Yes 0.65 0.35
Insert Chart = No 0.65 0.35
Node Name Mouse over insert
State Yes No
Line = Yes 0.85 0.15
Line = No 0.15 0.85
Node Name Mouse over Line
State Yes No
Line = Yes 0.85 0.15
Line = No 0.15 0.85
Node Name Highlight Cells
State Yes No
Line = Yes 0.85 0.15
Line = No 0.15 0.85

8. 8
P(I, L, MOI, MOL, H) = P(I)*(P(L|I)*P(MOI|L,I)*P(MOL|L,I)*P(H|L,I). Where I = Insert
Chart, L=Line, MOI = Mouse over insert, MOL = Mouse over line and H = Highlight cells.
Hence the model can now answer based on any inference such as
P (L= Yes | MOL = Yes, H= Yes) = P (L= Yes | H = Yes). P (MOL = Yes | L = Yes, H = Yes)
P (MOL = Yes | H = yes)
P (L= Yes | MOL = Yes, H= Yes) is the posterior probability, P (L= Yes | H = Yes) is the prior
probability, P (MOL = Yes | L = Yes, H = Yes) is the likelihood and gives the probability of
evidence assuming L= yes and H = Yes. P (MOL = Yes | H = yes) is the expectedness or how
expected the evidence is given H = yes.
The network was simulated using a software – BayesiaLab which calculates the posterior
probabilities based on the above formula:
Fig. 3

9. 9
Hence from the inference it is oberserved that if the user’s actions are “highlight cells”
and “Mouseover line chart”, the user model estimates that it is 64.09% likely that he would want
to Insert a chart and 96.98% that it would be a Line Chart. Similar simulations were carried out
for different evidences of user actions. This is a way that the assistant maps the assistance
required. But, there must be an additional network which must determine whether the User
“needs” this assistance or not.
4.1 Likelihood Network
One of the major reasons for the failure of the assistant was that there was no user
profiling and the assistant never considered the likelihood of the need for assistance. Though
Lumiere project included a network to assess the likelihood, it was not entirely incorporated onto
the Office suite. Translating the Bayesian computation onto an already existing software package
posed great difficulty. Hence some of the functionalities though had a bayesian background, was
not computationally the same when coded onto the software package.
Based on the negative reviews and the repeated mistakes made by Clippy, a network
(Based on the likelihood network by Lumiere team) with a minor change was built using
Bayesialabs and analysed how that minor addition could improve the likelihood.
Fig. 4 Fig. 5
Fig4 represents part of the network that was designed by the Lumiere team. Fig5.
represents the network that was based on the network from Fig4. With a minor addition of a node
“Historical Assistance Acknowledgement”. The reason behind including this node was that, the
network though successfully managed to elicit user’s intentions, it did not learn the likelihood

10. 10
probabilities over time of whether the user used the assistance or not. The “Historical Assistance
Acknowledgement” node captures the information of whether the User “Accepts” or “Denies”
the help offered by the Assistant.
User needs assistance is the decision making node that decides if user needs assistance.
The several factors that affect this decision are the remaining nodes present in the network.
“Pause after activity” node captures the information as to whether the user is idle above a
threshold time (Approximately 10seconds) or not. This information is form of a representation if
the user is unsure of how to move ahead. But the user could also have paused due to distraction
while doing something else, that information affects the “Pause” and is captured by “User
Distracted”. “Difficulty of current task” and the “User expertise” also determine the need for
assistance. Where, difficulty of current task is a predetermined by the software the user expertise
is provided by the user.
A sample data was used that intended to capture history of actions performed by the user,
and whether assistance was provided or not for that action. And also captured the information of
whether user accepted or denied the assistance which was only included in the second network
that was a modification of the Lumiere. The analysis was confined to one goal of “Chart
function” for easier analysis and comparison. For a similar inference in each of the network the
posteriors of both of the networks were calculated. The results are shown below:
Fig. 6

11. 11
Fig. 7
From Fig.6 and Fig.7 it can be seen that the posterior probability for assistance was lesser
for Fig.7 (36.36%) compared to Fig.6 (45.83%). Fig.7 can be claimed to perform better than
Fig.6 as Fig.6 only accounted for the number of times assistance was provided, but Fig.7 also
included the number of times the assistance was provided and was denied. Hence the second
network was able to learn that the user needed lesser assistance than the Assistant thought he
would. For further confirmation, few more data points were added and associated to the network:
Fig. 8

12. 12
Fig. 9
Hence with additional data points, Fig. 8 and Fig. 9 depict the network that associated the
additional data points and recalculated the posteriors. The additional data points were in such a
way that the user denied most of the assistance provided by the Assistant to show more precisely
how differently it would perform if not for “Historical Assistance Acknowledgement”.
Without the presence of “Historical Assistance Acknowledgement”, the network in Fig. 8
computes a posterior of 79.25% whereas the network in Fig. 9 accounts for the number of times
assistance is denied by the user reducing the posterior to 50%. Hence the network created
performs better than how it presently functions, and this would in turn help the Assistant be more
smart in learning the User’s expertise level. Wherein, if the user denies the assistance for a
particular action repeatedly, it reduces the likelihood of offering assistance.
4.2 Summary
A more precise network to capture the user’s decision and need for assistance was
created, that performed better than that of the present model that is used for the Office Assistant.
From the simulation of the two types of network, it was evident that the present network
repeatedly prompted for assistance despite the user denying it. The second network though,
captured this information. This change into the backend of the Assistant could make it smarter
and less annoying. An intelligent assistant would be highly desirable as timely need for help
when provided could change the impression that it had earlier created.

13. 13
4.3 Key Assumptions
The data used behind the all the networks built in this paper are not the actually used in
network of the Office Assistant. Hence the prior probabilities computed might not be the same.
They are just a representation of how the data could be, to explain the working of the network.
The search for user’s intentions considered here are only for user related actions and not
query based search. The integration of both the searches as mentioned earlier in the paper has not
been focused on.
4.3 Difficulties in implementing
Tracking user’s actions were inherently difficult to integrate with the software the first
time it was attempted to integrate with Lumiere project. Hence the ability to track the decision
made by the user for assistance provided and the complexity involved to store this information is
to be assessed.
The paper does not take into account of how much information must be stored, i..e if the
system keep storing user actions indefinitely, it could reduce the efficiency of the search as it had
to process all the information to come up with a likelihood.

14. 14
REFERENCES
[1] Luke Swartz, WHY PEOPLE HATE THE PAPERCLIP: LABELS, APPEARANCE,
BEHAVIOR AND SOCIAL RESPONSES TO USER INTERFACE AGENTS,
Symbolic Systems Program, Stanford University, June 12 2003
[2] E. Horvitz, J. Breese, D. Heckerman, D. Hovel, and K. Rommelse. The Lumiere Project:
Bayesian User Modeling for Inferring the Goals and Needs of Software Users. Proceedings of
the Fourteenth Conference on Uncertainty in Artificial Intelligence, July 1998.
[3] Michael Y.K. Kwan, K.P. Chow, Frank Y.W. Law, Pierre K.Y. Lai, Computer Forensics
using Bayesian Network: A Case Study.

15. 15
EXECUTIVE SUMMARY
The case study is about the much hated Microsoft Office Assistant – Clippy, its inception,
demise and the technical flaws behind its failure. Clippy based its inferences for providing
assistance to the user based on user’s recent actions or query based help request based on
Bayesian networks.
Microsoft Office assistant was part of Lumiere research that was established by Microsoft
Researchers from the Decision Theory & Adaptive Systems Group to study and improve human
computer interaction using Bayesian methods. The Lumiere prototypes explored the combination
of a Bayesian perspective on integrating information from user background, user actions, and
program state, along with a Bayesian analysis of the words in a user’s query.
Clippy eventually became one of the most hated inventions – Time, 2010. It apparently
prompted to offer assistance despite the user not really needing one. On secondary research from
the reasons behind why people hated Clippy it was certain that Clippy popping up incessantly
with help messages is what irked users the most. When looked at the technical background of
this problem it was found that Clippy had excluded some of the features from the Lumiere
project on its integration to the Office suite. The main aspects are: 1) No user persistent profiles
2) Separation between user interface events and word-based queries; for word based queries the
engine ignored any context and user actions 3) The automatic facility of providing assistance
based on the likelihood that a user may need assistance was not employed.
A sample network to explain the working of Clippy; how Clippy inferred which
assistance to provide based on user’s actions was built using a software - Bayesia Labs. As a
solution to the problem of Clippy not being to estimate the likelihood of user needing assistance
accurately, a modification to the likelihood network proposed by Lumiere was built with an
inclusion of node “Historical acknowledgement of assistance” which captured information on
response of the user to the assistance provided by Clippy : Whether user accepted or rejected the
assistance. This would be prior to further computation of the likelihood for a similar action in the
future, but now Clippy would have more information as to whether User would need the
assistance based on user’s earlier response to the same. Lumiere’s original network and the
modified network were simulated and found that the modified network performed better. This
solution could possibly make Clippy smarter and be able to decide when User needs help more
precisely.