1. 1
Living the user experience
Gary P. Smith
EG A/S
Industrivejsyd 13c
7400 Herning, Danmark
Telephone number, +45 99 28 32 51
gary.smith@eg.dk
ABSTRACT
In this paper I will discuss the lived experience in design
ethnography and use context from the perspective of the observer
shadowing the observer and how living the user experience can
bring you closer to the use context.
First I will argue that although video shadowing can capture a
wealth of data about users and use context in a design
ethnographic field study, I will show through examples, that there
is an experience beyond what the camera can see, in relation to a
plethora of different aspects of individual, social and cultural
experiences.
Second, what does this mean for a designer to have been there
where things happened?
Author Keywords
Design, ethnography, video, user, sensory, context
INTRODUCTION
Within software UI design, interaction design plays an important
role in conjunction with supporting activities; we need to identify
the users of a proposed interactive system. Ethnography has
become popular within interaction design because if products are
to be used in a wide variety of environments, a designer must
know the context and ecology of those environments.
“By bringing the user into the design problems we can focus on
the users needs, both general and specific.”
(Newman & Lamming, 1995:29)
One of the problems with traditional ethnographic studies is that
they take months or years to complete where as, “Design
ethnographers count their field studies in days rather than
months” (Buur & Ylirisku, 2007:46)
During a recent field study our design team carried out the method
of video shadowing (Buur & Ylirisku, 2007:65), which was spread
over a one-week period, following each day a group of different
users. The field studies focus was “working with the body”, this
could be people that work with there own bodies, carpenters and
an electrician or work with other peoples bodies, such as a
pathologist or a spinning instructor.
By capturing user experiences through this method our design
team was able to gather a record of what they do.
“Observing is an intense and tiring activity, but however tired
you are it is important to write up experiences and observations at
the end of each day. If not valuable information is lost as the next
day’s events override your previous days impressions”
(Preece, Rogers, Sharp 2007:328)
In contrast as software UI designer much of the theory and
methods implemented within my work are based around HCI
theory, which revolves around information and cognitive data
gathering.
“Focus groups uncover people’s /perceptions/ about their needs
and their values. This does not mean that they uncover what
people /actually/ need or what really /is/ valuable to them-
however-knowing perceptions of needs is as important as knowing
the needs themselves” (Preece & Rogers, 2007)
The insight it gave about users and its effectiveness as a design
tool, compared to personas, which we have used in the past, from
my experience, are more than often very open for interpretation
by the design team and even more so by the developers.
Although the method of video shadowing had been effective and
had brought me closer to end users, it began to highlight some
concerns I have had with HCI methods for data gathering; Much
of my previous work has been on desktop computing, but my
current and future work is evolving within the mobile space,
which has a much wider context. So this will become an important
issue in the future, when convincing management about using
these new methods.
Controlling the environment?
I am often testing people’s interactions with new interfaces, but
these are confined to environments, which are often artificially
constructed and controlled. In direct contrast to my first field
study with crafts people, which was dynamic and unconstrained.
Although this may seem trivial or even obvious now, at the time it
did not.
I arrived at 07.00am ready to begin the days video shadowing; it
started slowly by following the crafts person while he stocked his
van in preparation for the day’s activities.
At around 09:00am I began to miss the convenience of the office
coffee machine with the constant bombardment of body signals
telling me that I had missed my first cup of coffee. Although I was
experiencing my body’s apparent erg for the office coffee
machine, at the time I just ignored it as being a consequence of
doing the shadowing method, and thus being too occupied with
the task at hand to have time. It was only when we reached
12:00am and parked into a small lay by on our way to the next
customer that Peter said “OK, now we eat, its lunch time”.
It was at this point my body signals, which I had been ignoring,
caught up with me and I really began to observe my self and my
experiences within the environment. My environment had
changed and new rules applied.
Was my preparation poor? Had I forgotten something so basic as
a sandwich box? What had I just experienced?

2. 2
First Living Experience
Within the first hour of my video shadow I followed an electrician
on his daily rounds to a private residents, he was carrying a
ladder, a toolbox and materials to carry out his job.
This was my first field experience with a crafts person and I was
not prepared for something so simple, as taking off a pair of
shoes, when entering the house. I was so busy trying to observe
him and capture him on video that I was left with no alternative; I
pulled them off by using my feet.
Fig 1.0 Hands free environment
This action would become a problem later in the video shadow
when on numerous occasions we would have to go back and
fourth between his van to pick up extra equipment, it is not easy
putting your shoes on and tying your laces while holding a camera
with one hand. “I learned but could not alter my situation” and
adapt to the environment.
At some point during this taking off and putting on of my shoes I
began to reflect over my own experience within the context.
At the time it was a more passive, if not subconscious checklist or
active self-monitoring of my own experience in relation to my
surrounding context.
The electrician I was observing was guiding me through his
context. Observing the video only, without living the experience
shows the electrician carrying his tools.
But although he actually takes his shoes off when entering the
house, it is not captured on film. Is this interaction obvious or
more importantly is data lost? Although this may seem obvious it
is part of the tradesman’s context and it highlights something even
more important. The craftsman’s context has a working code of
ethics. Some of these ethics would unfold as the day progressed…
• When and how do you go to the toilet?
• Do you use the customers, or do you wait and go
elsewhere?
Peter said that he would only ever use a customer’s toilet in an
absolute emergency and then only to take a pee, otherwise he
would drive back to use his office.
By observing myself, and my own experiences from the
environment I was able to experience and learn from it that simple
things like tying your shoelaces and going to the toilet in this
context were in fact a hindrance, which they are not when
working in the comfort of an office. But it did not stop hear, I
began thinking about how more practical the electricians clothing
was compared to my own.
Although I was not carrying out the work I was already learning
how to adapt to this new environment, by observing how he
navigated his context, through my self-awareness.
I was holding a camera, and he was holding tools. I began to think
how his pair of clogs would solve my problem of untying my
shoelaces, because my hands were full. This is when I realized I
was having a living experience.
Self-observation: Different people, different cultures, if English
people do not wear clogs? What do they wear?
DEGREE OF PARTICIPATION
Something that became apparent after my field studies was the
degree of participation one has when observing others within
there own environments. From the outset I was worried that the
camera would be very intrusive and taping a complete working
shift might be too much.
What was interesting was how the days video shadowing
unfolded, at the start of the recording I felt a sense of self-
awareness that he was very aware that he was being filmed, such
as a quick look directly into the camera.
At the start when meeting fellow crafts people who were
unprepared by my presence which for them was out of the
ordinary could be sensed and seen by the way they stared into the
camera on our first encounters or moved slowly out of the lenses
view, Peter would quickly react almost as a reflex on each
encounter and explain who I was and what I was doing and then
quickly proceed about his daily routine.
“Whether the ethnographer is doing a form of participant
observation or interviewing, she or he is still participating in a
material, sensorial and social environment” (Pink 2009:105)
What was interesting about these encounters was how they
changed with each set of new encounters, Peters explanations of
my presence got shorter and until eventually he no longer gave an
explanation he just mumbled out a non-intelligible word or
sentence as he proceeded about his chores, he had avoided
answering there question entirely, this had a kind of numbing
effect, and seemed to play down my presence.
“Video may provoke some people to make faces, others to clean
up their speech, and yet others to move cautiously in front of the
camera. This effect – what scientists call bias – may wear off as
people become familiar with the presence of the camera” (Buur,
Ylirisku 2007:48)
My previous video observation work has been carried out in test
labs, these are controlled environments, and much of the work is
as a passive observer where we are interested in seeing how a user
will interact with a specific interface.
Fig 2 How much control do we have over our participation
This was different I was telling myself that I had to remain a fly
on the wall and keep a passive observer roll, but the fact that I was
there and following the crafts person, I was participating.

3. 3
“It is difficult to be a truly passive observer if you are in the field,
simply because you cant avoid interaction” (Sharp, Rogers, Preece
2007:326)
I was seeing how he carried out the job, at times he would even
explain to me in detail about specific methods or why he did the
things he did.
I was learning, I felt like an apprentice on his first day at work,
this was when I realized I was participating without even realizing
it
Video Rec. vs. Lived Experience
On another occasion while shadowing an electrician outside fixing
a street lamp onto the side of a building, I remember how bitterly
cold it was standing and holding onto a camera. I remember
alternating the camera from one hand to the other while placing
my free hand in my coat pocket to warm it up. At the same time
complaining about my own discomfort and fatigue of holding a
small video camera.
Fig 3 Sensing: Heavy, Cold, Tired and Balancing
The design team when reviewing the video could see the crafts
person standing on a ladder fixing the lamp in place, but really
could not comprehend or appreciate the weight of the object he
was holding, how heavy it became while he was holding it over an
extended period of time, effected by the objects temperature, or
the chill factor as the wind blows while standing still and confined
to the top of a ladder, and yet still maintaining a steady balance.
Experience beyond what the camera can see
In our design team sessions while discussing and reviewing the
video footage I found that my own experiences during the field
study more immersive than footage recorded in the field by other
team members, there clips felt more detached and distanced. I felt
that something was missing; I had not lived their experience. I
also began to observe an apparent enthusiasm for there own clips,
in contrast to other member’s clips.
Was this detachment from each other’s clips mutual? , And did
they feel that there was a connection between this detachment and
the lived experience. Would living each other’s experiences
reduce or remove this feeling of detachment.
This apparent detachment I was observing, made me ask the
following questions to the design team:
• Do you have a stronger attachment to your own clips?
• Why do you feel a better attachment to your own clips,
than those of other team members?
Members of my design team said they did feel a feeling of
detachment while watching other member’s clips compared to
there own, and interestingly they did feel that it was easy to recall
details from there own lived experiences.
It became apparent to us all that the video seemed not only to be a
memory trigger of our lived experiences, but it allowed us to
recall personal experiences, details that could be recollected
which were beyond what the camera lens had actually recorded.
From living the experience it became clear that we could use the
video as a form of Dictaphone, so that you could use it to index
personal experiences and recall them later.
We also discussed in a future field study, how it would have been
interesting to carry out a video shadow over a number of days
following each others users, so that each member of the design
team could experience a similar lived experience, to see if this
would close the detachment between each other’s clips.
Living Experience – Sensory ethnography
During the review of my video footage, while preparing clips for a
video card game (Buur, Ylirisku 2007:105), I felt a strong need to
explain my experiences to the design team; I felt that the short
clips did not convey enough empathy. Before I carried out the
field study I had already envisioned how an electrician went about
his daily work, I believed at the time they worked with electrical
cables and fuses all day. I never realized how far from the truth
this was.
I decided to produce a video portrait to allow other members of
my design team, to feel empathy for an electrician, by putting
them into his shoes, and to show them how physically hard his
work was.
It was during the presentation of this video portrait and the
playing of the video card game, that it became apparent that the
lived experience had been a multisensoriality of experience,
perception, knowing and practice. Which I felt had not been
captured by the video camera.
As an interactions designer I decided to contact an anthropologist
for current background literature for doing sensory ethnography as
I thought it was important to understand and enrich my knowledge
about the senses.
Sensory ethnography
Pink 2009, suggests an approach to doing sensory ethnography
where “vision does not dominate the way we experience our
environments” (Pink 2009:14), however (Pink 2009:28), explains
that there is some disagreement among scholars of the senses
regarding how phenomenological understandings might be
employed.
• What human perception involves
• Interconnections between the senses
• Relationship between perception and culture
This is due to how neurobiologists are challenging conventional
thinking, that are senses are not modular functions but in fact are
cross-modal interaction between the senses.
Much of the theory concentrates on re-thinking the ethnographic
process through reflexive attention to the ‘sensoriality’ of the
experience, by outlining a sensory ethnographic methodology.
Where “Embodiment” implies an integration of mind and body,

4. 4
and “Emplacement” is used to suggest the interrelationship of
body, mind and environment.
Pink 2009, suggests that often moments of sensory learning are
not planned, “Rather, these are often unplanned instances whereby
the researcher arrives at an understanding of other peoples
memories and meaning through their own embodied experiences
and/or attending to other people’s practices, subjectivities and
explanations.”(Pink 2009:65), this is the case during my field
study example described in “Video Rec. vs. Lived Experience” of
alternating my hands between my warm pocket and the cold
camera, while recording the electrician standing on a ladder, it
was through this engagement that I was able to reflect, and
conceptualize meaning through my own emplacement. The fact
that the camera felt cold by physical sensation of touch bridged a
gap beyond my own physical touch and touching a conscious
connection with the crafts person’s.
“However, an understanding of the senses as essentially
interconnected suggests how (audio) visual images and recordings
can evoke, or invite memories of the multisensoriality of the
research encounter.” (Pink 2009:101)
Fig 4 Confined, heavy, large, difficult to hold
I agree with Pink (2009:107) that the power of video as a
technique is invaluable to help researches use video to
communicate aspects of these experiences to others, or from my
own experience of how the video can evoke, or invite memories
from the lived experience, by observing my own ego, through act
of becoming aware of my self, I am able to use video as an
expressive medium for personal learning.
Although it would be impossible to cover in such a short paper,
Pink explains how we in the modern western culture have divided
the senses into vision, hearing, touch taste and smell, where as
other cultures do not necessarily divide them in the same way.
She goes on to explain a method which researchers can develop so
that they can be “of two sensorial” about things.
“Being able to operate complete awareness in two perceptual
systems of sensory orders simultaneously (the sensory order of
ones own culture and that of the culture being studied)” (Pink
2009:51)
The idea behind this is to break down our experiences into
sensory categories, and if we are aware of which categories we are
using they can help us to understand and give meaning to different
types of sensation.
“It does nevertheless imply that in applied research attention to
the senses can lead to an appreciation of what is important to
others”(Pink 2009:59)
Although these methods can help me in a field study to observe
users, I feel that there is a greater issue to address on how to
bridge what I will call a “gap of detachment” between members of
the design team who have lived the experience and those who
have not. I will come back to this later in a proposal to
“Annotating the experience”.
Living Experience – Mobile UI for crafts
people?
The software company I work for has developed an ERP1
system
over the last 25 years called ASPECT4. It is used within many
different industries. The front end to this system is designed for a
Windows desktop environment, not a mobile platform.
While writing this paper I was approached by our builders supply
department who have been thinking about moving parts of there
product into the mobile space.
He said: “I have spoken with your department head and he has
recommended that we speak to you about a new mobile UI we are
in the process of designing, which will be used for crafts people to
register there hours, through a web based solution using the
mobile browser opera.”
Fig: 5 Hands free… workflow is never interrupted
It was during his informal presentation, I asked him “have you
ever been out in the field with a crafts person while they are
working.”
I said to him from my lived experience, I could not see crafts
people using such a solution. “My lived experience….”,
It was at this point I began to experience that I had moved beyond
my previous personal role as a designer thinking in information
and cognition theory, founded in my HCI background, and was
reflecting over the lived experience, which was a much stronger
and a more personal experience, which I could now relate to.
In Fig 4.0 the electrician is busy working when he receives a call,
he takes the call but in no way does it become a hindrance in his
workflow he just continues working, while taking the call.
He then needs to check a date, so he says he will phone the person
back later, he then rips a piece of cardboard off of a screw box,
1
ERP – Enterprise resource planning

5. 5
writes down there telephone number and details. This is put into
his back pocket and he continues working.
Later on in his workflow, on returning to his van, the cardboard in
his pocket becomes a form of physical post-it which reminds him
to check his calendar, with a mobile PC mounted in the vans
cabin; the cardboard is used as a physical reference, and like a
post-it is thrown away when finished with.
Although the video documents him speaking and working at the
same time, I kept reflecting over the tying of shoes laces as
described in one of the my previous living experience, it then be
came apparent to me that this was a central aspect of this crafts
person’s context, and I could relate and begin to understand the
importance it meant to him that his hands are constantly free at all
times to carry out his work.
What was even more interesting was that I could explain this lived
experience without even showing the video footage to my
colleague I felt comfortable when I said to him that I could not see
crafts people using such a solution.
As an interesting side note to this, I am in the process of building
a new house, and when I attended one of the building project
meetings, I asked several of the crafts people, including funny
enough the electrician from my field study if they would use such
a system, and they all said “no”. When asked why they said that
from their experience they were more of a hindrance because they
got in the way of doing the job. They said: “Pen and Paper is easy
and reliable”.
Although the lived experience had highlighted how important a
hands free environment was for a craftsman, the personal benefit
as a designer to make a cost effective decision was even greater.
Annotating the experiences
During much of this paper I have focused on the living
experience, and explained how the design team had experienced
the same detachment while reviewing clips by other members.
So how can I reduce this gap of detachment between my lived
experience and how the design team perceives my clips when
viewing them?
After much reflection I wondered if it would be possible in some
way to annotate parts of my lived experiences onto the video
clips, allowing the design team to experience some of the
researchers embodied experience and practice, and vice versa
when viewing there clips.
I propose the following two methods as a possible solution to
reducing some of the detachment experienced by the design team:
• Annotating the video during the field study
• Annotating the video after the field study
Annotating during a field study
In the field studies I have carried out, using video shadowing
(Buur, Ylirisku 2007:65) it was done by using a single camera and
a built in microphone, I propose an alternative to this setup where
an extra microphone is added.
The first microphone would be used to capture the dialog of the
user being shadowed and the second microphone would capture
the observers dialogue. The observer would then record their
thoughts/observations/experiences in the way one would use a
Dictaphone to record notes, by speaking aloud.
This would obviously require that the observe:
• Becoming aware, of ones own thoughts and actions.
• Catching them self’s in the act of thinking and/or
reacting
• Comprehending the source and utility of such thoughts
and actions
The users and observers dialogue would be recorded on separate
tracks with the same time code, this would allow the design team
to swap between these tracks, or hear both tracks at the same time.
Advantages:
• The observer would describe their embodied
experiences and practices.
Disadvantages:
• Would an observer have the same problems associated
with users during a speaking-aloud test, where the
cognitive load could become too large and the observer
stops talking.
Annotating after a field study
Pink 2009 - “Visualising Emplacement”, I could not help thinking
about a question which had been raised about, to what extent
embodied experiences and practices can be represented with
(audio)visual media. An example is given about a researcher who
films an actor’s transfiguration when putting on makeup, the lens
only captures surface elements, but it is represented powerfully in
the following reflexive text:
“The color, smell, and textures of these objects [the make-up and
accessories] were familiar to me I moved so that I had a direct
view of Iemoto’s face reflected in the mirror. As I watched her put
on a habutai (a purple silk fabric to cover the hair under a wig)
and begin to apply layers of makeup, an empathetic sensation
came over me as if I could feel the makeup on my face too.” (Pink
2009:104,Hahn 2007:147)
If this can be written as a reflexive text, then why can this not be
annotated as a separate audio track over the original video clip?
I have seen this method used to great effect on DVD’s by
directors, when they explain to film enthusiasts, what they where
thinking when filming specific shots, and what they wanted to
convey to the audience.
When an observer has completed their field study they could
annotate the video with a separate audio track this would be done
during the reviewing of the footage.
They would then speak aloud-about there thoughts and memories
invoked by watching the clips.
Advantages:
• After completing a field study you may see things that
at the time were disjointed, or at the time unexplainable.
• The cognitive load would be less than out in the field.
Disadvantages:
• Data will be lost; we have to rely on their memory.

6. 6
CONCLUSION
Using the method of video shadowing alone is an effective
method to gather a record of what users do, when edited together
with a video interview it has aloud me to create a video portrait to
provoke empathy to other members of my design team.
It has also shown me that the video captured contains a trigger to
invoke memories from the field study and that even more
importantly that there is an experiences even beyond what the
camera can see.
This living experience is so profound that I was shocked that only
one days filming could give me so much insight into a particular
use context.
It has also shown me that an observer requires a better
understanding about how to observe their senses.
Therefore preparation is required by any observer on how tap into
this resource, pink 2009 chapter on “Preparing for sensory
research” is a good starting point.
What does this mean for a designer to have been out there where
things happen?
Personally as a designer it has only enforced the importance of
doing field studies, and getting away from the clinical
environment of the office, because living the experience allows a
greater and wider knowledge to understanding what users really
do. In the long term this knowledge will allow for better
innovation because my ideas will be based on user knowledge and
experience.
The mobile UI is a good example from my own working
environment where time and money can be saved, based on a
personal experience.
From my field study experiences I believe that annotating the
video would allow the possibility for members of a design team
who were not involved in the actual field study to obtain a better
understanding of the users context by observing the observer
through there audio dialogue. Thus reducing some of the
detachment experienced by the design team.
ACKNOWLEDGMENTS
I would like to thank Wendy Gun for literature from an
anthropology perspective.
Thank you to Peter Gull for allowing me to follow him while he
was working, using the video shadow method and permission to
use the material in this paper.
I would like to give a big thanks to Henrick, Louise and Kim the
members of my field study group for their knowledge and input.
Thank you prof. Jacob Burr, for your patients and knowledge,
your book was inspiring, and your video methods are now a part
of a new and developing tool case.
Finally thanks to my wife and boys for their support, for the many
hours I have been away from home.
REFERENCES
[1] Interaction design – Sharp Rogers Preece
ISBN 978-0-470-01866-8.
[2] User Centered Product Design – J. Buur, T.Binder
[3] Designing with video – Salu Ylirisku, Jakob Buur
ISBN 978-1-84628-960-6
[4] Design of Everyday things - Donald A. Norman
ISBN 13-978-0-465-06710-7
[5] Pink,S.2009. Doing Sensory Ethnography. London: Sage
Publications.
Literature background reading
Russell, C. 1999. Experimental ethnography: The Age of
Film in the Age of Video, Chapter 3.The body as main
attraction
USA: Duke University Press.