Visualisations of
Video.
Ben Kraal
People + Systems Lab
QUT
What I’m going to talk about today is how to turn video into data, make that data useful and get
some insights from it.
[fade in] In other words, how to create visualisations of video.

Hi, I’m Ben.
@bjkraal

I work at the
People + Systems
Lab, QUT.
We do Design Research.

These are some of the projects that have been associated with the P+S Lab in the past. We’re
focussed on learning about context, activity, culture and people.

context activity
culture people
And we use those as our basis to move our focus from objects to doing design research about
systems and experiences.

early design stage
before design started finished artifact
design and
development
This is an (overly simple) explanation of how design research fits into practice.
On the left, you can do design research before you’ve started design. On the right, you can do DR
with a finished artefact. In the middle is practice. Practice is where you’ll find techniques like user
research in the early stage of design or usability testing in the development phases.

Visualisations of
Video.
What I’m going to talk about today is how to turn video into data, make that data useful and get
some insights from it.
[fade in] In other words, how to create visualisations of video.

UX is off the desktop
and in the world.

Designers will do
more research.

Use Video to
collect data.

Video makes data. Lots of data. This is the tapes from two small studies. At P+S we will often use
two cameras concurrently to capture two different views simultaneously. Depending on how we
record the footage we either mix the two cameras picture-in-picture or we use the power of our
analysis software to play the footage simultaneously.
Purely from a logistical point of view, we generate hundreds of gigabytes of video each year,
pushing into terabytes, and that’s before we do anything with it.

Visualise your video.
The best way to deal with all this video is to to turn it into data and then create visualisations to
describe that data.

More than you ever
wanted to know
about
I’m going to use a case study to explain these ideas. The case study is about a project we started a
few years ago. It’s about Leg Ulcers.
All about Leg Ulcers:
* sores
* venous insufficiency (varicose veins etc)
* compression therapy
* nurses
* assessed by instrumenting healthy legs and testing whether achieves ideal pressure

Here’s a still from some of the footage I took.
Nurse on the floor. Patient on the chair. Patient has her left leg completely bandaged and the right
is partially done.

Visualise your video.
Back to visualising your video.

Code
your
video.
Before you can visaulise your video, you need to code it.
Coding data is a practice that comes from the social sciences. A lot of the time they can code
transcripts of interviews. Coding video is quite similar.

Tag
your
video.
Another way to consider coding data is to think about tagging it.

Tag parts of
your
video.
Or, more fully, to think about tagging parts of it.

Turn video into data.
And this tagging, or coding, is what turns video into data.

We use
Noldus Observer.

Noldus Observer looks like this.
Coding scheme in the centre.
The way it works is to allow you to establish a coding scheme and then as the parts of the video
that are described in the coding scheme occur, the software makes it very easy to track.

But you don’t have to.
Now, we use Observer, but you don’t have to.

Visualisations make
video “sit still”.

Represent the data in
different ways.
And let you represent data in different ways.

25:00
24:00
23:00
22:00
21:00
20:00
Eg, like time-series charts. Like this one that describes some bandaging activity.
19:00
18:00
17:00
16:00
15:00
14:00
13:00
12:00
11:00
10:00
09:00
08:00
07:00
06:00
05:00
04:00
03:00
02:00
01:00
00:00
doing
planning
reacting
materials

Aggregate your data.
And then you can graph it, or slice and dice it in new and interesting ways.

planning
doing
reacting
materials
00:00
01:00
02:00
03:00
04:00
05:00
06:00
07:00
08:00
09:00
10:00
11:00
12:00
13:00
14:00
15:00
16:00
17:00
18:00
19:00
20:00
21:00
22:00
23:00
24:00
25:00
This chart describes 25 minutes of activity. So you can see the power already. 25 minutes of video
on one slide that you can take in and see the shape of in 30 seconds.
Now, to explain this a little better, I’m going to make up a fictional example.

00:00:00 00:05:00
This shows 10 minutes of (fictional!) bandaging activity.
But that’s not really very exciting.

00:00:00 00:05:00
stocking undercast type 2 bandage stocking
It’s much more interesting if we can describe the activity in more detail.
Eg: which bandaging materials are being used and for how long.
We’ll assign colours to different bandage types. Yellow can be a thin cotton stocking. Orange can be
undercast. Purple can be a type 2 bandage. And then another thin cotton stocking to hold it all
together.
[these should fade in.]

00:00:00 00:05:00
stocking undercast type 2 bandage stocking
If you wanted, you could code for other behaviours, or activities, too. You might code for taping
bandages down. [fade in]
Or for using scissors to cut bandage to length. (stocking comes in a roll, so you cut it before you
use it.)
Or tearing it to length, as you can do with undercast.

Visualisations let you
make comparisons.

Compare tasks
across observations.

Or
within observations.

Combine
levels of detail.

Ask questions about
activity and task
relationships.

Visualisations add
dimension
to analysis.
Visualisations add dimension to analysis.
And let you add dimensions to analysis.

00:00:00 00:05:00
stocking undercast type 2 bandage stocking
Which brings us back to here.
We didn’t find that knowing where bandages were fixed, taped or torn was useful, so we stopped
looking for it. [fade out]
And we even found that looking only at the bandaging activity wasn’t that useful (because the way
that bandages are used is proscribed, so we weren’t going to learn about non-standard practices,
nor did we want to), so we took a longer view.

00:00:00 00:10:00
preparing
doing
s uc t2 s
[This should start with just the vertical bars and the 00.00.00. Zoom the 10.00 in.
Taking this back to a wider view, of ten minutes, and bringing the same bandaging activity [click] in
I can introduce some other codes, for activity at a higher level than bandages. First I’m introduce a
code that encompasses all the bandaging activity: doing [fade in].
Which leaves all this time before the bandaging took place. Now, in the video of the nurse she
spends quite a lot of time before doing the bandaging preparing the work space. So we can call that
preparing.
Everyone with me?

planning
doing
reacting
materials
00:00
01:00
02:00
03:00
04:00
05:00
06:00
07:00
08:00
09:00
10:00
11:00
12:00
13:00
14:00
15:00
16:00
17:00
18:00
19:00
20:00
21:00
22:00
23:00
24:00
25:00
Now, you’ve seen this before, but now I trust you can follow it.
[talk through chart]
Now, there’s something on here that I haven’t explained yet, and that’s the code that’s shown in
pink about 3/4 the way through. And right now it doesn’t matter, and I’ll get to it later.

Expertise.
The chart I’ve just shown you was of a nurse who demonstrated high expertise in executing the
application of the bandages.

Less Expertise.
Now let me show you one with slightly less expertise.

00:00:00 00:10:00
p p preparing
d doing d
s uc t2 s
[fade in from l-r and t-b] Preparing, doing bandaging, preparing next two, then doing. Then
preparing last one, then doing.
We say this demonstrates less expertise because pattern of preparing and doing is broken up.
There is not a demonstration of the nurse being able to fully imagine the end-to-end bandaging
process and so prepare everything before executing it. Instead, we see a partial sort of preparation
and doing.

25:00
24:00
23:00
Now, what I showed hypothetically before is seen in this actual chart. Let me zoom it in.
22:00
21:00
20:00
19:00
18:00
17:00
16:00
15:00
14:00
13:00
12:00
11:00
10:00
09:00
08:00
07:00
06:00
05:00
04:00
03:00
02:00
01:00
00:00
doing
planning
reacting
materials

planning
doing
reacting
materials
00:00
01:00
02:00
03:00
04:00
05:00
06:00
07:00
08:00
09:00
10:00
11:00
12:00
13:00
14:00
Now, this is a different nurse and a different set of bandaging materials. As you can see, it’s
basically the same pattern with four different types of compression therapy working together. And,
during the bandaging you can see that the nurse takes some brief pauses in the middle to gather
the next materials she needs.
Now, because she only takes breaks after completing each bandage, we say that she demonstrates
experience because she knows the individual component tasks, but she doesn’t pull them together
as fluidly as the first nurse.

Less Expertise.
So that was a demonstration of lower expertise.

Inexperience.
Now, what inexperience looks like. I’ll show you a real chart and not to a hypothetical.

planning
doing
reacting
materials
00:00
01:00
02:00
03:00
04:00
05:00
06:00
07:00
08:00
09:00
10:00
11:00
12:00
13:00
14:00
First, in this chart, there’s no initial preparation because the activity started before I turned on the
camera. ;) But in the video it’s possible to see that the nurse has prepared most of what she needs
on a small tray.
She begins and then has a break in the “doing” with no corresponding turn to “preparing” -- she’s
asking another nurse for help. And that’s what the pink is, not “asking for help” but “breaking away”
or shifting focus from the completion of the bandaging activity to some other activity. Then she
resumes the activity and there is a large break at about the 5 minute mark. She’s demonstrating the
bandage shown in green to the patient. Then she begins but soon after has a large focus shift
*while doing* the bandaging. She’s winding it up the patient’s leg, deciding that it’s not right and
winding it back down again.
Then at the end she has to stop and gather the final material before applying it.

planning
doing
reacting
materials
00:00
01:00
02:00
03:00
04:00
05:00
06:00
07:00
08:00
09:00
10:00
11:00
12:00
13:00
14:00
15:00
16:00
17:00
18:00
19:00
20:00
21:00
22:00
23:00
24:00
25:00
planning
doing
reacting
materials
00:00
01:00
02:00
03:00
04:00
05:00
06:00
07:00
08:00
09:00
10:00
11:00
12:00
13:00
14:00
15:00
16:00
17:00
18:00
19:00
20:00
21:00
22:00
23:00
24:00
25:00
planning
doing
reacting
materials
00:00
01:00
02:00
03:00
04:00
05:00
06:00
07:00
08:00
09:00
10:00
11:00
12:00
13:00
14:00
15:00
16:00
17:00
18:00
19:00
20:00
21:00
22:00
23:00
24:00
25:00
And to go back, here’s all three charts at once. The nice regular chart of the nurse who
demonstrated high expertise, the slightly less regular and more “busy” chart of the demonstration
of slightly less expertise and the chart of the inexperienced nurse.
I should say that we did do more analysis than just these three observations. I’ve shown these three
because they illustrate the story.
Also, the signifigance and innovation of this approach needs to be underscored. Creating these
visualisations is not especially hard, but using them in this way for analysis is new and adding the
extra dimensions to describe activities at a higher level while maintaining the lower-level
descriptions is where the power lies.

So
we did this research.
Now what?
From doing the research we learned three things.

Expertise is visible
in the organisation
of activity.
First, we learned that expertise is visible in the organisation of activity.
This was something that the expert nurses could see as well, but they had no way to describe it.
They just “knew” if someone was good. Maybe they saw something else, too, but our new heuristic
is very useful, not least because it means that there is a shortcut to seeing bandaging expertise in
the field, rather than in a lab.

Change
the training?
the assessment?
The second thing is we were able to ask whether the nurses should change their training or
assessment for compression therapy.

Develop a new
treatment?
And finally, we were able to start asking questions about how it would be possible to develop new
treatments that were, for example, less labour-intensive, while still taking into account all the really
valuable knowledge that the nurses have about what they do.
Investigating these existing practices in order to develop something new isn’t something we’ve
completely pursued, so I can’t describe what the new product or system would look like.

Design research with
video is hard.
To bring this to a close,
Design research with video is hard. It takes a long time and you end up with a lot of unwieldy data.

Create visualisations
to work with video.
One way to work with that data is to create visualisations of the video in order to turn it into
something that’s easier to work with.
Creating the visualistions takes time, but if you’re going to really work the data and look for
something new, taking that time is worth it.
When doing design research, particularly of this type which is of an existing product or situation but
with a view to develop something new, this kind of in-depth analysis is valuable and can lead to
significant insights. But you can’t have the insights without the research and analysis.

Understand and
planning
doing
analyse interaction
reacting
materials
00:00
01:00
02:00
03:00
04:00
05:00
06:00
07:00
08:00
09:00
10:00
11:00
12:00
13:00
14:00
15:00
16:00
17:00
18:00
19:00
20:00
21:00
22:00
23:00
24:00
25:00
through visualisation.
Ben Kraal
People + Systems Lab, QUT
@bjkraal
This is the one thing I want you to take away.
You can understand and analyse interaction through visualisation of activities.

Further Reading
Popovic, Vesna and Kraal, Ben J. (2008) Focus-
Shift: Interaction and Expertise Level.
In: Design Research Society 2008, 16-19 July 2008,
Sheffield, UK.
http://eprints.qut.edu.au/14673/
Kraal, Ben J. and Popovic, Vesna (2007) Looking
for Expertise in Physical Interactions.
In: OZCHI 2007 - Australasian Computer-Human
Interaction Conference, November 28-30, Adelaide,
Australia.
http://eprints.qut.edu.au/11083/