This document summarizes research using low-level interaction data to analyze user behavior in interactive media experiences. The researchers collected mouse movements, keystrokes, scrolling actions and other data from over 26,000 user events to explore whether users were experimenting with a cooking media experience or using it as intended. They extracted micro-behaviors, standardized the data, trained a model to predict intent, and found they could detect technical difficulties like video issues that caused spikes in 'back' and 'next' button events. Further work is still needed, including a user study for ground truth and real-time detection methods.
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Using Low-Level Interaction Data to Explore User Behaviour in Object Based Media Experiences
1. Using Low-Level Interaction Data to Explore
User Behaviour in Object Based Media
Experiences
Jonathan Carlton1,2, Andy Brown1, John Keane2 and Caroline Jay2
1 BBC Research and Development, Salford Quays, Manchester, UK.
2 School of Computer Science, University of Manchester, Manchester, UK.
@JonoCX
4. Low-Level Interaction Data
Mouse movements, keystrokes, scrolling actions, etc…
Large sample size and remote data collection
Used successfully in many different contexts
5. What Data do we have?
26,717 distinct events
5,533 distinct events
6. Experimenting vs. Intended Usage
“Can it be determined, from low-level interaction data, whether a user
was previewing, reviewing, or experimenting with the experience, or
whether they were using it to support the cooking process as
intended?”
7. Experimenting vs. Intended Usage
Extracted micro-behaviours and sessions
Standardised the data
Trained a model with single-session participants and testing data
Predicted intended/experimenting for multi-session participants
8. Experimenting vs. Intended Usage
Extracted micro-behaviours and sessions
Standardised the data
Trained a model with single-session participants and testing data
Predicted intended/experimenting for multi-session participants
9. Experimenting vs. Intended Usage
Extracted micro-behaviours and sessions
Standardised the data
Trained a model with single-session participants and testing data
Predicted intended/experimenting for multi-session participants
10. Indicators of Technical Difficulties
“Are there detectable indicators of technical difficulties, specifically
video-related issues, present in the click-based interaction data?”
16. Summary & Next steps
Promising leads
Plenty of work left to do
A user study to establish ground truth
Real-time detection
17. Summary & Next steps
Promising leads
Plenty of work left to do
A user study to establish ground truth
Real-time detection
Editor's Notes
TRADITIONAL MEDIA
Media is captured using traditional tools -> linear programme is produced -> broadcast to everyone -> same content is played back on all devices
OBJECT BASED MEDIA
Media is captured using new and traditional tools plus metadata is produced and recorded -> packaged as a collection of objects -> broadcast to everyone accompanied by the metadata (describes how they can be assembled) -> individual devices assemble the objects according to the metadata, producing the best experience for the viewer in the content of their devices, environment, and preference.
An example of where object-based media has been used is CAKE (Cook Along Kitchen Experience).
It will adjust the content based on your confidence level and cooking context (how many people you’re cooking for)
Explain the picture.
The data gathered from CAKE is the focus in this talk, but the Origami experience has just launched and I encourage you to check it out – it has more features such as changing the camera angle.
A user experience study was carried out on CAKE to assess if it better supported the cooking process vs a traditional, linear format (single video and written instructions).
What do I mean by low-level interaction data? It’s anything from mouse movements and keystrokes on a keyboard to gestures on a touch-based device.
Many benefits such as the large sample size (lots of data is generated per user!) and the ability to collect remotely.
Has been used in many different application areas to assess things such as raising alerts for mild cognitive impairments, identifying navigation difficulties, and it’s been well explored in the security area (biometric authentication systems)
Interaction data was collected during this study, both low-level events (mouse events and keystrokes) and button click events.
This data forms the basis of the experiments presented in this talk.
We noticed that several users in the experiment recorded multiple sessions, so we proposed the following research question and formulate an experiment to answer it.
Session defined as where two consecutive events, recorded by the same user, are 40minutes apart.
Detecting this allows us to assign weight to the feedback left by people.
Micro behaviours are essentially statistics about the underlying data; total time, idle time, etc.
Session defined as where two consecutive events, recorded by the same user, are 40minutes apart.
Standardised the data to remove large variances in some of the features.
Trained a model with participants who only recorded a single session (intended) and data collected during the testing of the study (experimenting)
Used the data from participants with multiple sessions (more than one) and predicted which were their ‘real’ and ‘exploring’ sessions.
Found that the model was able to pick out sessions as ‘exploring’ but could not assess performance, we have no ground truth.
Some of the participants (10) reported technical problems with CAKE, with 8 reported video-related issues (videos not working or buffering).
Can we find indicators of these video-related issues in the interaction data?
Finding these indicators opens up the possibility of real-time detection and presenting corrective solutions to the user, i.e. why not try switching to the written view?
First step was to look at the data from a statistical perspective, so we used the same statistics extracted for the previous experiment, to find differences between atypical and typical.
A simple plot but informative, it demonstrates that the video-related issues did not effect the overall length of the session – my initial impression was that it would.
A graph that shows the number of events recorded over time for each of the atypical participants.
It’s a bit of an indecipherable mess but…!
If we compare that to the typical version of the graph (it’s zoomed), you can see that it’s roughly the same.
We found that this doesn’t really tell us much, in terms of indicators.
Explored the sequences of events recorded by the atypical participants and noticed this trend (it’s common).
This plot is for ONE of the participants.
Back and next button are buttons that control the stepping through the experience.
We then explored the frequency of these events occurring in per minute windows.
It’s clear from these two plots that there is a clear difference between the behaviours of the two groups.
However, there is no ground truth – we cannot definitively state that these behaviours are indicators.
We’ve identified some promising leads in these two experiments, with the potential to have major impact.
So there is plenty of work left to do – just getting start!
A user study designed around collecting the interaction data and synthetically injecting technical problems with the experience to measure how the user reacts.
We’ve identified some promising leads in these two experiments, with the potential to have major impact.
So there is plenty of work left to do – just getting start!
A user study designed around collecting the interaction data and synthetically injecting technical problems with the experience to measure how the user reacts.
Leading to real-time detection of these types of behaviours.
