Dynamics and Interaction Steven Strachan, Roderick Murray-Smith Hamilton Institute, NUI Maynooth & Department of Computing Science, University of Glasgow [email_address] http://www.dcs.gla.ac.uk/~rod http://www.dcs.gla.ac.uk/~rod/Videos.html With John Williamson, Parisa Eslambolchilar, Andy Crossan, Vuokko Lantz, James Kelly, Stephen Brewster, Andrew Ramsay May 2007

Dynamics & Interaction group Led by Roderick Murray-Smith Two-campus group. Based in The Hamilton Institute & The University of Glasgow Dr Murray-Smith spent 7 years at Daimler-Benz research in Berlin, followed by M.I.T. and Technical University of Denmark. Exploring the overlap between control theory, machine learning, probabilistic reasoning and human-computer interaction design. Leading group in novel forms of interaction design for mobile interaction. Act as consultants for Nokia, Samsung and Microsoft. Current staff: Dr. Steven Strachan (NUIM) Dr. John Williamson (GU) Andrew Ramsay (GU) Stephen Hughes (GU)

Led by Roderick Murray-Smith

Two-campus group. Based in The Hamilton Institute & The University of Glasgow

Dr Murray-Smith spent 7 years at Daimler-Benz research in Berlin, followed by M.I.T. and Technical University of Denmark.

Exploring the overlap between control theory, machine learning, probabilistic reasoning and human-computer interaction design.

Leading group in novel forms of interaction design for mobile interaction. Act as consultants for Nokia, Samsung and Microsoft.

Current staff:

Dr. Steven Strachan (NUIM)

Dr. John Williamson (GU)

Andrew Ramsay (GU)

Stephen Hughes (GU)

Control in HCI? More and more devices are incorporating inertial sensing… What new kinds of interface can we develop with this new sensing capability? Think of user and device as being in a loop of control…

More and more devices are incorporating inertial sensing…

What new kinds of interface can we develop with this new sensing capability?

Think of user and device as being in a loop of control…

From ‘look-and-feel’ to ‘ handling qualities’? Moving from discrete event -based systems like this: To continuous control like this?

Moving from discrete event -based systems like this:

To continuous control like this?

Dynamics & Statistics in HCI? Why introduce dynamics – is that not harder? We can only control what we can perceive . Dependent on feedback, so upper limits on the speed of change of display. Large steps in entropy are unnatural & error-prone Dynamics allows us to slip in ‘intelligence’ which couldn’t be done with a static interaction technique Why uncertain interaction? Uncertainty in user’s mind about what to do next, and system uncertain about user’s intentions. With mobile devices, interaction with the user is now continuous instead of discrete, and input devices are noisier. ‘ Honest’ interfaces Can lead to smoother interaction, with user behaviour regularised appropriately (e.g. K ö rding & Wolpert)

Why introduce dynamics – is that not harder?

We can only control what we can perceive .

Dependent on feedback, so upper limits on the speed of change of display.

Large steps in entropy are unnatural & error-prone

Dynamics allows us to slip in ‘intelligence’ which couldn’t be done with a static interaction technique

Why uncertain interaction?

Uncertainty in user’s mind about what to do next, and system uncertain about user’s intentions.

With mobile devices, interaction with the user is now continuous instead of discrete, and input devices are noisier.

‘ Honest’ interfaces

Can lead to smoother interaction, with user behaviour regularised appropriately (e.g. K ö rding & Wolpert)

Mobile Sensing: MESH hardware description Built by MLE Palpable Machines group. Modality Enhancing Sensor-pack for Handhelds Designed for the IPAQ range of pocket PCs Physical design same as the PCMCIA expansion jacket Triple-Axis acceleration sensing MEMS Accelerometers Orientation sense and gesture capture High Fidelity Vibrotactile Display Sample based, Non-Volatile sample storage,Audio bus-driven option Actuator – VBW32 rewound Triple-axis magnetometer Orthogonal Magneto-Resistive elements Capacitive sensing GPS

Built by MLE Palpable Machines group.

Modality Enhancing Sensor-pack for Handhelds

Designed for the IPAQ range of pocket PCs

Physical design same as the PCMCIA expansion jacket

Triple-Axis acceleration sensing

MEMS Accelerometers

Orientation sense and gesture capture

High Fidelity Vibrotactile Display

Sample based, Non-Volatile sample storage,Audio bus-driven option

Actuator – VBW32 rewound

Triple-axis magnetometer

Orthogonal Magneto-Resistive elements

Capacitive sensing

GPS

Next sensor pack.. SHAKE Our next generation pack… Bluetooth, wireless and compact Accelerometers, gyros, magnetometers and haptic feedback. Use for head, device or bimanual gestures.

Our next generation pack…

Bluetooth, wireless and compact

Accelerometers, gyros, magnetometers and haptic feedback.

Use for head, device or bimanual gestures.

Multimodal feedback from active inquiry Model-based interaction design – physical interaction with abstract concepts

Shoogle - Informative Shaking Shake the phone to feel (and hear) content discreetly Only produces feedback when stimulated Simple physical model of objects in a box Movement from accelerometers Impact with edges produces sound and vibration VIDEO

Shake the phone to feel (and hear) content discreetly

Only produces feedback when stimulated

Simple physical model of objects in a box

Movement from accelerometers

Impact with edges produces sound and vibration

BodySpace Using constraints in the environment

BodySpace With MLE Palpable machines group – Jussi Angesleva, Ian Oakley, Sile O’Modhrain

With MLE Palpable machines group – Jussi Angesleva, Ian Oakley, Sile O’Modhrain

Example Application Gesture controlled music player Volume is controlled at the hip. Track switching is controlled at the ear. VIDEO

Gesture controlled music player

Volume is controlled at the hip.

Track switching is controlled at the ear.

Location-Aware Interaction

gps Tunes Whereable computing…. Location-Aware audio feedback Quickening useful – don’t want user to run back and forth to get gradient! Use tilt and bearing to get rapid exploration

Whereable computing….

Location-Aware audio feedback

Quickening useful – don’t want user to run back and forth to get gradient!

Use tilt and bearing to get rapid exploration

GPS Navigation Problem GPS is useful but inaccurate Inaccuracy varies in a complex way Reflections, shadowing, poor coverage Could use hybrid positioning General problem – accurate representation of belief and trustworthiness

GPS is useful but inaccurate

Inaccuracy varies in a complex way

Reflections, shadowing, poor coverage

Could use hybrid positioning

General problem – accurate representation of belief and trustworthiness

Uncertain Display Poor displays lead to poor control Norman's example of The Royal Majesty “ precise” position

Poor displays lead to poor control

Norman's example of The Royal Majesty

Uncertainty in GPS Navigation Represent and display the true uncertainty of the navigation system – make it “honest” realistic display should regularise control behaviour Incorporate models environment models user models Monte Carlo sampling is a convenient statistical technique for dealing with uncertainty

Represent and display the true uncertainty of the navigation system – make it “honest”

realistic display should regularise control behaviour

Incorporate models

environment models

user models

Monte Carlo sampling is a convenient statistical technique for dealing with uncertainty

Particle GPS Browsing Map browsing; include uncertainty about where we are Project forward, find likely locations in the future. Show all the possible places we might be, given a map of the area User can scan around and project further into the future with inertial control. Target points are sonified. Video Demonstration

Map browsing; include uncertainty about where we are

Project forward, find likely locations in the future.

Show all the possible places we might be, given a map of the area

User can scan around and project further into the future with inertial control.

Target points are sonified.

Relevance to the mobile internet… Mobile devices are by definition ‘mobile’. Traditional methods of interaction with the internet are not good in this mobile domain. We should acknowledge our current location or context and use this to our advantage. Introduce the location-aware internet…

Mobile devices are by definition ‘mobile’.

Traditional methods of interaction with the internet are not good in this mobile domain.

We should acknowledge our current location or context and use this to our advantage.

Introduce the location-aware internet…

Scenario…

Scenario… you are here

Scenario… you are here you are here

Scenario… you are here you are here you are here

Scenario… you are here you are here

Scenario… you are here you are here

Social interactions?… Real-time interaction with people in your social network…

Social interactions?…

Real-time interaction with people in your social network…

Outlook Dynamics allow intelligence to be sandwiched into an interface ‘ look-and-feel’ of an interface, ‘noisy channel’, or in control terms, the ‘adaptive handling qualities’? The mobile internet has the potential to become a highly-interactive, embodied and location-aware internet…

Dynamics allow intelligence to be sandwiched into an interface

‘ look-and-feel’ of an interface, ‘noisy channel’, or in control terms, the ‘adaptive handling qualities’?

The mobile internet has the potential to become a highly-interactive, embodied and location-aware internet…