Putting the Human Back in the Loop: Keynote Talk at IS-EUD 2023 Cagliari

Putting the Human
Back in the Loop
Antti Oulasvirta
Keynote, IS-EUD
June 7, 2023
https://cg3hci.dmi.unica.it/iseud2023/

“Design is where the action is”
Allen Newell at CHI 1985

So how does one know
what is good for someone?
To design is to add value

The foundational
technical problem

Putting the Human Back in the Loop - Antti Oulasvirta Cagliari June 7, 2023
Optimal design
<latexit sha1_base64="By1QtCmC3SL2Yll2dZFkd6ngNzA=">AAACkXicfVHfS9xAEN6Lttr0h2d9bB+WHoKVciSlan0QjlZBkFILPT24pMdkMxcXN5uwuxGPJS/+NX2t/43/TTdnCq1KBxY+vvlmduabpBRcmyC46XgLi48eLy0/8Z8+e/5ipbv68kQXlWI4ZIUo1CgBjYJLHBpuBI5KhZAnAk+T889N/vQCleaF/G5mJcY5ZJJPOQPjqEn3dfpjk+7RCFSWw+XEpjTiku7XNNtI3066vaAfzIPeB2ELeqSN48lqJ47SglU5SsMEaD0Og9LEFpThTGDtR5XGEtg5ZDh2UEKOOrbzNWq67piUTgvlnjR0zv5dYSHXepYnTpmDOdN3cw35UG5cmenH2HJZVgYlu/1oWglqCtp4QlOukBkxcwCY4m5Wys5AATPOOd+P9tEto/CLa/y1RAWmUJu2tax2y2XRuwb9T8jlH6FDDwptdDCqbdRMnyT2oK7XKV6Wbi7nyQWICn3fdwcJ79p/H5y874fb/a1vH3qDT+1plskr8oZskJDskAE5JMdkSBi5Ij/JL3LtrXm73sBrtV6nrVkj/4R39BtOycis</latexit>
d⇤
= arg max
d2D
g(d)
Find the design
…from a finite set of designs …
…that maximizes the goodness function

Optimal design
d⇤
= arg max
d2D
g(d)
Find the design
…from a finite set of designs …
…that maximizes the goodness function
Generative design
Adaptive interfaces
Personalization
Recommendations
Calibration

Solutions to this challenge would
enable computers that can better
serve users, taking their needs,
capabilities, and situations into
account

They would help designers
reach beyond intuition and
empirical evaluation

Every computational approach
has been tried on this problem

Oulasvirta et al. IEEE Proc. 2020
Combinatorial optimization
Example: generating menus with
human performance models
Computer-generated menus are 25% more usable
Model
Objectives
Constraints
Design task
Design solution
Solver

Underlying models are weak,
which hampers application
Towards Real-time Perceptual Optimisation of Sca�erplots ELEC-E7861 Research Project in Human–Computer Interaction,
Figure 6: The same data set as in Fig. 4, optimised with the
algorithm presented in this paper, in ⇠17 seconds. Aspect
Figure 7: A random data set with 15 625 data points, opti-
mised (in ⇠16 seconds) for correlation estimation using the
algorithm presented in this paper.
Micallef et al. 2017 IEEE TVCG
Objective function
Lots of fiddling required to
set up an objective function

Cognitive architectures
GLEAN3 – Kieras et al. 1999
Programs
Processor
Modules

Cognitive architectures
GLEAN3 – Kieras et al. 1999
Programs
Processor
Modules
Basically rule-based AI
New design, new
production system

Reinforcement learning
RL agents simulating users
State
Action
Reward
Jokinen et al. CHI’21
Ikkala et al. UIST’22
Reward functions very
hard to engineer
Scope limited to
sensorimotor
performance

Supervised learning
Input vector
Prediction vector
Data
Real human data Predicted by model
Fails with OODs
Jiang et al. CHI 2023

All methods fail to capture users’
tacit preferences
All methods fail to
capture tacit
preferences and
individual differences

The field is in an impasse
?

“The race is between the
tortoise of cumulative
science and the hare of
intuitive design”
– Allen Newell at CHI‘85
https://upload.wikimedia.org/wikipedia/commons/2/27/Frans_Snyders_
-_Fable_of_the_hare_and_the_tortoise.jpg
The Aesopian
race of HCI

Bayesian optimization
… “asks the user” to learn the
objective function

It uses two elements to tackle the
exploration/exploitation problem
1. A probabilistic surrogate model
• a prior distribution that
captures beliefs about the
behavior of the unknown
objective function
2. An acquisition function
• describes how optimal a
query is
Shahriari et al. 2016

It acquires and updates, iteratively
Step n
Step n+1
Step n+2

It is proposed to automate tasks
that normally require an expert
“design problems are fraught with choices,
choices that are often complex and high
dimensional, with interactions that make
them difficult for individuals to reason
about. “
(p. 148)
A global method
Derivative-free
Sample-efficient
Black box

The method is popular in ML and
applied engineering
https://pubs.acs.org/doi/abs/10.1021/acsami.1c16506
Hyperparameter tuning
It needs an “oracle” that
tells how good a design is
But what if the
“oracle” is a human?

https://upload.wikimedia.org/wikipedia/commons/f/fa/John_William
_Waterhouse_-_Consulting_the_Oracle_-_Christie%27s.jpg

Human-in-the-loop applications
Optimizer
Chan et al. CHI’22; submitted

Preparing an optimizer
Define design variables
(<20)
Define objectives
(<10)
Set up a testing
environment

The method is sample-efficient
6 times fewer iterations needed than with manual design
Brochu et al. 2010
Significant savings
over manual editing

What makes tactility good?
Liao et al. CHI’20

Optimizing tactile experience
Liao et al. CHI’20

The method can optimize UIs
5-10% improvements in pointing throughput achievable
Kim et al. CHI’20

A typical surrogate model in that
study
Kim et al. CHI’20
The method tackles
variability and
noise in user inputs

It can optimize for tacit preferences
Zhu et al. UMAP’23
Which SHAP-based explanation of “dog” is the best?

”Hyperparameter tuning with
humans” Zhu et al. UMAP’23
Optimal accuracy—explainability tradeoffs found with BO

It can handle multiple
objectives
User A
User B

Example: Tailoring a VR keyboard
Bayesian optimization for speed (WPM) and accuracy
Shen et al. ISMAR 22

It also supports
designer-in-the-loop optimization
It reduces cognitive effort in design
Chan et al. CHI’22; submitted

It can adapt using sensor data
Kim et al. 2016 Plos One

It can work with implicit signals
AdaptiFont - Kadner et al. CHI’22
Measure reading speed Optimize font

It can optimize for groups
(as well as individuals)
“Global GP” exploits information from multiple users to compute a
an optimal compromise
Liao et al. submitted

Getting started with code is quick
GPyOpt

It may have potential applications
in end-user development
Speculation warning!
CoScripter Scaffidi et al. 2010
Generative design
Adaptive interfaces
Personalization
Recommendations
Calibration

But it fails as an
interactive method…

Agency is
fundamental to
creative activities

Bayesian optimization
“experiments” on people

Chan
et
al.
CHI’22
Human-in-the-loop methods may
diminish agency

The reason: “Optimal design”
does not consider interaction
d⇤
= arg max
d2D
g(d)
Would keep asking the
user forever
Assumes that users do
not change
Users have no control
over process
Shortcomings
Users cannot express
their knowledge

Interactive Bayesian Optimization

From optimal experiments to
optimal interaction
Users have knowledge
Users also learn
Users have limited time
Principles we need to recognize

Colella et al. UMAP’21
Let users
exaggerate

Exploit ”common knowledge”
from other users
Warm-start GP uses a prior learned from other users

Yields a significant improvement
in efficiency

Cooperative interfaces for BO
Mo et al. submitted
Designers using
the UI explored
more and had high
agency

Infer users’ goals without asking
directly
Estimate users’ goal from slider manipulations and use that to
drive a Bayesian optimizer
Koyama et al UIST’22

Examples

“Human first, machine last”
Let an expert explore first, use BO
to drive the rest
Significant savings over human-
only and machine-only approaches
You can find an optimal handover
point empirically

We need to recast the problem
of optimal design

WIP: Sequential decision-
making with latent rewards
<latexit sha1_base64="uoIjEr73vYJO0XYlZq1QSBS8w7Q=">AAAC2HicfVFNbxMxEHWWr7J8pXDkYhFVSUoV7SK+LkgVUIkLIiCSRsRL5HUmW6v27tb2Vo2MJW6IK/+Kn8Cv4Ao3vOkioK0YydLTmzfjmTdpKbg2UfStFZw7f+HipbXL4ZWr167faK/fHOuiUgxGrBCFmqRUg+A5jAw3AialAipTAbvp/rM6v3sISvMif2uWJSSSZjlfcEaNp2ZtNn5vSck3XU/38ROMSZXPvRyMPXBE0iNMUp5lxOI3Pb110Md3McmolBQTXcmZ1V2Hhz3dxR/wKv2nW7fflLpw1u5Eg2gV+DSIG9BBTQxn662EzAtWScgNE1TraRyVJrFUGc4EuJBUGkrK9mkGUw9zKkEnduWGwxuemeNFofzLDV6xf1dYKrVeytQrJTV7+mSuJs/KTSuzeJxYnpeVgZwdf7SoBDYFrq3Fc66AGbH0gDLF/ayY7VFFmfGOhiF5Dn4ZBS9941clKGoKtWkJVZn32fnlMrJVo/8Jef5b6NGZQkt2Js6Sevo0tTvObWA4Kv1c3pNDKioIw/og8Un7T4PxvUH8cPDg9f3O9tPmNGvoNrqDeihGj9A2eoGGaIQY+oq+ox/oZ/Au+Bh8Cj4fS4NWU3ML/RPBl1/iA+M8</latexit>
V ⇡⇤
(s) = max
q
⇢
R(s, q) +
X
s0
P(s0
|s, q)V ⇡⇤
(s0
)
<latexit sha1_base64="lLgJyvGKt4bwKSHBLsuXq1DqI6A=">AAAC+nicfVHLbhMxFPUMrzK8EtjRjUVUlFZpNEG8NkgVUIkNoiDSRoqHyOO5mVj1eCa2pyQys+RH2CG2/AzfwE/gSQaptBVXsnx07rkPH8eF4NqE4S/Pv3T5ytVrG9eDGzdv3b7Tat891HmpGAxZLnI1iqkGwSUMDTcCRoUCmsUCjuLjV3X+6ASU5rn8aJYFRBlNJZ9yRo2jJq2v2MWHF0F9kRhSLi1z7XS1YjBOu8mnna7uzbe3e/ghJgYWxvJpRTDJ4nxhDaiMSyrwvARdt6wwIU3tLmbd+amy3MxAfeYa1s0JyKQZNml1wn64CnweDBrQQU0cTNpeRJKclRlIwwTVejwICxNZqgxnwg0gpYaCsmOawthBSTPQkV35VeEtxyR4mit3pMEr9nSFpZnWyyx2yoyamT6bq8mLcuPSTJ9HlsuiNCDZetC0FNjkuDYfJ1wBM2LpAGWKu10xm1FFmbNRBwF5De4xCt66xu8KUNTkascSqtKMLir3uJT0avQ/IZd/hQ5dKLRkf1RZUm8fx3a/qrYwLAq3l/PkhIoSgiBwHzI4a/95cPioP3jaf/L+cWfvZfM1G2gTPUBdNEDP0B56gw7QEDH022t7971N/4v/zf/u/1hLfa+puYf+Cf/nHz1M6lc=</latexit>
R =
(
g(d⇤
(s, q)), if terminal question
c(q), otherwise
Choose a sequence of questions that maximizes subjective payoff,
i.e. the benefits of the best design minus the costs of asking
De Peuter et al. AI Magazine 2023

“Optimize anything”
A vision for end-user optimization

Objectives
Min CdA
Max avg W
Min RPE
Objectives
Tactile experience
Durability
Price
Optimize
Saddle height
Stem length
Bar width
Optimize
Material
Thickness
Texture
Material
design
Bike
fitting

Characteristics of the 99%
• Do not know optimization
• Have both objective and
subjective measurements
• Rely on wonky measurements
• Have plenty of prior
knowledge
• Have limited time
Algorithms and UI
design must
converge

Landing page
Optimize Anything!
Let AI help you find the best solution
Three steps:
1. Define. Tell us what you want to
optimize (5 mins)
2. Optimize. Let AI help you find the
best options. (Stop when you want.)
3. Results. We’ll present you the best
options with their tradeoffs.
Learn how X
optimized
their Y (sports)
Learn how X
optimized
their Y
(engineering)
Learn how X
optimized
their Y
(design)

How it works
You AI
A solution
idea
Evaluation Ready
Skip this
one
More like
this
AI will propose you solutions one at a time. You evaluate them
and tell the AI. You can always propose solutions and steer the AI.
I want us
to try
this:…
Learn why this
method works

1. Define: Decisions
Let us know what you want to optimize
What factors do you need to
decide?
Describe each factor that you want to decide.
Examples: “saddle height”, “material thickness”,
“lamp color”.
1: [name ] [units] [min][max]
Next
Example case
Example case
+ add Tips

1. Define: Objectives
Let us know what you want to optimize
Which measurements do you
want to optimize?
Describe your objectives. You can include also
subjective measurements, even opinions.
Examples: “fatigue”, “fuel efficiency”, “price”.
1: [name ] [type] [units] [min/max]
Ready
Example case
Example case
+ add Tips

One question before we start…
Are there known good/bad solutions we should include?
Known solutions
Variable 1 (Saddle height): [type value]
Variable 2 (Stem length): [type value]
[good / bad solution]
Example case
Example case
+ add Tips
Yes, some
No, let’s
start

2. Optimize
Let AI suggest solutions with you
1st solution idea
[saddle = 18cm] [stem length = 11cm]
I’m done
Tips
Skip. I know
it’s not good
I want to
evaluate this
[Give a memorable name to this idea]
CdA = [enter measurement]
W = [enter measurement]
RPE = [enter measurement]
Give me the
next one
I want to
refine this

3. Results
Here are the best options we found
Option 1
[stem length = 9cm] [saddle = 17cm]
This option is great in CdA and W, but
weaker in RPE.
Restart
Show all
results in a
single plot
Option 2
[stem length = 12cm] [saddle=13cm]
This option is great in RPE, but weaker
in CdA and W.
Visualize
Visualize

ML methods were not developed for
interaction with humans

Bayesian optimization is great for design
• It works for cases where human input is scarce but
informative
• Great for personalizing interactions
But…
• Ironically, users lack agency
• This can be alleviated by
• by interface design for cooperation
• by rethinking the optimization problem
• by stronger user models

As a field we need to converge
algorithms with design…
• We need not just apply algorithms but appropriate
them and make them human-centric
• What assumptions do they make about people?
• How can interfaces optimally support these?
• We need to design interactions that require minimum
technical knowledge

Also big thanks to my collaborators Per Ola Kristensson,
Liwei Chan, Yi-Chi Liao, Samuel Kaski, Sebastiaan De
Peuter, John Dudley, Tomi Peltola, Jukka Corander, Antti
Kangasrääsiö, John Williamson, Suyog Chandramouli,
Yifan Zhu, Jukka Corander, Byungjoo Lee, Sunjun Kim
Notebook to teach BO for HCI students
https://github.com/oulasvir/bayesianoptimization/blob/mas
ter/Introduction_to_Bayesian_Optimization.ipynb
Thank you!
SIGGRAPH tutorial by Igarashi et al.
https://dl.acm.org/doi/pdf/10.1145/3450508.3464551

Putting the Human Back in the Loop: Keynote Talk at IS-EUD 2023 Cagliari

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Putting the Human Back in the Loop: Keynote Talk at IS-EUD 2023 Cagliari

Similar to Putting the Human Back in the Loop: Keynote Talk at IS-EUD 2023 Cagliari (20)

Recently uploaded

Recently uploaded (20)

Putting the Human Back in the Loop: Keynote Talk at IS-EUD 2023 Cagliari