[TITLE]
Breaking the Boundaries of Human-in-the-Loop Optimization
[ABSTRACT]
Human-in-the-loop optimization (HILO) has emerged as a principled solution for parametric optimization, leveraging artificial intelligence techniques to generate designs while incorporating human evaluation of design qualities. While HILO has demonstrated success across various fields such as robotics, engineering, machine learning, and human-computer interaction, it still faces two significant limitations. First, it primarily focuses on single-objective tasks, leading to narrow application scope. Second, HILO usually starts with no prior information, resulting in low efficiency.
This talk introduces a range of computational augmentations aimed at addressing these limitations. We explore advanced artificial intelligence methods to tackle multi-objective tasks and investigate various strategies for constructing prior models to enhance the efficiency of HILO. By showcasing HILO's capability in addressing real-world problems and discussing its role in the future, this talk hopes to encourage the audience to apply HILO to their own challenges and domains.
[BIO]
Yi-Chi Liao is a Ph.D. candidate researching in computational interaction at Aalto University, under the supervision of Prof. Antti Oulasvirta. His research primarily revolves around optimization-based interface design and user modeling using reinforcement learning. Yi-Chi's previous works were published at top-tier conferences in the human-computer interaction field, such as CHI and UIST. For more information about Yi-Chi, please visit his website at http://yichiliao.com.
Breaking the Boundaries of Human-in-the-Loop Optimization
1. Saarland University, July 6th, 2023
Breaking the Boundaries of
Human-in-the-Loop Optimization
Yi-Chi Liao. Ph.D. Candidate, UI Group @ Aalto University
Website: http://yichiliao.com | Twitter: @yichiliao | Email: yi-chi.liao@aalto.
f
1
2. • Ph.D. dissertation:
• Human-in-the-Loop Design Optimization
• Current research group:
• User Interfaces Group at Aalto University, led by Prof Antti Oulasvirta
• Research interests:
• Optimization (inference-free); human-in-the-loop optimization, simulation-based optimization
• User modelling via reinforcement learning
• My tools:
• Bayesian optimization, reinforcement learning, meta-learning, and physics simulation.
• Am joining in November!
Yi-Chi Liao
2
3. Three goals
• Introduce human-in-the-loop optimization (HILO)
• Introduce the recent advancements in HILO
• Turn you into a Bayesian optimization user
3
4. Let's play a game! Here's how it works:
You can choose any floating-point number between 0 and 1.2.
Each number corresponds to a specific amount of money in return.
The challenge is to find the number that yields the highest amount of money!
4
5. Design optimization
0 1
0
3
What is the best location to place the optical sensor?
x ~ [0,1]
y ~ [0,3]
[Optimal Sensor Position for a Computer Mouse, CHI ’20]
5
15. Manual optimization
[ThirdHand: Wearing a Robotic Arm to Experience Rich Force Feedback, Liao et al., 2015]
What is the optimal torque value(s) that maximizes the gaming experience?
15
16. Manual optimization
What is the optimal way to split the strokes?
[EdgeVib: E
ff
ective Alphanumeric Character Output Using a Wrist-Worn Tactile Display, Liao et al., 2015]
16
17. Manual optimization
[Outside-In: Visualizing Out-of-Sight Regions-of-Interest in a 360 Video Using Spatial Picture-in-Picture Preview, Lin et al., 2017]
What is the optimal position, size, and tilt angle of the window?
17
18. Manual optimization
Design and
prototype
Analysis User testing
• Pros:
• It is easy and feasible
• Cons:
• It does not guarantee the result
• The developers may introduce bias
• Can not adapt instantly
18
19. “Don’t ever make the mistake [of thinking] that you can
design something better than what you get from ruthless
massively parallel trial-and-error with a feedback cycle.”
— Linus Torvalds
19
21. Human-in-the-loop optimization (HILO)
Interactive system
paremeterized by x
Computational
optimizer
User
Design candidate,
x ∈ X
Objective-function
value, y ∈ Y
Interaction
Designer
Design space, X
Objective space, Y
Optimal design, x
21
25. Bayesian optimization —
1.Surrogate model of the target
problem: a Gaussian Process
regression
2.Acquisition function: a
function estimates the utility
of selecting a design choice
25
26. 26
1.Able to (roughly) predict the
outcome of a design choice
2.Able to identify the design
choice that may yield the most
improvements within a huge
design space
Bayesian optimization —
1.Surrogate model of the target
problem: a Gaussian Process
regression
2.Acquisition function: a
function estimates the utility
of selecting a design choice
37. Surrogate model: Gaussian Process
• GP is a generalized form of multivariate Gaussian (normal)
distribution.
• Covariance function (kernel):
• White noise kernel
• Exponentiated quadratic kernel
• Periodic kernel
37
39. Acquisition function
• AF quantifies the utility, potential, or worth of different x points based
on the GP surrogate model.
• Computing AF is way cheaper than actual evaluation!
39
40. Acquisition function
• AF quantifies the utility, potential, or worth of different x points based
on the GP surrogate model.
• Computing AF is way cheaper than actual evaluation!
• Common acquisition functions:
• Expected Improvements (EI)
• Probability of Improvements (PI)
• Upper Confidence Bound (UCB)
40
41. Why is BO so good (mathematically)?
41
• BO is a general tool with the least assumption of the problem.
• BO is clearly explainable and transparent.
• BO is a computational framework allows for easy customization & extension.
44. [Bayesian optimization explains human active search, Borji and Itti, 2013]
[Generalization guides human exploration in vast decision spaces, Wu et al., 2017]
44
BO is the best model/framework to explain human’s
parametric decision-making!
45. When Bayesian optimization performs poorly?
45
• GP will not work well when dealing with
non-continuous function
• More sample points will be needed in high-
dimensional (input/output) function
• Gets too expensive when handling with huge
amount of data: GP
fi
tting time ~ O(m^3)
where m is number of observations
46. Bayesian optimization for human-involved
interactive systems
[Human-in-the-loop optimization of exoskeleton assistance during walking, 2017]
46
47. [A Bayesian Interactive Optimization Approach to Procedural Animation Design, Siggraph SCA ’10]
47
Bayesian optimization for human-involved
interactive systems
48. 48
[Designing Engaging Games Using Bayesian Optimization, Khajah et al., 2016]
Bayesian optimization for human-involved
interactive systems
49. Interested in BO for HCI? Also check out Yuki’s works!!!
49
https://koyama.xyz/
50. Is HILO / BO perfect?
There are boundaries!
• HILO only addresses single-objective problems
• HILO reduces the designer’s agency and ownership in the process
• HILO only optimizes for one user/task at a time
• HILO only works for prototype-free interactions (software interfaces)
• HILO is still not efficient enough!
50
53. From a single objective to multiple objectives
53
Can we come up with a new
acquisition function?
54. From a single objective to multiple objectives
54
[Single- and Multiobjective Evolutionary Design Optimization Using
Gaussian Random Field Metamodels, Emmerich, 2005]
[Pareto frontier learning with expensive correlated objectives,
Shah & Ghahramani, 2016]
55. Short duration,
high amplitude
Long duration,
high amplitude
Long duration,
low amplitude
Short duration,
low amplitude
Short duration,
high amplitude
Long duration,
high amplitude
Long duration,
low amplitude
Short duration,
low amplitude
What is the optimal set of tacton (tactile icon)?
55
[Interaction Design With Multi-Objective Bayesian Optimization, Liao et al., 2023]
56. Short duration,
high amplitude
Long duration,
high amplitude
Long duration,
low amplitude
Short duration,
low amplitude
Short duration,
high amplitude
Long duration,
high amplitude
Long duration,
low amplitude
Short duration,
low amplitude
What is the optimal set of tacton (tactile icon)?
Length indicates duration
Set 1
56
[Interaction Design With Multi-Objective Bayesian Optimization, Liao et al., 2023]
57. Short duration,
high amplitude
Long duration,
high amplitude
Long duration,
low amplitude
Short duration,
low amplitude
Short duration,
high amplitude
Long duration,
high amplitude
Long duration,
low amplitude
Short duration,
low amplitude
What is the optimal set of tacton (tactile icon)?
Length indicates duration
Color indicates vibration amplitude
Set 1
Set 2
57
[Interaction Design With Multi-Objective Bayesian Optimization, Liao et al., 2023]
58. Short duration,
high amplitude
Long duration,
high amplitude
Long duration,
low amplitude
Short duration,
low amplitude
Short duration,
high amplitude
Long duration,
high amplitude
Long duration,
low amplitude
Short duration,
low amplitude
What is the optimal set of tacton (tactile icon)?
Length indicates duration
Color indicates vibration amplitude
Set 1
Set 2
58
Set 3
[Interaction Design With Multi-Objective Bayesian Optimization, Liao et al., 2023]
59. Short duration,
high amplitude
Long duration,
high amplitude
Long duration,
low amplitude
Short duration,
low amplitude
Short duration,
high amplitude
Long duration,
high amplitude
Long duration,
low amplitude
Short duration,
low amplitude
[Interaction Design With Multi-Objective Bayesian Optimization, Liao et al., 2023]
What are the design objectives?
59
61. [Investigating Positive and Negative Qualities of Human-in-the-Loop Optimization for Designing Interaction Techniques, Chan et al., 2022]
61
62. [Investigating Positive and Negative Qualities of Human-in-the-Loop Optimization for Designing Interaction Techniques, Chan et al., 2022]
What is the optimal design?
Imagine there are just 10 discrete levels in each parameter,
the number of possible combinations is already 10^4!
62
63. [Investigating Positive and Negative Qualities of Human-in-the-Loop Optimization for Designing Interaction Techniques, Chan et al., 2022]
What are the design objectives?
63
64. [Investigating Positive and Negative Qualities of Human-in-the-Loop Optimization for Designing Interaction Techniques, Chan et al., 2022]
What are the design objectives?
E
ffi
ciency & accuracy
64
65. [Investigating Positive and Negative Qualities of Human-in-the-Loop Optimization for Designing Interaction Techniques, Chan et al., 2022]
65
66. [Investigating Positive and Negative Qualities of Human-in-the-Loop Optimization for Designing Interaction Techniques, Chan et al., 2022]
Does multi-objective BO really perform better
than human designers?
66
67. How do designers perceive BO as a tool?
[Interaction Design With Multi-Objective Bayesian Optimization, Liao et al., 2023]
67
68. [Investigating Positive and Negative Qualities of Human-in-the-Loop Optimization for Designing Interaction Techniques, Chan et al., 2022]
Shortcomings of HILO?
68
70. A cooperative system where designers can steer the
design when needed
70
[Cooperative Multi-Objective Bayesian Design Optimization, Mo et al., submitted to TiiS 2023]
71. A cooperative system where designers can steer the
design when needed
71
[Cooperative Multi-Objective Bayesian Design Optimization, Mo et al., submitted to TiiS 2023]
Results:
The Cooperative BO reaches
comparable user performances
with less full evaluations (more
e
ffi
ciently).
The designers felt more engaging
in the process than fully relying on
BO.
72. Boundaries of current HILO
• HILO only addresses single-objective problems
• HILO reduces the designer’s agency and ownership in the process
• HILO only optimizes for one user/task at a time
• HILO only works for prototype-free interactions (software interfaces)
• HILO is still not efficient enough!
72
73. Boundaries of current HILO
• HILO only addresses single-objective problems
• HILO reduces the designer’s agency and ownership in the process
• HILO only optimizes for one user/task at a time
• HILO only works for prototype-free interactions (software interfaces)
• HILO is still not efficient enough!
73
74. From a single user to a population of users
74
[Practical Approaches to Group-Level Multi-Objective Bayesian Optimization in Interaction Technique Design, Liao et al., submitted 2023]
Aggregate
Sample over
design space
Global Pareto-optimal
performances
y2
y1
Mean prediction of y1
Mean prediction of y2
x
y
P1
y
x
x
y
x
y
Global GP
y
x
P2
P3 P4
75. Boundaries of current HILO
• HILO only addresses single-objective problems
• HILO reduces the designer’s agency and ownership in the process
• HILO only optimizes for one user/task at a time
• HILO only works for prototype-free interactions (software interfaces)
• HILO is still not efficient enough!
75
76. From prototyping to physical emulation
76
[Button Simulation and Design via FDVV Models, Liao et al., 2020]
77. From prototyping to physical emulation
77
[Button Simulation and Design via FDVV Models, Liao et al., 2020]
78. Boundaries of current HILO
• HILO only addresses single-objective problems
• HILO reduces the designer’s agency and ownership in the process
• HILO only optimizes for one user/task at a time
• HILO only works for prototype-free interactions (software interfaces)
• HILO is still not efficient enough!
78
79. Time-efficiency issue
• Bayesian optimization is mainly seen as a design tool. That is, it is
used to derive the optimal design parameter prior to deploying the
interface/interaction on end-users.
79
80. Time-efficiency issue
• Bayesian optimization is mainly seen as a design tool. That is, it is
used to derive the optimal design parameter prior to deploying the
interface/interaction on end-users.
• Why?
• It takes ~ 10 x “NO of design parameters” iterations to converge.
This takes 40 - 60 mins
80
81. But why? Isn’t BO the best optimization tool?
• Yes! It is the upper bound of optimization methods!
• But it suffers from the cold-start problem
81
83. Can we boost the efficiency of BO?
Yes! By augmenting BO with prior information!
83
84. 84
[Practical Approaches to Group-Level Multi-Objective Bayesian Optimization in Interaction Technique Design, Liao et al., submitted 2023]
Extract
representative
points
x
y
All observations across users
Initialize
BO
Mean prediction of y1
Mean prediction of y2
Observed y1
Observed y2
x
y
Warm-Start GP BO with a new user
x
y
New samples
Warm-start Bayesian optimization
88. 88
Current HILO
Single-
objective Lack agency
Cooperative
AI
Only for
1 user/task
Population
model
Aggregate
Sam
desig
Mean prediction of y1
Mean prediction of y2
x
y
P1
y
x
x
y
x
y
Global GP
y
x
P2
P3 P4
Pareto frontier
learning
90. 90
Current HILO
Single-
objective
Pareto frontier
learning
Lack agency
Cooperative
AI
Only for
1 user/task
Population
model
Only GUI
Physical
emulation
Low e
ffi
ciency
Constructing
prior
Extract
representative
points
x
y
All observations across users
Initialize
BO
Mean prediction of y1
Mean prediction of y2
Observed y1
Observed y2
x
y
Warm-Start GP BO with a new user
x
y
New samples
91. My advisor -
Prof. Antti Oulasvirta
Prof. Sunjun Kim
Dr. Kashyap Todi Dr. John Dudley Prof. Per Ola Kristensson Prof. Byungjoo Lee
Prof. Liwei Chan Dr. Aditya Acharya Antti Keurulainen Prof. Andrew Howes
Dr. Aakar Gupta Dr. Ruta Desai
George B. Mo
Dr. Tanya Jonker Dr. Hrvoje Benko Alec Pierce
Aini Putkonen Dr. Hee-Seung Moon Aleksi Ikkala
Amazing people who
made my Ph.D. studies
a wonderful journey!
91
92. 92
How to get Yi-Chi
http://yichiliao.com
Twitter: @yichiliao
Breaking the Boundaries of Human-in-the-Loop Optimization