Bayesian optimization can be used to efficiently solve optimization problems defined by human preferences. The researchers propose using BO as an assistant that learns a designer's preferences from their manipulations of sliders without requiring explicit feedback. They extract relative preference data from the slider interactions to use preferential Bayesian optimization. This allows the designer freedom while still benefiting from the intelligent sampling of BO. They demonstrate this approach for photo color enhancement and procedural material design.

1. BO as Assistant: Using Bayesian Optimization

5. De
fi
ned by
human
preference

6. "Which do you like?"
"I like this one."
✔︎
🤖
System
🤔
Designer
solves optimization problems with
perceptual objective functions
(e.g., preference)
Human-in-the-loop
optimization
Ask
Evaluate

7. "Which do you like?"
"I like this one."
✔︎
💪
BO
🤔
Designer
Ask
Evaluate
• Intelligent sampling strategy
• "Exploration and exploitation"
• Able to
fi
nd a good solution within
a small number of iterations
(sample e
ffi
cient)
• Less burden on the evaluator
Bayesian optimization (BO)

8. Previous Work: Human-in-the-Loop Bayesian Optimization
8
Melody Generation
Zhou+
IUI 2020
Photographic Lighting
Yamamoto+
UIST 2022
→ Successfully used in previous work to enable e
ffi
cient search
Koyama+
SIGGRAPH 2017
Koyama+
SIGGRAPH 2020
Photo Enhancement & 3D Computer Graphics

9. Problem & Motivation

10. "Which do you like?"
"I like this one."
✔︎
🤖
Bayesian
Optimization
(BO)
😢
Designer
I want to explore
freely!
Problem: No freedom for designer,
reducing agency and creativity
fi
t from BO’s
intelligence while providing the
designer with freedom?

11. Our Framework: "BO as Assistant"

13. Learning the designer’s preference

14. • There is no explicit loop
• BO requires no explicit inputs
• BO just "assists" the designer

20. Technical Background:
Preferential Bayesian Optimization (PBO)

21. Bayesian Optimization (BO)
• A global “black-box” optimization algorithm

22. ✔︎
🤖
PBO
🤔
Designer
Ask
Evaluate
• A variant of BO [Brochu+, NIPS 2007]
• Runs with relative preference data
(rather than absolute scoring)
• Better suited for human-in-the-loop
Q: Can we extract relative preference
data implicitly from slider manipulation?

23. Our Technique:
Extraction of Preferential Data for PBO

24. A Closer Look: Slider Manipulation Session
24
0.0 1.0
Design
goal

25. A Closer Look: Slider Manipulation Session
25
The photo looks too dark.
Let’s manipulate "brightness".
0.0 1.0
Design
goal
Brightness

26. A Closer Look: Slider Manipulation Session
26
Let’s increase the value.
0.0 1.0
Design
goal
Press

27. A Closer Look: Slider Manipulation Session
27
Move
0.0 1.0
Design
goal
Still dark. Increase more.

28. A Closer Look: Slider Manipulation Session
28
Move
0.0 1.0
Design
goal
Oh! Too much!!
Let’s decrease the value.

29. A Closer Look: Slider Manipulation Session
29
Move & Release
0.0 1.0
Design
goal
Now the photo looks nice.
Let’s stop.

30. A Closer Look: Slider Manipulation Session
Observations:

31. Estimated preference
Predicted standard deviation
Estimation uncertainty
Acquisition function
Priority in sampling
→ Please refer to the paper for details

32. 32
: Data points extracted from slider manipulation
: Suggestions generated by BO
Predicted mean
Estimated preference
Predicted standard deviation
Estimation uncertainty
Acquisition function
Priority in sampling
→ Please refer to the paper for details

33. 33
: Data points extracted from slider manipulation
: Suggestions generated by BO
Predicted mean
Estimated preference
Predicted standard deviation
Estimation uncertainty
Acquisition function
Priority in sampling
→ Please refer to the paper for details

34. 34
: Data points extracted from slider manipulation
: Suggestions generated by BO
Predicted mean
Estimated preference
Predicted standard deviation
Estimation uncertainty
Acquisition function
Priority in sampling
→ Please refer to the paper for details

35. Demonstrations

36. Photo Color Enhancement
• #Parameters: 12
• Brightness
• Contrast
• Saturation
• Lift (RGB)
• Gamma (RGB)
• Gain (RGB)

37. x5

38. x5

39. Procedural Material Design
• #Parameters: 8
• Threshold
• Noise Scale
• Noise Detail
• Noise Roughness
• Noise Distortion
• Ambient Occlusion
• Bump Strength
• Peel Boundary Strength
• Design goal: Rusted metal with peeled painting