[CHI 2016] SelPh: Progressive Learning and Support of Manual Photo Color Enhancement

Yuki Koyama
Yuki KoyamaPh.D. Student at The University of Tokyo
SelPh
Progressive Learning and Support of
Manual Photo Color Enhancement
Yuki Koyama, Daisuke Sakamoto, Takeo Igarashi
[CHI 2016] SelPh: Progressive Learning and Support of Manual Photo Color Enhancement
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Parameter 

tweaking
- Based on 

the designer’s

aesthetic

preference
3
Photo Color Enhancement
[Adobe Photoshop]
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Tedious
4
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 5
Enhancing Many Photos
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■Could be tedious
■Need manual enhancement of each photo 

one by one, independently
6
Enhancing Many Photos
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Tedious
7Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
Solution?: Auto-Enhancement
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 8
Auto-Enhancement is NOT Perfect
Manual edit
(ground truth)
Original Auto
[Adobe Photoshop]
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 9
Auto-Enhancement is NOT Perfect
Different
Manual edit
(ground truth)
Original Auto
[Adobe Photoshop]
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 10
Auto-Enhancement is NOT Perfect
Different
Manual edit
(ground truth)
Original Auto
[Adobe Photoshop]
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 11
Auto-Enhancement is NOT Perfect
One-by-one manual tweaking is necessary
Manual edit
(ground truth)
Original Auto
[Adobe Photoshop]
Our Approach
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 13
Our Approach
Achieving 

automatic enhancement
Supporting 

manual enhancement
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 14
Our Approach
Achieving 

automatic enhancement
Supporting 

manual enhancement
✗ ✓
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 15
Our Approach
Achieving 

automatic enhancement
Supporting 

manual enhancement
New workflow concept:
Self-reinforcing color enhancement
✗ ✓
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 16
Enhancement with a “Self-Reinforcing” System
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 17
Enhancement with a “Self-Reinforcing” System
Target
photos
User’s
edits
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 18
Enhancement with a “Self-Reinforcing” System
Target
photos
User’s
edits
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 19
Enhancement with a “Self-Reinforcing” System
Target
photos
User’s
edits
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 20
Enhancement with a “Self-Reinforcing” System
Target
photos
User’s
edits
…
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 21
Enhancement with a “Self-Reinforcing” System
Learn
Target
photos
User’s
edits
Preference
model
…
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 22
Enhancement with a “Self-Reinforcing” System
Learn
Support
Target
photos
User’s
edits
Preference
model
…
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 23
Enhancement with a “Self-Reinforcing” System
Learn
Support
Update
Learn
Target
photos
User’s
edits
Preference
model
…
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 24
Enhancement with a “Self-Reinforcing” System
Learn
Support
Update
Learn
Support
Target
photos
User’s
edits
Preference
model
…
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 25
Enhancement with a “Self-Reinforcing” System
Learn
Support
Update
Learn
Support
Update
Learn
Target
photos
User’s
edits
Preference
model
…
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 26
Enhancement with a “Self-Reinforcing” System
Learn
Support
Update
Learn
Support
Update
Learn
Support
Target
photos
User’s
edits
Preference
model
…
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 27
Enhancement with a “Self-Reinforcing” System
Learn
Support
Update
Learn
Support
Update
Learn
Support
Update
Learn
Target
photos
User’s
edits
Preference
model
…
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 28
Enhancement with a “Self-Reinforcing” System
…
…
Learn
Support
Update
Learn
Support
Update
Learn
Support
Update
Learn
Support
Update
Target
photos
User’s
edits
Preference
model
…
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Implicit
- The user can work as usual
- No explicit training phase
■ Progressive
- The more photos are
enhanced, the more useful
the system becomes
29
“Self-Reinforcing” System
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Implicit
- The user can work as usual
- No explicit training phase
■ Progressive
- The more photos are
enhanced, the more useful
the system becomes
30
“Self-Reinforcing” System
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Implicit
- The user can work as usual
- No explicit training phase
■ Progressive
- The more photos are
enhanced, the more useful
the system becomes
31
“Self-Reinforcing” System
Related Work
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 33
UIs for Design Exploration
Design Galleries
[Marks+, SIGGRAPH97]
■ Design Galleries
- Sampling-based selection
■ Our approach
- Direct manipulation of each
parameter using slider
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 34
UIs for Design Exploration
Design Galleries
[Marks+, SIGGRAPH97]
■ Design Galleries
- Sampling-based selection
■ Our approach
- Direct manipulation of each
parameter using slider
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 35
UIs for Design Exploration
Design Galleries
[Marks+, SIGGRAPH97]
■ Design Galleries
- Sampling-based selection
■ Our approach
- Direct manipulation of each
parameter using slider
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 36
UIs for Design Exploration
■ VisOpt Slider

[Koyama+, UIST14]
- Visualization
- Optimization
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 37
UIs for Design Exploration
■ VisOpt Slider

[Koyama+, UIST14]
- Visualization
- Optimization
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 38
UIs for Design Exploration
VisOpt Slider
[Koyama+, UIST14]
Adopted into our prototype system
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 39
UIs for Design Exploration
VisOpt Slider
[Koyama+, UIST14]
■ [Koyama+, UIST14]
- Training Data: Crowdsourcing
- Interaction: Pre-computing
■ This work
- Training Data: Editing history
- Interaction: Progressive / Implicit
Adopted into our prototype system
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 40
UIs for Design Exploration
VisOpt Slider
[Koyama+, UIST14]
■ [Koyama+, UIST14]
- Training Data: Crowdsourcing
- Interaction: Pre-computing
■ This work
- Training Data: Editing history
- Interaction: Progressive / Implicit
Adopted into our prototype system
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 41
UIs for Design Exploration
VisOpt Slider
[Koyama+, UIST14]
■ [Koyama+, UIST14]
- Training Data: Crowdsourcing
- Interaction: Pre-computing
■ This work
- Training Data: Editing history
- Interaction: Progressive / Implicit
Adopted into our prototype system
A Prototype System
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
SelPh
A self-reinforcing system
for photo enhancement
43
A Prototype System
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
1. Goodness Visualization
2. Interactive Optimization
3. Auto-Enhancement
4. Variable Confidence
5. Reference Photos
44
User Support Functions
[CHI 2016] SelPh: Progressive Learning and Support of Manual Photo Color Enhancement
1. Visualization
2. Optimization
3. Auto-Enhancement
4. Variable Confidence
5. Reference Photos
Self-Reinforcement Algorithm
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 52
Interactive enhancement
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 53
Interactive enhancement
Push the
“Next” button
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 54
Interactive enhancement
Push the
“Next” button
Update the distance metric
d( , ) = ?
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 55
Interactive enhancement
Push the
“Next” button
Update the preference model
Update
Update the distance metric
d( , ) = ?
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 56
Interactive enhancement
Push the
“Next” button
Show the
next photo
Update the preference model
Update
Update the distance metric
d( , ) = ?
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 57
Interactive enhancement
Push the
“Next” button
Show the
next photo
Update the preference model
Update
Update the distance metric
d( , ) = ?
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 58
Interactive enhancement
Push the
“Next” button
Show the
next photo
Update the preference model
Update
Update the distance metric
d( , ) = ?
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 59
Distance Metric Learning
d( , ) = ?
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ We adopt techniques by [Kapoor+, IJCV14]
- Learning a personalized distance metric from editing history
60
Distance Metric Learning
d( , ) = ?
d( , ) d( , )is as equivalent to as possibleS.t.
[CHI 2016] SelPh: Progressive Learning and Support of Manual Photo Color Enhancement
User Study
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Qualitative evaluation of our approach / system
- How photographers enhance photos
- How they are satisfied
63
Purpose
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Participants: 8 photographers
■ Pre-task: Take 100 photos
■ Main-tasks:
- Enhance the first 50 photos using { Baseline
- Enhance the other 50 photos using { SelPh | Baseline }
■ Post-task: Interviews and questionnaires
64
Study Method
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Participants: 8 photographers
■ Pre-task: Take 100 photos
■ Main-tasks:
- Enhance the first 50 photos using { Baseline
- Enhance the other 50 photos using { SelPh | Baseline }
■ Post-task: Interviews and questionnaires
65
Study Method
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Participants: 8 photographers
■ Pre-task: Take 100 photos
■ Main-tasks:
- Enhance the first 50 photos using { Baseline
- Enhance the other 50 photos using { SelPh | Baseline }
■ Post-task: Interviews and questionnaires
66
Study Method
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Participants: 8 photographers
■ Pre-task: Take 100 photos
■ Main-tasks:
- Enhance the first 50 photos using { Baseline | SelPh }
- Enhance the other 50 photos using { SelPh | Baseline }
■ Post-task: Interviews and questionnaires
67
Study Method
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Participants: 8 photographers
■ Pre-task: Take 100 photos
■ Main-tasks:
- Enhance the first 50 photos using { Baseline | SelPh }
- Enhance the other 50 photos using { SelPh | Baseline }
■ Post-task: Interviews and questionnaires
68
Study Method
Prepared by limiting the
user support functions
of SelPh
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Participants: 8 photographers
■ Pre-task: Take 100 photos
■ Main-tasks:
- Enhance the first 50 photos using { Baseline | SelPh }
- Enhance the other 50 photos using { SelPh | Baseline }
■ Post-task: Interviews and questionnaires
69
Study Method
Prepared by limiting the
user support functions
of SelPh
Results
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 71
Post-Task Questionnaires
Q1#
Q2#
Q3#
Q4#
Q5#
Q6#
Q1#
Q2#
Q3#
Q4#
Q5#
Q6#
Q7#
Q8#
Q2#
Q3#
Q4#
Q5#
Q6#
Q7#
Q8#
Q9#
Q1
Visualization of goodness on sliders was useful
compared to the absence of it.
Q2
Interactive optimization of slider values was useful
compared to the absence of it.
Q5
Auto-enhancement in SelPh was more useful than
that in commercial software.
Q7
Reference photos in SelPh were more useful than
those in Baseline.
Q8 Confidence value was useful.
Strongly
disagree
Strongly
agree
About user support functions — Positive
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 72
Post-Task QuestionnairesQ6#
Q7#
Q8#
Q9#
Q10#
Q11#
Q9
As the task proceeds, I felt that the system learns my
preference or intent.
Q10 The learning result reflected my preference or intent.
Q11
It is preferable for the system to learn my preference
or intent.
Strongly
disagree
Strongly
agree
About overall approach — Positive
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ About Overall Experience
- “The functions [in SelPh] […] evoke the feeling of collaborating
with another me.”
- Using the baseline system, “[I felt] lonely.” In contrast, “there
is interaction with the [self-reinforcing] system,” thus
“executing the task [with SelPh] was fun.”
73
Comments from Interviews (1/2)
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ About Confidence Value
- “[According to the confidence value, I] decided to use or not
to use the optimization and the auto-enhancement”
- “I could trust the system more [by knowing the confidence]”
- “It was an enjoyable experience to do the task [while knowing
the confidence]”
- “[I felt] humanity from the confidence”
74
Comments from Interviews (2/2)
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ About Confidence Value
- “[According to the confidence value, I] decided to use or not
to use the optimization and the auto-enhancement”
- “I could trust the system more [by knowing the confidence]”
- “It was an enjoyable experience to do the task [while knowing
the confidence]”
- “[I felt] humanity from the confidence”
75
Comments from Interviews (2/2)
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ About Confidence Value
- “[According to the confidence value, I] decided to use or not
to use the optimization and the auto-enhancement”
- “I could trust the system more [by knowing the confidence]”
- “It was an enjoyable experience to do the task [while knowing
the confidence]”
- “[I felt] humanity from the confidence”
76
Comments from Interviews (2/2)
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ About Confidence Value
- “[According to the confidence value, I] decided to use or not
to use the optimization and the auto-enhancement”
- “I could trust the system more [by knowing the confidence]”
- “It was an enjoyable experience to do the task [while knowing
the confidence]”
- “[I felt] humanity from the confidence”
77
Comments from Interviews (2/2)
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 78
Quantitative Results
Marginally faster [p < .10] Significantly smaller [p < .05]
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 79
Quantitative Results
Marginally faster [p < .10] Significantly smaller [p < .05]
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 80
Quantitative Results
Marginally faster [p < .10] Significantly smaller [p < .05]
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ All the participants were satisfied with self-reinforcing
color enhancement
■ The inclusion of the confidence value makes SelPh
more trustworthy and enjoyable to use
81
Summary & Lessons Learned
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ All the participants were satisfied with self-reinforcing
color enhancement
■ The inclusion of the confidence value makes SelPh
more trustworthy and enjoyable to use
82
Summary & Lessons Learned
Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ All the participants were satisfied with self-reinforcing
color enhancement
■ The inclusion of the confidence value makes SelPh
more trustworthy and enjoyable to use
83
Summary & Lessons Learned
Conclusion
■ Self-reinforcing system for manual photo enhancement
- Progressively and implicitly learning the user’s preference
■ A prototype system: SelPh
- Five user support functions (e.g., variable confidence)
■ Qualitative user study
- Photographers prefer the proposed workflow
■ Self-reinforcing system for manual photo enhancement
- Progressively and implicitly learning the user’s preference
■ A prototype system: SelPh
- Five user support functions (e.g., variable confidence)
■ Qualitative user study
- Photographers prefer the proposed workflow
■ Self-reinforcing system for manual photo enhancement
- Progressively and implicitly learning the user’s preference
■ A prototype system: SelPh
- Five user support functions (e.g., variable confidence)
■ Qualitative user study
- Photographers prefer the proposed workflow
■ Self-reinforcing system for manual photo enhancement
- Progressively and implicitly learning the user’s preference
■ A prototype system: SelPh
- Five user support functions (e.g., variable confidence)
■ Qualitative user study
- Photographers prefer the proposed workflow
Paper / Videos / Software / Source Codes
are available at
http://koyama.xyz/project/SelPh/
SelPh
Progressive Learning and Support of
Manual Photo Color Enhancement
Yuki Koyama, Daisuke Sakamoto, Takeo Igarashi
1 of 90

Recommended

Crowd-Powered Parameter Analysis for Visual Design Exploration (UIST 2014) by
Crowd-Powered Parameter Analysis for Visual Design Exploration (UIST 2014)Crowd-Powered Parameter Analysis for Visual Design Exploration (UIST 2014)
Crowd-Powered Parameter Analysis for Visual Design Exploration (UIST 2014)Yuki Koyama
5K views77 slides
[VRST 2017] Computational Design of Hand-Held VR Controllers Using Haptic Sha... by
[VRST 2017] Computational Design of Hand-Held VR Controllers Using Haptic Sha...[VRST 2017] Computational Design of Hand-Held VR Controllers Using Haptic Sha...
[VRST 2017] Computational Design of Hand-Held VR Controllers Using Haptic Sha...Eisuke Fujinawa
2.8K views51 slides
[SIGGRAPH 2017] Sequential Line Search for Efficient Visual Design Optimizati... by
[SIGGRAPH 2017] Sequential Line Search for Efficient Visual Design Optimizati...[SIGGRAPH 2017] Sequential Line Search for Efficient Visual Design Optimizati...
[SIGGRAPH 2017] Sequential Line Search for Efficient Visual Design Optimizati...Yuki Koyama
15.7K views91 slides
Introduction by
IntroductionIntroduction
IntroductionInstitute for Community Design
239 views47 slides
Embracing Digital Creation in the AP Classroom by
Embracing Digital Creation in the AP ClassroomEmbracing Digital Creation in the AP Classroom
Embracing Digital Creation in the AP ClassroomNicoleGraham48
52 views37 slides
Hcc presentation deck for corporates by
Hcc presentation deck for corporatesHcc presentation deck for corporates
Hcc presentation deck for corporatesHappyCameraClub
762 views34 slides

More Related Content

More from Yuki Koyama

[UIST 2022] BO as Assistant: Using Bayesian Optimization for Asynchronously G... by
[UIST 2022] BO as Assistant: Using Bayesian Optimization for Asynchronously G...[UIST 2022] BO as Assistant: Using Bayesian Optimization for Asynchronously G...
[UIST 2022] BO as Assistant: Using Bayesian Optimization for Asynchronously G...Yuki Koyama
90 views44 slides
[SIGGRAPH 2020] Sequential Gallery for Interactive Visual Design Optimization by
[SIGGRAPH 2020] Sequential Gallery for Interactive Visual Design Optimization[SIGGRAPH 2020] Sequential Gallery for Interactive Visual Design Optimization
[SIGGRAPH 2020] Sequential Gallery for Interactive Visual Design OptimizationYuki Koyama
371 views90 slides
Precomputed Optimal One-Hop Motion Transition for Responsive Character Animat... by
Precomputed Optimal One-Hop Motion Transition for Responsive Character Animat...Precomputed Optimal One-Hop Motion Transition for Responsive Character Animat...
Precomputed Optimal One-Hop Motion Transition for Responsive Character Animat...Yuki Koyama
467 views67 slides
Decomposing Images into Layers with Advanced Color Blending [Pacific Graphics... by
Decomposing Images into Layers with Advanced Color Blending [Pacific Graphics...Decomposing Images into Layers with Advanced Color Blending [Pacific Graphics...
Decomposing Images into Layers with Advanced Color Blending [Pacific Graphics...Yuki Koyama
606 views70 slides
[CHI 2018] OptiMo: Optimization-Guided Motion Editing for Keyframe Character ... by
[CHI 2018] OptiMo: Optimization-Guided Motion Editing for Keyframe Character ...[CHI 2018] OptiMo: Optimization-Guided Motion Editing for Keyframe Character ...
[CHI 2018] OptiMo: Optimization-Guided Motion Editing for Keyframe Character ...Yuki Koyama
326 views78 slides
[IPSJ-ONE 2017] コンピュテーションがつなぐ人とデザイン by
[IPSJ-ONE 2017] コンピュテーションがつなぐ人とデザイン[IPSJ-ONE 2017] コンピュテーションがつなぐ人とデザイン
[IPSJ-ONE 2017] コンピュテーションがつなぐ人とデザインYuki Koyama
316 views52 slides

More from Yuki Koyama(11)

[UIST 2022] BO as Assistant: Using Bayesian Optimization for Asynchronously G... by Yuki Koyama
[UIST 2022] BO as Assistant: Using Bayesian Optimization for Asynchronously G...[UIST 2022] BO as Assistant: Using Bayesian Optimization for Asynchronously G...
[UIST 2022] BO as Assistant: Using Bayesian Optimization for Asynchronously G...
Yuki Koyama90 views
[SIGGRAPH 2020] Sequential Gallery for Interactive Visual Design Optimization by Yuki Koyama
[SIGGRAPH 2020] Sequential Gallery for Interactive Visual Design Optimization[SIGGRAPH 2020] Sequential Gallery for Interactive Visual Design Optimization
[SIGGRAPH 2020] Sequential Gallery for Interactive Visual Design Optimization
Yuki Koyama371 views
Precomputed Optimal One-Hop Motion Transition for Responsive Character Animat... by Yuki Koyama
Precomputed Optimal One-Hop Motion Transition for Responsive Character Animat...Precomputed Optimal One-Hop Motion Transition for Responsive Character Animat...
Precomputed Optimal One-Hop Motion Transition for Responsive Character Animat...
Yuki Koyama467 views
Decomposing Images into Layers with Advanced Color Blending [Pacific Graphics... by Yuki Koyama
Decomposing Images into Layers with Advanced Color Blending [Pacific Graphics...Decomposing Images into Layers with Advanced Color Blending [Pacific Graphics...
Decomposing Images into Layers with Advanced Color Blending [Pacific Graphics...
Yuki Koyama606 views
[CHI 2018] OptiMo: Optimization-Guided Motion Editing for Keyframe Character ... by Yuki Koyama
[CHI 2018] OptiMo: Optimization-Guided Motion Editing for Keyframe Character ...[CHI 2018] OptiMo: Optimization-Guided Motion Editing for Keyframe Character ...
[CHI 2018] OptiMo: Optimization-Guided Motion Editing for Keyframe Character ...
Yuki Koyama326 views
[IPSJ-ONE 2017] コンピュテーションがつなぐ人とデザイン by Yuki Koyama
[IPSJ-ONE 2017] コンピュテーションがつなぐ人とデザイン[IPSJ-ONE 2017] コンピュテーションがつなぐ人とデザイン
[IPSJ-ONE 2017] コンピュテーションがつなぐ人とデザイン
Yuki Koyama316 views
[CEDEC2016] 大規模学習を用いたCGの最先端研究の紹介 - 前半 by Yuki Koyama
[CEDEC2016] 大規模学習を用いたCGの最先端研究の紹介 - 前半[CEDEC2016] 大規模学習を用いたCGの最先端研究の紹介 - 前半
[CEDEC2016] 大規模学習を用いたCGの最先端研究の紹介 - 前半
Yuki Koyama1.9K views
視覚デザイン探索のためのクラウドソーシングを活用したパラメタ空間解析(WISS 2014) by Yuki Koyama
視覚デザイン探索のためのクラウドソーシングを活用したパラメタ空間解析(WISS 2014)視覚デザイン探索のためのクラウドソーシングを活用したパラメタ空間解析(WISS 2014)
視覚デザイン探索のためのクラウドソーシングを活用したパラメタ空間解析(WISS 2014)
Yuki Koyama3.5K views
こだわり物理エンジン (Innovative Technologies 2013 @ DC EXPO) by Yuki Koyama
こだわり物理エンジン (Innovative Technologies 2013 @ DC EXPO)こだわり物理エンジン (Innovative Technologies 2013 @ DC EXPO)
こだわり物理エンジン (Innovative Technologies 2013 @ DC EXPO)
Yuki Koyama934 views
View-Dependent Control of Elastic Rod Simulation for 3D Character Animation (... by Yuki Koyama
View-Dependent Control of Elastic Rod Simulation for 3D Character Animation (...View-Dependent Control of Elastic Rod Simulation for 3D Character Animation (...
View-Dependent Control of Elastic Rod Simulation for 3D Character Animation (...
Yuki Koyama9.3K views
Real-Time Example-Based Elastic Deformation (SCA '12) by Yuki Koyama
Real-Time Example-Based Elastic Deformation (SCA '12)Real-Time Example-Based Elastic Deformation (SCA '12)
Real-Time Example-Based Elastic Deformation (SCA '12)
Yuki Koyama744 views

Recently uploaded

NTGapps NTG LowCode Platform by
NTGapps NTG LowCode Platform NTGapps NTG LowCode Platform
NTGapps NTG LowCode Platform Mustafa Kuğu
474 views30 slides
Evaluation of Quality of Experience of ABR Schemes in Gaming Stream by
Evaluation of Quality of Experience of ABR Schemes in Gaming StreamEvaluation of Quality of Experience of ABR Schemes in Gaming Stream
Evaluation of Quality of Experience of ABR Schemes in Gaming StreamAlpen-Adria-Universität
44 views34 slides
Generative AI: Shifting the AI Landscape by
Generative AI: Shifting the AI LandscapeGenerative AI: Shifting the AI Landscape
Generative AI: Shifting the AI LandscapeDeakin University
78 views55 slides
Cencora Executive Symposium by
Cencora Executive SymposiumCencora Executive Symposium
Cencora Executive Symposiummarketingcommunicati21
174 views14 slides
This talk was not generated with ChatGPT: how AI is changing science by
This talk was not generated with ChatGPT: how AI is changing scienceThis talk was not generated with ChatGPT: how AI is changing science
This talk was not generated with ChatGPT: how AI is changing scienceElena Simperl
34 views13 slides
"Surviving highload with Node.js", Andrii Shumada by
"Surviving highload with Node.js", Andrii Shumada "Surviving highload with Node.js", Andrii Shumada
"Surviving highload with Node.js", Andrii Shumada Fwdays
59 views29 slides

Recently uploaded(20)

NTGapps NTG LowCode Platform by Mustafa Kuğu
NTGapps NTG LowCode Platform NTGapps NTG LowCode Platform
NTGapps NTG LowCode Platform
Mustafa Kuğu474 views
This talk was not generated with ChatGPT: how AI is changing science by Elena Simperl
This talk was not generated with ChatGPT: how AI is changing scienceThis talk was not generated with ChatGPT: how AI is changing science
This talk was not generated with ChatGPT: how AI is changing science
Elena Simperl34 views
"Surviving highload with Node.js", Andrii Shumada by Fwdays
"Surviving highload with Node.js", Andrii Shumada "Surviving highload with Node.js", Andrii Shumada
"Surviving highload with Node.js", Andrii Shumada
Fwdays59 views
Don’t Make A Human Do A Robot’s Job! : 6 Reasons Why AI Will Save Us & Not De... by Moses Kemibaro
Don’t Make A Human Do A Robot’s Job! : 6 Reasons Why AI Will Save Us & Not De...Don’t Make A Human Do A Robot’s Job! : 6 Reasons Why AI Will Save Us & Not De...
Don’t Make A Human Do A Robot’s Job! : 6 Reasons Why AI Will Save Us & Not De...
Moses Kemibaro38 views
Business Analyst Series 2023 - Week 4 Session 8 by DianaGray10
Business Analyst Series 2023 -  Week 4 Session 8Business Analyst Series 2023 -  Week 4 Session 8
Business Analyst Series 2023 - Week 4 Session 8
DianaGray10180 views
Optimizing Communication to Optimize Human Behavior - LCBM by Yaman Kumar
Optimizing Communication to Optimize Human Behavior - LCBMOptimizing Communication to Optimize Human Behavior - LCBM
Optimizing Communication to Optimize Human Behavior - LCBM
Yaman Kumar39 views
Innovation & Entrepreneurship strategies in Dairy Industry by PervaizDar1
Innovation & Entrepreneurship strategies in Dairy IndustryInnovation & Entrepreneurship strategies in Dairy Industry
Innovation & Entrepreneurship strategies in Dairy Industry
PervaizDar139 views
Digital Personal Data Protection (DPDP) Practical Approach For CISOs by Priyanka Aash
Digital Personal Data Protection (DPDP) Practical Approach For CISOsDigital Personal Data Protection (DPDP) Practical Approach For CISOs
Digital Personal Data Protection (DPDP) Practical Approach For CISOs
Priyanka Aash171 views
The Power of Heat Decarbonisation Plans in the Built Environment by IES VE
The Power of Heat Decarbonisation Plans in the Built EnvironmentThe Power of Heat Decarbonisation Plans in the Built Environment
The Power of Heat Decarbonisation Plans in the Built Environment
IES VE85 views
Cocktail of Environments. How to Mix Test and Development Environments and St... by Aleksandr Tarasov
Cocktail of Environments. How to Mix Test and Development Environments and St...Cocktail of Environments. How to Mix Test and Development Environments and St...
Cocktail of Environments. How to Mix Test and Development Environments and St...
The Role of Patterns in the Era of Large Language Models by Yunyao Li
The Role of Patterns in the Era of Large Language ModelsThe Role of Patterns in the Era of Large Language Models
The Role of Patterns in the Era of Large Language Models
Yunyao Li104 views
Enabling DPU Hardware Accelerators in XCP-ng Cloud Platform Environment - And... by ShapeBlue
Enabling DPU Hardware Accelerators in XCP-ng Cloud Platform Environment - And...Enabling DPU Hardware Accelerators in XCP-ng Cloud Platform Environment - And...
Enabling DPU Hardware Accelerators in XCP-ng Cloud Platform Environment - And...
ShapeBlue120 views
Bronack Skills - Risk Management and SRE v1.0 12-3-2023.pdf by ThomasBronack
Bronack Skills - Risk Management and SRE v1.0 12-3-2023.pdfBronack Skills - Risk Management and SRE v1.0 12-3-2023.pdf
Bronack Skills - Risk Management and SRE v1.0 12-3-2023.pdf
ThomasBronack31 views
The Power of Generative AI in Accelerating No Code Adoption.pdf by Saeed Al Dhaheri
The Power of Generative AI in Accelerating No Code Adoption.pdfThe Power of Generative AI in Accelerating No Code Adoption.pdf
The Power of Generative AI in Accelerating No Code Adoption.pdf
Saeed Al Dhaheri44 views

[CHI 2016] SelPh: Progressive Learning and Support of Manual Photo Color Enhancement

  • 1. SelPh Progressive Learning and Support of Manual Photo Color Enhancement Yuki Koyama, Daisuke Sakamoto, Takeo Igarashi
  • 3. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 ■ Parameter 
 tweaking - Based on 
 the designer’s
 aesthetic
 preference 3 Photo Color Enhancement [Adobe Photoshop]
  • 4. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 ■ Tedious 4
  • 5. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 5 Enhancing Many Photos
  • 6. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 ■Could be tedious ■Need manual enhancement of each photo 
 one by one, independently 6 Enhancing Many Photos
  • 7. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 ■ Tedious 7Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 Solution?: Auto-Enhancement
  • 8. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 8 Auto-Enhancement is NOT Perfect Manual edit (ground truth) Original Auto [Adobe Photoshop]
  • 9. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 9 Auto-Enhancement is NOT Perfect Different Manual edit (ground truth) Original Auto [Adobe Photoshop]
  • 10. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 10 Auto-Enhancement is NOT Perfect Different Manual edit (ground truth) Original Auto [Adobe Photoshop]
  • 11. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 11 Auto-Enhancement is NOT Perfect One-by-one manual tweaking is necessary Manual edit (ground truth) Original Auto [Adobe Photoshop]
  • 13. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 13 Our Approach Achieving 
 automatic enhancement Supporting 
 manual enhancement
  • 14. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 14 Our Approach Achieving 
 automatic enhancement Supporting 
 manual enhancement ✗ ✓
  • 15. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 15 Our Approach Achieving 
 automatic enhancement Supporting 
 manual enhancement New workflow concept: Self-reinforcing color enhancement ✗ ✓
  • 16. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 16 Enhancement with a “Self-Reinforcing” System
  • 17. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 17 Enhancement with a “Self-Reinforcing” System Target photos User’s edits
  • 18. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 18 Enhancement with a “Self-Reinforcing” System Target photos User’s edits
  • 19. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 19 Enhancement with a “Self-Reinforcing” System Target photos User’s edits
  • 20. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 20 Enhancement with a “Self-Reinforcing” System Target photos User’s edits …
  • 21. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 21 Enhancement with a “Self-Reinforcing” System Learn Target photos User’s edits Preference model …
  • 22. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 22 Enhancement with a “Self-Reinforcing” System Learn Support Target photos User’s edits Preference model …
  • 23. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 23 Enhancement with a “Self-Reinforcing” System Learn Support Update Learn Target photos User’s edits Preference model …
  • 24. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 24 Enhancement with a “Self-Reinforcing” System Learn Support Update Learn Support Target photos User’s edits Preference model …
  • 25. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 25 Enhancement with a “Self-Reinforcing” System Learn Support Update Learn Support Update Learn Target photos User’s edits Preference model …
  • 26. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 26 Enhancement with a “Self-Reinforcing” System Learn Support Update Learn Support Update Learn Support Target photos User’s edits Preference model …
  • 27. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 27 Enhancement with a “Self-Reinforcing” System Learn Support Update Learn Support Update Learn Support Update Learn Target photos User’s edits Preference model …
  • 28. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 28 Enhancement with a “Self-Reinforcing” System … … Learn Support Update Learn Support Update Learn Support Update Learn Support Update Target photos User’s edits Preference model …
  • 29. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 ■ Implicit - The user can work as usual - No explicit training phase ■ Progressive - The more photos are enhanced, the more useful the system becomes 29 “Self-Reinforcing” System
  • 30. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 ■ Implicit - The user can work as usual - No explicit training phase ■ Progressive - The more photos are enhanced, the more useful the system becomes 30 “Self-Reinforcing” System
  • 31. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 ■ Implicit - The user can work as usual - No explicit training phase ■ Progressive - The more photos are enhanced, the more useful the system becomes 31 “Self-Reinforcing” System
  • 33. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 33 UIs for Design Exploration Design Galleries [Marks+, SIGGRAPH97] ■ Design Galleries - Sampling-based selection ■ Our approach - Direct manipulation of each parameter using slider
  • 34. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 34 UIs for Design Exploration Design Galleries [Marks+, SIGGRAPH97] ■ Design Galleries - Sampling-based selection ■ Our approach - Direct manipulation of each parameter using slider
  • 35. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 35 UIs for Design Exploration Design Galleries [Marks+, SIGGRAPH97] ■ Design Galleries - Sampling-based selection ■ Our approach - Direct manipulation of each parameter using slider
  • 36. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 36 UIs for Design Exploration ■ VisOpt Slider
 [Koyama+, UIST14] - Visualization - Optimization
  • 37. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 37 UIs for Design Exploration ■ VisOpt Slider
 [Koyama+, UIST14] - Visualization - Optimization
  • 38. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 38 UIs for Design Exploration VisOpt Slider [Koyama+, UIST14] Adopted into our prototype system
  • 39. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 39 UIs for Design Exploration VisOpt Slider [Koyama+, UIST14] ■ [Koyama+, UIST14] - Training Data: Crowdsourcing - Interaction: Pre-computing ■ This work - Training Data: Editing history - Interaction: Progressive / Implicit Adopted into our prototype system
  • 40. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 40 UIs for Design Exploration VisOpt Slider [Koyama+, UIST14] ■ [Koyama+, UIST14] - Training Data: Crowdsourcing - Interaction: Pre-computing ■ This work - Training Data: Editing history - Interaction: Progressive / Implicit Adopted into our prototype system
  • 41. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 41 UIs for Design Exploration VisOpt Slider [Koyama+, UIST14] ■ [Koyama+, UIST14] - Training Data: Crowdsourcing - Interaction: Pre-computing ■ This work - Training Data: Editing history - Interaction: Progressive / Implicit Adopted into our prototype system
  • 43. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 SelPh A self-reinforcing system for photo enhancement 43 A Prototype System
  • 44. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 1. Goodness Visualization 2. Interactive Optimization 3. Auto-Enhancement 4. Variable Confidence 5. Reference Photos 44 User Support Functions
  • 52. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 52 Interactive enhancement
  • 53. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 53 Interactive enhancement Push the “Next” button
  • 54. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 54 Interactive enhancement Push the “Next” button Update the distance metric d( , ) = ?
  • 55. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 55 Interactive enhancement Push the “Next” button Update the preference model Update Update the distance metric d( , ) = ?
  • 56. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 56 Interactive enhancement Push the “Next” button Show the next photo Update the preference model Update Update the distance metric d( , ) = ?
  • 57. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 57 Interactive enhancement Push the “Next” button Show the next photo Update the preference model Update Update the distance metric d( , ) = ?
  • 58. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 58 Interactive enhancement Push the “Next” button Show the next photo Update the preference model Update Update the distance metric d( , ) = ?
  • 59. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 59 Distance Metric Learning d( , ) = ?
  • 60. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 ■ We adopt techniques by [Kapoor+, IJCV14] - Learning a personalized distance metric from editing history 60 Distance Metric Learning d( , ) = ? d( , ) d( , )is as equivalent to as possibleS.t.
  • 63. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 ■ Qualitative evaluation of our approach / system - How photographers enhance photos - How they are satisfied 63 Purpose
  • 64. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 ■ Participants: 8 photographers ■ Pre-task: Take 100 photos ■ Main-tasks: - Enhance the first 50 photos using { Baseline - Enhance the other 50 photos using { SelPh | Baseline } ■ Post-task: Interviews and questionnaires 64 Study Method
  • 65. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 ■ Participants: 8 photographers ■ Pre-task: Take 100 photos ■ Main-tasks: - Enhance the first 50 photos using { Baseline - Enhance the other 50 photos using { SelPh | Baseline } ■ Post-task: Interviews and questionnaires 65 Study Method
  • 66. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 ■ Participants: 8 photographers ■ Pre-task: Take 100 photos ■ Main-tasks: - Enhance the first 50 photos using { Baseline - Enhance the other 50 photos using { SelPh | Baseline } ■ Post-task: Interviews and questionnaires 66 Study Method
  • 67. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 ■ Participants: 8 photographers ■ Pre-task: Take 100 photos ■ Main-tasks: - Enhance the first 50 photos using { Baseline | SelPh } - Enhance the other 50 photos using { SelPh | Baseline } ■ Post-task: Interviews and questionnaires 67 Study Method
  • 68. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 ■ Participants: 8 photographers ■ Pre-task: Take 100 photos ■ Main-tasks: - Enhance the first 50 photos using { Baseline | SelPh } - Enhance the other 50 photos using { SelPh | Baseline } ■ Post-task: Interviews and questionnaires 68 Study Method Prepared by limiting the user support functions of SelPh
  • 69. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 ■ Participants: 8 photographers ■ Pre-task: Take 100 photos ■ Main-tasks: - Enhance the first 50 photos using { Baseline | SelPh } - Enhance the other 50 photos using { SelPh | Baseline } ■ Post-task: Interviews and questionnaires 69 Study Method Prepared by limiting the user support functions of SelPh
  • 71. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 71 Post-Task Questionnaires Q1# Q2# Q3# Q4# Q5# Q6# Q1# Q2# Q3# Q4# Q5# Q6# Q7# Q8# Q2# Q3# Q4# Q5# Q6# Q7# Q8# Q9# Q1 Visualization of goodness on sliders was useful compared to the absence of it. Q2 Interactive optimization of slider values was useful compared to the absence of it. Q5 Auto-enhancement in SelPh was more useful than that in commercial software. Q7 Reference photos in SelPh were more useful than those in Baseline. Q8 Confidence value was useful. Strongly disagree Strongly agree About user support functions — Positive
  • 72. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 72 Post-Task QuestionnairesQ6# Q7# Q8# Q9# Q10# Q11# Q9 As the task proceeds, I felt that the system learns my preference or intent. Q10 The learning result reflected my preference or intent. Q11 It is preferable for the system to learn my preference or intent. Strongly disagree Strongly agree About overall approach — Positive
  • 73. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 ■ About Overall Experience - “The functions [in SelPh] […] evoke the feeling of collaborating with another me.” - Using the baseline system, “[I felt] lonely.” In contrast, “there is interaction with the [self-reinforcing] system,” thus “executing the task [with SelPh] was fun.” 73 Comments from Interviews (1/2)
  • 74. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 ■ About Confidence Value - “[According to the confidence value, I] decided to use or not to use the optimization and the auto-enhancement” - “I could trust the system more [by knowing the confidence]” - “It was an enjoyable experience to do the task [while knowing the confidence]” - “[I felt] humanity from the confidence” 74 Comments from Interviews (2/2)
  • 75. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 ■ About Confidence Value - “[According to the confidence value, I] decided to use or not to use the optimization and the auto-enhancement” - “I could trust the system more [by knowing the confidence]” - “It was an enjoyable experience to do the task [while knowing the confidence]” - “[I felt] humanity from the confidence” 75 Comments from Interviews (2/2)
  • 76. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 ■ About Confidence Value - “[According to the confidence value, I] decided to use or not to use the optimization and the auto-enhancement” - “I could trust the system more [by knowing the confidence]” - “It was an enjoyable experience to do the task [while knowing the confidence]” - “[I felt] humanity from the confidence” 76 Comments from Interviews (2/2)
  • 77. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 ■ About Confidence Value - “[According to the confidence value, I] decided to use or not to use the optimization and the auto-enhancement” - “I could trust the system more [by knowing the confidence]” - “It was an enjoyable experience to do the task [while knowing the confidence]” - “[I felt] humanity from the confidence” 77 Comments from Interviews (2/2)
  • 78. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 78 Quantitative Results Marginally faster [p < .10] Significantly smaller [p < .05]
  • 79. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 79 Quantitative Results Marginally faster [p < .10] Significantly smaller [p < .05]
  • 80. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 80 Quantitative Results Marginally faster [p < .10] Significantly smaller [p < .05]
  • 81. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 ■ All the participants were satisfied with self-reinforcing color enhancement ■ The inclusion of the confidence value makes SelPh more trustworthy and enjoyable to use 81 Summary & Lessons Learned
  • 82. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 ■ All the participants were satisfied with self-reinforcing color enhancement ■ The inclusion of the confidence value makes SelPh more trustworthy and enjoyable to use 82 Summary & Lessons Learned
  • 83. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 ■ All the participants were satisfied with self-reinforcing color enhancement ■ The inclusion of the confidence value makes SelPh more trustworthy and enjoyable to use 83 Summary & Lessons Learned
  • 85. ■ Self-reinforcing system for manual photo enhancement - Progressively and implicitly learning the user’s preference ■ A prototype system: SelPh - Five user support functions (e.g., variable confidence) ■ Qualitative user study - Photographers prefer the proposed workflow
  • 86. ■ Self-reinforcing system for manual photo enhancement - Progressively and implicitly learning the user’s preference ■ A prototype system: SelPh - Five user support functions (e.g., variable confidence) ■ Qualitative user study - Photographers prefer the proposed workflow
  • 87. ■ Self-reinforcing system for manual photo enhancement - Progressively and implicitly learning the user’s preference ■ A prototype system: SelPh - Five user support functions (e.g., variable confidence) ■ Qualitative user study - Photographers prefer the proposed workflow
  • 88. ■ Self-reinforcing system for manual photo enhancement - Progressively and implicitly learning the user’s preference ■ A prototype system: SelPh - Five user support functions (e.g., variable confidence) ■ Qualitative user study - Photographers prefer the proposed workflow
  • 89. Paper / Videos / Software / Source Codes are available at http://koyama.xyz/project/SelPh/
  • 90. SelPh Progressive Learning and Support of Manual Photo Color Enhancement Yuki Koyama, Daisuke Sakamoto, Takeo Igarashi