This document describes a system called SelPh that aims to support manual photo color enhancement. It proposes a "self-reinforcing" workflow where a user's manual edits are used to progressively train a preference model, which then helps support further manual enhancements. The system visualizes enhancement goodness, provides interactive optimization, allows auto-enhancements, indicates enhancement confidence levels, and references similar photos. A prototype was created and evaluated in a user study with photographers to understand how the system could help with enhancing many photos.
3. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Parameter
tweaking
- Based on
the designer’s
aesthetic
preference
3
Photo Color Enhancement
[Adobe Photoshop]
4. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Tedious
4
5. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 5
Enhancing Many Photos
6. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■Could be tedious
■Need manual enhancement of each photo
one by one, independently
6
Enhancing Many Photos
7. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Tedious
7Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
Solution?: Auto-Enhancement
8. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 8
Auto-Enhancement is NOT Perfect
Manual edit
(ground truth)
Original Auto
[Adobe Photoshop]
9. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 9
Auto-Enhancement is NOT Perfect
Different
Manual edit
(ground truth)
Original Auto
[Adobe Photoshop]
10. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 10
Auto-Enhancement is NOT Perfect
Different
Manual edit
(ground truth)
Original Auto
[Adobe Photoshop]
11. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 11
Auto-Enhancement is NOT Perfect
One-by-one manual tweaking is necessary
Manual edit
(ground truth)
Original Auto
[Adobe Photoshop]
13. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 13
Our Approach
Achieving
automatic enhancement
Supporting
manual enhancement
14. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 14
Our Approach
Achieving
automatic enhancement
Supporting
manual enhancement
✗ ✓
15. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 15
Our Approach
Achieving
automatic enhancement
Supporting
manual enhancement
New workflow concept:
Self-reinforcing color enhancement
✗ ✓
16. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 16
Enhancement with a “Self-Reinforcing” System
17. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 17
Enhancement with a “Self-Reinforcing” System
Target
photos
User’s
edits
18. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 18
Enhancement with a “Self-Reinforcing” System
Target
photos
User’s
edits
19. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 19
Enhancement with a “Self-Reinforcing” System
Target
photos
User’s
edits
20. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 20
Enhancement with a “Self-Reinforcing” System
Target
photos
User’s
edits
…
21. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 21
Enhancement with a “Self-Reinforcing” System
Learn
Target
photos
User’s
edits
Preference
model
…
22. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 22
Enhancement with a “Self-Reinforcing” System
Learn
Support
Target
photos
User’s
edits
Preference
model
…
23. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 23
Enhancement with a “Self-Reinforcing” System
Learn
Support
Update
Learn
Target
photos
User’s
edits
Preference
model
…
24. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 24
Enhancement with a “Self-Reinforcing” System
Learn
Support
Update
Learn
Support
Target
photos
User’s
edits
Preference
model
…
25. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 25
Enhancement with a “Self-Reinforcing” System
Learn
Support
Update
Learn
Support
Update
Learn
Target
photos
User’s
edits
Preference
model
…
26. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 26
Enhancement with a “Self-Reinforcing” System
Learn
Support
Update
Learn
Support
Update
Learn
Support
Target
photos
User’s
edits
Preference
model
…
27. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 27
Enhancement with a “Self-Reinforcing” System
Learn
Support
Update
Learn
Support
Update
Learn
Support
Update
Learn
Target
photos
User’s
edits
Preference
model
…
28. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 28
Enhancement with a “Self-Reinforcing” System
…
…
Learn
Support
Update
Learn
Support
Update
Learn
Support
Update
Learn
Support
Update
Target
photos
User’s
edits
Preference
model
…
29. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Implicit
- The user can work as usual
- No explicit training phase
■ Progressive
- The more photos are
enhanced, the more useful
the system becomes
29
“Self-Reinforcing” System
30. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Implicit
- The user can work as usual
- No explicit training phase
■ Progressive
- The more photos are
enhanced, the more useful
the system becomes
30
“Self-Reinforcing” System
31. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Implicit
- The user can work as usual
- No explicit training phase
■ Progressive
- The more photos are
enhanced, the more useful
the system becomes
31
“Self-Reinforcing” System
33. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 33
UIs for Design Exploration
Design Galleries
[Marks+, SIGGRAPH97]
■ Design Galleries
- Sampling-based selection
■ Our approach
- Direct manipulation of each
parameter using slider
34. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 34
UIs for Design Exploration
Design Galleries
[Marks+, SIGGRAPH97]
■ Design Galleries
- Sampling-based selection
■ Our approach
- Direct manipulation of each
parameter using slider
35. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 35
UIs for Design Exploration
Design Galleries
[Marks+, SIGGRAPH97]
■ Design Galleries
- Sampling-based selection
■ Our approach
- Direct manipulation of each
parameter using slider
36. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 36
UIs for Design Exploration
■ VisOpt Slider
[Koyama+, UIST14]
- Visualization
- Optimization
37. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 37
UIs for Design Exploration
■ VisOpt Slider
[Koyama+, UIST14]
- Visualization
- Optimization
38. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 38
UIs for Design Exploration
VisOpt Slider
[Koyama+, UIST14]
Adopted into our prototype system
39. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 39
UIs for Design Exploration
VisOpt Slider
[Koyama+, UIST14]
■ [Koyama+, UIST14]
- Training Data: Crowdsourcing
- Interaction: Pre-computing
■ This work
- Training Data: Editing history
- Interaction: Progressive / Implicit
Adopted into our prototype system
40. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 40
UIs for Design Exploration
VisOpt Slider
[Koyama+, UIST14]
■ [Koyama+, UIST14]
- Training Data: Crowdsourcing
- Interaction: Pre-computing
■ This work
- Training Data: Editing history
- Interaction: Progressive / Implicit
Adopted into our prototype system
41. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 41
UIs for Design Exploration
VisOpt Slider
[Koyama+, UIST14]
■ [Koyama+, UIST14]
- Training Data: Crowdsourcing
- Interaction: Pre-computing
■ This work
- Training Data: Editing history
- Interaction: Progressive / Implicit
Adopted into our prototype system
43. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
SelPh
A self-reinforcing system
for photo enhancement
43
A Prototype System
44. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
1. Goodness Visualization
2. Interactive Optimization
3. Auto-Enhancement
4. Variable Confidence
5. Reference Photos
44
User Support Functions
52. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 52
Interactive enhancement
53. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 53
Interactive enhancement
Push the
“Next” button
54. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 54
Interactive enhancement
Push the
“Next” button
Update the distance metric
d( , ) = ?
55. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 55
Interactive enhancement
Push the
“Next” button
Update the preference model
Update
Update the distance metric
d( , ) = ?
56. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 56
Interactive enhancement
Push the
“Next” button
Show the
next photo
Update the preference model
Update
Update the distance metric
d( , ) = ?
57. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 57
Interactive enhancement
Push the
“Next” button
Show the
next photo
Update the preference model
Update
Update the distance metric
d( , ) = ?
58. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 58
Interactive enhancement
Push the
“Next” button
Show the
next photo
Update the preference model
Update
Update the distance metric
d( , ) = ?
59. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 59
Distance Metric Learning
d( , ) = ?
60. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ We adopt techniques by [Kapoor+, IJCV14]
- Learning a personalized distance metric from editing history
60
Distance Metric Learning
d( , ) = ?
d( , ) d( , )is as equivalent to as possibleS.t.
63. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Qualitative evaluation of our approach / system
- How photographers enhance photos
- How they are satisfied
63
Purpose
64. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Participants: 8 photographers
■ Pre-task: Take 100 photos
■ Main-tasks:
- Enhance the first 50 photos using { Baseline
- Enhance the other 50 photos using { SelPh | Baseline }
■ Post-task: Interviews and questionnaires
64
Study Method
65. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Participants: 8 photographers
■ Pre-task: Take 100 photos
■ Main-tasks:
- Enhance the first 50 photos using { Baseline
- Enhance the other 50 photos using { SelPh | Baseline }
■ Post-task: Interviews and questionnaires
65
Study Method
66. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Participants: 8 photographers
■ Pre-task: Take 100 photos
■ Main-tasks:
- Enhance the first 50 photos using { Baseline
- Enhance the other 50 photos using { SelPh | Baseline }
■ Post-task: Interviews and questionnaires
66
Study Method
67. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Participants: 8 photographers
■ Pre-task: Take 100 photos
■ Main-tasks:
- Enhance the first 50 photos using { Baseline | SelPh }
- Enhance the other 50 photos using { SelPh | Baseline }
■ Post-task: Interviews and questionnaires
67
Study Method
68. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Participants: 8 photographers
■ Pre-task: Take 100 photos
■ Main-tasks:
- Enhance the first 50 photos using { Baseline | SelPh }
- Enhance the other 50 photos using { SelPh | Baseline }
■ Post-task: Interviews and questionnaires
68
Study Method
Prepared by limiting the
user support functions
of SelPh
69. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Participants: 8 photographers
■ Pre-task: Take 100 photos
■ Main-tasks:
- Enhance the first 50 photos using { Baseline | SelPh }
- Enhance the other 50 photos using { SelPh | Baseline }
■ Post-task: Interviews and questionnaires
69
Study Method
Prepared by limiting the
user support functions
of SelPh
71. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 71
Post-Task Questionnaires
Q1#
Q2#
Q3#
Q4#
Q5#
Q6#
Q1#
Q2#
Q3#
Q4#
Q5#
Q6#
Q7#
Q8#
Q2#
Q3#
Q4#
Q5#
Q6#
Q7#
Q8#
Q9#
Q1
Visualization of goodness on sliders was useful
compared to the absence of it.
Q2
Interactive optimization of slider values was useful
compared to the absence of it.
Q5
Auto-enhancement in SelPh was more useful than
that in commercial software.
Q7
Reference photos in SelPh were more useful than
those in Baseline.
Q8 Confidence value was useful.
Strongly
disagree
Strongly
agree
About user support functions — Positive
72. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 72
Post-Task QuestionnairesQ6#
Q7#
Q8#
Q9#
Q10#
Q11#
Q9
As the task proceeds, I felt that the system learns my
preference or intent.
Q10 The learning result reflected my preference or intent.
Q11
It is preferable for the system to learn my preference
or intent.
Strongly
disagree
Strongly
agree
About overall approach — Positive
73. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ About Overall Experience
- “The functions [in SelPh] […] evoke the feeling of collaborating
with another me.”
- Using the baseline system, “[I felt] lonely.” In contrast, “there
is interaction with the [self-reinforcing] system,” thus
“executing the task [with SelPh] was fun.”
73
Comments from Interviews (1/2)
74. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ About Confidence Value
- “[According to the confidence value, I] decided to use or not
to use the optimization and the auto-enhancement”
- “I could trust the system more [by knowing the confidence]”
- “It was an enjoyable experience to do the task [while knowing
the confidence]”
- “[I felt] humanity from the confidence”
74
Comments from Interviews (2/2)
75. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ About Confidence Value
- “[According to the confidence value, I] decided to use or not
to use the optimization and the auto-enhancement”
- “I could trust the system more [by knowing the confidence]”
- “It was an enjoyable experience to do the task [while knowing
the confidence]”
- “[I felt] humanity from the confidence”
75
Comments from Interviews (2/2)
76. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ About Confidence Value
- “[According to the confidence value, I] decided to use or not
to use the optimization and the auto-enhancement”
- “I could trust the system more [by knowing the confidence]”
- “It was an enjoyable experience to do the task [while knowing
the confidence]”
- “[I felt] humanity from the confidence”
76
Comments from Interviews (2/2)
77. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ About Confidence Value
- “[According to the confidence value, I] decided to use or not
to use the optimization and the auto-enhancement”
- “I could trust the system more [by knowing the confidence]”
- “It was an enjoyable experience to do the task [while knowing
the confidence]”
- “[I felt] humanity from the confidence”
77
Comments from Interviews (2/2)
78. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 78
Quantitative Results
Marginally faster [p < .10] Significantly smaller [p < .05]
79. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 79
Quantitative Results
Marginally faster [p < .10] Significantly smaller [p < .05]
80. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 80
Quantitative Results
Marginally faster [p < .10] Significantly smaller [p < .05]
81. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ All the participants were satisfied with self-reinforcing
color enhancement
■ The inclusion of the confidence value makes SelPh
more trustworthy and enjoyable to use
81
Summary & Lessons Learned
82. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ All the participants were satisfied with self-reinforcing
color enhancement
■ The inclusion of the confidence value makes SelPh
more trustworthy and enjoyable to use
82
Summary & Lessons Learned
83. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ All the participants were satisfied with self-reinforcing
color enhancement
■ The inclusion of the confidence value makes SelPh
more trustworthy and enjoyable to use
83
Summary & Lessons Learned
85. ■ Self-reinforcing system for manual photo enhancement
- Progressively and implicitly learning the user’s preference
■ A prototype system: SelPh
- Five user support functions (e.g., variable confidence)
■ Qualitative user study
- Photographers prefer the proposed workflow
86. ■ Self-reinforcing system for manual photo enhancement
- Progressively and implicitly learning the user’s preference
■ A prototype system: SelPh
- Five user support functions (e.g., variable confidence)
■ Qualitative user study
- Photographers prefer the proposed workflow
87. ■ Self-reinforcing system for manual photo enhancement
- Progressively and implicitly learning the user’s preference
■ A prototype system: SelPh
- Five user support functions (e.g., variable confidence)
■ Qualitative user study
- Photographers prefer the proposed workflow
88. ■ Self-reinforcing system for manual photo enhancement
- Progressively and implicitly learning the user’s preference
■ A prototype system: SelPh
- Five user support functions (e.g., variable confidence)
■ Qualitative user study
- Photographers prefer the proposed workflow
89. Paper / Videos / Software / Source Codes
are available at
http://koyama.xyz/project/SelPh/