This document describes a system called SelPh that aims to support manual photo color enhancement. It proposes a "self-reinforcing" workflow where a user's manual edits are used to progressively train a preference model, which then helps support further manual enhancements. The system visualizes enhancement goodness, provides interactive optimization, allows auto-enhancements, indicates enhancement confidence levels, and references similar photos. A prototype was created and evaluated in a user study with photographers to understand how the system could help with enhancing many photos.

1. SelPh
Progressive Learning and Support of
Manual Photo Color Enhancement
Yuki Koyama, Daisuke Sakamoto, Takeo Igarashi

3. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Parameter
tweaking
- Based on
the designer’s
aesthetic
preference
3
Photo Color Enhancement
[Adobe Photoshop]

4. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Tedious
4

5. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 5
Enhancing Many Photos

6. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■Could be tedious
■Need manual enhancement of each photo
one by one, independently
6
Enhancing Many Photos

7. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Tedious
7Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
Solution?: Auto-Enhancement

8. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 8
Auto-Enhancement is NOT Perfect
Manual edit
(ground truth)
Original Auto
[Adobe Photoshop]

9. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 9
Auto-Enhancement is NOT Perfect
Different
Manual edit
(ground truth)
Original Auto
[Adobe Photoshop]

10. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 10
Auto-Enhancement is NOT Perfect
Different
Manual edit
(ground truth)
Original Auto
[Adobe Photoshop]

11. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 11
Auto-Enhancement is NOT Perfect
One-by-one manual tweaking is necessary
Manual edit
(ground truth)
Original Auto
[Adobe Photoshop]

12. Our Approach

13. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 13
Our Approach
Achieving
automatic enhancement
Supporting
manual enhancement

14. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 14
Our Approach
Achieving
automatic enhancement
Supporting
manual enhancement
✗ ✓

15. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 15
Our Approach
Achieving
automatic enhancement
Supporting
manual enhancement
New workflow concept:
Self-reinforcing color enhancement
✗ ✓

16. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 16
Enhancement with a “Self-Reinforcing” System

17. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 17
Enhancement with a “Self-Reinforcing” System
Target
photos
User’s
edits

18. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 18
Enhancement with a “Self-Reinforcing” System
Target
photos
User’s
edits

19. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 19
Enhancement with a “Self-Reinforcing” System
Target
photos
User’s
edits

20. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 20
Enhancement with a “Self-Reinforcing” System
Target
photos
User’s
edits
…

21. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 21
Enhancement with a “Self-Reinforcing” System
Learn
Target
photos
User’s
edits
Preference
model
…

22. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 22
Enhancement with a “Self-Reinforcing” System
Learn
Support
Target
photos
User’s
edits
Preference
model
…

23. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 23
Enhancement with a “Self-Reinforcing” System
Learn
Support
Update
Learn
Target
photos
User’s
edits
Preference
model
…

24. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 24
Enhancement with a “Self-Reinforcing” System
Learn
Support
Update
Learn
Support
Target
photos
User’s
edits
Preference
model
…

25. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 25
Enhancement with a “Self-Reinforcing” System
Learn
Support
Update
Learn
Support
Update
Learn
Target
photos
User’s
edits
Preference
model
…

26. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 26
Enhancement with a “Self-Reinforcing” System
Learn
Support
Update
Learn
Support
Update
Learn
Support
Target
photos
User’s
edits
Preference
model
…

27. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 27
Enhancement with a “Self-Reinforcing” System
Learn
Support
Update
Learn
Support
Update
Learn
Support
Update
Learn
Target
photos
User’s
edits
Preference
model
…

28. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 28
Enhancement with a “Self-Reinforcing” System
…
…
Learn
Support
Update
Learn
Support
Update
Learn
Support
Update
Learn
Support
Update
Target
photos
User’s
edits
Preference
model
…

29. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Implicit
- The user can work as usual
- No explicit training phase
■ Progressive
- The more photos are
enhanced, the more useful
the system becomes
29
“Self-Reinforcing” System

30. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Implicit
- The user can work as usual
- No explicit training phase
■ Progressive
- The more photos are
enhanced, the more useful
the system becomes
30
“Self-Reinforcing” System

31. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Implicit
- The user can work as usual
- No explicit training phase
■ Progressive
- The more photos are
enhanced, the more useful
the system becomes
31
“Self-Reinforcing” System

32. Related Work

33. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 33
UIs for Design Exploration
Design Galleries
[Marks+, SIGGRAPH97]
■ Design Galleries
- Sampling-based selection
■ Our approach
- Direct manipulation of each
parameter using slider

34. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 34
UIs for Design Exploration
Design Galleries
[Marks+, SIGGRAPH97]
■ Design Galleries
- Sampling-based selection
■ Our approach
- Direct manipulation of each
parameter using slider

35. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 35
UIs for Design Exploration
Design Galleries
[Marks+, SIGGRAPH97]
■ Design Galleries
- Sampling-based selection
■ Our approach
- Direct manipulation of each
parameter using slider

36. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 36
UIs for Design Exploration
■ VisOpt Slider
[Koyama+, UIST14]
- Visualization
- Optimization

37. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 37
UIs for Design Exploration
■ VisOpt Slider
[Koyama+, UIST14]
- Visualization
- Optimization

38. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 38
UIs for Design Exploration
VisOpt Slider
[Koyama+, UIST14]
Adopted into our prototype system

39. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 39
UIs for Design Exploration
VisOpt Slider
[Koyama+, UIST14]
■ [Koyama+, UIST14]
- Training Data: Crowdsourcing
- Interaction: Pre-computing
■ This work
- Training Data: Editing history
- Interaction: Progressive / Implicit
Adopted into our prototype system

40. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 40
UIs for Design Exploration
VisOpt Slider
[Koyama+, UIST14]
■ [Koyama+, UIST14]
- Training Data: Crowdsourcing
- Interaction: Pre-computing
■ This work
- Training Data: Editing history
- Interaction: Progressive / Implicit
Adopted into our prototype system

41. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 41
UIs for Design Exploration
VisOpt Slider
[Koyama+, UIST14]
■ [Koyama+, UIST14]
- Training Data: Crowdsourcing
- Interaction: Pre-computing
■ This work
- Training Data: Editing history
- Interaction: Progressive / Implicit
Adopted into our prototype system

42. A Prototype System

43. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
SelPh
A self-reinforcing system
for photo enhancement
43
A Prototype System

44. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
1. Goodness Visualization
2. Interactive Optimization
3. Auto-Enhancement
4. Variable Confidence
5. Reference Photos
44
User Support Functions

46. 1. Visualization

47. 2. Optimization

48. 3. Auto-Enhancement

49. 4. Variable Confidence

50. 5. Reference Photos

51. Self-Reinforcement Algorithm

52. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 52
Interactive enhancement

53. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 53
Interactive enhancement
Push the
“Next” button

54. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 54
Interactive enhancement
Push the
“Next” button
Update the distance metric
d( , ) = ?

55. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 55
Interactive enhancement
Push the
“Next” button
Update the preference model
Update
Update the distance metric
d( , ) = ?

56. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 56
Interactive enhancement
Push the
“Next” button
Show the
next photo
Update the preference model
Update
Update the distance metric
d( , ) = ?

57. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 57
Interactive enhancement
Push the
“Next” button
Show the
next photo
Update the preference model
Update
Update the distance metric
d( , ) = ?

58. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 58
Interactive enhancement
Push the
“Next” button
Show the
next photo
Update the preference model
Update
Update the distance metric
d( , ) = ?

59. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 59
Distance Metric Learning
d( , ) = ?

60. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ We adopt techniques by [Kapoor+, IJCV14]
- Learning a personalized distance metric from editing history
60
Distance Metric Learning
d( , ) = ?
d( , ) d( , )is as equivalent to as possibleS.t.

62. User Study

63. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Qualitative evaluation of our approach / system
- How photographers enhance photos
- How they are satisfied
63
Purpose

64. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Participants: 8 photographers
■ Pre-task: Take 100 photos
■ Main-tasks:
- Enhance the first 50 photos using { Baseline
- Enhance the other 50 photos using { SelPh | Baseline }
■ Post-task: Interviews and questionnaires
64
Study Method

65. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Participants: 8 photographers
■ Pre-task: Take 100 photos
■ Main-tasks:
- Enhance the first 50 photos using { Baseline
- Enhance the other 50 photos using { SelPh | Baseline }
■ Post-task: Interviews and questionnaires
65
Study Method

66. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Participants: 8 photographers
■ Pre-task: Take 100 photos
■ Main-tasks:
- Enhance the first 50 photos using { Baseline
- Enhance the other 50 photos using { SelPh | Baseline }
■ Post-task: Interviews and questionnaires
66
Study Method

67. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Participants: 8 photographers
■ Pre-task: Take 100 photos
■ Main-tasks:
- Enhance the first 50 photos using { Baseline | SelPh }
- Enhance the other 50 photos using { SelPh | Baseline }
■ Post-task: Interviews and questionnaires
67
Study Method

68. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Participants: 8 photographers
■ Pre-task: Take 100 photos
■ Main-tasks:
- Enhance the first 50 photos using { Baseline | SelPh }
- Enhance the other 50 photos using { SelPh | Baseline }
■ Post-task: Interviews and questionnaires
68
Study Method
Prepared by limiting the
user support functions
of SelPh

69. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ Participants: 8 photographers
■ Pre-task: Take 100 photos
■ Main-tasks:
- Enhance the first 50 photos using { Baseline | SelPh }
- Enhance the other 50 photos using { SelPh | Baseline }
■ Post-task: Interviews and questionnaires
69
Study Method
Prepared by limiting the
user support functions
of SelPh

70. Results

71. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 71
Post-Task Questionnaires
Q1#
Q2#
Q3#
Q4#
Q5#
Q6#
Q1#
Q2#
Q3#
Q4#
Q5#
Q6#
Q7#
Q8#
Q2#
Q3#
Q4#
Q5#
Q6#
Q7#
Q8#
Q9#
Q1
Visualization of goodness on sliders was useful
compared to the absence of it.
Q2
Interactive optimization of slider values was useful
compared to the absence of it.
Q5
Auto-enhancement in SelPh was more useful than
that in commercial software.
Q7
Reference photos in SelPh were more useful than
those in Baseline.
Q8 Confidence value was useful.
Strongly
disagree
Strongly
agree
About user support functions — Positive

72. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 72
Post-Task QuestionnairesQ6#
Q7#
Q8#
Q9#
Q10#
Q11#
Q9
As the task proceeds, I felt that the system learns my
preference or intent.
Q10 The learning result reflected my preference or intent.
Q11
It is preferable for the system to learn my preference
or intent.
Strongly
disagree
Strongly
agree
About overall approach — Positive

73. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ About Overall Experience
- “The functions [in SelPh] […] evoke the feeling of collaborating
with another me.”
- Using the baseline system, “[I felt] lonely.” In contrast, “there
is interaction with the [self-reinforcing] system,” thus
“executing the task [with SelPh] was fun.”
73
Comments from Interviews (1/2)

74. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ About Confidence Value
- “[According to the confidence value, I] decided to use or not
to use the optimization and the auto-enhancement”
- “I could trust the system more [by knowing the confidence]”
- “It was an enjoyable experience to do the task [while knowing
the confidence]”
- “[I felt] humanity from the confidence”
74
Comments from Interviews (2/2)

75. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ About Confidence Value
- “[According to the confidence value, I] decided to use or not
to use the optimization and the auto-enhancement”
- “I could trust the system more [by knowing the confidence]”
- “It was an enjoyable experience to do the task [while knowing
the confidence]”
- “[I felt] humanity from the confidence”
75
Comments from Interviews (2/2)

76. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ About Confidence Value
- “[According to the confidence value, I] decided to use or not
to use the optimization and the auto-enhancement”
- “I could trust the system more [by knowing the confidence]”
- “It was an enjoyable experience to do the task [while knowing
the confidence]”
- “[I felt] humanity from the confidence”
76
Comments from Interviews (2/2)

77. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ About Confidence Value
- “[According to the confidence value, I] decided to use or not
to use the optimization and the auto-enhancement”
- “I could trust the system more [by knowing the confidence]”
- “It was an enjoyable experience to do the task [while knowing
the confidence]”
- “[I felt] humanity from the confidence”
77
Comments from Interviews (2/2)

78. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 78
Quantitative Results
Marginally faster [p < .10] Significantly smaller [p < .05]

79. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 79
Quantitative Results
Marginally faster [p < .10] Significantly smaller [p < .05]

80. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 80
Quantitative Results
Marginally faster [p < .10] Significantly smaller [p < .05]

81. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ All the participants were satisfied with self-reinforcing
color enhancement
■ The inclusion of the confidence value makes SelPh
more trustworthy and enjoyable to use
81
Summary & Lessons Learned

82. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ All the participants were satisfied with self-reinforcing
color enhancement
■ The inclusion of the confidence value makes SelPh
more trustworthy and enjoyable to use
82
Summary & Lessons Learned

83. Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016
■ All the participants were satisfied with self-reinforcing
color enhancement
■ The inclusion of the confidence value makes SelPh
more trustworthy and enjoyable to use
83
Summary & Lessons Learned

84. Conclusion

85. ■ Self-reinforcing system for manual photo enhancement
- Progressively and implicitly learning the user’s preference
■ A prototype system: SelPh
- Five user support functions (e.g., variable confidence)
■ Qualitative user study
- Photographers prefer the proposed workflow

86. ■ Self-reinforcing system for manual photo enhancement
- Progressively and implicitly learning the user’s preference
■ A prototype system: SelPh
- Five user support functions (e.g., variable confidence)
■ Qualitative user study
- Photographers prefer the proposed workflow

87. ■ Self-reinforcing system for manual photo enhancement
- Progressively and implicitly learning the user’s preference
■ A prototype system: SelPh
- Five user support functions (e.g., variable confidence)
■ Qualitative user study
- Photographers prefer the proposed workflow

88. ■ Self-reinforcing system for manual photo enhancement
- Progressively and implicitly learning the user’s preference
■ A prototype system: SelPh
- Five user support functions (e.g., variable confidence)
■ Qualitative user study
- Photographers prefer the proposed workflow

89. Paper / Videos / Software / Source Codes
are available at
http://koyama.xyz/project/SelPh/

90. SelPh
Progressive Learning and Support of
Manual Photo Color Enhancement
Yuki Koyama, Daisuke Sakamoto, Takeo Igarashi