Experiments in Artist Assistant AI discusses research on developing AI tools to assist artists. The main areas of research are a recolorization assistant tool that can recolor images based on a reference image using deep learning models trained on histogram analogies. The second area is an assessment assistant tool that uses a GAN to assess the color quality of images by the degree of improvement achieved by the GAN's color enhancement network. Further research directions are proposed to make tools faster, able to generate more detailed images, and help make images more aesthetically pleasing by analyzing which parts of images are most critical and what visual elements impact perceived beauty.

1. Experiments in
Artist Assistant AI
Lee Gunhee (이건희)
Computer Graphics Laboratory
POSTECH

2. Contents
1. What is Artist Assistant AI?
2. Main Research 1: Recolorize Assistant Tool
3. Main Research 2: Assessment Assistant Tool
4. Further Research
2

3. What is Artist Assistant AI(AAA)?
• AAA helps artist to realize their imagination
▫ It will help to make contents faster (computer games/animation/learning tools…)
• Main Components in AAA
▫ Fast – helps to embody faster
▫ Detailed – makes more detailed/perfect image
▫ Aesthetic – helps to look more pleasing
• Related Area
▫ Inverse Rendering, Evolutionary Search,
Interactive Evolutionary Computation,
Computational Aesthetics, Computational Design ….
3
[InverseRenderNet, CVPR 2019]
[Interactive Evolutionary Computation, Hideyuki Takagi]

4. What is Artist Assistant AI(AAA)?
• Related Works
▫ Academic Field (Computer)
 [GauGAN] Semantic Image Synthesis with Spatially-Adaptive Normalization, CVPR, 2019
 Coloring With Limited Data: Few-Shot Colorization via Memory Augmented Networks, CVPR, 2019
 Effective Aesthetics Prediction with Multi-level Spatially Pooled Features, CVPR, 2019
 [STROTSS] Style Transfer by Relaxed Optimal Transport and Self-Similarity, CVPR, 2019
 …
▫ Academic Field (Bio)
 Deep image reconstruction from human brain activity. Preprint at bioRxiv, 240317, 2017
 GenerativeAdversarialNetworksConditionedonBrainActivityReconstructSeenImages.PreprintatbioRxiv,304774,2018
 …
▫ Personal projects
 Nono Martinez Alonso, Suggestive Drawing among Human and Artificial Intelligence, 2017
 Eyal Gruss, WanderGAN, ICCC, 2019
 …
4

5. Representative Works
• Deep Learning based approach
▫ Image-to-image translation
▫ Generative Adversarial Network(GAN) / VAE
 Fast – Yes
 Detailed – Far to go
 Controllable
▫ Latent Space Entanglement(DC-IGN, InfoGAN, BetaVAE …)
▫ Conditional Information (MUNIT, StyleGAN …)
 Aesthetic – Target data based
▫ Style Transfer
 Fast – Feed-forward network(Yes), Optimization (No)
 Detailed – Mainly(Far to go), STROTSS(Good)
 Aesthetic – Target data based
5
[DC-IGN, NIPS 2015]
[StyleGAN, CVPR 2019]

6. Personal Experiment
• Interactive Evolutionary Computation
▫ Generation Path – Computer generates outputs based on the user input
▫ Evaluation Path – User selects/evaluates the best outputs
• Color-Centered Assistant AI
▫ Generation Path – Recolorize Assistant Tool
▫ Evaluation Path – Assessment Assistant Tool (Helps user to select the best outputs)
6
[Interactive Evolutionary Computation, Hideyuki Takagi]

7. Main Research 1 - Recolorize Assistant Tool
Master Thesis: Deep Image Recolorization using Histogram Analogy

8. Example based Image Recolorization
• Definition
▫ Converting an input image according to a reference image that has desired color characteristics
• Motivation
▫ The input and reference image correlation can be diverse
 Strong relevance – High similarity in contents and configuration
 Weak relevance – High similarity in contents, less correlations in object configuration
 Irrelevance – Dissimilar contents
8
[Overall flow]
ReferenceInput Output(Ours)
Strong
Relevance
Weak
Relevance
Irrelevance
[Our example output]

9. Key Idea
• Histogram-Driven Image Recolorization
▫ Feed-forward deep learning framework that learns histogram analogy between input and reference image
▫ Histogram analogy can be applied either uniformly or adaptively
 Uniform histogram analogy are applied when strongly relevant/Irrelevant case
 Adaptive histogram analogy are applied for weakly relevant case using semantic information
9

10. Histogram-Driven Image Recolorization
• Framework Overview
▫ Composed of two networks: histogram encoding network(HEN) and image recolorization network(IRN)
 HEN encodes histogram of both input and reference image in Lab color space
 IRN recolorizes the input image based on the encoded histogram feature
▫ Jointly train HEN and IRN with semantic replacement module
10

11. Histogram-Driven Image Recolorization
• Objective Function
▫ Image and corresponding encoded histogram pair are defined as [Input 𝐼𝑠, 𝐻𝑠], [Reference 𝐼𝑡, 𝐻𝑡], [Output 𝐼𝑡, 𝐻𝑡]
▫ Minimize the following objective function
 Image loss, ℒ 𝑖𝑚𝑎𝑔𝑒
 Image loss enforces network to learn histogram analogy between inference image and reference image
 Histogram loss, ℒℎ𝑖𝑠𝑡
 Histogram loss enforces HEN to output similar encoded histogram between inference histogram and reference histogram
 Multi-scale loss, ℒ 𝑚𝑢𝑙𝑡𝑖
 Multi-scale loss explicitly enforces decoder to output reference image at each level
11
ℒ 𝑖𝑚𝑎𝑔𝑒 = 𝑀𝑆𝐸 𝐼𝑡, 𝐼𝑡
ℒ 𝑚𝑢𝑙𝑡𝑖 =
1
𝐷
𝑑=1
𝐷
𝑀𝑆𝐸( 𝐼𝑡
𝑑
, 𝐼𝑡
𝑑
)
ℒ 𝑡𝑜𝑡𝑎𝑙 = ℒ 𝑖𝑚𝑎𝑔𝑒 + 𝜆1ℒℎ𝑖𝑠𝑡 + 𝜆2ℒ 𝑚𝑢𝑙𝑡𝑖 ℒℎ𝑖𝑠𝑡 = 𝑀𝑆𝐸( 𝐻𝑡, 𝐻𝑡)

12. Histogram-Driven Image Recolorization
• Semantic Replacement
12
{𝑆𝑡 𝑙
}
IRN
Output
Target Histogram Initialization
HEN
Global
Histogram Encoding
𝐸𝑠
Global
Histogram Encoding
HEN
𝐸𝑡
Segment-wise
Histogram Encoding
HEN
{𝐸𝑡 𝑙
′
}
Ref 𝐼𝑡
Input 𝐼𝑠
{𝑆𝑠 𝑙
} Hist 𝐻𝑠
Hist 𝐻𝑡
Hist {𝐻𝑡 𝑙
′
}
Segment-wise Semantic Replacement
Input Ref

13. Experimental Results
• Qualitative evaluation
13

14. Application
• Palette based Recolorization
▫ Using a reference image in the form of a palette, recolor input image
▫ We can see the different result by changing the major color of the palette
14

15. Application
• Image Editing with Histogram Modification
▫ By changing the histogram information, the result also changed.
▫ Interactive histogram modification is possible
▫ Only needs input image (no reference)
15
IRN
HEN
HENHistogram
Modification
[Overall flow]
Input Reference Output AB Space
Blue
Increased
Red
Decreased

16. Main Research 2 – Assessment Assistant Tool
Company Project: Color Image Assessment using GAN

17. Computational Aesthetics
• Image Assessment
▫ Two objectives
 1. Image Aesthetics (IA)
 Motivation: Defining aesthetic score
 2. Image quality assessment (IQA)
 Gives the score of the image by the degree of corruption
 Motivation: Doesn’t care about a type of corruption, rather care about mimicking the perceptual attribute
▫ Dataset
 1. AVA Dataset / AADB Dataset
 Score range from 1 to 10 by multiple people
 2. TID 2013 Dataset
 Contains the 24 different number of distortions (Image Acquisition/Image Compression/Data transmission …)
 Five level of distortions each
 Also contains the color related distortions (contrast change/ change of color saturation)
17

18. Computational Aesthetics
• Feature: A hint for a definition of ‘good’ photo
▫ High-level describable feature (Hand-crafted/explainable feature)
 Colorfulness (Hasler, 2003): Color should be diverse
 Color Harmony (Moon Spencer, 1944)
 Saturation (Datta, 2006)
 Hue Count (Ke, 2006): Color should be simple
 …
▫ Implicit Features
 Bag-of-Visual-Words (Csurka,2004)
 Fisher Vector(Jaakkola,1999)
▫ Deep-learning Features
 Convolution feature
18
Colorfulness Hue Count
?

19. Computational Aesthetics
• Image Assessment
▫ NIMA: Neural image assessment, TIP, 2018
 Goal: Both 1)Image Quality 2)Image Aesthetics
 Ordered class: Cross entropy loss → Normalized Earth mover’s distance (Hou et al., 2016)
 Limitation:
 Binary classification
 Explainability
19
Network Performance on AVA Dataset

20. Color Assessment using GAN
• Key Idea
▫ The degree of improvement depends on the color quality of each image
 Good colored images require little improvement, but bad colored images require large improvement
▫ Based on the degree of improvement of each image, calculate the final score
20
Color
Enhancement
Network
Final Score
Degree of
Improvement
Color
Enhancement
Network
Degree of
Improvement
Final Score
Normal
Image
Good-Colored
Image

21. Color Assessment using GAN
• Image color improvement and evaluation based on GAN
▫ Unsupervised Learning requires no additional information other than video
▫ Network learns data distribution of target data set
• Modify system to output the target image as it is given to the generator
21
Discriminator
Generator
Real
Fake
Real
Loss
Fake
Target
Image
Source
Image
Target
Image
Modified GAN Network

22. Color Assessment using GAN
• Result Example
▫ Unlike previous technique, color score is continuous
▫ Image color enhancement and assessment happen at the same time
22
13.29
23.64
31.87
46.29
Before After Color Score Before After Color Score

23. Further Research
From three perspectives: Fast, Detailed, Aesthetic

24. Possible Research Area
• Main Components in AAA
▫ Fast – helps to embody faster
▫ Detailed – makes more detailed/perfect image
▫ Aesthetic – helps to look more pleasing
• Analysis
▫ Fast - Is this tool smart enough?
▫ Detailed - Which part is critical in artwork?
▫ Aesthetic – What makes it beautiful?
• Current Work
▫ Limitation of current SOTA method
▫ …
24

25. [Fast] Is this tool smart enough?
• Two elements of creation
▫ 1) Search well 2) Modify well
25
100%
0%
Final
Result
Data?
Interactive
Modification
Source
Searching
Resemblance
Number of intervention

26. [Fast] Is this tool smart enough?
• Naïve tool analysis model
▫ 1) Select sample images to reproduce
▫ 2) Multiple artists check this to look as similar as possible with the minimal intervention
▫ 3) Graph shows how much effort is required
26
100%
Resemblance
Number of
Intervention1
Complexity
Homogeneous Plane
Artwork/Model
Complexity
H
L
100%
Resemblance
Number of
Intervention
1

27. [Fast] Is this tool smart enough?
• Smarter tool makes converge faster
▫ Smarter tool would make resemblance faster no matter how the image looks
▫ Like graph below, we might analyze the degree of smart of each tool
27
100%
Resemblance
1
<Smarter Tool>
Number of
Intervention
<Ideal Tool>
100%
Resemblance
1 Number of
Intervention
100%
Resemblance
Number of
Intervention
1
<Ordinary Tool>

28. [Detailed] Which part is critical in drawing?
• There is also a visually insignificant part
▫ Less important parts can be presented in less detail
• Calculate perceptual similarity between current and final frame
▫ Assumption: Completed Image == Image in imagination
▫ Measure: LPIPS (CVPR, 2018)
28
LPIPS Path Example LPIPS Network

29. [Detailed] Which part is critical in drawing?
• X coordinate: number of frame, Y coordinate: LPIPS between current frame and last frame
• Video1- Girl
▫ https://www.youtube.com/watch?v=DFVB4zNitAE&list=UUTkzB2yMMQXU8V9CQczyZWg&index=2
• Video2- Cat
▫ https://www.youtube.com/watch?v=8Z8xYv6gM7Q&list=UUTkzB2yMMQXU8V9CQczyZWg
29

30. [Detailed] Which part is critical in drawing?
• Through this analysis
▫ We can see what makes perceptual gap
 The larger the slope, the larger the visual difference
 Large region matters
 Big contrast matters
▫ Different artist might have different graph patterns
 We may distinguish ‘Drawing Style’ of each artist
30
The effect of
cat whiskers

31. [Aesthetic] What makes it beautiful?
• Color Classification
▫ Goal: Test whether the network using convolution can properly classify images according to the color
information.
▫ Method: Among FiveK dataset, classify Expert B, C, and E using VGG Network
▫ Results: Accuracy on each dataset B: 75%, C: 50%, E: 64%
• Grad-CAM
▫ Method: Through Grad-CAM, visualize the heat-map that affected the classification
▫ Result: The criteria for determining the dataset are vague
31Object Classification Result using Grad-CAM

32. [Aesthetic] What makes it beautiful?
• Grad-CAM Visualization Result
32
B C EB C E
More research
should be done!

33. GauGAN, 2019
• Does semantic segmentation is enough for generating artwork?
▫ Instance segmentation …
33
By Colie Wertz
Input Segmentation Generation Output Retouched Output

34. Thank you