PR-122: Can-Creative Adversarial Networks paper : https://arxiv.org/abs/1706.07068

1. visionNoob
(Jaewon Lee)
PR-122
CAN: Creative Adversarial Networks,
Generating "Art" by Learning About Styles and Deviating from
Style NormsAhmed Elgammal, Bingchen Liu, Mohamed Elhoseiny, Marian Mazzone, (2017)
1
https://arxiv.org/abs/1706.07068

2. 2
∗This paper is an extended version of a paper published on the eighth International Conference on
Computational Creativity (ICCC), held in Atlanta, GA, June 20th-June 22nd, 2017.
The Art and Artificial Intelligence Laboratory at Rutgers: https://sites.google.com/site/digihumanlab/home

3. 3
Figure 1: Example of images generated by CAN. The generated images vary from simple abstract ones to
complex textures and compositions. (256 x 256)

4. 4
Figure 1: Example of images generated by CAN. The generated images vary from simple abstract ones to
complex textures and compositions. (256 x 256)

5. 5
If we teach the machine about art and art styles and
force it to generate novel images that do not follow
established styles,
what would it generate?

6. 6
Would it generate something
that is aesthetically appealing to humans?
CAN : Top ranked by human subjects 1

7. 7
Would that be considered “art”?
CAN : Top ranked by human subjects 2

8. t
8
Art Generating Agent
The agent tries to generate that is novel, but not too novel.
https://www.psychologytoday.com/intl/blog/in-the-brain-the-beholder/201405/its-acquired-taste-how-knowledge-drives-aesthetics?amp
https://news.artnet.com/market/google-inceptionism-art-sells-big-439352
too novel
novel
deepdream
not
nobel
“dazzling, druggy, and creepy”

9. 9
Background
Wundt curve
Novelty
Surprisingness
Complexity
Ambiguity
Puzzlingness
Computational Curiosity, Qiong Wu Published 2015 in ArXiv

10. t
10
Art Generating Agent
The agent tries to generate that is novel, but not too novel.
https://www.psychologytoday.com/intl/blog/in-the-brain-the-beholder/201405/its-acquired-taste-how-knowledge-drives-aesthetics?amp
https://news.artnet.com/market/google-inceptionism-art-sells-big-439352
“dazzling, druggy, and creepy”
too novel
novel
deepdream
not
nobel

11. 11
Dataset
WikiArt Dataset (2015)
81,449 paintings from 1,119 artists (from 15th~21st century) with style label
https://www.wikiart.org/

12. 12
GAN is Emulative and not Creative
Recap - Vanilla GAN
min
𝐺
max
𝐷
𝑉 𝐷, 𝐺 = 𝔼 𝑥~𝑝 𝑑𝑎𝑡𝑎
[log 𝐷 𝑥 ] + 𝔼 𝑧~𝑝 𝑧
[log(1 − 𝐷 𝐺(𝑧 ))]
https://blog.openai.com/generative-models/

13. 13
min
𝐺
max
𝐷
𝑉 𝐷, 𝐺 = 𝔼 𝑥~𝑝 𝑑𝑎𝑡𝑎
[log 𝐷 𝑥 ] + 𝔼 𝑧~𝑝 𝑧
[log(1 − 𝐷 𝐺(𝑧 ))]
min
𝐺
max
𝐷
𝑉 𝐷, 𝐺 =
𝔼 𝑥, 𝑐~𝑝 𝑑𝑎𝑡𝑎
[log 𝐷𝑟 𝑥 + log 𝐷𝑐 𝑐 = 𝑐|𝑥 ] +
𝔼 𝑧~𝑝 𝑧
[log(1 − 𝐷𝑟 𝐺(𝑧 )) −
𝑘=1
𝐾
1
𝐾
log(𝐷𝑐 𝑐 𝑘 𝐺 𝑧 + (1 −
1
𝑘
) log(1 − 𝐷𝑐(𝑐 𝑘|𝐺 𝑧 ) ]
recap - Vanilla GAN loss
𝑐 : style label from the the data distribution 𝑝 𝑑𝑎𝑡𝑎
K : # of style classes
𝐷𝑟 〮 : discriminate between real art and generated images
𝐷𝑐(〮) : discriminate between different style categories and estimate posterior (i.e., 𝐷𝑐 𝑐 𝑘 〮 = 𝑝(𝑐 𝑘|〮))
style classification
style ambiguity
maximizes the entropy of p(c|x)
CAN : Loss function

14. 14
CAN : Loss function
Training Discriminator
Training Generator
max
𝐷
𝑉 𝐷, 𝐺 = 𝔼 𝑥, 𝑐~𝑝 𝑑𝑎𝑡𝑎
[log 𝐷𝑟 𝑥 + log 𝐷𝑐 𝑐 = 𝑐|𝑥 ] + # For Training data
𝔼 𝑧~𝑝 𝑧
[log(1 − 𝐷 𝐺(𝑧 ))] # For Generated data
𝔼 𝑧~𝑝 𝑧
[log(1 − 𝐷 𝐺(𝑧 )) −
𝑘=1
𝐾
1
𝐾
log(𝐷𝑐 𝑐 𝑘 𝐺 𝑧 + (1 −
1
𝑘
) log(1 − 𝐷𝑐(𝑐 𝑘|𝐺 𝑧 ) ]
min
𝐺
𝑉 𝐷, 𝐺 =
style classification
style ambiguity

15. 15

16. 16
Experiments
CAN implementation : https://github.com/mlberkeley/Creative-Adversarial-Networks

17. Experiments
17
Basically…
Assessing the creativity of artifacts generated
by the machine is an open and hard question.
Dataset : WikiArt dataset
Baseline 1 : original DCGAN (64 x 64)
Baseline 2 : original DCGAN (256 x 256)
Baseline 3 : CAN (w.o. style ambiguity) (256 x 256)
Proposed : CAN (256 x 256)

18. 18
We also do not see any recognizable figures.
Many of the images seems abstract.
Is that simply because it fails to emulate the art distribution or
Is it because it tried to generate novel images?
Is it at all creative?
Figure 1: Example of images generated by CAN. The generated images vary from simple abstract ones to
complex textures and compositions. (256 x 256)

19. 19
Qualitative Result
DCGAN 64 x 64 DCGAN 256 x 256 CAN 256 x 256
(w.o. style ambiguity)
CAN 256 x 256

20. 20
Quantitative Result
Experiment 1 (Amazone Mturk)
test the ability of the system to generate art that human users
could not distinguish from top creative art that is being generated by artists today
Q1: Do you think the work is created by an artist or generated by a computer?
The user has to choose one of two answers: artist or computer.
Q2: The user asked to rate how they like the image
in a scale 1 (extremely dislike) to 5 (extremely like).

21. 21
Figure 8: Art Basel Set: a collection of 25
paintings selected from Art Basel 2017 art fair.
Shamir, Lior, Jenny Nissel, and Ellen Winner. "Distinguishing between abstract art by artists vs. children and animals: Comparison
between human and machine perception." ACM Transactions on Applied Perception (TAP) 13.3 (2016): 17

22. 22
Quantitative Result
Experiment 2 (Amazone MTurk)
Q1 How do you like this image: 1-extremely dislike ~ 5-extremely like.
Q2 Rate the novelty of the image: 1-extremely not novel, ~ 5-extremely novel.
Q3 Do you find the image surprising: 1-extremely not surprising ~ 5-extremely surprising.
Q4 Rate the ambiguity of the image. I find this image: 1-extremely not ambiguous ~ 5-extremely ambiguous.
Q5 Rate the complexity of the image. I find this image: 1-extremely simple ~ 5-extremely complex
Q6 Do you think the image is created by an artist or generated by computer?

23. 23
Quantitative Result
Experiment 3
judge aspects related to whether the images generated by CAN can be considered art
Q1: As I interact with this painting, I start to see the artist’s intentionality:
it looks like it was composed very intentionally.
Q2: As I interact with this painting, I start to see a structure emerging.
Q3: Communication: As I interact with this painting, I feel that it is communicating with me.
Q4: Inspiration: As I interact with this painting, I feel inspired and elevated.

24. 24
Quantitative Result
Experiment 4 CAN vs sc-CAN (sophisticated art-educated subjects)
evaluate the effect of adding the style ambiguity loss to the CAN model
Q1 Which image do you think is more novel?
Q2 Which image do you think is more aesthetically appealing?
Result:
CAN images are more novel than sc-CAN : 59.47%
CAN images are more aesthetically appealing than sc-CAN : 60%

25. Q&A
25
Figure 5: Example of images generated by CAN. Top: Images ranked high in “likeness”
according to human subjects. Bottom: Images ranked the lowest by human subjects.