This document discusses creative AI and multimodality. It begins by looking at current possibilities for creative AI, including appropriating standard neural networks for creative use, reinforcement learning approaches that frame creativity as a game, recurrent neural networks, sequence-to-sequence models that treat creativity as translation, autoencoders, attention-based models, and generative adversarial networks. It also discusses needs for creative AI, including developing a system that marries a creative process with creative outputs using minimal human input data but with its own style and the ability for human-level supervision to enable rapid experimentation. The document frames creative AI as a "brush" that can be used for painting.

1. Creative AI &
multimodality:
looking ahead
Roelof Pieters
@graphific
Imperial College London,
1 Dec 2015
roelof@graph-technologies.comhttp://artificialexperience.com/http://www.csc.kth.se/~roelof/

2. AICreative

3. AI
I kinda expect the audience to know AI & Machine Learning
Let’s move on shall we ?

4. AI
All references to:
- Arxiv or
- GitXiv if the “code” or “dataset” is available
Collaborative Open Computer Science
more info (Medium)

5. AI > today’s focus

6. AI > today’s focus

7. “Deep learning is a set of
algorithms in machine learning
that attempt to learn in multiple
levels, corresponding to
different levels of abstraction.”

8. AI > today’s focus
use of several modes (media) to
create a single artifact.
Multimodality
“Mode”
Socially and culturally shaped
resource for making meaning.
— Gunther Kress

9. Creativity

10. Creativity
• Many definitions: philosophical, sociological, historical,
practical

11. Creativity
1. Making unfamiliar combinations of familiar ideas.
2. Explore a structured conceptual space
3. (Radically) transforming ones structured conceptual space
“Exploration”
“Remix”
“The Creative Mind”
— Margaret Boden
“Transformation”

12. • Skill
• Appreciation
• Imagination
• Learning
• Innovation
• Accountability,
• Subjectivity
• Intentionality.
Creativity > “Traits” software has to exhibit in order to
avoid easy criticism of being “non-creative”.
(Simon Colton)

13. • Skill
• Appreciation
• Imagination
• Learning
• Innovation
• Accountability,
• Subjectivity
• Intentionality
Creativity > software traits

14. • Skill
• Appreciation
• Imagination
• Learning
• Innovation
• Accountability,
• Subjectivity
• Intentionality
Creativity > software traits

15. • Skill
• Appreciation
• Imagination
• Learning
• Innovation
• Accountability,
• Subjectivity
• Intentionality
Creativity > software traits

16. • Skill
• Appreciation
• Imagination
• Learning
• Innovation
• Accountability,
• Subjectivity
• Intentionality
Creativity > software traits

17. • Skill
• Appreciation
• Imagination
• Learning
• Innovation
• Accountability,
• Subjectivity
• Intentionality
Creativity > software traits

18. • Skill
• Appreciation
• Imagination
• Learning
• Innovation
• Accountability,
• Subjectivity
• Intentionality
Creativity > software traits

19. • Skill
• Appreciation
• Imagination
• Learning
• Innovation
• Accountability,
• Subjectivity
• Intentionality
Creativity > software traits

20. • Skill
• Appreciation
• Imagination
• Learning
• Innovation
• Accountability,
• Subjectivity
• Intentionality
Creativity > software traits

21. AICreative

22. Creative AI > Current possibilities
• Appropriating “standard” nets for creative use
• Reinforcement Learning: Creativity as a Game
• RNNs/LSTMs/GRUs
• Sequence-to-Sequence: Creativity as a Translation Task
• Auto-Encoders
• Attention-based Models
• Generative Adversarial Nets

23. Creative AI > Current possibilities
• Appropriating “standard” nets for creative use
• Reinforcement Learning: Creativity as a Game
• RNNs/LSTMs/GRUs
• Sequence-to-Sequence: Creativity as a Translation Task
• Auto-Encoders
• Attention-based Models
• Generative Adversarial Nets

24. Creative AI > Current possibilities > Appropriating “standard” nets for creative use Deep Dream
see also: www.csc.kth.se/~roelof/deepdream/

25. Creative AI > Current possibilities > Appropriating “standard” nets for creative use Deep Dream
see also: www.csc.kth.se/~roelof/deepdream/ codeyoutubeRoelof Pieters 2015

26. Creative AI > Current possibilities > Appropriating “standard” nets for creative use Deep Dream
see also: www.csc.kth.se/~roelof/deepdream/
C.M.Kosemen &
Roelof Pieters (2015)
Gizmodo

27. Creative AI > Current possibilities > Appropriating “standard” nets for creative use
Leon A. Gatys, Alexander S. Ecker, Matthias Bethge , 2015.
A Neural Algorithm of Artistic Style (GitXiv)
Style Net

29. Gene Kogan, 2015. Why is a Raven Like a Writing Desk? (vimeo)

30. Creative AI > Current possibilities
• Appropriating “standard” nets for creative use
• Reinforcement Learning: Creativity as a Game
• RNNs/LSTMs/GRUs
• Sequence-to-Sequence: Creativity as a Translation Task
• Auto-Encoders
• Attention-based Models
• Generative Adversarial Nets

32. Creative AI > Current possibilities > Reinforcement Learning
• AMN: Emilio Parisotto, Jimmy Lei Ba, Ruslan Salakhutdinov 2015, Actor-Mimic:
Deep Multitask and Transfer Reinforcement Learning (arxiv)
• DQN: Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Rusu, Andrei A., Veness,
Joel, Bellemare, Marc G., Graves, Alex, Riedmiller, Martin, Fidjeland, Andreas K.,
Ostrovski, Georg, Petersen, Stig, Beattie, Charles, Sadik, Amir, Antonoglou, Ioannis,
King, Helen, Kumaran, Dharshan, Wierstra, Daan, Legg, Shane, and Hassabis,
Demis. Human-level control through deep reinforcement learning. Nature, 518(7540):
529–533, 2015.

33. Creative AI > Current possibilities > Reinforcement Learning
Ardi Tampuu, Tambet Matiisen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, Raul Vicente, 2015
Multiagent Cooperation and Competition with Deep Reinforcement Learning (GitXiv)
(YouTube)

34. Reinforcement Learning
Ning Xie, Hirotaka Hachiya, Masashi Sugiyama, 2013 ,
Artist Agent: A Reinforcement Learning Approach to Automatic
Stroke Generation in Oriental Ink Painting (Paper, Lecture,
YouTube)

35. (YouTube)

36. Ning Xie, Hirotaka Hachiya, Masashi Sugiyama, 2013
Artist Agent: A Reinforcement Learning Approach to Automatic Stroke Generation
in Oriental Ink Painting (Paper, Lecture, YouTube)

37. Creative AI > Current possibilities
• Appropriating “standard” nets for creative use
• Reinforcement Learning: Creativity as a Game
• RNNs/LSTMs/GRUs
• Sequence-to-Sequence: Creativity as a Translation Task
• Auto-Encoders
• Attention-based Models
• Generative Adversarial Nets

38. Creative AI > Current possibilities
• Appropriating “standard” nets for creative use
• Reinforcement Learning: Creativity as a Game
• RNNs/LSTMs/GRUs
• Sequence-to-Sequence: Creativity as a Translation Task
• Auto-Encoders
• Attention-based Models
• Generative Adversarial Nets

39. Creative AI > Current possibilities
• Appropriating “standard” nets for creative use
• Reinforcement Learning: Creativity as a Game
• RNNs/LSTMs/GRUs
• Sequence-to-Sequence: Creativity as a Translation Task
• Auto-encoders
• Attention-based Models
• Generative Adversarial Nets

40. Creative AI > Current possibilities
• Standard (“denoising”) Autoencoders
• Variational Autoencoder (VAE) / Stochastic Gradient VB
• Deep Convolutional Inverse Graphics Network
• Variational RNN (VRNN)
Vincent et al, 2010. Stacked Denoising Autoencoders: Learning Useful Representations in
a Deep Network with a Local Denoising Criterion (paper) (code)

41. Creative AI > Current possibilities
• Standard “denoising” Autoencoders
• Variational Autoencoder (VAE) / Stochastic Gradient VB
• Deep Convolutional Inverse Graphics Network
• Variational RNN (VRNN)
• Diederik P Kingma, Max Welling, 2013.
Auto-Encoding Variational Bayes (GitXiv)

42. Creative AI > Current possibilities
• Standard “denoising” Autoencoders
• Variational Autoencoder (VAE)
• Deep Convolutional Inverse Graphics Network (modified VAE)
• Variational RNN (VRNN)
Tejas D. Kulkarni, Will Whitney, Pushmeet Kohli, Joshua B. Tenenbaum, 2015
Deep Convolutional Inverse Graphics Network (GitXiv)

43. Creative AI > Current possibilities
• Standard “denoising” Autoencoders
• Variational Autoencoder (VAE)
• Deep Convolutional Inverse Graphics Network
• Variational RNN (VRNN) (VAE at every time step)
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron Courville, Yoshua Bengio, 2015
A Recurrent Latent Variable Model for Sequential Data (GitXiv)
VAEVAEVAE

44. Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron Courville, Yoshua Bengio , 2015.
A Recurrent Latent Variable Model for Sequential Data (GitXiv) (Audio Samples)

45. Creative AI > Current possibilities
• Appropriating “standard” nets for creative use
• Reinforcement Learning: Creativity as a Game
• RNNs/LSTMs/GRUs
• Sequence-to-Sequence: Creativity as a Translation Task
• Auto-Encoders
• Attention-based Models
• Generative Adversarial Nets

46. Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, Daan Wierstra, 2015
DRAW: A Recurrent Neural Network For Image Generation (GitXiv)
Variational Auto-Encoder
Deep Recurrent Attentive Writer
(DRAW) Network

47. (YouTube)

48. Creative AI > Current possibilities
• Appropriating “standard” nets for creative use
• Reinforcement Learning: Creativity as a Game
• RNNs/LSTMs/GRUs
• Sequence-to-Sequence: Creativity as a Translation Task
• Auto-Encoders
• Attention-based Models
• Generative Adverserial Nets

49. Emily Denton, Soumith Chintala, Arthur Szlam, Rob Fergus, 2015.
Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks (GitXiv)

50. Alec Radford, Luke Metz, Soumith Chintala , 2015.
Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (GitXiv)

51. Alec Radford, Luke Metz, Soumith Chintala , 2015.
Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (GitXiv)

52. ”turn” vector created from four averaged samples of faces looking
left vs looking right.
Alec Radford, Luke Metz, Soumith Chintala , 2015.
Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (GitXiv)

53. walking through the manifold

54. top: unmodified samples
bottom: same samples dropping out ”window” filters

55. Autonomy Supervision
Creativity?
- unsupervised training
- generator/discrimator
- latent/z space
- auto encoders
- multimodality
- query - target/class

56. Creativity?
Process Result

57. Creative AI > Needs as I see it
Creative AI as a
“tool”
or “brush” to paint
with

58. A system which marries the need for a creative
process with the need for a creative output
• with as less human input as possible (data)
• with its own style
• with the possibility for human level supervision
for rapid experimentation
Creative AI > a “brush”

59. A system which marries the need for a creative
process with the need for a creative output
• with as less human input as possible ( )
• with its own style
• with the possibility for human level supervision
for rapid experimentation
Creative AI > a “brush”
data

60. Creative AI > a “brush” > data
• reuse nets as much as possible
• combining unsupervised & supervised
• multiple modalities
• plug in external knowledge bases

61. Creative AI > a “brush” > data input
• unlabeled & labeled data
• external knowledge bases (dbpedia, wikipedia)
• one-shot learning
• zero-shot learning
Richard Socher, Milind Ganjoo, Hamsa Sridhar, Osbert Bastani, Christopher D. Manning, Andrew Y. Ng, 2013
Zero-Shot Learning Through Cross-Modal Transfer
a zero-shot model that can predict both seen and unseen classes

62. Creative AI > a “brush” > data input
Richard Socher, Milind Ganjoo, Hamsa Sridhar, Osbert Bastani, Christopher D. Manning, Andrew Y. Ng, 2013
Zero-Shot Learning Through Cross-Modal Transfer
(slides)

63. Creative AI > a “brush” > data input
Richard Socher, Milind Ganjoo, Hamsa Sridhar, Osbert Bastani, Christopher D. Manning, Andrew Y. Ng, 2013
Zero-Shot Learning Through Cross-Modal Transfer
(slides)

64. Creative AI > a “brush” > data input
Richard Socher, Milind Ganjoo, Hamsa Sridhar, Osbert Bastani, Christopher D. Manning, Andrew Y. Ng, 2013
Zero-Shot Learning Through Cross-Modal Transfer
(slides)

65. A system which marries the need for a creative
process with the need for a creative output
• with as less human input as possible (data)
• with its own style
• with the possibility for human level
for rapid experimentation
Creative AI > a “brush”
supervision

66. Creative AI > a “brush” > data
• “rich” latent (“z”) space
• easy user supervision over output:
• priors
• constrain network (units, layers, etc)
• guided input
• mixed input
• latent space

67. Creative AI > a “brush” > data
• “rich” latent (“z”) space
• easy user supervision over output:
• priors
• constrain network (units, layers, etc)
• guided input
• mixed input
• latent space

68. Creative AI > a “brush” > data
Deep Dream
Alexander Mordvintsev, Christopher Olah, Mike Tyka, 2015.
Inceptionism: Going Deeper into Neural Networks
Google Research Blog

69. Creative AI > a “brush” > data
Deep Dream
Roelof Pieters, 2015 DeepDream - Class visualization Experiment (link)

70. Roelof Pieters, 2015 DeepDream - Class visualization Experiment (link)

71. Creative AI > a “brush” > data
• “rich” latent (“z”) space
• easy user supervision over output:
• priors
• constrain network (units, layers, etc)
• guided input
• mixed input
• latent space

72. Creative AI > a “brush” > data
Deep Dream
Roelof Pieters, 2015 DeepDream - Overview of standard bvlc googlenet (inception) layers (link)
Constrain Layers

73. Creative AI > a “brush” > data
Deep Dream
Roelof Pieters, 2015 Single Unit Activations (early layer) (Flickr Album)
Constrain Units

74. Creative AI > a “brush” > data
• “rich” latent (“z”) space
• easy user supervision over output:
• priors
• constrain network (units, layers, etc)
• guided input
• mixed input
• latent space

75. Creative AI > a “brush” > data
Deep Dream
Roelof Pieters, 2015 DeepDream Video (GitHub)

76. Creative AI > a “brush” > data
• “rich” latent (“z”) space
• easy user supervision over output:
• priors
• constrain network (units, layers, etc)
• guided input
• mixed input
• latent space

77. Creative AI > a “brush” > data
Style Net
Roelof Pieters (graphific) (tweet) Roelof Pieters (graphific) (tweet)

78. Creative AI > a “brush” > data
• “rich” latent (“z”) space
• easy user supervision over output:
• priors
• constrain network (units, layers, etc)
• guided input
• mixed input
• latent space

79. Image -> Text
“A person riding a motorcycle on a dirt road.”???

80. Image -> Text
“Two hockey players are fighting over the puck.”???

81. Image -> Text
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron
Courville, Ruslan Salakhutdinov, Richard Zemel, Yoshua Bengio,
Show, Attend and Tell: Neural Image Caption Generation with
Visual Attention (arxiv) (info) (code)
Andrej Karpathy Li Fei-Fei , 2015.
Deep Visual-Semantic Alignments for Generating Image Descriptions (pdf) (info) (code)
Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan ,
2015. Show and Tell: A Neural Image Caption Generator (arxiv)

82. Text -> Image “A stop sign is flying in blue skies.”
“A herd of elephants flying in the blue skies.”
Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, Ruslan Salakhutdinov, 2015.
Generating Images from Captions with Attention (arxiv) (examples)

83. Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, Ruslan Salakhutdinov, 2015.
Generating Images from Captions with Attention (arxiv) (examples)
Text -> Image

84. Subhashini Venugopalan, Marcus Rohrbach, Jeff Donahue, Raymond Mooney,
Trevor Darrell, Kate Saenko , 2015. Sequence to Sequence -- Video to Text (GitXiv)
Video -> Text

85. A system which marries the need for a creative
process with the need for a creative output
• with as less human input as possible (data)
• with its own style
• with the possibility for human level supervision
for
Creative AI > a “brush”
rapid experimentation

86. Creative AI > a “brush” > rapid experimentation

87. Widening
Deepening
Tianqi Chen, Ian Goodfellow, Jonathon Shlens, 2015. Net2Net: Accelerating Learning via
Knowledge Transfer (arxiv) / code (torch)
Reusing Nets:
Bigger Net

88. Teacher and Student net Hint training
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta,
Yoshua Bengio, 2014. FitNets: Hints for Thin Deep Nets (arxiv)
Knowledge distillation
SVHN Error
MNIST Error
Reusing Nets:
Smaller Net

89. Hashed Net
Wenlin Chen, James T. Wilson, Stephen Tyree, Kilian Q. Weinberger, Yixin Chen, 2015.
Compressing Neural Networks with the Hashing Trick (arxiv)
Shrinking Nets:
Hashing

90. Song Han, Huizi Mao, William J. Dally, 2015. Deep Compression: Compressing Deep Neural
Networks with Pruning, Trained Quantization and Huffman Coding (arxiv)
Shrinking Nets:
Pruning,
Quantization &
Huffman coding

91. Creative AI > a “brush” > rapid experimentation
• experiments need “tooling”, specialised design
software to
• try things
• explore latent spaces (z-space)
• push the AI in the right direction
• be surprised by AI

92. Creative AI > a “brush” > rapid experimentation
human-machine collaboration

93. Creative AI > a “brush” > rapid experimentation
(YouTube, Paper)

94. Creative AI > a “brush” > rapid experimentation
(YouTube, Paper)

95. Creative AI > a “brush” > rapid experimentation
(Vimeo, Paper)

96. Creative AI > a “brush” > rapid experimentation
• Advertising and marketing
• Architecture
• Crafts
• Design: product, graphic and fashion design
• Film, TV, video, radio and photography
• IT, software and computer services
• Publishing
• Museums, galleries and libraries
• Music, performing and visual arts

97. Questions?
love letters? existential dilemma’s? academic questions? gifts? find me at:
www.csc.kth.se/~roelof/
roelof@kth.se