This document discusses using generative AI models for creative work such as generating movie scripts and content. It describes several generative models including restricted Boltzmann machines, deep generative models, variational autoencoders, and hierarchical LSTMs. It proposes using these models to help optimize the movie production process by automatically generating schedules, budgets, storyboards, and dialog from input scripts to help filmmakers explore creative options. An example project of a movie conceived by humans but with AI-generated dialog is briefly described.

1. Generative AI for
Creative Work
Debajyoti (Deb) Ray, CTO, End Cue

2. Generative Models
Model the Joint Probability
of Inputs and Outputs
Can generate both Inputs and Outputs given hidden paramete
Restricted Boltzmann Machine:
Learns probability distribution ov
set of inputs.

3. Deep Generative
Models

4. Wake-Sleep Algorithm
Perceptual Systems can also
Imagine
Hinton et al, Science 1995

5. Generated Creatives
Tensorflow trained on ImageNet -> Sampled data
Google Magenta project -> generating music with
RNNs

6. Language
Feedforward Networks Recurrent Networks
Statele
ss
Maintains
state

7. Long-Short Term Memory Models
RNNs Cannot encode
long-range dependencies
LSTMs encode long-range
dependencies with gated
memory

8. Maximum Likelihood
The _ quick _ brown _ fox
Learn conditional probabilities: P (fox | “The quick brown
Weight matrix W is large. ~ 100,000 tokens per corpus.
Generate text: Maximize likelihood of generated sentences

9. A LSTM Movie
SUNSPRING 2016
Produced by End Cue
LSTM trained on Sci-fi scrip

10. Optimizing Content Creation
Script
Financing
Greenlighting PreProduction
Principal
Photography
Post-
Production
Agile StoryWriter Agile Producer
Distribution
Forecasting

11. May 2017 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
PreProduction
Schedule
Location Scouting
Shooting Schedule
Finalize Script
AOSD
ShotLists and StoryBoard
Script Breakdown
Casting

12. PreProduction : Script
Breakdown

13. PreProduction : Story Board
Text -> Image Classifier
Script -> Suggested Shot; Find image in archive that best match

14. PreProduction : Story Board
Trained on: Scripts -> Film Images (canonical shots in scenes)
Archival footage can be used for training algorithms. Licensed for Storyboar

15. PreProduction : Scheduler

16. Variables that impact the optimal schedule:
• Cast schedules
• Location availability
• Equipment and Props (e.g. vehicles)
• Weather
• Cost and budget constraints
AI to solve the combinatorial optimization problem:
We realized up to 20% cost and time efficiencies. Data: Run optimization re
PreProduction : Scheduler

17. PreProduction : Scheduler

19. Ingest Script  Automatically generate a Story Interaction Graph
Agile StoryWriter

20. Remove Node: Action Sequence was adding significantly to budget
Agile StoryWriter
Butterfly Effects: What other nodes are impacted from the change?

21. Explore What If Scenarios
Agile Content Creation
Script
Greenlighting
Decision
Pre-Production
Financing
Forecasting
Schedules,
Budget
Distribution
PredictionsIterations
Eliminate
Scenes

22. Language Models
Why is language hard? Discrete space.
Cats are good pets
Bats are good pets
Images & Sound : Continuous space
Gradient search is not effective for language learning.

23. Generative Adversarial
Networks
Minimax game between two
Deep Neural Networks:
Generator G
Discriminator D
Generates sharp images that resemble the training data.

24. Generative Adversarial
Networks
Drawbacks:
- Cannot control what to generate (some advances with Conditional GANs)
- Cannot generate categorical data
- Cannot access latent features
Not very successful in discrete space e.g. text generation
- Learning converges to a specific part of the distribution

25. Autoencoders
Utilize Latent variables
Marginal distribution is intract
Intuition: Learn Latent Concepts. Words -> Concepts -> Words

26. Variational Autoencoders
Calculating is infeasible, so we use variational lower
KL-divergence with the variational posterior
A normal distribution
parameterized by a
Deep Neural Network.

27. Variational Autoencoders
Tensorflow Visualization
For end-to-end VAE with two hidden
encoder / decoder layers.

28. Hierarchical LSTMs
Standard Autoencoder with
Long-Short Term Memory model
Hierarchical Autoencoder with
Long-Short Term Memory model

29. Generative Active Learning
U
Labeled
Training
Data
Labeled
New Instances
& Repeat
Select Examples
to Label or Rank
Unlabeled
Generated
Data

30. AI Generated Content
Assistant Writer, Assistant Producer & Assistant Director
• Writer sets up the context. AI generates the narrative and dialogues.
• Producer enters ATL parameters. AI generates budget, schedule.
• AI recommends Shot Lists and Storyboards.

31. A Variational Autoencoder
Movie
Movie conceived by and
Storyline written by Human
AI generated dialogues
Stay tuned for public release..

32. Thank You!
• AI team @ End Cue
• Walter Kortschak and Andrew Kortschak @ End Cue.
• We are Hiring! Email me at deb@endcue.com
• Production Team at End Cue
• Production Team at GreenCard Pictures
Nathan Crockett