The document discusses the application of TensorFlow Lite in art and design, highlighting computer vision tasks, generative art techniques, and various AI models like GANs. It provides examples of implementations such as style transfer and image recognition, alongside useful resources and datasets. The presentation aims to demonstrate how AI can assist and inspire artists and designers in creating new forms of digital art.

1. AI for Art & Design
with Lite
Margaret Maynard-Reid, Sept 4 2020
#TFUGIndiaSummit
@margaretmz

2. #TFUGIndiaSummit | @margaretmz | #TFLite
Topics
● Computer vision tasks
● Generative art & design
● Datasets
2
AI for Art & Design TensorFlow Lite Samples & tutorials
● Samples & tutorials
● Community
● Tools & resources
● What is TFLite?
● How to make apps

3. Computer Vision Tasks
Complexity
Image classification
(single-label)
Classify an image to
a class
Examples
Painting style or
artist
Van Gogh
Image classification
(multi-label)
Classify an image to
multiple classes
Examples
Movie poster genre
action, sci-fi
Feature extraction
Extracting latent
features of an
image with CNN
models
Examples
Visual search
find similar fashion
Object detection
Identify one or
multiple objects
within an image and
their locations with
bounding boxes.
detect UI elements
Segmentation
Classify whether
each pixel of the
image belongs to a
certain class
segment UI elements
Generative models
(GANs)
Two or more models
trained simultaneously
Examples
- Generate new images
- Super res
- Image-to-image
3

4. Generative Art & Design
AI for Art & Design with TFLite

5. #TFUGIndiaSummit | @margaretmz | #TFLite
Deep Dream 2015
5

6. Use CNN neural networks
Content image + style image = styled image Content Style Generated
Neural Style Transfer: 50 Shades of Miaw (link)
A few more samples of generated images…
Style Transfer 2015

7. #TFUGIndiaSummit | @margaretmz | #TFLite
Generative Adversarial Networks 2014
What are GANs?
Generative Adversarial Networks (GANs)
have at least two network models which
compete against each other...
7

8. #TFUGIndiaSummit | @margaretmz | #TFLite
GANs variations
GANs are fun but difficult to train!
● 2014 Ian Goodfellow’s original GAN’s paper
● 2015 DCGAN (Deep Convolutional GAN) (paper)
● 2014 cGAN (Conditional GAN) - Generate images conditioned on a class
(paper)
● 2016 Pix2Pix – paired image to image translation (paper)
● 2017 CycleGAN – unpaired Image to image translation
● 2018 StyleGAN – Generate images with some control on image characteristics
(paper)
● Many other GAN variations…
8

9. #TFUGIndiaSummit | @margaretmz | #TFLite
● Training datasets must be paired
● A type of conditional GAN
Link to TensorFlow tutorial
Pix2Pix 2016
9

10. #TFUGIndiaSummit | @margaretmz | #TFLite
CycleGAN 2017
● Training datasets are unpaired
● Horses <--> zebras
● Summer <--> winter
● Photos to paintings (Monet, Van Gogh etc.)
Link to TensorFlow tutorial
10

11. #TFUGIndiaSummit | @margaretmz | #TFLite
BigGAN 2018
Large scale GAN that generates high fidelity class-conditional images.
Link to paper | TF Hub biggan512, biggan-256, biggan-128
11

12. #TFUGIndiaSummit | @margaretmz | #TFLite
Control the styles of the generated images
● Pose
● Face shape
● Age
● Gender
● Hair color
4x4 --> 1024x1024
8GPUs trained for 1 week
Current examples: faces, bedrooms and cats
Link to video
StyleGAN 2018
12

13. #TFUGIndiaSummit | @margaretmz | #TFLite
Sketch2Code
Microsoft: Turn your whiteboard sketches to working code in seconds with Sketch2Code
13

14. #TFUGIndiaSummit | @margaretmz | #TFLite
Fashion++
Facebook AI: “Fashion++: Minimal Edits for Outfit Improvement” (news | paper)
14

15. Datasets
Art
The Art Institute of Chicago - 5000 high res images
Best Artworks of All Time
Art Images: Drawing/Painting/Sculptures/Engravings
Overwatch Heroes Recognition
Chinese Fine Art
Museum of Modern Art Collection
15
UI/UX Design
● Rico mobile app datasets
● Kaggle
○ Common Mobile/Web App Icons
○ Icons-50

16. TensorFlow Lite
AI for Art & Design with TFLite

17. #TFUGIndiaSummit | @margaretmz | #TFLite
What is TensorFlow Lite?
A framework with a set of tools for
deploying ML to mobile & embedded:
● A converter
● An interpreter
● Ops / kernels
● Interface to hardware
accelerations:
○ NN AIP
○ Edge TPu
● A bunch of other tools...
17

18. #TFUGIndiaSummit | @margaretmz | #TFLite
Why on device?
18
It’s all about that camera!

19. #TFUGIndiaSummit | @margaretmz | #TFLite
How to make a TensorFlow Lite app?
● ML model:
○ Train / find a model
○ Convert to .tflite format
○ Run inference in Python
○ Benchmark (optional)
○ Add model metadata
● Build Android app
○ UI navigation
○ Input image: gallery or camera
○ Run model inference
○ Display result in UI
19

20. #TFUGIndiaSummit | @margaretmz | #TFLite
What is an ML model?
• A complex mathematical function
• A representation of learnings from
training data
20

21. #TFUGIndiaSummit | @margaretmz | #TFLite
How to get an ML model?
● Train your own
● Transfer learning
● Download from TensorFlow Hub
● Use ML Kit
21

22. https://pair.withgoogle.com/

23. Apps and Tutorials
AI for Art & Design with TFLite

24. #TFUGIndiaSummit | @margaretmz | #TFLite
ML Kit Digital Ink Recognition
Draw & recognize emojis
Blog post:
https://developers.googleblog.com/2020/08/digital-ink-
recognition-in-ml-kit.html
Sample app:
https://developers.google.com/ml-kit/vision/digital-ink-r
ecognition
24

25. Part 2: ML Model Binding from Android Studio
TensorFlow Lite Model Maker
Part 1: TensorFlow Lite Model Maker
● Image/Text classification
25

26. #TFUGIndiaSummit | @margaretmz | #TFLite
Style transfer
Google Arts & Culture app:
https://play.google.com/store/apps/details?id=com.google.android.apps.cultural
Blog post:
4/2/2020 - Transform your photo in the style of an iconic artist
26

27. 27
Selfie2Anime
● U-GAT-IT paper
● Image to image translation
● Written in TF1 code
● Unpaired selfie and anime images
● 50 & 100 epochs checkpoints
● Collaboration with Sayak & Khanh

28. #TFUGIndiaSummit | @margaretmz | #TFLite
Whitebox CartoonGAN
● Faster & smaller
● Not just for selfies
● Link to project repo
● Link to TF Hub models
● Collaboration with Sayak & Khanh
28

29. #TFUGIndiaSummit | @margaretmz | #TFLite
Segmentation + Style Transfer
Demo by Khanh LeViet TFLite inference in Python,
to be implemented on mobile...
29

30. Community & resources
AI for Art & Design with TFLite

31. #TFUGIndiaSummit | @margaretmz | #TFLite
2020 E2E TFLite Tutorials
31

32. Awesome TensorFlow Lite
A curated list of models, apps, tutorials and
learning resources…
Thanks to the ML GDE and Android community!
Support of TensorFlow Lite team
https://github.com/margaretmz/awesome-tflite
32

33. #TFUGIndiaSummit | @margaretmz | #TFLite
Magenta
https://magenta.tensorflow.org/ | https://github.com/magenta
33

34. https://ami.withgoogle.com/
34

35. #TFUGIndiaSummit | @margaretmz | #TFLite
AI & the Future of UX
https://uxdesign.cc/ai/home
35

36. #TFUGIndiaSummit | @margaretmz | #TFLite
AI for Art & Design
AI will assist and inspire artists and designers
Everyone can now make art and design!
36

37. #TFUGIndiaSummit | @margaretmz | #TFLite
Follow me on Twitter, Medium or GitHub to learn more about
deep learning, TensorFlow and on-device ML.
@margaretmz
@margaretmz
margaretmz
Thank you!
37