Recent machine learning developments saw a breakthrough in generating images. So-called Diffusion Models can create photo-realistic images from noise. With the help of an input text (prompt) we can guide the generation and produce matching images. This technology opened new doors for creating digital art, modifying existing images, and creating stunning visual experiences. In the talk, we will find out how these algorithms work, introduce Stable Diffusion (a concrete implementation), and find out what its use cases are. We will see how text can be used to generate matching outputs but also take a look at more experimental features such as creating images from edges, outlines, or depth maps. We will mainly focus on the open source text-to-image model Stable Diffusion, which has set new standards in image generation. With it also comes an active community that keeps it open source and accessible for everyone.

1. “Standing on top of the highest mountain, looking down to the other peaks, alps.
An award-winning landscape photo of South Tyrol”

2. Image Generation with
Diffusion Models
How computers imagine our world
Markus Pobitzer

3. Overview
Diffusion Models

4. Stable Diffusion (SD)
(Text) Guidance Stable Diffusion Output
“Majestic royal ship on
a calm sea, oil
painting, …”

5. Inpainting with Stable Diffusion
Input Mask Output
Original image from: https://unsplash.com/@overture_creations

6. Billions
of Image-Text pairs
256
Nvidia A100 (40GB) GPUs
150 000
GPU Hours
Training Stable Diffusion (v1)
Source: SD Model Card, https://github.com/CompVis/stable-diffusion/blob/main/Stable_Diffusion_v1_Model_Card.md

7. • DALL-E
• Imagen
• Midjourney
• Stable Diffusion
Image Generation Applications
unreleased
or
behind API
training code
and
model weights
available

8. Beyond Text Guidance
Input (Edges) Outputs

9. “Sign with SFSCON written on it” Real or Fake?
Challenges
Shortcomings

11. Try it out!
Generate the image on the left on your
device thanks to Google Colab.
Scan the QR-Code or visit:
github.com/Markus-Pobitzer/SFSCON-2023