My presentation on PyCon Ireland "Python for Computer Vision" gives an introduction into deep learning and computer vision and a list of references for learning more.

1. PYTHON
for
Computer Vision
Tatiana Gabruseva, PhD

2. Agenda
We want to make
computers see
What is Computer Vision?
Cameras everywhere:
- smart phones
- baby monitors
- security cameras
- IR cameras
LiDARs …

3. AgendaWhat is Computer Vision?
Making computers to gain
high-level understanding
from images and videos
We want to make
computers see

4. AgendaWhat is Computer Vision?
Vision begins with eyes,
but it truly takes place in
the brain (Dr Fei Fei Li)
Making computers to gain high-level
understanding from images and videos
By seeing we really mean understanding

5. Agenda Computer Vision Tasks
Classification
What is that?
people

6. Agenda Computer Vision Tasks
Identification
Who/what exactly
is that?
John Travolta
Uma Thurman

7. Agenda
Object
Detection
Where is that?
Computer Vision Tasks

8. Agenda
Segmentation
Where exactly is
that?
Computer Vision Tasks

9. Agenda
Pose
estimation
What is it doing?
Computer Vision Tasks
dancing

10. Agenda
Action
recognition
What’s happening?
Computer Vision Tasks
dancingA man and a woman

11. Why computer vision?

12. Biomedical
applications
- Classify cancer histograms
- Select individual cells

13. Self-driving cars
Detect and select cars, road signs,
traffic lights, pedestrian…

14. Security
- Identification
- Action recognition

15. Satellite imaging
- - Object detection
- Detect fires
- - Find people

16. Art
- Style transfer
- Animation

17. Medical
diagnostics
- Pneumonia detection
- Anomaly classification
- Organs/tumor segmentation

18. For example…

20. Classification problem
Plant Can

21. How? Supervised training
Can
Bicycle
Ball
Tree Flower
Cat

22. Get the data

23. ~ 15 million images
in
~ 22 000 categories

24. http://www.image-net.org/

26. Image from a computer point of view
11 13 14 15 15 16 18 20 12 15
88 33 24 16 17 17 19 25 23 34
79 31 22 34 43 46 23 22 21 31
11 13 14 15 15 16 18 20 12 15
88 33 24 16 17 17 19 25 23 34
79 31 22 34 43 46 29 22 21 31
17 13 14 15 15 17 18 40 52 15
58 33 57 16 47 57 19 25 23 34
79 37 22 34 47 46 29 27 21 31
73 33 23 33 87 46 23 83 21 81
11 13 14 15 15 16 18 20 12 15
88 33 24 16 17 17 19 25 23 34
79 31 22 34 43 46 23 22 21 31
11 13 14 15 15 16 18 20 12 15
88 33 24 16 17 17 19 25 23 34
79 31 22 34 43 46 29 22 21 31
17 13 14 15 15 17 18 40 52 15
58 33 57 16 47 57 19 25 23 34
79 37 22 34 47 46 29 27 21 31
73 33 23 33 87 46 23 83 21 81
11 13 14 15 15 16 18 20 12 15
88 33 24 16 17 17 19 25 23 34
79 31 22 34 43 46 23 22 21 31
11 13 14 15 15 16 18 20 12 15
88 33 24 16 17 17 19 25 23 34
79 31 22 34 43 46 29 22 21 31
17 13 14 15 15 17 18 40 52 15
58 33 57 16 47 57 19 25 23 34
79 37 22 34 47 46 29 27 21 31
73 33 23 33 87 46 23 83 21 81

27. Supervised training
Label: tree
Machine Learning
Algorithm
Input Output
tower cat tree car can ball …
probability
tree
tower
cat
car
can
ball

28. Supervised training
Label: tree
Input Output
tower cat tree car can ball …
probability
tree
tower
cat
car
can
ball

29. Neural Networks
cs231n.github.io
W1
W2
Wn
bias
Input 1
Input 2
Input n
∑
Neuron-like unit
f()

30. Neural Networks
24 million nodes
140 million parameters
15 billion connections
*How we teach computers to understand pictures | Fei Fei Li

31. Supervised training
Label: tree
Input Output
tower cat tree car can ball …
probability
Mismatch between true labels and outputs →
Loss function (true labels, predictions)

32. Minimize loss function
deeplearning.ai | Andrew Ng
Gradient descent

33. Supervised training
Label: tree
Input Output
tower cat tree car can ball …
probability
output

34. Supervised training
Label: tree
Input Output
output

35. Supervised training
Label: tree
Input Output
tower cat tree car can ball …
probability
output

36. Supervised training
Label: tree
Input Output
output
tower cat tree car can ball …
probability

37. Supervised training
Label: tree
Input Output
output
tower cat tree car can ball …
probability

38. Supervised training – Deep learning
Label: tree
Input Output
dog cat tree car can ball …
probability
output

39. Supervised training – Deep learning
Label: tree
Input Output
output

40. Inference / validation
Unseen image
Input
cat 93 %
Output
dog cat tree car can ball …
probability

41. Test

42. PYTHON is a language of
Machine Learning

43. PYTHON deep learning frameworks

44. PYTHON frameworks
TORCHVISION
Albumentations

45. Easy-peasy pipeline
STEP 1. Get the data
CIFAR 10 Dataset
import torch
from torchvision.datasets import CIFAR10
from torch.utils.data import DataLoader
train_set = CIFAR10(root= './data’, train=True,
download=True)
test_set = CIFAR10(root= './data’, train=False,
download=True)
trainloader = DataLoader(train_set, batch_size=4,
shuffle=True, num_workers=2)
testloader = DataLoader(test_set, batch_size=4,
shuffle=False, num_workers=2)

46. Easy-peasy pipeline
STEP 2. Pre-process data
STEP 1. Get the data
Resize and augment
import cv2
image = cv2.imread('image.jpg’)
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
image = cv2.resize(image, (256, 256))

47. Augment image Albumentations
Horizontal Flip Crop Median Blur
Contrast Hue / Saturation / Value Gamma

48. Augment image Albumentations
Horizontal Flip Crop Median Blur
Contrast Hue / Saturation / Value Gamma
import cv2
import albumentations as A
transform = A.Compose([
A.RandomCrop(256, 256),
A.HorizontalFlip(p=0.5),
A.HueSaturationValue(p=0.2),
A.Normalize()
])
augmented = transform(image=image)
augmented_image = augmented[ 'image’ ]

49. Easy-peasy pipeline
STEP 1. Get the data
STEP 3. Setup model
from torchvision.models import models
model = models.resnet18(pretrained=True)
STEP 2. Pre-process data
STEP 4. Choose loss and optimizer
from torch import nn, optim
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

50. Easy-peasy pipeline
STEP 1. Get the data
STEP 3. Setup model
STEP 2. Pre-process data
STEP 4. Loss and optimizer
STEP 5. Train model
# loop over data multiple times (epochs)
for epoch in range(num_epochs):
epoch_losses = []
for data in trainloader:
# get inputs; data is a list of [inputs, labels]
inputs, labels = data
# zero the parameter gradients
optimizer.zero_grad()
# get model output, get criterion, optimize
outputs = model(inputs)
loss = criterion(outputs, labels)
loss.backward()
optimizer.step()
epoch_losses.append(loss.detach().cpu().numpy())
# print loss history
print("Epoch {}, Train Loss: {}“.format(epoch,
np.mean(epoch_losses)))
print("Finished Training“)

51. Easy-peasy pipeline
STEP 1. Get the data
STEP 3. Setup model
STEP 2. Pre-process data
STEP 4. Loss and optimizer
STEP 5. Train model
correct, total = 0, 0
model.eval()
with torch.no_grad():
# loop over test data
for data in testloader:
images, labels = data
# get model output
outputs = model(images)
# select most probable prediction
_, predicted = torch.max(outputs.data, 1)
total += labels.size(0)
correct += (predicted == labels).sum().item()
print("Accuracy: {}“.format(100 * correct / total))
STEP 6. Run inference
bird cat deer dog frog ship …
probability

52. Illusion

53. Illusion

54. Welcome
to the REAL world

55. Occlusions

56. Different poses

57. Intra-class variation

58. Illumination variations

60. Augmentation

61. Confusing images

62. What computer vision can not do
http://karpathy.github.io/2012/10/22/state-of-computer-vision/
Mirrors
Physics
3D structure of the scene
Status of identified people
What’s on peoples’ mind
Intentions
Humour

63. References:
In this presentation I used materials from:
“WALL-E” | Steven Spielberg
“Pulp fiction” | Quentin Tarantino
Imagenet, http://image-net.org
How we teach computers to understand pictures | Fei Fei Li
https://en.wikipedia.org/wiki/Neuron#/media/File:Blausen_0657_MultipolarNeuron.png
cs231n.github.io
deeplearning.ai | Andrew Ng
MIT 6.S094: Computer Vision | Lex Fridman
https://github.com/albu/albumentations
https://pytorch.org/docs/stable/torchvision/models.html
https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html
http://karpathy.github.io/2012/10/22/state-of-computer-vision/
MIT Deep Learning Basics: Introduction and Overview | Lex Fridman

64. Tatiana Gabruseva, PhD