Updated version of the Practical Guide to Deep Learning with some actual projects, and our process of working through deep learning projects - less end to end, more joining several models together

1. DEEP LEARNING IN PRACTICE
Tess Ferrandez – Microsoft - @TessFerrandez

2. TIM
MITYA
YANA
VITO
CLAUS
TESS

3. SHOTS ON GOAL

4. DETECTING CANCER

5. SHOPLIFTING

6. DEEP LEARNING
What’s so magical about it

9. int EstimatePrice(...){
price = 100000 +
67000 * area_in_sqm +
200000 * has_pool +
100000 * new_kitchen +
5000 * neighborhood_quality;
return price;
}
Price = b + w1*area_in_sqm + w2*has_pool + ...

10. [LINEAR REGRESSION]

13. GIVEN ENOUGH SAMPLES,
A NEURAL NETWORK WILL
FIND THE PATTERN

14. GIVEN ENOUGH SAMPLES,
A NEURAL NETWORK WILL
FIND THE PATTERN

15. SEQUENCESPACE
CONVOLUTIONAL NN RECURRENT NNDENSE NN

18. [ 0.01949719, 0.09399229, -0.01618082, -0.00876935, 0.03146157, 0.06853894, 0.00096175, -0.06854118, -0.04771797, -0.05296798, 0.02119147, 0.00511259, 0.13726683
INTERMEDIATE REPRESENTATIONEMBEDDINGSECRET CODE

19. RECOMMEND A BOOK

20. -1.0 1.0
ADULTYOUTH

21. ADULT
FICTION
(-0.6, 0.4) *
(0.7, 0.9) *

22. FICTION ADULT
MATH REFERENCES
US-CENTRIC
CHICK LIT
FUNNY
SCI-FI
LAWYERS
WOULD BRAD PITT PLAY A
CHARACTER IN THE MOVIE?
EMBEDDING
[ 0.01949719, 0.09399229, -0.01618082, -0.00876935, 0.03146157, 0.06853894, 0.00096175, -0.06854118, -0.04771797, -0.05296798, 0.02119147, 0.00511259, 0.1372668

23. WORD
EMBEDDINGS

24. [ 0.01949719, 0.09399229, -0.01618082, -0.00876935, 0.03146157, 0.06853894, 0.00096175, -0.06854118, -0.04771797, -0.05296798, 0.02119147, 0.00511259, 0.1372668

25. [FACENET]
T-SNE
Projection of
128D to 2D

26. FACE RECOGNITION DEMO

27. SEGMENTATION
encoder decoder

28. encoder decoder

29. encoder decoder

30. IN PRACTICE
The secrets behind the magic

37. Time for the Epoch
Training data
Validation data

38. MODEL LOSS ACCURACY
BASIC 0.2507 91.05%

40. OOPSIE DOOPSIE!
We’re overfitting

41. Chihuahua the movie
[DATA AUGMENTATION]

43. [DROPOUT]

45. MODEL LOSS ACCURACY
BASIC 0.2507 91.05%
AUGMENTATION 0.1988 93.68%

49. MODEL LOSS ACCURACY
BASIC 0.2507 91.05%
AUGMENTATION 0.1988 93.68%
TRANSFER LEARN 0.01253 99.47%

51. APPLIED MACHINE LEARNING
When the magic is gone, and we’re left with Software Engineering

52. UNDERSTAND THE BUSINESS NEEDS

53. WHAT IS THE PROBLEM?
HOW WILL THE MODEL BE
USED? / REQUIREMENTS
HOW IS IT DONE TODAY?
IS IT FEASIBLE?
ETHICAL CONCERNS
UNDERSTAND THE BUSINESS NEEDS

54. UNDERSTAND THE
BUSINESS NEEDS
MINE CLEAN EXPLORE

56. UNDERSTAND THE
BUSINESS NEEDS
MINE CLEAN EXPLORE
ENGINEER MODEL DEPLOY

57. LOTS OF LABLED SAMPLES and
NO CONSEQUENTIAL DECISIONS
SHOTS ON GOAL

58. LOUD CROWD
GOAL VISIBLE
SPEED/DIRECTION PLAYER DENSITY
PLAYER POSES

59. SCENE CHANGES
GOAL IN VIEW
NEGATIVE SAMPLING

60. 5S VIDEOS - AROUND ACTION
NEGATIVE SAMPLES FROM ATTACKS

61. VGG EMBEDDINGS

62. 90+%
MODEL
ACCURACY
GRASS?
GOAL?
SCENE CHANGE

63. ShotNoShot
https://github.com/tyiannak/pyAudioAnalysis
AUDIO

64. PEOPLE CLUSTERS - SIZES

65. https://github.com/fizyr/keras-retinanet
model_path =
'c:/Tess/source/vision_samples/models/resnet50_coco_best_v2.1.0.h5'
model = models.load_model(model_path, backbone_name='resnet50’)
image_path = 'C:/Tess/source/vision_samples/data/images/basket_image.jpg'
image = read_image_bgr(image_path)
image = preprocess_image(image)
image, scale = resize_image(image)
# process image
boxes, scores, labels = model.predict_on_batch(np.expand_dims(image,
axis=0))
from keras_retinanet import models
from keras_retinanet.utils.image import read_image_bgr, preprocess_image,
resize_image

66. GOAL / NO GOAL

67. SCENE CHANGE DETECTION

68. FOCUSED OPTICAL FLOW ON PLAYERS

69. DETECTING CANCER
VERY FEW POSITIVE SAMPLES
EXTREME ACCURACY NEEDS
POTENTIAL FOR BIAS

70. HARD TO DIFFERENTIATE
ONLY PARTIALLY LABLED
EXTREMELY LARGE IMAGES

71. COLOR SEGMENTATION

72. CONVEX HULL

73. SHOPLIFTING
VERY FEW POSITIVE SAMPLES
VERY FEW SAMPLES PER ACTION TYPE
VERY SENSITIVE TO BIAS

74. COVERED FACES
ALONE
MEN 20-40 HOODIES
SHOPLIFTING POSES
MEN 20-40 HOODIESCOVERED FACES
ALONE

75. 12:32:00CHRISTMAS
FISH EYE
DETECT not PREDICT HUD ARTIFACTS

76. NEGATIVE SAMPLES FROM SAME VIDEOS
PEOPLE SHOPPING

77. POSE DETECTION

78. BACKGROUND SUBTRACTION

79. CLASSIFICATION AT THE BOX LEVEL

80. A LITTLE
DOMAIN KNOWLEDGE
GOES A LONG WAY

81. KISSKeep it Simple …

83. DEEP LEARNING IN PRACTICE
Tess Ferrandez – Microsoft - @TessFerrandez