Doug Sillars discusses using AI and machine learning to simplify image preparation for the web. He describes how object detection can be used for cropping, blurring objects, object removal, and generating alt text. Sillars also provides examples of using these techniques like detecting and adding sunglasses to images. He concludes that image processing with AI and ML can automate tasks like cropping, blurring, object removal, and alt text generation for image optimization.

1. Using AI to Power Your Images
Doug Sillars
Python Hrvatska
February 11, 2020

2. Contact Me:
@DougSillars
Doug.Sillars@gmail.com
www.dougsillars.com
Doug Sillars
Freelance Developer Relations
Performance Audits: Web/Native
Workshops:
Performance/Images/Video/AR/ML
http://bit.ly/HighPerformanceAndroidApps

3. Images Dominate the Web
HTTPArchive mobile websites 04/2019

4. Image Prep for the Web

5. Image Prep for the Web

6. Image Prep for the Web

7. Image Prep for the Web

8. Image Prep for the Web

9. Shortcuts for Image Prep?

10. Shortcuts for Image Prep?
Ensure You are delivering the optimal image

11. Shortcuts: Taking screenshots of Photoshop

12. Shortcuts for Image Prep?

13. Images on the web named screen*shot
426,000 screenshots live on the web
April 2019

14. Images on the web named screen*shot
106,000 sites with Screen Shots
April 2019

15. Images on the web named screen*shot
5M sites tested
106,000 sites with Screen Shots

16. Images on the web named screen*shot
5M sites tested
106,000 sites with Screen Shots 2.1%
of the
web!
=

17. Image Formats – Screenshots -> JPG
https://dougsillars.com/2019/02/10/using-screenshots-in-production/
Sometimes our Shortcuts have lasting effects

18. Shortcuts for Image Prep?

19. Shortcuts for Image Prep?

20. Shortcuts for Image Prep?
https://res.cloudinary.com/dougsillars/image/upload/c_crop,h_1000,w_1000,q_auto,f_auto/v1558503538/doug_headshot_alps
_tfzado.jpg

21. Shortcuts for Image Prep?
https://res.cloudinary.com/dougsillars/image/upload/c_crop,h_1000,w_1000,q_auto,f_auto/v1558503538/doug_headshot_alps
_tfzado.jpg
120 KB

22. Shortcuts for Image Prep?

23. Shortcuts for Image Prep?
https://res.cloudinary.com/dougsillars/image/upload/c_crop,h_1080,w_1000,q_auto,f_auto/
v1558504239/IMG_20160527_131042692_HDR_c21a3k.jpg

24. Shortcuts for Image Prep
Manual crop

25. Shortcuts for Image Prep
Manual crops are not a shortcut

26. Shortcuts for Image Prep: Detection of Objects
https://res.cloudinary.com/dougsillars/image/upload/g_auto,c_crop,w_1000,h_1000/
v1558504239/IMG_20160527_131042692_HDR_c21a3k.jpg

27. Image Cropping in Production: Twitter

28. Image Cropping in Production: Twitter
https://blog.twitter.com/engineering/en_us/topics/infrastructure/2018/Smart-Auto-Cropping-of-Images.html

29. Object Detection Using Machine Learning
• ImageAI library
• Tensorflow based
• Need Image Model Library
detector = ObjectDetection()
detector.setModelTypeAsRetinaNet()
detector.setModelPath( os.path.join(execution_path , "resnet50_coco_best_v2.0.1.h5"))
detector.loadModel()
detections = detector.detectObjectsFromImage(input_image=os.path.join(execution_path , "berwickontweed.jpg"),
output_image_path=os.path.join(execution_path , "resnet.jpg"))
resnet=plt.imread('resnet.jpg')
https://medium.com/deepquestai/train-object-detection-ai-with-6-lines-of-code-6d087063f6ff

30. Object Detection Using Machine Learning

31. Object Detection Using Machine Learning

32. Object Detection with Machine Learning

33. Object Detection
• Cropping

34. Object Detection
• Cropping
• Blurring objects

35. Object Detection
• Cropping
• Blurring objects
• Object Removal
http://bit.ly/bootsphoto

36. Object Detection
• Cropping
• Blurring objects
• Object Removal
• Alt Text Creation

37. Object Detection
• Cropping
• Blurring objects
• Object Removal
• Alt Text Creation
Alt = “A Person and 2 dogs”

38. If the Object is not Modeled…

39. If the Object is not Modeled…
•person : 99.72
•dog : 88.67
•dog : 96.92

40. …Try a new library
• https://llama.vision

41. …Try a new library
• https://llama.vision

42. Object Detection Using Machine Learning

43. ML ALT text in production
alt="Image may contain:
bridge, sky, tree,
outdoor, water and
nature"

44. ML ALT text in production
alt="Image may
contain: dog"

45. ML ALT text in production

46. …Or you can train a model

47. Google Object Detection

48. Google Object Detection: Model Creation

49. Google Object Detection: Model Creation

50. Google Object Detection: Model Creation

51. Google Object Detection: Model Creation

52. Train Your Model
Precision: How often Model’s
predictions were correct. In this case
1/6 is a false positive
Recall: How often model found
known objects. 1/6 is a missed
prediction.

53. Deploy Model and Use

54. Deploy Model and Use

55. Model Results!!

56. Model Results!!

57. Image Fix 1: Cropping

58. Image Fix 1: Cropping

59. Image Fix 1: Cropping

60. Image Fix 2: Inpainting

61. Image Fix 2: Inpainting
Create a mask of the area

62. Image Fix 2: Inpainting
Create a mask of the area

63. Image Fix 2: Inpainting

64. Image Fix 2: Inpainting

65. InPaint: Works well on single color areas

66. InPaint: Works Not So well with lots of color

67. Tricking InPaint
Remove “outlier” colors from
the border
Use the inside as inpainting area

68. Tricking InPaint
Convert to Grayscale
colors are 0 – 255 (1D)
Kmeans to bucket the colors

69. Tricking InPaint

70. Tricking InPaint
10 most common gray colors found

71. Tricking InPaint
10 most common gray colors found
Frequency

72. Tricking InPaint 1. Loop through Color list

73. Tricking InPaint 2. Ignore the White Center

74. Tricking InPaint
3. If Color is an outlier
(lower than 25th percentile
Higher than 75th)
Mask and replace

75. Tricking InPaint

76. Tricking InPaint

77. Tricking InPaint

79. Sunglasses Project

80. Sunglasses Project

81. Manually edit 1,000s of glasses?

82. Sunglasses Project

83. Sunglasses Project

84. Sunglasses Project

85. Sunglasses Project
1. Remove Background

86. Sunglasses Project
1. Remove Background

87. Sunglasses Project
2. Make (slightly) transparent
https://res.cloudinary.com/dougsillars/image
/upload/e_make_transparent,o_70/v1558760
129/6658.png

88. Train a model!

89. Want to minimize frame capture

90. Train a model!

91. Beginning

92. Find the arms

93. Find the arms
Finding an Odd number: Fail

94. Modify the border around the arms

95. Inpaint and upload

96. Putting it all together
https://res.cloudinary.com/dougsillars/image/upload/v1558855691/IMG_20181207_163533_ibfm7a.jpg

97. Putting it all together
https://res.cloudinary.com/dougsillars/image/upload/c_crop,g_face,w_1500,h_1500/v1558855691/IMG_2018120
7_163533_ibfm7a.jpg

98. Putting it all together
https://res.cloudinary.com/dougsillars/image/upload/c_crop,g_face,w_1500,h_1500/v1558855691/IMG_2018120
7_163533_ibfm7a.jpg
l_6558_f,o_80,e_make_transparent/

99. Putting it all together
https://res.cloudinary.com/dougsillars/image/upload/c_crop,g_face,w_1500,h_1500/l_6558_f,o_80,e_make_transparent/
g_adv_eyes,fl_layer_apply/v1558855691/IMG_20181207_163533_ibfm7a.jpg

100. Demo Time
https://res.cloudinary.com/dougsillars/image/upload/c_crop,g_face,w_1500,h_1500/l_6558_f,o_80,e_make_transparent/
g_adv_eyes,fl_layer_apply/v1558855691/IMG_20181207_163533_ibfm7a.jpg

101. Conclusion
• Image processing with AI and ML can simplify image prep for the web
• Cropping
• Blurring
• Object removal
• Alt text

102. Love Building with Video and Images?
Media Developer Expert
mde-comm@cloudinary.com
Become a