Doug Sillars discusses using AI to optimize images for the web. He notes that images dominate web content and presents techniques like automatic cropping, object detection for cropping or blurring parts of images, and using machine learning to generate alt text for images. Sillars demonstrates applying these techniques to sample images, such as automatically adding sunglasses. He concludes that image processing with AI and ML can simplify tasks like image cropping, redaction, and accessibility.

1. Using AI to Power Your Images
Doug Sillars
Frankfurt AI
August 8, 2019

2. Contact Me:
@DougSillars
Doug.Sillars@gmail.com
www.dougsillars.com
Doug Sillars
Freelance Developer Relations
Performance Audits: Web/Native
Workshops:
Performance/Images/Video/AR/ML
http://bit.ly/HighPerformanceAndroidApps

3. Images Dominate the Web
HTTPArchive mobile websites 04/2019

4. Image Prep for the Web

5. Image Prep for the Web

6. Image Prep for the Web

7. Image Prep for the Web

8. Image Prep for the Web

9. Shortcuts for Image Prep?

10. Shortcuts for Image Prep?
Ensure You are delivering the optimal image

11. Shortcuts: Taking screenshots of Photoshop

12. Shortcuts for Image Prep?

13. Images on the web named screen*shot
426,000 screenshots live on the web
April 2019

14. Images on the web named screen*shot
106,000 sites with Screen Shots
April 2019

15. Images on the web named screen*shot
5M sites tested
106,000 sites with Screen Shots

16. Images on the web named screen*shot
5M sites tested
106,000 sites with Screen Shots 2.1%
of the
web!
=

17. Image Formats – Screenshots -> JPG
https://dougsillars.com/2019/02/10/using-screenshots-in-production/
Sometimes our Shortcuts have lasting effects

18. Shortcuts for Image Prep?

19. Shortcuts for Image Prep?

20. Shortcuts for Image Prep?
https://res.cloudinary.com/dougsillars/image/upload/c_crop,h_1000,w_1000,q_auto,f_auto/v1558503538/doug_headshot_alps
_tfzado.jpg

21. Shortcuts for Image Prep?
https://res.cloudinary.com/dougsillars/image/upload/c_crop,h_1000,w_1000,q_auto,f_auto/v1558503538/doug_headshot_alps
_tfzado.jpg
120 KB

22. Shortcuts for Image Prep?

23. Shortcuts for Image Prep?
https://res.cloudinary.com/dougsillars/image/upload/c_crop,h_1080,w_1000,q_auto,f_auto/
v1558504239/IMG_20160527_131042692_HDR_c21a3k.jpg

24. Shortcuts for Image Prep
Manual crop

25. Shortcuts for Image Prep
Manual crops are not a shortcut

26. Shortcuts for Image Prep: Detection of Objects
https://res.cloudinary.com/dougsillars/image/upload/g_auto,c_crop,w_1000,h_1000/
v1558504239/IMG_20160527_131042692_HDR_c21a3k.jpg

27. Image Cropping in Production: Twitter

28. Image Cropping in Production: Twitter
https://blog.twitter.com/engineering/en_us/topics/infrastructure/2018/Smart-Auto-Cropping-of-Images.html

29. Object Detection Using Machine Learning
• ImageAI library
• Tensorflow based
• Need Image Model Library
detector = ObjectDetection()
detector.setModelTypeAsRetinaNet()
detector.setModelPath( os.path.join(execution_path , "resnet50_coco_best_v2.0.1.h5"))
detector.loadModel()
detections = detector.detectObjectsFromImage(input_image=os.path.join(execution_path , "berwickontweed.jpg"),
output_image_path=os.path.join(execution_path , "resnet.jpg"))
resnet=plt.imread('resnet.jpg')

30. Object Detection Using Machine Learning

31. Object Detection Using Machine Learning

32. Object Detection with Machine Learning

33. Object Detection
• Cropping

34. Object Detection
• Cropping
• Blurring objects

35. Object Detection
• Cropping
• Blurring objects
• Object Removal
http://bit.ly/bootsphoto

36. Object Detection
• Cropping
• Blurring objects
• Object Removal
• Alt Text Creation

37. Object Detection
• Cropping
• Blurring objects
• Object Removal
• Alt Text Creation
Alt = “A Person and 2 dogs”

38. If the Object is not Modeled…

39. If the Object is not Modeled…
•person : 99.72
•dog : 88.67
•dog : 96.92

40. …Try a new library
• https://llama.vision

41. …Try a new library
• https://llama.vision

42. ML ALT text in production

43. ML ALT text in production
alt="Image may contain: sky, cloud, ocean, outdoor and
nature"

44. …Or you can train a model

45. Google Object Detection

46. Google Object Detection: Model Creation

47. Google Object Detection: Model Creation

48. Google Object Detection: Model Creation

49. Google Object Detection: Model Creation

50. Train Your Model
Precision: How often Model’s
predictions were correct. In this case
1/6 is a false positive
Recall: How often model found
known objects. 1/6 is a missed
prediction.

51. Deploy Model and Use

52. Deploy Model and Use

53. Model Results!!

54. Model Results!!

55. Image Fix 1: Cropping

56. Image Fix 1: Cropping

57. Image Fix 1: Cropping

58. Image Fix 2: Inpainting

59. Image Fix 2: Inpainting
Create a mask of the area

60. Image Fix 2: Inpainting
Create a mask of the area

61. Image Fix 2: Inpainting

62. Image Fix 2: Inpainting

63. InPaint: Works well on single color areas

64. InPaint: Works Not So well with lots of color

65. Tricking InPaint
Remove “outlier” colors from
the border
Use the inside as inpainting area

66. Tricking InPaint
Convert to Grayscale
colors are 0 – 255 (1D)
Kmeans to bucket the colors

67. Tricking InPaint

68. Tricking InPaint
10 most common gray colors found

69. Tricking InPaint
10 most common gray colors found
Frequency

70. Tricking InPaint 1. Loop through Color list

71. Tricking InPaint 2. Ignore the White Center

72. Tricking InPaint
3. If Color is an outlier
(lower than 25th percentile
Higher than 75th)
Mask and replace

73. Tricking InPaint

74. Tricking InPaint

75. Tricking InPaint

77. Sunglasses Project

78. Sunglasses Project

79. Manually edit 1,000s of glasses?

80. Sunglasses Project

81. Sunglasses Project

82. Sunglasses Project

83. Sunglasses Project
1. Remove Background

84. Sunglasses Project
1. Remove Background

85. Sunglasses Project
2. Make (slightly) transparent
https://res.cloudinary.com/dougsillars/image
/upload/e_make_transparent,o_70/v1558760
129/6658.png

86. Train a model!

87. Want to minimize frame capture

88. Train a model!

89. Beginning

90. Find the arms

91. Find the arms
Finding an Odd number: Fail

92. Modify the border around the arms

93. Inpaint and upload

94. Putting it all together
https://res.cloudinary.com/dougsillars/image/upload/v1558855691/IMG_20181207_163533_ibfm7a.jpg

95. Putting it all together
https://res.cloudinary.com/dougsillars/image/upload/c_crop,g_face,w_1500,h_1500/v1558855691/IMG_2018120
7_163533_ibfm7a.jpg

96. Putting it all together
https://res.cloudinary.com/dougsillars/image/upload/c_crop,g_face,w_1500,h_1500/v1558855691/IMG_2018120
7_163533_ibfm7a.jpg
l_6558_f,o_80,e_make_transparent/

97. Putting it all together
https://res.cloudinary.com/dougsillars/image/upload/c_crop,g_face,w_1500,h_1500/l_6558_f,o_80,e_make_transparent/
g_adv_eyes,fl_layer_apply/v1558855691/IMG_20181207_163533_ibfm7a.jpg

98. Demo Time
https://res.cloudinary.com/dougsillars/image/upload/c_crop,g_face,w_1500,h_1500/l_6558_f,o_80,e_make_transparent/
g_adv_eyes,fl_layer_apply/v1558855691/IMG_20181207_163533_ibfm7a.jpg

99. Conclusion
• Image processing with AI and ML can simplify image prep for the web
• Cropping
• Blurring
• Object removal
• Alt text

100. Love Building with Video and Images?
Media Developer Expert
mde-comm@cloudinary.com
Become a