Doug Sillars discusses using AI and machine learning to simplify image preparation for the web. He covers using object detection models to automatically crop images, blur private objects, and generate alt text. Sillars also demonstrates training custom models, such as one to add sunglasses to faces. With AI, tasks like removing backgrounds, making transparent overlays, and combining elements can be automated to save time in image editing workflows.

1. Using AI to Power Your Images
Doug Sillars
APIs and IPAs
Seattle
June 27, 2019

2. Contact Me:
@DougSillars
Doug.Sillars@gmail.com
www.dougsillars.com
Doug Sillars
Freelance Developer Relations
Performance Audits: Web/Native
Workshops:
Performance/Images/Video/AR/ML
http://bit.ly/HighPerformanceAndroidApps

3. Images Dominate the Web
HTTPArchive mobile websites 04/2019

4. Image Prep for the Web

5. Image Prep for the Web

6. Image Prep for the Web

7. Image Prep for the Web

8. Image Prep for the Web

9. Shortcuts for Image Prep?

10. Shortcuts for Image Prep?
Ensure You are delivering the optimal image

11. Shortcuts: Taking screenshots of Photoshop

12. Shortcuts for Image Prep?

13. Images on the web named screen*shot
426,000 screenshots live on the web
April 2019

14. Images on the web named screen*shot
106,000 sites with Screen Shots
April 2019

15. Images on the web named screen*shot
5M sites tested
106,000 sites with Screen Shots

16. Images on the web named screen*shot
5M sites tested
106,000 sites with Screen Shots 2.1%
of the
web!
=

17. Image Formats – Screenshots -> JPG
https://dougsillars.com/2019/02/10/using-screenshots-in-production/
Sometimes our Shortcuts have lasting effects

18. Shortcuts for Image Prep?

19. Shortcuts for Image Prep?

20. Shortcuts for Image Prep?
https://res.cloudinary.com/dougsillars/image/upload/c_crop,h_1000,w_1000,q_auto,f_auto/v1558503538/doug_headshot_alps
_tfzado.jpg

21. Shortcuts for Image Prep?
https://res.cloudinary.com/dougsillars/image/upload/c_crop,h_1000,w_1000,q_auto,f_auto/v1558503538/doug_headshot_alps
_tfzado.jpg
120 KB

22. Shortcuts for Image Prep?

23. Shortcuts for Image Prep?
https://res.cloudinary.com/dougsillars/image/upload/c_crop,h_1080,w_1000,q_auto,f_auto/
v1558504239/IMG_20160527_131042692_HDR_c21a3k.jpg

24. Shortcuts for Image Prep
Manual crop

25. Shortcuts for Image Prep
Manual crops are not a shortcut

26. Shortcuts for Image Prep: Detection of Objects
https://res.cloudinary.com/dougsillars/image/upload/g_auto,c_crop,w_1000,h_1000/
v1558504239/IMG_20160527_131042692_HDR_c21a3k.jpg

27. Image Cropping in Production: Twitter

28. Image Cropping in Production: Twitter
https://blog.twitter.com/engineering/en_us/topics/infrastructure/2018/Smart-Auto-Cropping-of-Images.html

29. Object Detection Using Machine Learning
• ImageAI library
• Tensorflow based
• Need Image Model Library
detector = ObjectDetection()
detector.setModelTypeAsRetinaNet()
detector.setModelPath( os.path.join(execution_path , "resnet50_coco_best_v2.0.1.h5"))
detector.loadModel()
detections = detector.detectObjectsFromImage(input_image=os.path.join(execution_path , "berwickontweed.jpg"),
output_image_path=os.path.join(execution_path , "resnet.jpg"))
resnet=plt.imread('resnet.jpg')

30. Object Detection Using Machine Learning

31. Object Detection Using Machine Learning

32. Object Detection with Machine Learning

33. Object Detection
• Cropping

34. Object Detection
• Cropping
• Blurring objects

35. Object Detection
• Cropping
• Blurring objects
• Object Removal
http://bit.ly/bootsphoto

36. Object Detection
• Cropping
• Blurring objects
• Object Removal
• Alt Text Creation

37. Object Detection
• Cropping
• Blurring objects
• Object Removal
• Alt Text Creation
Alt = “A Person and 2 dogs”

38. If the Object is not Modeled…

39. If the Object is not Modeled…
•person : 99.72
•dog : 88.67
•dog : 96.92

40. …Try a new library
• https://llama.vision

41. …Try a new library
• https://llama.vision

42. …Or you can train a model

43. Google Object Detection

44. Google Object Detection: Model Creation

45. Google Object Detection: Model Creation

46. Google Object Detection: Model Creation

47. Google Object Detection: Model Creation

48. Train Your Model
Precision: How often Model’s
predictions were correct. In this case
1/6 is a false positive
Recall: How often model found
known objects. 1/6 is a missed
prediction.

49. Deploy Model and Use

50. Deploy Model and Use

51. Model Results!!

52. Model Results!!

53. Image Fix 1: Cropping

54. Image Fix 1: Cropping

55. Image Fix 1: Cropping

56. Image Fix 2: Inpainting

57. Image Fix 2: Inpainting
Create a mask of the area

58. Image Fix 2: Inpainting
Create a mask of the area

59. Image Fix 2: Inpainting

60. Image Fix 2: Inpainting

61. InPaint: Works well on single color areas

62. InPaint: Works Not So well with lots of color

63. Tricking InPaint
Remove “outlier” colors from
the border
Use the inside as inpainting area

64. Tricking InPaint
Convert to Grayscale
colors are 0 – 255 (1D)
Kmeans to bucket the colors

65. Tricking InPaint

66. Tricking InPaint
10 most common gray colors found

67. Tricking InPaint
10 most common gray colors found
Frequency

68. Tricking InPaint 1. Loop through Color list

69. Tricking InPaint 2. Ignore the White Center

70. Tricking InPaint
3. If Color is an outlier
(lower than 25th percentile
Higher than 75th)
Mask and replace

71. Tricking InPaint

72. Tricking InPaint

73. Tricking InPaint

75. Sunglasses Project

76. Sunglasses Project

77. Manually edit 1,000s of glasses?

78. Sunglasses Project

79. Sunglasses Project

80. Sunglasses Project

81. Sunglasses Project
1. Remove Background

82. Sunglasses Project
1. Remove Background

83. Sunglasses Project
2. Make (slightly) transparent
https://res.cloudinary.com/dougsillars/image
/upload/e_make_transparent,o_70/v1558760
129/6658.png

84. Train a model!

85. Want to minimize frame capture

86. Train a model!

87. Beginning

88. Find the arms

89. Find the arms
Finding an Odd number: Fail

90. Modify the border around the arms

91. Inpaint and upload

92. Putting it all together
https://res.cloudinary.com/dougsillars/image/upload/v1558855691/IMG_20181207_163533_ibfm7a.jpg

93. Putting it all together
https://res.cloudinary.com/dougsillars/image/upload/c_crop,g_face,w_1500,h_1500/v1558855691/IMG_2018120
7_163533_ibfm7a.jpg

94. Putting it all together
https://res.cloudinary.com/dougsillars/image/upload/c_crop,g_face,w_1500,h_1500/v1558855691/IMG_2018120
7_163533_ibfm7a.jpg
l_6558_f,o_80,e_make_transparent/

95. Putting it all together
https://res.cloudinary.com/dougsillars/image/upload/c_crop,g_face,w_1500,h_1500/l_6558_f,o_80,e_make_transparent/
g_adv_eyes,fl_layer_apply/v1558855691/IMG_20181207_163533_ibfm7a.jpg

96. Demo Time
https://res.cloudinary.com/dougsillars/image/upload/c_crop,g_face,w_1500,h_1500/l_6558_f,o_80,e_make_transparent/
g_adv_eyes,fl_layer_apply/v1558855691/IMG_20181207_163533_ibfm7a.jpg

97. Conclusion
• Image processing with AI and ML can simplify image prep for the web
• Cropping
• Blurring
• Object removal
• Alt text

98. Love Building with Video and Images?
Media Developer Expert
mde-comm@cloudinary.com
Become a