Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Ai powered images

PyConWeb 2019

  • Be the first to comment

  • Be the first to like this

Ai powered images

  1. 1. Using AI to Power Your Images Doug Sillars PyConWeb May 26, 2019
  2. 2. Contact Me: @DougSillars Doug.Sillars@gmail.com www.dougsillars.com Doug Sillars Freelance Developer Relations Performance Audits: Web/Native Workshops: Performance/Images/Video/AR/ML http://bit.ly/HighPerformanceAndroidApps
  3. 3. Images Dominate the Web HTTPArchive mobile websites 04/2019
  4. 4. Image Prep for the Web
  5. 5. Image Prep for the Web
  6. 6. Image Prep for the Web
  7. 7. Image Prep for the Web
  8. 8. Image Prep for the Web
  9. 9. Shortcuts for Image Prep?
  10. 10. Shortcuts for Image Prep? Ensure You are delivering the optimal image
  11. 11. Shortcuts: Taking screenshots of Photoshop
  12. 12. Shortcuts for Image Prep?
  13. 13. Images on the web named screen*shot 426,000 screenshots live on the web April 2019
  14. 14. Images on the web named screen*shot 106,000 sites with Screen Shots April 2019
  15. 15. Images on the web named screen*shot 5M sites tested 106,000 sites with Screen Shots
  16. 16. Images on the web named screen*shot 5M sites tested 106,000 sites with Screen Shots 2.1% of the web! =
  17. 17. Image Formats – Screenshots -> JPG https://dougsillars.com/2019/02/10/using-screenshots-in-production/ Sometimes our Shortcuts have lasting effects
  18. 18. Shortcuts for Image Prep?
  19. 19. Shortcuts for Image Prep?
  20. 20. Shortcuts for Image Prep? https://res.cloudinary.com/dougsillars/image/upload/c_crop,h_1000,w_1000,q_auto,f_auto/v1558503538/doug_headshot_alps _tfzado.jpg
  21. 21. Shortcuts for Image Prep? https://res.cloudinary.com/dougsillars/image/upload/c_crop,h_1000,w_1000,q_auto,f_auto/v1558503538/doug_headshot_alps _tfzado.jpg 120 KB
  22. 22. Shortcuts for Image Prep?
  23. 23. Shortcuts for Image Prep? https://res.cloudinary.com/dougsillars/image/upload/c_crop,h_1080,w_1000,q_auto,f_auto/ v1558504239/IMG_20160527_131042692_HDR_c21a3k.jpg
  24. 24. Shortcuts for Image Prep Manual crop
  25. 25. Shortcuts for Image Prep Manual crops are not a shortcut
  26. 26. Shortcuts for Image Prep: Detection of Objects https://res.cloudinary.com/dougsillars/image/upload/g_auto,c_crop,w_1000,h_1000/ v1558504239/IMG_20160527_131042692_HDR_c21a3k.jpg
  27. 27. Image Cropping in Production: Twitter
  28. 28. Image Cropping in Production: Twitter https://blog.twitter.com/engineering/en_us/topics/infrastructure/2018/Smart-Auto-Cropping-of-Images.html
  29. 29. Object Detection Using Machine Learning • ImageAI library • Tensorflow based • Need Image Model Library detector = ObjectDetection() detector.setModelTypeAsRetinaNet() detector.setModelPath( os.path.join(execution_path , "resnet50_coco_best_v2.0.1.h5")) detector.loadModel() detections = detector.detectObjectsFromImage(input_image=os.path.join(execution_path , "berwickontweed.jpg"), output_image_path=os.path.join(execution_path , "resnet.jpg")) resnet=plt.imread('resnet.jpg')
  30. 30. Object Detection Using Machine Learning
  31. 31. Object Detection Using Machine Learning
  32. 32. Object Detection with Machine Learning
  33. 33. Object Detection • Cropping
  34. 34. Object Detection • Cropping • Blurring objects
  35. 35. Object Detection • Cropping • Blurring objects • Object Removal http://bit.ly/bootsphoto
  36. 36. Object Detection • Cropping • Blurring objects • Object Removal • Alt Text Creation
  37. 37. Object Detection • Cropping • Blurring objects • Object Removal • Alt Text Creation Alt = “A Person and 2 dogs”
  38. 38. If the Object is not Modeled…
  39. 39. If the Object is not Modeled… •person : 99.72 •dog : 88.67 •dog : 96.92
  40. 40. …Try a new library • https://llama.vision
  41. 41. …Try a new library • https://llama.vision
  42. 42. …Or you can train a model
  43. 43. Google Object Detection
  44. 44. Google Object Detection: Model Creation
  45. 45. Google Object Detection: Model Creation
  46. 46. Google Object Detection: Model Creation
  47. 47. Google Object Detection: Model Creation
  48. 48. Train Your Model Precision: How often Model’s predictions were correct. In this case 1/6 is a false positive Recall: How often model found known objects. 1/6 is a missed prediction.
  49. 49. Deploy Model and Use
  50. 50. Deploy Model and Use
  51. 51. Model Results!!
  52. 52. Model Results!!
  53. 53. Image Fix 1: Cropping
  54. 54. Image Fix 1: Cropping
  55. 55. Image Fix 1: Cropping
  56. 56. Image Fix 2: Inpainting
  57. 57. Image Fix 2: Inpainting Create a mask of the area
  58. 58. Image Fix 2: Inpainting Create a mask of the area
  59. 59. Image Fix 2: Inpainting
  60. 60. Image Fix 2: Inpainting
  61. 61. InPaint: Works well on single color areas
  62. 62. InPaint: Works Not So well with lots of color
  63. 63. Tricking InPaint Remove “outlier” colors from the border Use the inside as inpainting area
  64. 64. Tricking InPaint Convert to Grayscale colors are 0 – 255 (1D) Kmeans to bucket the colors
  65. 65. Tricking InPaint
  66. 66. Tricking InPaint 10 most common gray colors found
  67. 67. Tricking InPaint 10 most common gray colors found Frequency
  68. 68. Tricking InPaint 1. Loop through Color list
  69. 69. Tricking InPaint 2. Ignore the White Center
  70. 70. Tricking InPaint 3. If Color is an outlier (lower than 25th percentile Higher than 75th) Mask and replace
  71. 71. Tricking InPaint
  72. 72. Tricking InPaint
  73. 73. Tricking InPaint
  74. 74. Sunglasses Project
  75. 75. Sunglasses Project
  76. 76. Manually edit 1,000s of glasses?
  77. 77. Sunglasses Project
  78. 78. Sunglasses Project
  79. 79. Sunglasses Project
  80. 80. Sunglasses Project 1. Remove Background
  81. 81. Sunglasses Project 1. Remove Background
  82. 82. Sunglasses Project 2. Make (slightly) transparent https://res.cloudinary.com/dougsillars/image /upload/e_make_transparent,o_70/v1558760 129/6658.png
  83. 83. Train a model!
  84. 84. Want to minimize frame capture
  85. 85. Train a model!
  86. 86. Beginning
  87. 87. Find the arms
  88. 88. Find the arms Finding an Odd number: Fail
  89. 89. Modify the border around the arms
  90. 90. Inpaint and upload
  91. 91. Putting it all together https://res.cloudinary.com/dougsillars/image/upload/v1558855691/IMG_20181207_163533_ibfm7a.jpg
  92. 92. Putting it all together https://res.cloudinary.com/dougsillars/image/upload/c_crop,g_face,w_1500,h_1500/v1558855691/IMG_2018120 7_163533_ibfm7a.jpg
  93. 93. Putting it all together https://res.cloudinary.com/dougsillars/image/upload/c_crop,g_face,w_1500,h_1500/v1558855691/IMG_2018120 7_163533_ibfm7a.jpg l_6558_f,o_80,e_make_transparent/
  94. 94. Putting it all together https://res.cloudinary.com/dougsillars/image/upload/c_crop,g_face,w_1500,h_1500/l_6558_f,o_80,e_make_transparent/ g_adv_eyes,fl_layer_apply/v1558855691/IMG_20181207_163533_ibfm7a.jpg
  95. 95. Demo Time https://res.cloudinary.com/dougsillars/image/upload/c_crop,g_face,w_1500,h_1500/l_6558_f,o_80,e_make_transparent/ g_adv_eyes,fl_layer_apply/v1558855691/IMG_20181207_163533_ibfm7a.jpg
  96. 96. Conclusion • Image processing with AI and ML can simplify image prep for the web • Cropping • Blurring • Object removal • Alt text
  97. 97. Love Building with Video and Images? Media Developer Expert mde-comm@cloudinary.com Become a

×