Automatic Image Cropping - A journey from a Master Thesis to Production
•
1 like•176 views
The document discusses developing a neural network model for automatic image cropping. It proposes that properly cropped images can improve listing performance on online classifieds sites. The model uses a Deeply Supervised Salient Network (DSS) which improves on fully convolutional networks with deep supervision and short connections. Experiments were conducted on an online classifieds site by manually evaluating cropped images in different categories. The best performing category of engagement rings was selected for initial deployment. The system architecture includes components for image enhancement, cropping, hosting, and a frontend interface.
More Related Content
Similar to Automatic Image Cropping - A journey from a Master Thesis to Production
Similar to Automatic Image Cropping - A journey from a Master Thesis to Production (20)
More from Alexey Grigorev
More from Alexey Grigorev (20)
Recently uploaded
Recently uploaded (20)
Automatic Image Cropping - A journey from a Master Thesis to Production
- 1. A journey from a Master Thesis to Production Alexey Grigorev 11/12/2019 Automatic Image Cropping
- 2. ● Aishwarya Manjunath Shetty (Naspers, TU/e) ○ The master thesis student who did all the work ● Selçuk Sandıkcı (Naspers) ● Dmitri Jarnikov (Naspers, TU/e) ● Sandor Akszenovics (OLX Group) ● Carmine Paolino (OLX Group) Credit
- 3. ● Software Engineer in the past ● Masters in BI ● Doing ML for 6 years, now Data Scientist at OLX Group About me
- 4. Plan ● Cropping in online classifieds ● Saliency detection ● Neural networks for saliency detection ● Experiments ● Deployment
- 10. Hypothesis ● Properly cropped image ⇒ ● Better visibility in home feed and search results ⇒ ● Better overall listing quality ⇒ ● Better overall listing performance Properly cropped image ⇒ Good listing performance
- 11. Plan ● Cropping in online classifieds ● Saliency detection ● Neural networks for saliency detection ● Experiments ● Deployment
- 13. Saliency Detection 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.8 1 0.8 0 0 0 0.8 0.9 0.3 0 0 0 0 0 0 0 0 0 0 0 0 0 Pixels P(pixel is salient)
- 14. Image cropping
- 15. Background removal (let’s pretend it’s good)
- 16. Plan ● Cropping in online classifieds ● Saliency detection ● Neural networks for saliency detection ● Experiments ● Deployment
- 19. CNN → FC Replace with Conv Layer
- 20. Fully Convolutional Networks for Semantic Segmentation https://people.eecs.berkeley.edu/~jonlong/long_shelhamer_fcn.pdf FC network
- 21. FC network Fully Convolutional Networks for Semantic Segmentation https://people.eecs.berkeley.edu/~jonlong/long_shelhamer_fcn.pdf
- 22. There’s some spatial info as well
- 23. Deeply supervised salient object detection with short connections https://arxiv.org/abs/1611.04849 DSS network
- 24. Deeply supervised salient object detection with short connections https://arxiv.org/abs/1611.04849 DSS network CNN (e.g. VGG)
- 25. Deeply supervised salient object detection with short connections https://arxiv.org/abs/1611.04849 DSS network
- 26. Deeply supervised salient object detection with short connections https://arxiv.org/abs/1611.04849 DSS network Deep supervision
- 27. Deeply supervised salient object detection with short connections https://arxiv.org/abs/1611.04849 DSS network Short connections
- 28. Deeply supervised salient object detection with short connections https://arxiv.org/abs/1611.04849 DSS network Short connections
- 30. DSS Examples
- 31. DSS Examples
- 32. DSS Examples
- 33. DSS Examples
- 34. Plan ● Cropping in online classifieds ● Saliency detection ● Neural networks for saliency detection ● Experiments ● Deployment
- 35. Data driven! Plan: ● Use OLX.PT for the experiment ● Pick 200 images from each category ● Manually evaluate them ● Select the best one ● Go live there!
- 36. Experiments
- 37. Experiments
- 39. Evaluation results ● Good: 90% and above ○ Animals (dogs, horses, cats); cars, motorcycles; shoes ● 80%-90% ○ Electronics, cellphones, clothing ● Bad: 50% or worse ○ Real estate (offices, shops, garages, apartments)
- 40. Selecting the category ● Cars: too important ● Animals: unclear business opportunity ● Fashion: low liquidity, good cropping quality ⇒ Fashion
- 41. Plan ● Cropping in online classifieds ● Saliency detection ● Neural networks for saliency detection ● Experiments ● Deployment
- 43. Cropping service Cropper 204 No Content
- 44. Engagement rings
- 45. Engagement rings
- 71. Summary ● Proper cropping ⇒ Good performance
- 72. Summary ● Proper cropping ⇒ Good performance ● Saliency detection can be used for cropping
- 73. Summary ● Proper cropping ⇒ Good performance ● Saliency detection can be used for cropping ● DSS improve FC networks with: ○ Deep supervision: refining the location info at every granularity ○ Short connections: helping earlier levels with info from later
- 74. Summary ● Proper cropping ⇒ Good performance ● Saliency detection can be used for cropping ● DSS improve FC networks with: ○ Deep supervision: refining the location info at every granularity ○ Short connections: helping earlier levels with info from later ● Be data driven: select the best performing and most impactful category
- 75. Summary ● Proper cropping ⇒ Good performance ● Saliency detection can be used for cropping ● DSS improve FC networks with: ○ Deep supervision: refining the location info at every granularity ○ Short connections: helping earlier levels with info from later ● Be data driven: select the best performing and most impactful category ● Slack is a great way to sneak peak and validate the performance
- 76. Summary ● Proper cropping ⇒ Good performance ● Saliency detection can be used for cropping ● DSS improve FC networks with: ○ Deep supervision: refining the location info at every granularity ○ Short connections: helping earlier levels with info from later ● Be data driven: select the best performing and most impactful category ● Slack is a great way to sneak peak and validate the performance ● Kubernetes takes care of scaling
- 77. Summary ● Proper cropping ⇒ Good performance ● Saliency detection can be used for cropping ● DSS improve FC networks with: ○ Deep supervision: refining the location info at every granularity ○ Short connections: helping earlier levels with info from later ● Be data driven: select the best performing and most impactful category ● Slack is a great way to sneak peak and validate the performance ● Kubernetes takes care of scaling ● Let it crash - and let the infra restart the service
- 78. * * work in progress
- 79. Contact information ● http://alexeygrigorev.com & contact@alexeygrigorev.com ● https://github.com/alexeygrigorev ● https://www.linkedin.com/in/agrigorev ● https://www.slideshare.net/AlexeyGrigorev
- 80. Feedback (and link to the slides!) https://forms.gle/QUU46Pf48iStVhWL8
- 81. Questions?