Download to read offline
Uploaded byStuart Myles
Image Tagging at the Associated Press
Stuart Myles, director of information management at the Associated Press, discusses image recognition technology that identifies various subjects in images, requiring extensive example images for accurate pattern recognition. The AP aims to enhance search and auto-publishing efficiency, handling thousands of images daily and applying metadata through multiple methods, though initial efforts in 2017 faced challenges with existing tagging solutions. A hybrid approach combining off-the-shelf models with custom tagging has been adopted to improve accuracy and relevance in image tagging.
More Related Content
Similar to Image Tagging at the Associated Press
![[AWS Media Symposium 2019] Enhancing your Media Workflows with Amazon Machine...](https://cdn.slidesharecdn.com/ss_thumbnails/09enhancingyourmediaworkflowswithamazonmachinelearningshwetajain-191029011300-thumbnail.jpg?width=640&height=640&fit=bounds)
More from Stuart Myles
Image Tagging at the Associated Press
- 1. Stuart Myles Director ofInformation Management at the Associated Press
- 2. What is ImageRecognition? • Technology to recognize people, places, things or emotions in an image • Available as APIs, as well as open source software • Image recognition involves building a model to identify a set of topics • Topics can be anything you want - baseball, happy faces, drug use, war… • Requires lots of example images, so the software works out what patterns to look for • Consumers of image recognition services are often stock agencies • Off the shelf models are therefore available for concepts like • Graphic/NSFW • Celebrities • Emotion • General keywords – wedding, food • Many commercial companies also offer to create custom models – for a higher fee @smyles
- 3. Improve Search andAuto-Publishing • Improve search experience • In AP portals and in customer CMSes • Keywords to match more queries and so surface more content • Filters to narrow results to more relevant content • Simplify auto publishing • AP customers have fewer – or even no – editors to manually review content before publishing • Filters let customers fine-tune saved searches • Eliminate customer need to manually identify graphic content @smyles
- 4. AP Images Metadata •AP handles 3,000 – 4,000 images a day • Digitize 700 – 800 photos a month • AP Images has about 34 million photos • AP already applies metadata to images • Manually by photographers and editors • Mapped from third party feeds • Automatically based on photo text – such as caption – via AP’s tagging service • We manually keyword some archive images @smyles
- 5. Early Days: FirstHalf of 2017 • Early in 2017, we evaluated leading vendors • None offered custom tagging • Disappointing results • Too many keywords that do not apply to image • Inaccurate keywords scored with high confidence • Some were strong for stock images, not so good for news • Others were too generic @smyles
- 6.
- 7.
- 8. Technology Evolves: SecondHalf of 2017 • We evaluated open source software • Future option, but the software isn’t mature yet • Later in 2017, vendors upgraded their offerings • Most added custom tagging • Working with business and sales, we designed a new image taxonomy • Sports actions, NSFW filters, emotions, and image attributes • Complements the existing news taxonomy we apply to text content @smyles
- 9. A Hybrid Approach:Out of the Box + Custom • Use out-of-the-box tagging for most concepts • Train custom tagger for any concepts not covered (or covered well) by OOB tagger • Find example images for each concept e.g. “tackle” - anywhere from 500 to 5,000 examples per concept • Test the tagger to make sure it is accurate • Feed the tagger more examples where it underperforms • Proof image tagging in Production • High confidence tags accepted as-is • Ignore low confidence tags • Medium confidence tags reviewed by Editorial @smyles
- 10. Train and Test Management •We assembled training sets that we shared with the partner • And we held back test images • Testing for accuracy • Precision • Recall
- 12. Things We Learnt •Assembling test and train sets is arduous • But also where most of the value lies • Some concepts are difficult to distinguish • Dawn / Dusk, Happy / Jubilant • Perceived concepts are different than text subjects • May require some reorganization of our taxonomy and how we represent it @smyles
- 13.