Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics

What are the latest developments in visual search and image recognition? This talk, presented at Marketing Week Live in March 2017 (London, UK), offers a framework for visual search, discusses the Visual Data Gap, and examines what companies such as Google, Pinterest, and a range of startups are doing. It then offers examples of visual search analytics, and finally goes into areas that will be impacted by this, such as risk management, social listening, competitive intelligence, creative optimization, and customer service.

  • Login to see the comments

A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics

  1. 1. A picture’s worth a thousand hashtags: How image recognition will power the future of analytics David Berkowitz Chief Strategy Officer Sysomos @sysomos / @dberkowitz
  2. 2. About this presentation This talk was presented to Marketing Week Live in London in March 2017. A Texan version of this was delivered to W2O Group’s Pre-Commerce Summit during SXSW that month. If you prefer Frito pie to bangers and mash, I will gladly send you the W2O edition. Sources for information shown in slides are presented as links in the bottom- left corner. Further details about sources, where applicable, are also included in the notes field when downloading this presentation. To share your feedback or discuss this further, please contact me at Thank you. David
  3. 3. And now for something completely standard: an agenda • A brief history of nearly everything visual search • Why visual search matters • How Google, Pinterest, and others are deploying it • How marketers can use it • Numerous gratuitous references to all things British
  4. 4. Not British, but just a Chunnel ride away
  5. 5. How would you #HASHTAG the Mona Lisa?
  6. 6. #YOLO
  7. 7. Mona Lisa #Hashtags • #MonaLisa • #art • #painting • #woman • #lady • #smile • #smug • #Italian • #DaVinci • #epic • #outdoors • #masterpiece • #sky • #France • #LaGioconda • #badhairday • #beautiful • #Louvre • #Renaissance • #portrait This is a futile exercise. One can’t simply capture the Mona Lisa in hashtags. It points to the need for better ways to identify and analyze visual content. Text and hashtags alone don’t cut it.
  8. 8. Textual Media Visual Media We are here.
  9. 9. Drop in a bucket visual The Visual Data GapRecall how pictures are worth a thousand words? There is so much more data in images (‘the sun’) than there is in text (represented metaphorically by the planets).
  10. 10. A Short History of Nearly Everything Visual
  11. 11. Dual British/ American citizen
  12. 12. A challenge of Shakespearean proportions: “His reasons are as two grains of wheat hid in two bushels of chaff: you shall seek all day ere you find them, and when you have them, they are not worth the search.” -Bassanio, The Merchant of Venice
  13. 13. Number of object categories out there 15,000There are 15,000object categories Source: IEEE
  14. 14. Source: Computer Vision by Richard Szeliski For fun, I included a few examples of early attempts at machine- powered object recognition.
  15. 15. Source: Computer Vision by Richard Szeliski
  16. 16. An inflection point waiting to happen Jason Goldberg, Razorfish: “I’m strongly bullish on visual search. It solves a real problem consumers have… In the not- too-distant future, it’ll become a heavily used mainstream feature. I think the inflection point is at least a year away, but not two years." We’re approaching the inflection point, but it has taken longer than expected. This report is from November 2014.
  17. 17. Scanning products = Cool Facial scanning = Creepy
  18. 18. This is simply an enlarged, cropped version of the highlights from the previous chart.
  19. 19. You can’t always get what you want (with text search) • 74% of consumers say text-based keyword searches are inefficient for helping them find the right products online • 67% of consumers say quality of product images is very important in selecting and purchasing products • 90% of information transmitted to the brain is visual • Visual information is processed 60,000x faster than text Source: Slyce
  20. 20. The dress that inspired Google Image Search in 2001
  21. 21. The dress that inspired Google Image Search in 2001 “…People wanted more than just text. This first became apparent after the 2000 Grammy Awards, where Jennifer Lopez wore a green dress that, well, caught the world’s attention. At the time, it was the most popular search query we had ever seen. But we had no surefire way of getting users exactly what they wanted: JLo wearing that dress. Google Image Search was born.” -Eric Schmidt, Executive Chairman, Google Source: Project Syndicate
  22. 22. Solving the Clarissa problem My wife gave me this reference. As a kid, she always wanted to identify and shop for whatever Clarissa wore.
  23. 23. Applications for image recognition Source: Facebook
  24. 24. Source: The Verge
  25. 25. An introductory framework for visual search
  26. 26. Layers of image recognition A Deep Learning algorithm is presented with the images made up of simple pixels. The algorithm discovers simple regularities that are present across many/all images, like curves & lines. The algorithm discovers how these regularities are related to form higher-level concepts The system gains a high-level understanding of the original image… all automatically Source: GazeMetrix
  27. 27. A framework for visual search Scene Identification Intelligence Object Identification Intelligence Logo Identification Intelligence Image Identification Intelligence Category Identification Intelligence This notes some of the most important processes within visual search. Also note that identification and intelligence are two separate approaches. Examples follow.
  28. 28. What follows is an example using a real photo from Agnes, a Chinese tourist to the UK.
  29. 29. Here’s her photo. In each subsequent slide, you can see how the framework plays out and a sample finding that can be derived. Note the intelligence examples that follow are for illustrative purposes only; feel free to cite the framework, but not the data itself.
  30. 30. Category Identification: This is food and drink Category Intelligence: 7.2% of images posted at museums include food or drinks
  31. 31. Logo Identification: There is a Fanta logo, and the text in the top-right says Starbucks Logo Intelligence: Fanta logos are rarely paired with Starbucks; Fanta logos are most often seen with Coca-Cola and Adidas
  32. 32. Object Identification: This looks like fish and chips with a can of Fanta Lemon Object Intelligence: Fanta Lemon is the fourth most popular soda when paired with fish & chips
  33. 33. Image Identification: This is the same photo that appears on Agnes_Cin’s Flickr and public Facebook pages Image Intelligence: This image hasn’t been shared in any media outlets and hasn’t been shared publicly
  34. 34. Scene Identification: This photo seems to be taken outdoors during the day Scene Intelligence: 94% of photos at the British Museum are shot indoors, compared to 87% of museum photos worldwide
  35. 35. Spotlight: Google
  36. 36. This section is drawn (no pun intended) from
  37. 37. Spotlight: Pinterest
  38. 38. Pinterest: one of the world’s biggest search engines • 150 million monthly users • 75 billion pins • 2 billion searches/month • 97% of searches are unbranded Source: Pinterest
  39. 39. Pinterest: search pins from real-world images Source: Pinterest Examples from Pinterest’s new Visual Discovery follow. In the downloadable version of this talk, the next few visuals play as GIFs, and you can read more at the source below.
  40. 40. Pinterest Visual Discovery Source: Pinterest
  41. 41. Browse images and buy from them Source: Pinterest
  42. 42. I really liked the circled image here: Pinterest draws a connection between Big Ben and the Blue Mosque. Such errors are often more revealing.
  43. 43. Spotlight: Emerging Technologies
  44. 44. Toys R Us offers Slyce image detection for its catalog Source: Slyce
  45. 45. Visual search to complement textual search eMarketer: Do you think [visual search] will replace some types of searches, or do you think it will augment existing searches? Gierhart: It will probably augment. It’s adding a new utility to what was there before... There will still be contexts for both. Source: eMarketer (see a related video on YouTube)
  46. 46. Blippar: scan images for surprises about Planet Earth Source: Blippar
  47. 47. Source: Houzz Houzz has a Visual Match offering akin to Pinterest.
  48. 48. TheTake uses AI to identify products, locations in video Source: TheTake
  49. 49. What’s Possible with Image Analytics The images that follow are sample reports drawn from Sysomos. The data is again for illustrative purposes. Reach out if you want to dive deeper into any of this.
  50. 50. Visual analysis: understanding visual characteristics Logo recognition Object recognition Scene recognition Food recognition Color detection OCR: Search text within images
  51. 51. Visual analysis: understanding visual characteristics Logo recognition Object recognition Scene recognition Food recognition Color detection OCR: Search text within images
  52. 52. Visual analysis: understanding visual characteristics Logo recognition Object recognition Scene recognition Food recognition Color detection OCR: Search text within images
  53. 53. Audience analytics show growth and spikes
  54. 54. Brand affinities can highlight cross-promotion opportunities
  55. 55. Identify most popular objects, scenes
  56. 56. B2B applications for visual search Creative optimization Influencer marketing Rights management Crisis management Partnership ideation Competitive intelligence Customer service
  57. 57. Any questions?
  58. 58. So long, and thanks for all the fish. Let’s take tea! David Berkowitz Chief Strategy Officer Sysomos @sysomos / @dberkowitz