Background: Some early applications of Computer Vision in Retail arose from e-commerce use cases - but increasingly, it is being used in physical stores in a variety of new and exciting ways, such as: ● Optimizing merchandising execution, in-stocks and sell-thru ● Enhancing operational efficiencies, enable real-time customer engagement ● Enhancing loss prevention capabilities, response time ● Creating frictionless experiences for shoppers Abstract: This talk will cover the use of Computer Vision in Retail, the implications to the broader Consumer Goods industry and share business drivers, use cases and benefits that are unfolding as an integral component in the remaking of an age-old industry. We will also take a ‘peek under the hood’ of Computer Vision and Deep Learning, sharing technology design principles and skill set profiles to consider before starting your CV journey. Deep learning has matured considerably in the past few years to produce human or superhuman abilities in a variety of computer vision paradigms. We will discuss ways to recognize these paradigms in retail settings, collect and organize data to create actionable outcomes with the new insights and applications that deep learning enables. We will cover the basics of object detection, then move into the advanced processing of images describing the possible ways that a retail store of the near future could operate. Identifying various storefront situations by having a deep learning system attached to a camera stream. Such things as; identifying item stocks on shelves, a shelf in need of organization, or perhaps a wandering customer in need of assistance. We will also cover how to use a computer vision system to automatically track customer purchases to enable a streamlined checkout process, and how deep learning can power plausible wardrobe suggestions based on what a customer is currently wearing or purchasing. Finally, we will cover the various technologies that are powering these applications today. Deep learning tools for research and development. Production tools to distribute that intelligence to an entire inventory of all the cameras situation around a retail location. Tools for exploring and understanding the new data streams produced by the computer vision systems. By the end of this talk, attendees should understand the impact Computer Vision and Deep Learning are having in the Consumer Goods industry, key use cases, techniques and key considerations leaders are exploring and implementing today.

1. © Cloudera, Inc. All rights reserved.
COMPUTER VISION:
COMING TO A STORE
NEAR YOU
Brent Biddulph, Managing Director, Retail & CG, Cloudera
Florian Muellerklein, Data Scientist, Miner & Kasch

2. © Cloudera, Inc. All rights reserved. 2
AGENDA
Industry Trends
Business Drivers
Use Cases
How it Works
Bringing It All Together

3. © Cloudera, Inc. All rights reserved. 3
86% …OF RETAIL SALES
WILL STILL OCCUR IN-
STORE IN 2021.
(STATISTA, 2018)

4. © Cloudera, Inc. All rights reserved. 4
Mobile
Apps
ESLs,
Robots
Autonomous
Vehicles
Magic
Mirrors+
Shelf, Bin,
& Rack
Sensors
Surveillance
CamerasCompetitive
PricingSatellite
Images
Logistics &
Asset
TelematicsIn Home
Devices Frictionless
Checkout
RFID
Streams
SINGLE VIEW
OF CUSTOMER
FRICTIONLESS
COMMERCE
UBIQUITOUS
FULFILLMENT
RELEVANT
INTERACTIONS
OPERATIONAL
EFFICIENCY
Streaming Capabilities Enable New Insights,Right-Time Response
New Data Sources Extend Retail Intelligence = video, image data

5. © Cloudera, Inc. All rights reserved. 5
BUSINESS IMPROVEMENT
OPPORTUNITIES FOR
COMPUTER VISION IN-STORE
• Enabling Frictionless Commerce
• Improving Operational Efficiencies
• Improving Customer Experiences
• Reducing Fraud & Shrink
$410B
“AI / IOT ECONOMIC VALUE-
ADD IN RETAIL BY 2025,
WITH SOME OF THE MOST
VALUABLE USE CASES
DRIVEN BY STREAMING
VIDEO ANALYTICS.”
(MCKINSEY, 2017)
REAL-TIME PROCESSING –IMPROVED ACCURACY – DEEPER INSIGHTS

6. © Cloudera, Inc. All rights reserved. 6
ENABLING
FRICTIONLESS
COMMERCE
Amazon Go Store
• Computer Vision (100s
of cameras on shelves, ceiling)
• Sensor Fusion (cameras,
shelf scales)
• Deep Learning (advanced
pattern recognition, 100s of
algorithm’s)

7. © Cloudera, Inc. All rights reserved. 7
ENABLING
FRICTIONLESS
COMMERCE
Alibaba’s HEMA store
• Robots, apps, facial
recognition and
overhead conveyor
belts for fulfillment
delivery
• Using facial recognition
customers can pay
using their face - no QR
code nor credit card
swipe required

8. © Cloudera, Inc. All rights reserved. 8
IMPROVING
OPERATIONAL
EFFICIENCIES
Out-of-Stock &
Merchandising
Execution
• Out-of-Stock
Notifications
• Price and Promotion
Compliance
• Schematic & Display
Compliance
Photo Source: Retail Technology Corp.

9. © Cloudera, Inc. All rights reserved. 9
24% …OF AMAZON REVENUE
CAN BE ATTRIBUTED TO
BRICK & MORTAR STORE
OUT-OF-STOCKS.
(IHL, 2018)
OPPORTUNITY: IMPROVED OUT OF STOCK RESPONSE

10. © Cloudera, Inc. All rights reserved. 10
IMPROVING
OPERATIONAL
EFFICIENCIES
Advertising &
Promotional
Execution:
• Day-Part Promotional
Execution
• Audience and
Conversion Insights
• Proximity Marketing

11. © Cloudera, Inc. All rights reserved. 11© Cloudera, Inc. All rights reserved.
$174B
IN CPG ANNUAL TRADE PROMO SPEND.
OF THAT, 17% ($300M) SPENT ON SHELF,
YET 68% OF CONSUMER DECISIONS ARE
MADE AT THE SHELF.
IRI, 2016
OPPORTUNITY: IMPROVED MERCHANDISING EXECUTION INSIGHTS

12. © Cloudera, Inc. All rights reserved. 12
IMPROVING
CUSTOMER
EXPERIENCES
• Identify VIPs, ensure
high-touch CX
(‘clienteling’)
• Understand, respond
(anonymously) with
proactive customer
service opportunities

13. © Cloudera, Inc. All rights reserved. 13
• Simplifying the
dressing room
experience
• Assist to complete
the ‘look’ via cross-
sell of matching
shoes, accessories
• Reduce likelihood of
returns
IMPROVING
CUSTOMER
EXPERIENCES

14. © Cloudera, Inc. All rights reserved. 14© Cloudera, Inc. All rights reserved.
$260B
IN MERCHANDISE IS RETURNED TO
RETAILERS EACH YEAR.
AS MUCH AS 40% OF ONLINE
CLOTHING PURCHASES ARE RETURNED.
NRF, 2016
OPPORTUNITY: IMPROVED CX + MERCHANDISE RETURNS

15. © Cloudera, Inc. All rights reserved. 15
The use cases for CV
are promising…driving
meaningful business
impact, even creating
new revenue models
Frictionless Checkout
Loss Prevention
Neighborhood Insights Personalization
Customer Insights Customer Engagement
Dynamic Merchandising
Autonomous DeliverySource: Allure Source: Ahold/Delhaize
Source: Caper
Source: Orbital Insight
Source: Aura visionSource: Trigo-vision

16. © Cloudera, Inc. All rights reserved. 16
HOW IT WORKS AND HOW YOU CAN DO IT

17. © Cloudera, Inc. All rights reserved. 17
HOW DOES IMAGE RECOGNITION WORK?
Detect & Score
Convolutional neural
network analyzes
images and produces
actionable
representations
Compare
Representation manifolds can be mapped to find
matches, similar, or complementary images
Match & Alert
Users alerted and
shown matches

18. © Cloudera, Inc. All rights reserved. 18© Cloudera, Inc. All rights reserved.
IMPLEMENTING THESE TECHNIQUES IN YOUR COMPANY
• Options for implementing these functionalities in your store
• Purchase a point service for the desired functionality
• Whether it’s the whole solution or just part of the solution
• Hire a company to design and implement a custom solution
• Shameless plug for my company
• Implement the solutions yourself!

19. © Cloudera, Inc. All rights reserved. 19© Cloudera, Inc. All rights reserved.
IMPLEMENTING THESE TECHNIQUES IN YOUR COMPANY
• Need to define the problem and break it down into smaller parts
• There are a few CV modalities where machine learning shines
• Think of a modality as a smaller subproblem in machine perception
• May have multiple modalities within one problem
• A few different models may need to run
• Design supporting infrastructure for ML models
• Method to feed data into model(s)
• Methods to interpret model output(s)
• System to chain together multiple models
• Translate from model output to human interpretable results

20. © Cloudera, Inc. All rights reserved. 20
MACHINE LEARNING AND COMPUTER VISION

21. © Cloudera, Inc. All rights reserved. 21
DEEP LEARNING
• Many computer vision applications are now
becoming successful due to advances in
deep learning
• A deep learning model for computer vision is
specifically designed to exploit the spatial
dependencies in image data
• Flexible architecture designs to produce
desired output for given problem domain

22. © Cloudera, Inc. All rights reserved. 22
COMPUTER VISION PROBLEMS
• Computer vision problems can often be broken down into several problem
types
• Can make global inferences about an image
• Multiple spatially aware inferences
• Pixel level inference for very fine grained output needs
• Learning latent representations

23. © Cloudera, Inc. All rights reserved. 23
GLOBAL INFERENCE
• We may want a model to produce only a single output for a given image
• Identify the subject of the image (type of inventory, type of object)
• Identify affect of a person’s face
• Qualitative assessment of scene or object (does shelf need organization, is inventory
damaged)
• Infer distance between two objects
Gender
Affect
Eyeglasses
Headwear

24. © Cloudera, Inc. All rights reserved. 24
SPATIALLY AWARE INFERENCE
• We may want a model to produce many outputs for a given image
• Describes the locations and classifications for objects in an image
• Locate customers in your stores (counts, locations, heatmaps, time lingering, …)
• Locate products on a shelf
Aura Vision, https://auravision.ai

25. © Cloudera, Inc. All rights reserved. 25
PIXEL LEVEL INFERENCE
• We may want to produce an output for every
pixel in an image
• Classify each pixel as what object it’s a part of
• Produce mapping or overlay for the entire image
• Probability map of object location
• Depth map of a room

26. © Cloudera, Inc. All rights reserved. 26
LATENT REPRESENTATION LEARNING
• Finally, we may want generate
some kind of representation of
an image that allows us to
perform some useful math
• Representations capture semantic
information
• Quantify similarity metrics
between two images
• Train a model on a closely related
task that forces it to learn good
representations

27. © Cloudera, Inc. All rights reserved. 27© Cloudera, Inc. All rights reserved.
EXAMPLE USE CASE

28. © Cloudera, Inc. All rights reserved. 28
EXAMPLE USE CASE
• Some combination of these tools and pre/post model logic can be used to
build a real retain computer vision use-case
• Lets assume that we want to build a tool to help our customers by providing
clothing recommendations with CV
• They come into our store looking for a specific style of clothing
• Arrive with image of desired style
• Set up kiosk for clothing recommendations
• Augment magic mirror

29. © Cloudera, Inc. All rights reserved. 29
FASHION PRODUCT RECOMMENDATION EXAMPLE
• In order to create this application we’ll need two different machine learning
models and some logic around the model outputs
• Some capability to interpret the image and find the targeted piece of clothing
• Using a segmentation model
• Extract targeted article of clothing from image
• Create representation of clothing that allows it to be compared to others (model b)
• Create representations of entire catalog
• Matching algorithm to find closest match to query article

30. © Cloudera, Inc. All rights reserved. 30
Segmentoutthevariousarticlesofclothing

31. © Cloudera, Inc. All rights reserved. 31
Createsemanticrepresentationsforourentirecatalog

32. © Cloudera, Inc. All rights reserved. 32
Extractthetypeofclothingthatwewanttomatchon

33. © Cloudera, Inc. All rights reserved. 33
Createsemanticrepresentationsfromthoseextracted
Numeric vector
representing
visual patterns
and attributes

34. © Cloudera, Inc. All rights reserved. 34
Comparethoserepresentationstothoseofourcatalog
A
B
Best Match

35. © Cloudera, Inc. All rights reserved. 35
Puttingittogether

36. © Cloudera, Inc. All rights reserved. 36
DEMO

37. © Cloudera, Inc. All rights reserved. 37

38. © Cloudera, Inc. All rights reserved. 38
BRINGING IT ALL TOGETHER

39. © Cloudera, Inc. All rights reserved.39
WHATINDUSTRIALIZED“EDGETOAI”LOOKSLIKE
Streaming
Ingest
Batch Ingest
Machine
Learning Tools
BI Tools and
SQL Editors
Data Products
DATA, METADATA, SECURITY, GOVERNANCE, WORKLOAD
MANAGEMENT
MACHINE
LEARNING
DATA
ENGINEERING
DATA
WAREHOUSE
OPERATIONAL
DATABASE

40. © Cloudera, Inc. All rights reserved.
THANK YOU
Let’s Keep the Conversation Going…
Brent Biddulph
MD, Retail & CG, Cloudera
bbiddulph@cloudera.com
+1 425.273.6851
www.linkedin.com/in/brentbiddulph/
@brentbiddulph
Florian Muellerklein
Data Scientist, Miner & Kasch
fmuellerklein@minerkasch.com
+1 410.564.1720
www.linkedin.com/in/florian-muellerklein/
@mllrkln