1. Using Intelligent Natural User Interfaces
to Support Sales Conversations
Sven Gehring, Markus Löchtefeld, Florian Daiber, Matthias Böhmer, Antonio Krüger
DFKI // Innovative Retail Laboratory

2. Introduction
‣ Customers‘ shopping experience has become important
‣ Customers‘ demand for product information has increased
‣ During sales conversations...
- Gestures and mimics are of high importance
- Repetitive negotiation on product and amount
‣ Idea and motivation
- Recognizes gestures to support negotiation
- Enable customers to inform themselves about products

4. ‣ Informal Field Study
- At fresh food counter in a German supermarket
- Approx. 350 customers during one day
- Approx. 3.6 products/customer
- How do customers express desired products?
- How do they negotiate on desired quantities?
4

5. Interaction Protocol
(1) Customer points on product
(2) Employee also points on product and asks for confirmation
(3) After both agreed on the product...
(a) Customer tells how much he wants
(b) Customer gestures how much he wants
(4) Employe cuts off piece of product, wraps it and hands it over

6. ‣ Prototype Implementation
- Using a depth camera (Microsoft Kinect)
- Using a scale with two displays
- Using a real sales counter
6

7. orschungszentrum für Forschungszentrum für Künstliche Intelligenz
ligenz
Künstliche Intelligenz
Deutsches Künstliche Intelligenz
Deutsches Forschungszentrum für Künstliche Intelligenz
ermanyCampus D3University, Campus D3 2, Saarbrücken, Germany
ersity, Saarland 2, Saarbrücken, University, Campus D3 2, Saarbrücken, Germany
Saarbrücken, Germany Saarland Germany
tefeld, florian.daiber, Setup and Scenarios
Application
er, krueger}@dfki.de matthias.boehmer, krueger}@dfki.de matthias.boehmer, krueger
n.gehring, markus.loechtefeld, florian.daiber, matthias.boehmer, krueger}@dfki.de
matthias.boehmer, krueger}@dfki.de
{sven.gehring, markus.loechtefeld, florian.daiber,
ABSTRACT
micsDuring high conversations, gestures and mimics are of high
nversations, sales
are of gestures and mimics are of high
communicate information about a product. about a product.
boutimportance to communicate information
a product.
ures One prominent example for is thesales gestures is the meat
example for such sales gestures such meat
is the meat
maining spots in counter, which is one of the remaining spots in
nter,and cheese of the remaining spots in
which is one
withsupermarkets where sales persons interact with customers.
here customers.
sales persons interact with customers.
ts normally fol- at such counters in supermarkets normally fol-
uch Interactions supermarkets normally fol-
counters in
rotocol. item customer points at customerof
s at low The of protocol. The an item points at an item of
an a simple
d, in choice. The employee takes out the item and, in most of the
ployee takes out the item and, in most of the
most of the
amount the be productfit the amount the cus-the amount the cus-
ct needs to cus- to needs to be cut to fit
cases the cut
aboutOftenwantsambiguous about is ambiguous about what spe-
buy.tomer spe- to buy. Often it what spe-
what it is
e customer and the employees and talking
ees cific talking the customer are the employees are talking
are product
ow, about.arere-to now, there arein HCI few efforts in HCI re-
orts there Upjust a few efforts just a re-
in HCI
communication at the point of sale. at the point of sale. In this
of sale. Into enrich communication In this
search this
uresetup withreport and analyze The application scale with the counter, a scale The application setupscal
on paper weone scenario in scale thescenario in which an 1: The application setup with the counter, a wit
and 1: The applicationFigurewhich an counter, a setup
which an intel- counter, a with
analyze the setup 1: one intel- Figure intel- Figure 1:
ahdepthdisplays can a depth camera can support communication the counter. displays above depth camer
user ligent natural user interface aboveand a depth camera displays and a depth camera and a the counter
communication
interface and support two displays the counter. two above
with communication
two camera above the counter. with with two
‣ employee
merconversation. in a and employee in a sales conversation.and customer
s and Support sales talk between employee
between customer sales conversation.
e is able on track we report on our prototype that is able to track
t report to our prototype that is able to track
Furthermore,
esonlyto display not ply different:to display they needshopping when not only go People not only theysho
different: People when theyshopping when only goto display
t and using agesturescamera anda depth camera and different: People they need shopping when go ne
by go shopping only usingneed
pointing depth by go People not ply ply different:
‣ pointed at. items pointed
o time they consider something;consider shopping as a consider products athey their to shopping a
ething; from time to shoppingretrieve information from shopping as on consider time they c
ut itemsCustomers can as a at.time to time they
information abouttime they from something; on time to time from time own
something;
ular leisureshopping with(e.g.,leisure time activitypopular leisure timepopular leisureshopping with(e.g.,wh
vity (e.g., time activity the whole
popular shopping with the whole shopping withactivity (e.g., time activity the sh
(e.g., the whole
ds on weekends). During sales weekends). During saleson weekends). During weekends). During sales c
ly Author Keywords
uring sales conversations, gestures
family on conversations, gestures conversations, gestures
family family on sales conversations, gestu
inting. areConversation, Salesinfor-communicate and mimicscommunicateimportance of high importance
tion,Gesturalof high importance to of high Pointing.
importance to communicate Conversation, importance to are of high mimics are to communicate inf
mimics Sales Interaction, Pointing.
and mimics are infor- and infor-

8. Gesture Recognition
‣ Gesturing already happens during sales conversation
‣ We make use of natural gestures that already exist
- Pointing Gesture
- Size Gesture
‣ These gestures are well established and commonly understood

9. Preliminary Feedback
‣ Customers
- System helps to disambiguate the desired item and amount
- System could be useful when buying food in foreign countries
- People point differently (e.g. index finger vs. whole hand)
‣ Experts (e.g. sales personal and retail managers)
- Density of items is higher in real world settings
- Pieces are „traveling“ during the day
- Suggested system for self-service frozen food areas

10. ‣ Conclusion
10

11. In Collaboration with Hochschule für Gestaltung Schwäbisch Gmünd
http://vimeo.com/36467293
11

12. Conclusion
‣ Intelligent natural interface to ease sales conversations
- We studied the common interaction protocol
- We proposed a prototypical system to support conversation
- We reported preliminary feedback of customers and experts
‣ Future Work: User study in the supermarket / in the wild
- Use projection for feedback
- Effect on time of sales conversation?
- Effect on sales figures at freshness counter?
- Effect on user shopping experience?