Research on spatial cognition and blind
navigation suggests that a device aimed at helping blind people
to shop independently should provide the shopper with
effective interfaces to the locomotor and haptic spaces of the
supermarket. In this article, we argue that robots can act as
effective interfaces to haptic and locomotor spaces in modern
supermarkets.We also present the design and evaluation
of three product selection modalities—browsing, typing and
speech, which allow the blind shopper to select the desired
product from a repository of thousands of products.
RoboCart: Toward Robot-Assisted Navigation of Grocery Stores by the Visually ...Vladimir Kulyukin
This paper presents RoboCart, a proof-of-concept
prototype of a robotic shopping assistant for the visually
impaired. The purpose of RoboCart is to help visually impaired
customers navigate a typical grocery store and carry purchased
items. The hardware and software components of the system
are presented. For localization, RoboCart relies on RFID tags
deployed at various locations in the store. For navigation, Robo-
Cart relies on laser range finding. Experiences with deploying
RoboCart in a real grocery store are described. The current
status of the system and its limitations are outlined.
Passive Radio Frequency Exteroception in Robot-Assisted Shopping for the BlindVladimir Kulyukin
In 2004, the Computer Science Assistive Technology Laboratory (CSATL) of Utah State University (USU) started a project whose objective is to develop RoboCart, a robotic shopping assistant for the
visually impaired. RoboCart is a continuation of our previous work on RG, a robotic guide for the visually impaired in structured indoor environments. The determinism provided by exteroception of passive RFID-
enabled surfaces is desirable when dealing with dynamic and uncertain
environments where probabilistic approaches like Monte Carlo Markov
localization (MCL) may fail. We present the results of a pilot feasibility study with two visually impaired shoppers in Lee’s MarketPlace, a
supermarket in Logan, Utah.
Now, the perception ability of artificial intelligence robots has achieved significant progress. Around machine vision, robots can realize a series of functions like image recognition, target detection, and text recognition, which are widely used; around natural language processing, robots can perform basic speech understanding, machine translation, voice dialogue, etc.; around machine tactility, robots can realize various actions such as flexible object perception, grasping and pushing.
Teleassistance in Accessible Shopping for the BlindVladimir Kulyukin
In this paper, we present TeleShop, the teleassistance
module of ShopMobile 2, our mobile accessible shopping system for visually impaired (VI) and blind individuals
that we have been developing for the past several
years. TeleShop enables its users to obtain help from remote
sighted guides by transmitting images and voice from their
smartphones to the guides’ computers or phones. We have
successfully tested TeleShop in a laboratory study in which
a married couple (a blind husband and a sighted wife) used
it to retrieve grocery products and read nutrition facts from
product packages.
RoboCart: Toward Robot-Assisted Navigation of Grocery Stores by the Visually ...Vladimir Kulyukin
This paper presents RoboCart, a proof-of-concept
prototype of a robotic shopping assistant for the visually
impaired. The purpose of RoboCart is to help visually impaired
customers navigate a typical grocery store and carry purchased
items. The hardware and software components of the system
are presented. For localization, RoboCart relies on RFID tags
deployed at various locations in the store. For navigation, Robo-
Cart relies on laser range finding. Experiences with deploying
RoboCart in a real grocery store are described. The current
status of the system and its limitations are outlined.
Passive Radio Frequency Exteroception in Robot-Assisted Shopping for the BlindVladimir Kulyukin
In 2004, the Computer Science Assistive Technology Laboratory (CSATL) of Utah State University (USU) started a project whose objective is to develop RoboCart, a robotic shopping assistant for the
visually impaired. RoboCart is a continuation of our previous work on RG, a robotic guide for the visually impaired in structured indoor environments. The determinism provided by exteroception of passive RFID-
enabled surfaces is desirable when dealing with dynamic and uncertain
environments where probabilistic approaches like Monte Carlo Markov
localization (MCL) may fail. We present the results of a pilot feasibility study with two visually impaired shoppers in Lee’s MarketPlace, a
supermarket in Logan, Utah.
Now, the perception ability of artificial intelligence robots has achieved significant progress. Around machine vision, robots can realize a series of functions like image recognition, target detection, and text recognition, which are widely used; around natural language processing, robots can perform basic speech understanding, machine translation, voice dialogue, etc.; around machine tactility, robots can realize various actions such as flexible object perception, grasping and pushing.
Teleassistance in Accessible Shopping for the BlindVladimir Kulyukin
In this paper, we present TeleShop, the teleassistance
module of ShopMobile 2, our mobile accessible shopping system for visually impaired (VI) and blind individuals
that we have been developing for the past several
years. TeleShop enables its users to obtain help from remote
sighted guides by transmitting images and voice from their
smartphones to the guides’ computers or phones. We have
successfully tested TeleShop in a laboratory study in which
a married couple (a blind husband and a sighted wife) used
it to retrieve grocery products and read nutrition facts from
product packages.
This is a brief comparison of PermaMatrix vs. topsoil, mulch and compost. By hydraulically applying this organic biotic soil amendment containing Biochar, you will not only cut costs, but also cut pollution and create a sustainable environment for plants to thrive.
The Blind Leading the Blind: Toward Collaborative Online Route Information Ma...Vladimir Kulyukin
The long-term objective of our project is to discover the fundamental
principles underlying the collaborative production,
sharing, and management of route information by travelers
with visual impairments. The specific research hypothesis is
two-fold: 1) people with varied levels of vision loss and orientation
and mobility (O&M) skills will be able to form online
social networks that collaboratively manage large route
information collections for different geographic areas, and 2)
members of such networks will be able to successfully travel
through a variety of previously unknown indoor and outdoor
environments of varied complexity when provided with online
verbal route directions referencing landmarks and path
integration information salient to their particular vision and
skill level. In this paper, we report on the initial stage of our
project: an online survey whose objectives are to collect samples
of route descriptions from travelers with visual impairments
and to do the initial profiling of the target population.
The data collected so far provide valuable insights into what
travelers with visual impairments need to know about their
environments in order to travel independently and how they
may communicate that information to their fellow travelers in
the future.
RFID in Robot-Assisted Indoor Navigation for the Visually ImpairedVladimir Kulyukin
We describe how Radio Frequency Identification
(RFID) can be used in robot-assisted indoor navigation for
the visually impaired. We present a robotic guide for the
visually impaired that was deployed and tested both with
and without visually impaired participants in two indoor
environments. We describe how we modified the standard
potential fields algorithms to achieve navigation at moderate
walking speeds and to avoid oscillation in narrow spaces.
The experiments illustrate that passive RFID tags deployed
in the environment can act as reliable stimuli that trigger local
navigation behaviors to achieve global navigation objectives.
Surface-Embedded Passive RF Exteroception: Kepler, Greed, and Buffon’s NeedleVladimir Kulyukin
Surface-embedded passive radio frequency (PRF) exteroception is a
method whereby an action to be executed by a mobile unit is selected through a
signal received from a surface-embedded external passive RFID transponder. This
paper describes how Kepler’s hexagonal packing pattern is used to embed passive
RFID transponders into a carpet to create PRF surfaces. Proof-of-concepts
experiments are presented that show how such surfaces enable mobile robots to
reliably accomplish point-to-point navigation indoors and outdoors. Two greedy
algorithms are presented for automated design of PRF surfaces. A theoretical extension
of the classic Buffon’s Needle problem from computational geometry is
presented as a possible way to optimize the packing of RF transponders on a
surface.
For those who have missed Jaslin's sharing on how just 21 Days at Outward Bound Singapore changed her perspective of life on 11 November 2011, we are pleased to share her slideshow here.
Independent blind supermarket shopping is difficult at best. This paper presents
ShopTalk, a wearable small-scale system that enables a visually impaired shopper to
successfully retrieve specific products. ShopTalk uses exclusively commercial-off-theshelf
components and requires no instrumentation of the store. The system relies on the
navigation abilities of independent blind navigators and on the inherent supermarket
structure.
Robots as Interfaces to Haptic and Locomotor SpacesVladimir Kulyukin
Research on spatial cognition and navigation of the visually
impaired suggests that vision may be a primary sensory modality
that enables humans to align the egocentric (self to object) and
allocentric (object to object) frames of reference in space. In the
absence of vision, the frames align best in the haptic space. In the
locomotor space, as the haptic space translates with the body, lack
of vision causes the frames to misalign, which negatively affects
action reliability. In this paper, we argue that robots can function
as interfaces to the haptic and locomotor spaces in supermarkets.
In the locomotor space, the robot eliminates the necessity of
frame alignment and, in or near the haptic space, it cues the
shopper to the salient features of the environment sufficient for
product retrieval. We present a trichotomous ontology of spaces
in a supermarket induced by the presence of a robotic shopping
assistant and analyze the results of robot-assisted shopping
experiments with ten visually impaired participants conducted in
a real supermarket.
This is a brief comparison of PermaMatrix vs. topsoil, mulch and compost. By hydraulically applying this organic biotic soil amendment containing Biochar, you will not only cut costs, but also cut pollution and create a sustainable environment for plants to thrive.
The Blind Leading the Blind: Toward Collaborative Online Route Information Ma...Vladimir Kulyukin
The long-term objective of our project is to discover the fundamental
principles underlying the collaborative production,
sharing, and management of route information by travelers
with visual impairments. The specific research hypothesis is
two-fold: 1) people with varied levels of vision loss and orientation
and mobility (O&M) skills will be able to form online
social networks that collaboratively manage large route
information collections for different geographic areas, and 2)
members of such networks will be able to successfully travel
through a variety of previously unknown indoor and outdoor
environments of varied complexity when provided with online
verbal route directions referencing landmarks and path
integration information salient to their particular vision and
skill level. In this paper, we report on the initial stage of our
project: an online survey whose objectives are to collect samples
of route descriptions from travelers with visual impairments
and to do the initial profiling of the target population.
The data collected so far provide valuable insights into what
travelers with visual impairments need to know about their
environments in order to travel independently and how they
may communicate that information to their fellow travelers in
the future.
RFID in Robot-Assisted Indoor Navigation for the Visually ImpairedVladimir Kulyukin
We describe how Radio Frequency Identification
(RFID) can be used in robot-assisted indoor navigation for
the visually impaired. We present a robotic guide for the
visually impaired that was deployed and tested both with
and without visually impaired participants in two indoor
environments. We describe how we modified the standard
potential fields algorithms to achieve navigation at moderate
walking speeds and to avoid oscillation in narrow spaces.
The experiments illustrate that passive RFID tags deployed
in the environment can act as reliable stimuli that trigger local
navigation behaviors to achieve global navigation objectives.
Surface-Embedded Passive RF Exteroception: Kepler, Greed, and Buffon’s NeedleVladimir Kulyukin
Surface-embedded passive radio frequency (PRF) exteroception is a
method whereby an action to be executed by a mobile unit is selected through a
signal received from a surface-embedded external passive RFID transponder. This
paper describes how Kepler’s hexagonal packing pattern is used to embed passive
RFID transponders into a carpet to create PRF surfaces. Proof-of-concepts
experiments are presented that show how such surfaces enable mobile robots to
reliably accomplish point-to-point navigation indoors and outdoors. Two greedy
algorithms are presented for automated design of PRF surfaces. A theoretical extension
of the classic Buffon’s Needle problem from computational geometry is
presented as a possible way to optimize the packing of RF transponders on a
surface.
For those who have missed Jaslin's sharing on how just 21 Days at Outward Bound Singapore changed her perspective of life on 11 November 2011, we are pleased to share her slideshow here.
Independent blind supermarket shopping is difficult at best. This paper presents
ShopTalk, a wearable small-scale system that enables a visually impaired shopper to
successfully retrieve specific products. ShopTalk uses exclusively commercial-off-theshelf
components and requires no instrumentation of the store. The system relies on the
navigation abilities of independent blind navigators and on the inherent supermarket
structure.
Robots as Interfaces to Haptic and Locomotor SpacesVladimir Kulyukin
Research on spatial cognition and navigation of the visually
impaired suggests that vision may be a primary sensory modality
that enables humans to align the egocentric (self to object) and
allocentric (object to object) frames of reference in space. In the
absence of vision, the frames align best in the haptic space. In the
locomotor space, as the haptic space translates with the body, lack
of vision causes the frames to misalign, which negatively affects
action reliability. In this paper, we argue that robots can function
as interfaces to the haptic and locomotor spaces in supermarkets.
In the locomotor space, the robot eliminates the necessity of
frame alignment and, in or near the haptic space, it cues the
shopper to the salient features of the environment sufficient for
product retrieval. We present a trichotomous ontology of spaces
in a supermarket induced by the presence of a robotic shopping
assistant and analyze the results of robot-assisted shopping
experiments with ten visually impaired participants conducted in
a real supermarket.
User Intent Communication in Robot-Assisted Shopping for the BlindVladimir Kulyukin
The research reported in this chapter describes our work on robot-assisted shopping for the blind. In our previous research, we developed RoboCart, a robotic shopping cart for the visually impaired. RoboCart's operation includes four steps: 1) the blind shopper (henceforth the shopper) selects a
product; 2) the robot guides the shopper to the shelf with the product; 3) the shopper finds
the product on the shelf, places it in the basket mounted on the robot, and either selects
another product or asks the robot to take him to a cash register; 4) the robot guides the
shopper to the cash register and then to the exit.
Steps 2, 3, and 4 were addressed in our previous publications . In this paper, we focus on Step 1 that requires
the shopper to select a product from the repository of thousands of products, thereby
communicating the next target destination to RobotCart. This task becomes time critical in
opportunistic grocery shopping when the shopper does not have a prepared list of products.
If the shopper is stranded at a location in the supermarket selecting a product, the shopper
may feel uncomfortable or may negatively affect the shopper traffic.
ShopTalk: Toward Independent Shopping by People with Visual ImpairmentsVladimir Kulyukin
ShopTalk, a proof-of-concept system designed to assist individuals with visual impairments with finding shelved products
in grocery stores, is built on the assumption that simple
verbal route directions and layout descriptions can be
used to leverage the O&M skills of independent visually impaired travelers to enable them to navigate the store and
retrieve shelved products. This paper introduces ShopTalk
and summarizes experiments performed in a real-world supermarket.
Ergonomics-for-One in a Robotic Shopping Cart for the BlindVladimir Kulyukin
Assessment and design frameworks for human-robot teams
attempt to maximize generality by covering a broad range of
potential applications. In this paper, we argue that, in assistive
robotics, the other side of generality is limited applicability: it is
oftentimes more feasible to custom-design and evolve an
application that alleviates a specific disability than to spend
resources on figuring out how to customize an existing generic
framework. We present a case study that shows how we used a
pure bottom-up learn-through-deployment approach inspired by
the principles of ergonomics-for-one to design, deploy and
iteratively re-design a proof-of-concept robotic shopping cart for
the blind.
Page 1 of 14 ENS4152 Project Development Proposal a.docxkarlhennesey
Page 1 of 14
ENS4152 Project Development
Proposal and Risk Assessment Report
Baxter Research Robot: Solving a Rubik’s Cube
Chris Dawes
Student # 10282558
30 Mar 2015
Supervisor: Dr Alexander Rassau
Page 2 of 14
Abstract
Robotics is currently used to perform many tasks but many of these are simple repetition of a
predefined method. By combining AI with robotics we can greatly increase the applications of
robotics. An algorithm that combines the vision and servo systems of a Baxter Research Robot
with a solving solution for a Rubik’s cube will demonstrate that the use of even simple AI with
robotics allows complex tasks to be completed. Further integration of object recognition will
allow the task to be completed in a dynamic environment, and further increase the areas
robots are capable of working within.
1. Introduction
1.1. Motivation
The Baxter Research Robot by Rethink Robotics is a dual arm robot, with seven degrees of
freedom per arm, released in 2012. Developed to be affordable, flexible in its purpose, and
above all else safe, Baxter includes three cameras, one on each wrist and the other on its head,
and a screen for displaying information relating to Baxter’s current task. The robot is designed
to be a versatile research platform while containing the same hardware as its industry
counterpart, allowing research to translate into industrial applications (Rethink Robotics,
2015).
In general robotics artificial intelligence (AI) has been developed separately to robotics, but is
now starting to become integrated. Unfortunately current AI is fragmented as each application
focuses on one area, as opposed to making a true AI that thinks like a human (Bogue, 2014).
Current usable AI is more akin to ‘smart’ robotics where decisions are made and problems
solved by the robot in very specific applications. In industry, robots are expanding into areas
that require more flexibility allowing robots to fill many more positions in increasingly complex
areas (Hajduk, Jenčík, Jezný, & Vargovčík, 2013). Mobile robots are even becoming more
common place, allowing for dynamic and spread out workspaces. These are all due to adding
sensing and analysis to robots allowing them to react to dynamic environments.
To further robotics in industry, multi robot work cells have been designed that combine
several robots working on the same part while cooperatively performing either one task, such
as welding and the required handling, or multiple tasks at the same time (Hajduk, Jenčík, Jezný,
& Vargovčík, 2013). The number of activities these work cells can perform increases
Page 3 of 14
dramatically, as the complexity of the task or tasks can be higher while the robots don’t need
to be capable of performing the whole task individually.
For performing more human tasks, dual arm robots have begun to emerge (Hajduk, Jenčík,
Jezný, & Vargovčík, 2013 ...
Page 1 of 14 ENS4152 Project Development Proposal a.docxsmile790243
Page 1 of 14
ENS4152 Project Development
Proposal and Risk Assessment Report
Baxter Research Robot: Solving a Rubik’s Cube
Chris Dawes
Student # 10282558
30 Mar 2015
Supervisor: Dr Alexander Rassau
Page 2 of 14
Abstract
Robotics is currently used to perform many tasks but many of these are simple repetition of a
predefined method. By combining AI with robotics we can greatly increase the applications of
robotics. An algorithm that combines the vision and servo systems of a Baxter Research Robot
with a solving solution for a Rubik’s cube will demonstrate that the use of even simple AI with
robotics allows complex tasks to be completed. Further integration of object recognition will
allow the task to be completed in a dynamic environment, and further increase the areas
robots are capable of working within.
1. Introduction
1.1. Motivation
The Baxter Research Robot by Rethink Robotics is a dual arm robot, with seven degrees of
freedom per arm, released in 2012. Developed to be affordable, flexible in its purpose, and
above all else safe, Baxter includes three cameras, one on each wrist and the other on its head,
and a screen for displaying information relating to Baxter’s current task. The robot is designed
to be a versatile research platform while containing the same hardware as its industry
counterpart, allowing research to translate into industrial applications (Rethink Robotics,
2015).
In general robotics artificial intelligence (AI) has been developed separately to robotics, but is
now starting to become integrated. Unfortunately current AI is fragmented as each application
focuses on one area, as opposed to making a true AI that thinks like a human (Bogue, 2014).
Current usable AI is more akin to ‘smart’ robotics where decisions are made and problems
solved by the robot in very specific applications. In industry, robots are expanding into areas
that require more flexibility allowing robots to fill many more positions in increasingly complex
areas (Hajduk, Jenčík, Jezný, & Vargovčík, 2013). Mobile robots are even becoming more
common place, allowing for dynamic and spread out workspaces. These are all due to adding
sensing and analysis to robots allowing them to react to dynamic environments.
To further robotics in industry, multi robot work cells have been designed that combine
several robots working on the same part while cooperatively performing either one task, such
as welding and the required handling, or multiple tasks at the same time (Hajduk, Jenčík, Jezný,
& Vargovčík, 2013). The number of activities these work cells can perform increases
Page 3 of 14
dramatically, as the complexity of the task or tasks can be higher while the robots don’t need
to be capable of performing the whole task individually.
For performing more human tasks, dual arm robots have begun to emerge (Hajduk, Jenčík,
Jezný, & Vargovčík, 2013.
Page 1 of 14 ENS4152 Project Development Proposal a.docxjakeomoore75037
Page 1 of 14
ENS4152 Project Development
Proposal and Risk Assessment Report
Baxter Research Robot: Solving a Rubik’s Cube
Chris Dawes
Student # 10282558
30 Mar 2015
Supervisor: Dr Alexander Rassau
Page 2 of 14
Abstract
Robotics is currently used to perform many tasks but many of these are simple repetition of a
predefined method. By combining AI with robotics we can greatly increase the applications of
robotics. An algorithm that combines the vision and servo systems of a Baxter Research Robot
with a solving solution for a Rubik’s cube will demonstrate that the use of even simple AI with
robotics allows complex tasks to be completed. Further integration of object recognition will
allow the task to be completed in a dynamic environment, and further increase the areas
robots are capable of working within.
1. Introduction
1.1. Motivation
The Baxter Research Robot by Rethink Robotics is a dual arm robot, with seven degrees of
freedom per arm, released in 2012. Developed to be affordable, flexible in its purpose, and
above all else safe, Baxter includes three cameras, one on each wrist and the other on its head,
and a screen for displaying information relating to Baxter’s current task. The robot is designed
to be a versatile research platform while containing the same hardware as its industry
counterpart, allowing research to translate into industrial applications (Rethink Robotics,
2015).
In general robotics artificial intelligence (AI) has been developed separately to robotics, but is
now starting to become integrated. Unfortunately current AI is fragmented as each application
focuses on one area, as opposed to making a true AI that thinks like a human (Bogue, 2014).
Current usable AI is more akin to ‘smart’ robotics where decisions are made and problems
solved by the robot in very specific applications. In industry, robots are expanding into areas
that require more flexibility allowing robots to fill many more positions in increasingly complex
areas (Hajduk, Jenčík, Jezný, & Vargovčík, 2013). Mobile robots are even becoming more
common place, allowing for dynamic and spread out workspaces. These are all due to adding
sensing and analysis to robots allowing them to react to dynamic environments.
To further robotics in industry, multi robot work cells have been designed that combine
several robots working on the same part while cooperatively performing either one task, such
as welding and the required handling, or multiple tasks at the same time (Hajduk, Jenčík, Jezný,
& Vargovčík, 2013). The number of activities these work cells can perform increases
Page 3 of 14
dramatically, as the complexity of the task or tasks can be higher while the robots don’t need
to be capable of performing the whole task individually.
For performing more human tasks, dual arm robots have begun to emerge (Hajduk, Jenčík,
Jezný, & Vargovčík, 2013.
It's a really a great privilege,proud & honor as well that I instated a PPT on Robotics power presentation which was amazed so many people and hope u people also 'll be obliged & benefited.....................................My good wishes with ye always . Thank you.
Intelligent Robotics Navigation System: Problems, Methods, and Algorithm IJECEIAES
This paper set out to supplement new studies with a brief and comprehensible review of the advanced development in the area of the navigation system, starting from a single robot, multi-robot, and swarm robots from a particular perspective by taking insights from these biological systems. The inspiration is taken from nature by observing the human and the social animal that is believed to be very beneficial for this purpose. The intelligent navigation system is developed based on an individual characteristic or a social animal biological structure. The discussion of this paper will focus on how simple agent’s structure utilizes flexible and potential outcomes in order to navigate in a productive and unorganized surrounding. The combination of the navigation system and biologically inspired approach has attracted considerable attention, which makes it an important research area in the intelligent robotic system. Overall, this paper explores the implementation, which is resulted from the simulation performed by the embodiment of robots operating in real environments.
A novel visual tracking scheme for unstructured indoor environmentsIJECEIAES
In the ever-expanding sphere of assistive robotics, the pressing need for advanced methods capable of accurately tracking individuals within unstructured indoor settings has been magnified. This research endeavours to devise a realtime visual tracking mechanism that encapsulates high performance attributes while maintaining minimal computational requirements. Inspired by the neural processes of the human brain’s visual information handling, our innovative algorithm employs a pattern image, serving as an ephemeral memory, which facilitates the identification of motion within images. This tracking paradigm was subjected to rigorous testing on a Nao humanoid robot, demonstrating noteworthy outcomes in controlled laboratory conditions. The algorithm exhibited a remarkably low false detection rate, less than 4%, and target losses were recorded in merely 12% of instances, thus attesting to its successful operation. Moreover, the algorithm’s capacity to accurately estimate the direct distance to the target further substantiated its high efficacy. These compelling findings serve as a substantial contribution to assistive robotics. The proficient visual tracking methodology proposed herein holds the potential to markedly amplify the competencies of robots operating in dynamic, unstructured indoor settings, and set the foundation for a higher degree of complex interactive tasks.
Eyes-Free Barcode Localization and Decoding for Visually Impaired Mobile Phon...Vladimir Kulyukin
An eyes-free barcode localization and decoding method is presented that enables visually impaired (VI) mobile phone users to decode MSI (Modified Plessy) barcodes on shelves and UPC barcodes on individual boxes, cans, and bottles. Simple and efficient barcode localization and decoding techniques are augmented with an interactive haptic feedback loop that allows the VI user to align the phone’s camera with a fixed surface in the pitch and yaw planes. The method is implemented on a Google Nexus One smart phone running Android 2.1. A laboratory study is presented in which the method was evaluated by one VI and four blindfolded sighted participants.
This paper reports results of artificial neural network for robot navigation tasks. Machine
learning methods have proven usability in many complex problems concerning mobile robots
control. In particular we deal with the well-known strategy of navigating by “wall-following”.
In this study, probabilistic neural network (PNN) structure was used for robot navigation tasks.
The PNN result was compared with the results of the Logistic Perceptron, Multilayer
Perceptron, Mixture of Experts and Elman neural networks and the results of the previous
studies reported focusing on robot navigation tasks and using same dataset. It was observed the
PNN is the best classification accuracy with 99,635% accuracy using same dataset.
Similar to Robot-Assisted Shopping for the Blind: Issues in Spatial Cognition and Product Selection (20)
Digitizing Buzzing Signals into A440 Piano Note Sequences and Estimating Fora...Vladimir Kulyukin
Digitizing Buzzing Signals into A440 Piano Note Sequences and Estimating Forager Traffic Levels from Images in Solar-Powered, Electronic Beehive Monitoring
Many problems in information retrieval and related fields depend on a reliable measure of the distance or similarity between objects that, most frequently, are represented
as vectors. This paper considers vectors of bits. Such data structures implement entities as diverse as bitmaps that indicate the occurrences of terms and bitstrings indicating the presence
of edges in images. For such applications, a popular distance measure is the Hamming distance. The value of the Hamming distance for information retrieval applications is limited by the
fact that it counts only exact matches, whereas in information retrieval, corresponding bits that are close by can still be considered to be almost identical. We define a "Generalized
Hamming distance" that extends the Hamming concept to give partial credit for near misses, and suggest a dynamic programming algorithm that permits it to be computed efficiently.
We envision many uses for such a measure. In this paper we define and prove some basic properties of the :Generalized Hamming distance," and illustrate its use in the area of object
recognition. We evaluate our implementation in a series of experiments, using autonomous robots to test the measure's effectiveness in relating similar bitstrings.
Adapting Measures of Clumping Strength to Assess Term-Term SimilarityVladimir Kulyukin
Automated information retrieval relies heavily on statistical regularities that emerge as terms are deposited to produce text. This paper examines statistical patterns expected of a pair of
terms that are semantically related to each other. Guided by a conceptualization of the text generation process, we derive measures of how tightly two terms are semantically associated.
Our main objective is to probe whether such measures yields reasonable results. Specifically, we examine how the tendency of a content bearing term to clump, as quantified by previously
developed measures of term clumping, is influenced by the presence of other terms. This
approach allows us to present a toolkit from which a range of measures can be constructed.
As an illustration, one of several suggested measures is evaluated on a large text corpus built from an on-line encyclopedia.
Vision-Based Localization and Scanning of 1D UPC and EAN Barcodes with Relaxe...Vladimir Kulyukin
V. Kulyukin & T. Zaman. "Vision-Based Localization and Scanning of 1D UPC and EAN Barcodes with Relaxed Pitch, Roll, and Yaw Camera A lignment Constraints." International Journal of Image Processing (IJIP), V olume (8) : Issue (5) : 2014, pp. 355-383.
Toward Blind Travel Support through Verbal Route Directions: A Path Inference...Vladimir Kulyukin
The work presented in this article continues our investigation of such assisted navigation solutions where the
main emphasis is placed not on sensor sets or sensor fusion algorithms but on the ability of the travelers to interpret and
contextualize verbal route directions en route. This work contributes to our investigation of the research hypothesis that
we have formulated and partially validated in our previous studies: if a route is verbally described in sufficient and
appropriate amount of detail, independent VI travelers can use their O&M and problem solving skills to successfully
follow the route without any wearable sensors or sensors embedded in the environment.
In this investigation, we temporarily put aside the issue of how VI and blind travelers successfully interpret route
directions en route and tackle the question of how those route directions can be created, generated, and maintained by
online communities. In particular, we focus on the automation of path inference and present an algorithm that may be used
as part of the background computation of VGI sites to find new paths in the previous route directions written by online
community members, generate new route descriptions from them, and post them for subsequent community editing.
Wireless Indoor Localization with Dempster-Shafer Simple Support FunctionsVladimir Kulyukin
A mobile robot is localized in an indoor environment
using IEEE 802.11b wireless signals. Simple support
functions of the Dempster-Shafer theory are used to combine evidence
from multiple localization algorithms. Emperical results
are presented and discussed. Conclusions are drawn regarding
when the proposed sensor fusion methods may improve performance
and when they may not.
This paper examines the appropriateness of natural language dialogue (NLD) with assistive robots. Assistive robots are defined in terms of an existing human-robot interaction taxonomy. A
decision support procedure is outlined for assistive technology
researchers and practitioners to evaluate the appropriateness of
NLD in assistive robots. Several conjectures are made on when
NLD may be appropriate as a human-robot interaction mode.
A Wearable Two-Sensor O&M Device for Blind College StudentsVladimir Kulyukin
A major problem for visually impaired college students is independent campus
navigation. Many universities, such as Utah State University (USU), have no Orientation
and Mobility (O&M) instructors. Thus, visually impaired undergraduates must rely on
their friends, siblings, and even parents to learn their way around a large campus, which
reduces their sense of independence. This paper describes a wearable two-sensor O&M
device for visually impaired USU undergraduates and presents a single subject feasibility
test that estimates how a visually impaired navigator can use the device to learn new
routes on the USU campus.
The ability to recreate computational results with minimal effort and actionable metrics provides a solid foundation for scientific research and software development. When people can replicate an analysis at the touch of a button using open-source software, open data, and methods to assess and compare proposals, it significantly eases verification of results, engagement with a diverse range of contributors, and progress. However, we have yet to fully achieve this; there are still many sociotechnical frictions.
Inspired by David Donoho's vision, this talk aims to revisit the three crucial pillars of frictionless reproducibility (data sharing, code sharing, and competitive challenges) with the perspective of deep software variability.
Our observation is that multiple layers — hardware, operating systems, third-party libraries, software versions, input data, compile-time options, and parameters — are subject to variability that exacerbates frictions but is also essential for achieving robust, generalizable results and fostering innovation. I will first review the literature, providing evidence of how the complex variability interactions across these layers affect qualitative and quantitative software properties, thereby complicating the reproduction and replication of scientific studies in various fields.
I will then present some software engineering and AI techniques that can support the strategic exploration of variability spaces. These include the use of abstractions and models (e.g., feature models), sampling strategies (e.g., uniform, random), cost-effective measurements (e.g., incremental build of software configurations), and dimensionality reduction methods (e.g., transfer learning, feature selection, software debloating).
I will finally argue that deep variability is both the problem and solution of frictionless reproducibility, calling the software science community to develop new methods and tools to manage variability and foster reproducibility in software systems.
Exposé invité Journées Nationales du GDR GPL 2024
Earliest Galaxies in the JADES Origins Field: Luminosity Function and Cosmic ...Sérgio Sacani
We characterize the earliest galaxy population in the JADES Origins Field (JOF), the deepest
imaging field observed with JWST. We make use of the ancillary Hubble optical images (5 filters
spanning 0.4−0.9µm) and novel JWST images with 14 filters spanning 0.8−5µm, including 7 mediumband filters, and reaching total exposure times of up to 46 hours per filter. We combine all our data
at > 2.3µm to construct an ultradeep image, reaching as deep as ≈ 31.4 AB mag in the stack and
30.3-31.0 AB mag (5σ, r = 0.1” circular aperture) in individual filters. We measure photometric
redshifts and use robust selection criteria to identify a sample of eight galaxy candidates at redshifts
z = 11.5 − 15. These objects show compact half-light radii of R1/2 ∼ 50 − 200pc, stellar masses of
M⋆ ∼ 107−108M⊙, and star-formation rates of SFR ∼ 0.1−1 M⊙ yr−1
. Our search finds no candidates
at 15 < z < 20, placing upper limits at these redshifts. We develop a forward modeling approach to
infer the properties of the evolving luminosity function without binning in redshift or luminosity that
marginalizes over the photometric redshift uncertainty of our candidate galaxies and incorporates the
impact of non-detections. We find a z = 12 luminosity function in good agreement with prior results,
and that the luminosity function normalization and UV luminosity density decline by a factor of ∼ 2.5
from z = 12 to z = 14. We discuss the possible implications of our results in the context of theoretical
models for evolution of the dark matter halo mass function.
Richard's aventures in two entangled wonderlandsRichard Gill
Since the loophole-free Bell experiments of 2020 and the Nobel prizes in physics of 2022, critics of Bell's work have retreated to the fortress of super-determinism. Now, super-determinism is a derogatory word - it just means "determinism". Palmer, Hance and Hossenfelder argue that quantum mechanics and determinism are not incompatible, using a sophisticated mathematical construction based on a subtle thinning of allowed states and measurements in quantum mechanics, such that what is left appears to make Bell's argument fail, without altering the empirical predictions of quantum mechanics. I think however that it is a smoke screen, and the slogan "lost in math" comes to my mind. I will discuss some other recent disproofs of Bell's theorem using the language of causality based on causal graphs. Causal thinking is also central to law and justice. I will mention surprising connections to my work on serial killer nurse cases, in particular the Dutch case of Lucia de Berk and the current UK case of Lucy Letby.
Deep Behavioral Phenotyping in Systems Neuroscience for Functional Atlasing a...Ana Luísa Pinho
Functional Magnetic Resonance Imaging (fMRI) provides means to characterize brain activations in response to behavior. However, cognitive neuroscience has been limited to group-level effects referring to the performance of specific tasks. To obtain the functional profile of elementary cognitive mechanisms, the combination of brain responses to many tasks is required. Yet, to date, both structural atlases and parcellation-based activations do not fully account for cognitive function and still present several limitations. Further, they do not adapt overall to individual characteristics. In this talk, I will give an account of deep-behavioral phenotyping strategies, namely data-driven methods in large task-fMRI datasets, to optimize functional brain-data collection and improve inference of effects-of-interest related to mental processes. Key to this approach is the employment of fast multi-functional paradigms rich on features that can be well parametrized and, consequently, facilitate the creation of psycho-physiological constructs to be modelled with imaging data. Particular emphasis will be given to music stimuli when studying high-order cognitive mechanisms, due to their ecological nature and quality to enable complex behavior compounded by discrete entities. I will also discuss how deep-behavioral phenotyping and individualized models applied to neuroimaging data can better account for the subject-specific organization of domain-general cognitive systems in the human brain. Finally, the accumulation of functional brain signatures brings the possibility to clarify relationships among tasks and create a univocal link between brain systems and mental functions through: (1) the development of ontologies proposing an organization of cognitive processes; and (2) brain-network taxonomies describing functional specialization. To this end, tools to improve commensurability in cognitive science are necessary, such as public repositories, ontology-based platforms and automated meta-analysis tools. I will thus discuss some brain-atlasing resources currently under development, and their applicability in cognitive as well as clinical neuroscience.
Comparing Evolved Extractive Text Summary Scores of Bidirectional Encoder Rep...University of Maribor
Slides from:
11th International Conference on Electrical, Electronics and Computer Engineering (IcETRAN), Niš, 3-6 June 2024
Track: Artificial Intelligence
https://www.etran.rs/2024/en/home-english/
Observation of Io’s Resurfacing via Plume Deposition Using Ground-based Adapt...Sérgio Sacani
Since volcanic activity was first discovered on Io from Voyager images in 1979, changes
on Io’s surface have been monitored from both spacecraft and ground-based telescopes.
Here, we present the highest spatial resolution images of Io ever obtained from a groundbased telescope. These images, acquired by the SHARK-VIS instrument on the Large
Binocular Telescope, show evidence of a major resurfacing event on Io’s trailing hemisphere. When compared to the most recent spacecraft images, the SHARK-VIS images
show that a plume deposit from a powerful eruption at Pillan Patera has covered part
of the long-lived Pele plume deposit. Although this type of resurfacing event may be common on Io, few have been detected due to the rarity of spacecraft visits and the previously low spatial resolution available from Earth-based telescopes. The SHARK-VIS instrument ushers in a new era of high resolution imaging of Io’s surface using adaptive
optics at visible wavelengths.
Salas, V. (2024) "John of St. Thomas (Poinsot) on the Science of Sacred Theol...Studia Poinsotiana
I Introduction
II Subalternation and Theology
III Theology and Dogmatic Declarations
IV The Mixed Principles of Theology
V Virtual Revelation: The Unity of Theology
VI Theology as a Natural Science
VII Theology’s Certitude
VIII Conclusion
Notes
Bibliography
All the contents are fully attributable to the author, Doctor Victor Salas. Should you wish to get this text republished, get in touch with the author or the editorial committee of the Studia Poinsotiana. Insofar as possible, we will be happy to broker your contact.
2. 238 Intel Serv Robotics (2008) 1:237–251
shopper has (1) a means to accurately access the pose (loca- 1.2 Robot-assisted shopping
tion and orientation) and to obtain adequate topological
knowledge or (2) a means to reliably maneuver the haptic Can robots function as effective interfaces to the haptic and
space in the locomotor space. The first choice ensures accu- locomotor spaces in the supermarket? We believe that this
rate frame alignment and leaves to the shopper the maneu- question can be answered in the affirmative. Several reasons
vering of the haptic space in the locomotor space. The second justify our belief. Traditional navigation aids, such as guide
choice guarantees reliable maneuvering but takes no position dogs and white canes, can act as interfaces to the haptic space
on frame alignment. in the environment by enhancing the blind individuals per-
Either solution requires an effective interface to the ception around the body. However, neither guide dogs nor
locomotor space. The shopper can accomplish the last white canes can effectively interface to locomotor spaces,
subtask—product retrieval—if the shopper has a means to because they cannot help their users with macro-navigation,
maneuver the haptic space in the vicinity of a target prod- which requires functional topological knowledge of the envi-
uct until the product is within the haptic space. To guarantee ronment. It is true that sighted guides ensure the reliable
independence, any assistive shopping device for the visually maneuvering of the haptic space, but only at the expense
impaired must necessarily address both subtasks and, conse- of independence. Loss of independence translates into loss
quently, provide the shopper with effective interfaces to the of privacy. Robot-assisted shopping experiments as reported
haptic and locomotor spaces in supermarkets. in [21,23], indicate that the visually impaired shoppers are
Several research projects are related to our research. In usually not willing to use store staffers when shopping for
[2], the authors have developed a shopping aid consisting personal hygiene items, medicine, and other products that
of a barcode scanner and a laptop secured to a shopping require discretion.
cart. After the user scans the product’s barcode, the laptop Our central research hypothesis is that, in order to function
accesses the corresponding data files and displays them on as an effective interface to the haptic and locomotor spaces in
the laptop in an enlarged font. Although the device helps the supermarket, the robot must satisfy a two-fold objective:
the visually impaired to make an informed product choice, in the locomotor space, the robot must eliminate the neces-
it does not help them shop independently. This is because it sity of frame alignment and, in or near the haptic space, the
does not provide the shopper with an interface to the loco- robot must cue the shopper to the salient features of the envi-
motor space to navigate within the store. It does not appear ronment sufficient for product retrieval.
to be possible for white cane users and guide dog handlers
to navigate safely with a large shopping cart in the front. In 1.2.1 RoboCart: a robotic shopping assistant
addition, the device provides only the product specific infor-
mation. It does not help the shopper find the product on the The overall design of RoboCart is based on the principles of
shelf. ergonomics-for-one [23] and reflects the dual interface func-
In [27], the authors propose a wearable device consisting tionality through two modules: locomotor and haptic. The
of an earpiece, display visor, bluetooth connectivity, and locomotor module consists of a Pioneer 2DX mobile robotic
RFID scanner, all mounted on an eyewear. However, the base from ActivMedia, Inc. upon which a wayfinding toolkit
proposed device is limited to product identification only. is fitted in a polyvinyl chloride (PVC) pipe structure. A shop-
Another system, Trinetra [28], has its functionality limited ping basket is mounted upon the PVC structure, as shown
to product identification using a text-to-speech enabled cell- in Fig. 1. The current Robocart setup is slightly unstable
phone and a pen-like barcode reader. Such systems are not, due to a small robotic base and possibly heavy top, when
in and of themselves, sufficient for independent shopping for the basket is full with products. However we feel that, as a
the visually impaired. However, they can be incorporated as proof-of-concept prototype, the setup is sufficient to conduct
a part of the interface to the haptic space in systems like feasibility studies. Using a numeric keypad, the shopper can
RoboCart. select a product through an appropriate product selection
To accomplish the subtask of product selection, the shop- interface. Once the shopper confirms the selection, RoboCart
per should be able to—speedily, accurately and comfort- guides the shopper to the vicinity of the product.
ably—convey her intent to the shopping device. An example The haptic module consists of a wireless omni-directional
of the shopper intent would be: “Take me to <product>”, barcode reader shown in Fig. 1. The reader is ergonomically
where <product> can be any product from a repository modified with a plastic structure that helps the blind shop-
of thousands of products. Thus, the interface to the prod- per align the barcode reader with the shelf. After RoboCart
uct repository (catalogue) should be designed such that (1) brings the shopper in the vicinity of the product, RoboCart
the shopper can rapidly and comfortably select the desired uses the shopper’s egocentric frame of reference to instruct
product and (2) the shopper should be able to maintain an the shopper through synthetic speech on how to find the prod-
adequate level of discretion and comfort in a supermarket. uct, e.g., Honey Nut Cheerios is on the top shelf to your right.
123
3. Intel Serv Robotics (2008) 1:237–251 239
Fig. 1 a RoboCart hardware, b Shopper scanning a barcode, c Modified barcode reader
The shopper finds the shelf and uses the barcode to scan the
barcodes on that shelf. The product name of each scanned
barcode is read to the shopper.
RoboCart software architecture is the popular three-tier
architecture consisting of the sense, plan, and act modules.
RoboCart’s shared state is updated and checked by these
modules and the user interface module.The connectivity of
different components in RoboCart’s architecture is shown in
Fig. 2.
The remainder of our article is organized as follows. In
Sect. 2, we talk about spaces in a supermarket. In Sect. 3,
we present the design and implementation of RoboCart. In
Sect. 4, we discuss the product selection problem and present
our solution to it. In Sect. 5, we present the results of longitu-
dinal robot-assisted shopping experiments with ten visually
impaired participants in a real supermarket. In Sect. 5, we Fig. 2 RoboCart’s component connectivity
present the results of the experiments with the three different
product selection interfaces: browsing, typing, and speech.
In Sect. 6, we discuss our findings from field trials and user
feedback. In Sect. 7, we present and discuss several user vs. locomotor) currently used by many researchers on spatial
comments. In Sect. 8, we present our conclusions. cognition of the visually impaired. Our trichotomy is cer-
tainly incomplete. We developed it solely for the purpose
of describing and analyzing the interactions between visu-
ally impaired shoppers and RoboCart. We believe that the
2 Spaces in the supermarket trichotomous ontology can be used to evaluate any assisted
shopping device for the visually impaired.
2.1 Spaces and interfaces RoboCart assists the shopper in two stages. It first guides
the shopper into the vicinity of the desired product and then
Spatial ontologies come about when we attempt to categorize instructs the shopper on how to maneuver the haptic space
space according to the ways we interact with it [40]. Freund- within that vicinity. In the first stage, RoboCart interfaces the
schuh and Egenhofer [7] give a comprehensive review of shopper to the locomotor space, guiding the shopper to the
previous work on categorization of space and distinguish six required aisle section and interacting with other shoppers by
categories based on manipulability, locomotion, and size and asking them to yield the way when a passage is blocked. In
use their ontology to describe previous ontologies of space in the second stage, RoboCart cues the shopper to some salient
the geography literature. We contribute to this line of research features of the environment near the haptic interface through
a trichotomous ontology of spaces in a supermarket. This tri- voice instructions grounded in the shopper’s egocentric frame
chotomy is an extension of the dichotomous ontology (haptic of reference.
123
4. 240 Intel Serv Robotics (2008) 1:237–251
To describe how the visually impaired shopper interacts performance may also depend on the intrinsic complexity
with the supermarket space using RoboCart, we introduce of the target space. The complexity of the target space can
the category of target space. The target space is the shopper- be characterized by the number of products, the product
centric subspace of the locomotor space in which the shopper density, the homogeneity of product types, the number
perceives the target product to be. The target space is always of shelves, package sizes, product layouts, presence of other
defined with respect to a specific target product. Haptic cues shoppers, etc.
in the target space act as external reference points during the
shopper’s maneuvering of the haptic space in the target space
until the haptic space contains the product.
3 Design of RoboCart
2.2 Inside the target space
3.1 Locomotor module
RoboCart eliminates the necessity of frame alignment by
restricting the target space to the searchable vicinity of the Autonomous indoor navigation was necessary to realize the
product. In the target space, the shopper is left with the task interface to the locomotor space. Since we had already devel-
of retrieving the product from the shelf and placing it into oped an orientation free RFID based navigation algorithm,
RoboCart’s basket. It seems reasonable to conjecture that we decided we use that in the supermarket. However, our
the shopper’s performance in the target space depends on the approach failed in unstructured open spaces. To deal with
shopper’s knowledge of the target space, the shopper’s sen- unstructured open spaces, we decided to use laser-based
sory, cognitive, and physical abilities, and the complexity of Monte Carlo Markov localization (MCL) [6], as it was
the target space. already implemented in ActivMedias Laser Mapping and
The knowledge of the target space includes knowledge of Navigation software.
distances, product shapes, neighboring products, the spatial Navigation in RoboCart is a partial realization of Kupi-
arrangement of shelves, and other haptic cues that may be ers´ 6 Spatial Semantic Hierarchy (SSH) [25]. The SSH is
c
unique to the shopper. If the robot is consistent overtime in a model to represent spatial knowledge. According to SSH,
how it sets up the target space with respect to a given prod- spatial knowledge can be represented in five levels: sensory,
uct and verbally orients the shopper in the target space, then, control, causal, topological and metric. Sensory level is the
as the shopper gets more exposure to the target space, the interface to the robot’s sensory system. RoboCarts primary
shopper’s performance should improve. sensors are a laser range finder, a camera, and an RFID reader.
The target space instructions given by the robot to the The control level represents the environment in terms of
shopper need not be exhaustive. It is unreasonable to expect control laws which have trigger and termination conditions
the robot designer to equip the robot with the knowledge of associated with them. The causal level describes the environ-
every possible haptic cue that may assist the shopper with ment in terms of views and actions. Views specify triggers;
retrieving the product. Instead, our assumption is that with actions specify control laws. For example, follow-hall can
time the shopper learns many haptic cues, such as level dif- be a control law triggered by start-of-hall and terminated by
ferences among shelf levels, bad barcodes, and protruding end-of-hall. The topological level of the SSH is a higher level
labels, that may be valid only for that shopper. Thus, the of abstraction, consisting of places, paths and regions, and
robot ensures the consistency of the target space and orients their connectivity, order and containment relationships. The
the shopper inside that target space. These haptic cues are metrical level describes a global metric map of the environ-
cognitive indices into the target space used by the shopper in ment within a single frame of reference.
product retrieval. The robot also emits audio beacons to help Several problems with MCL localization were discovered
the shopper with placing the found product into the basket. [10]. First, the robot’s ability to accurately localize rapidly
In the case of RoboCart, the audio beacon is the sonar click- deteriorated in the presence of heavy shopper traffic. Sec-
ing sound. RoboCart does not use sonars but turns them to ond, MCL sometimes failed due to wheel slippage on a wet
further orient the shopper in the target space. floor or due to the blind shopper inadvertently pulling on the
The shopper’s performance in the target space may also be handle. Third, since MCL relies exclusively on odometery to
conditional on the shopper’s sensory and physical abilities. localize itself along a long uniform hallway that lacks unique
The literature on the haptic space performance of the visually laser range signatures, it would frequently get lost in an aisle.
impaired [19,26,42] suggests that the degree of vision may Fourth, MCL localization frequently failed in the store lobby,
affect the efficiency of product retrieval. In addition, the shop- because the lobby constantly changed its layout due to pro-
per’s abilities to bend and scan products on bottom shelves, to motion displays, flower stands, product boxes. Finally, once
hold products, and to find barcodes on the shelves are likely MCL fails, it either never recovers, or recovers after a long
to contribute to the shopper’s performance. The shopper’s drift.
123
6. 242 Intel Serv Robotics (2008) 1:237–251
using non-speech audio messages called earcons to navigate
a menu hierarchy. In [44], the authors propose a new audi-
tory representation called spearcons. Spearcons are created
by speeding up a phrase until it is not recognized as speech.
Both approaches are however suitable for navigating menus,
and not large object hierarchies. Another approach for brows-
ing object hierarchies [35] uses conversational gestures like
open-object, parent, which are associated with specific nav-
igation actions. Work done on auditory interaction objects
[8,20], outlines the requirements for an auditory interaction
object that supports navigation of hierarchies.
In our implementation, we follow the functional require-
ments for accessible tree navigation tool given in [38]. In
that, the participants are required to find six objects from a
large object hierarchy. However, the evaluation is done only Fig. 3 a Prediction tree, b Possible search strings
to check for successful completion of the task, and not the
speed of completion.
In [5], the authors present a spoken user interface in which simply all entries i, such that Pi &S = S, where Pi is the
the task of invoking responses from the system is treated one N -bit vector of the ith product, S is the N -bit search vector,
of retrieval from the set of all possible responses. In our case, and ‘&’ is the bit-wise and operation.
the responses are the product names. They use the Spoken- This is a simple approach with two obvious problems: (1)
Query system [45] which is effective for searching spoken the users will need to type complete words, which is tedious
queries in large databases, is robust to environment noise and using just a numeric keypad; and (2) the search will fail if a
is effective as an user interface. word is spelled incorrectly. To solve the first problem, we use
In [37], the authors propose the use of subset languages word prediction where the whole word is predicted by look-
for interacting with collaborative agents. The advantage of ing at the partial word entered by the user. However, instead of
using subset language is that it can easily be characterized in having the user make a choice from a list of predicted words,
a grammar for a speech recognition system. In our case, the or waiting for the user to type the whole word, we search
subset language is nothing more that just the names of the the repository for all prediction options. To solve the second
products. To make the problem easier, we developed a gram- problem, we do not use the spell checker, but instead provide
mar where each rule consisted of a single word. The user the user with continuous feedback. Every time the user types
is thus limited to speaking one word at a time. This greatly a character, the number of results returned is informed to the
simplifies the grammar and reduces the misrecognitions. In user. The user obviously chooses not to type a character that
addition, it obviates the need for the user having to learn a returns zero results. At any point, the user can choose not to
complicated subset language. type the remaining word and continue to type the next word.
We implemented and tested three non-visual interfaces The predictions of partially typed words form a tree.
for product selection. The first one is a browsing interface, Figure 3 shows the prediction tree and the resultant search
where, given the product categories, the shopper browses strings when the user types “deo so ola”. The sharp-
through the complete hierarchy to find the desired product. cornered rectangles represent the keywords in the repository,
For the typing and speech interfaces, we employ an infor- also called keyword nodes. The round-cornered rectangles
mation retrieval based approach. In the typing based inter- are the partial search words entered by the user, also called
face, the shopper is required to type the search string using the partial nodes. Keyword nodes are nothing but all possi-
a numeric keypad. In the speech based interface, speech rec- ble extensions of their (parent) partial node, as found in the
ognition is used to form the search string. keyword repository.
A search string is associated with each keyword consisting
4.1 Searching the product repository of all keywords from the root to that keyword node. The
prediction subtree is terminated at the keyword node where
Each entry in the product repository is represented by an the associated search string returns zero results. For exam-
N -dimensional vector where N is the total number of unique ple, in Fig. 3a, the subtree rooted at solution, along the path
keywords in the repository. Thus, each vector is an N -bit vec- deodorant-solution will be terminated since the search string
tor with a bit set if the corresponding keyword exists in the “deodorant solution” returns zero results. Figure 3b shows the
product string. The search vector obtained from the search possible search string for the prediction tree in Fig. 3a. The
string is also an N -bit vector. The result of the search is numbers in the parentheses indicate the number of results
123
7. Intel Serv Robotics (2008) 1:237–251 243
returned for those search strings. The number after the ‘:’ 5.1.1 Participants
in the partial nodes indicates the total results returned by all
search strings corresponding to its children (keyword) nodes. Ten visually impaired participants from various locations in
At any point, the user can press the Return key to start Utah were recruited for the experiments through the Utah
browsing through the list of search results. We use Micro- Chapter of the National Federation of the Blind (NFB) in Salt
soft’s Speech API (SAPI) for the speech interface. The user Lake City, UT. The Utah NFB Chapter provided us with a list
is required to speak one word at a time, which is recognized of visually impaired Utah residents. Each individual on the
correctly is appended to the search string. Microsoft’s SAPI list was first contacted by e-mail. The e-mail briefly described
provides a set of alternates for every word recognized. These RoboCart and the experiments and asked the addressee if
are the alternate keywords which replace the predicted key- he or she would be interested in participating in the experi-
words in the prediction tree, and are thus incorporated in the ments. A brief phone interview was conducted with all those
search. However Microsoft SAPI did not provide alternates who responded positively. The inclusion criteria were: (1) the
in case of the grammar used for our product repository. Our participant must be ambulatory; (2) the participant may not
speech interface is just a simple version of the SILO (Speech have any hearing or cognitive impairments; (3) the partici-
In List Out) approach proposed in [5]. pant must understand English; and (4) the participant must
be willing to travel to Logan, UT, for a period of 2 days. Ten
participants were thus selected. Each of the selected partici-
5 Experiments pants was transported to Logan, UT, by vehicle for a period
of 2 days and was paid a $90 honorarium.
5.1 Robot-assisted shopping experiments
5.1.2 Procedure
The robot-assisted shopping experiments conducted with ten
visually impaired participants in Lee’s Market Place, a The procedure consisted of three stages. First, the individual
supermarket in Logan, UT. We formulated several research was given a 30 min introduction to RoboCart in our labora-
hypotheses to test how well RoboCart functions as haptic tory. The participant was trained in using RoboCarts keypad
and locomotor interface to the supermarket. As is often the and used RoboCart to navigate a short route in the office space
case with studies involving visually impaired participants, it around the laboratory. The participant was then asked to take
is not feasible to test in a statistically significant way all con- a technology readiness survey [11,33] which was used to cal-
tributing factors in a single study [1]. The main reason is that culate the participants Technology Readiness Index (TRI).
the visually impaired population in the US is not distributed Second, the participant was driven to Lees MarketPlace, a
evenly, with the majority living in just a few urban areas. supermarket in Logan, UT, and asked to use RoboCart to shop
Therefore, our hypotheses below address only a fraction of for several products. Twelve products were chosen from two
the factors outlined in the previous section. aisles: four products from bottom shelves, four products from
middle shelves, and four from top shelves. In Lees Market-
Hypothesis 1 If the robot is consistent overtime in how it
Place, each aisle consists of several shelf sections. A shelf
sets up the target space with respect to a given product and
section spans 4–5 m in length and consists of five to eight
verbally orients the shopper in the target space, the shopper’s
shelves. The selected products were divided into four sets.
efficiency of maneuvering the haptic space in the target space
Set 1 included three products on top shelves; Set 2 included
increases with experience in the target space where experi-
three products on middle shelves; Set 3 included three prod-
ence is measured as the number of shopping iterations.
ucts on bottom shelves; Set 4 included one product on a top
Hypothesis 2 The shopper’s efficiency of maneuvering the shelf, one product on a middle shelf, and one product on a
haptic space in the target space is inversely related to the bottom shelf. A single shopping trial consisted of the shopper
area of the target space. picking up RoboCart from the docking area near the super-
markets entrance, navigating to each of the three products
Hypothesis 3 The shopper’s efficiency of maneuvering the from a randomly chosen set, retrieving the products from the
haptic space in the target space is inversely related to the shelves, navigating to a designated cash register, unloading
complexity of the target space. the product on the belt, picking them up on the other side
Hypothesis 4 In the absence of any prior knowledge of the of the belt, and navigating back to the docking area near the
target space, minor differences in sensory abilities affect the entrance. Figure 4 shows the route along which the experi-
target space performance. ments were conducted.
Five shopping iterations were conducted for each product
Hypothesis 5 The location of the product on the shelf (top, set. The following measurements were taken during each run:
middle, bottom levels) affects the performance. (1) navigation time from location to location; (2) product
123
8. 244 Intel Serv Robotics (2008) 1:237–251
Fig. 4 Fuzzy areas in the grocery store environment
Fig. 5 Shopping Iteration vs. product selection time
retrieval time (time interval that starts when RoboCart stops
and instructs the shopper on where the product is in the tar-
that the shopper’s performance in the target space eventually
get space and ends when the shopper puts the product into
reaches an asymptote and becomes optimal given the partic-
RoboCarts basket); (3) the distance between the robot and the
ipant’s sensory, cognitive, and physical abilities.
product; and (4) the number of other shoppers encountered
To test Hypothesis 2, we measured the distance to the
on route. We also recorded observations regarding specific
product from where the robot stops. Since the description
haptic cues used by the participants to find products. Two
that RobotCart gives to the shopper after it brings the shopper
observers accompanied the participants on every run: the first
into the target space contains the direction and shelf number
observer was monitoring RoboCart; the second observer was
of the product (e.g., product X is on the third shelf on your
making and recording measurements and observations.
right), the distance can be considered as an accurate indica-
Third, each participant was given an exit questionnaire to
tor of the area of the target space. Surprisingly, we found a
assess the subjective impression of the shopping experience
very low correlation coefficient (Pearsons product moment)
with RoboCart. The questionnaire consisted of five questions
of 0.37 between the target space area and the shopper’s per-
whose answers were points on ten point Likert scales. If the
formance, which suggests that Hypothesis 2 may not hold
participants response was below 5, the participant was asked
for our sample. Our notes during the experiments and infor-
to comment on why he or she gave a low mark.
mal conversations with the participants after the experiments
suggest that this outcome may be explained by the presence
5.1.3 Results of various haptic cues in the target space. Certain haptic cues
in the target space help the shopper retrieve the target product
To test Hypothesis 1, the two populations were product retri- faster. One participant remembered the shape of the cooking
eval times for the first and the fifth iteration, respectively. oil spray can (one of the target products) and remembered
Each participant shopped for 12 products, which gave us 12 that it was located near a label protruding from the shelf.
items in each sample. The paired ttest at p = 0.01 was used Another participant remembered that a target product was
to compute the t statistics for all participants. The resulting P located ten barcodes to the left of a bad barcode which the
values for each of the t statistics are statistically significant at barcode reader could not read.
the P = 0.01 level. There appears to be sufficient evidence To test Hypothesis 3, we used the product density as the
to reject the null hypothesis that the shopper’s efficiency in measure of the target space complexity. The product density
the supermarket is not affected by the shopper’s exposure was computed as the number of products per foot between
to the target space. As Fig. 5 indicates, the product retrieval the robot and the target product on the correct shelf. The mea-
time reduces with every iteration. Thus, the participant’s effi- surement was motivated by our previous ergonomic studies
ciency in maneuvering the haptic space in the target space where it was found that the shoppers easily find the correct
appears to improve with experience. It is reasonable to expect shelf from the robot’s instructions but may have difficulties
123
9. Intel Serv Robotics (2008) 1:237–251 245
scanning the barcodes on the shelf. A general trend of We also tested whether the technology readiness index
decrease in efficiency with increasing complexity is observed. (TRI) [11,33] is an indicator of how well the shopper per-
Product retrieval time and target space complexity have a cor- forms with RoboCart. Each participant was given the TRI
relation coefficient of 0.7. survey. The survey consists of four groups of questions to be
To test Hypothesis 4, we compared how the participants answered on a Likert scale: optimism, innovativeness, dis-
performed with respect to each other. During the first shop- comfort, and insecurity. All four TRI components have low
ping iteration, the shopper does not have any knowledge correlation coefficients with performance: 0.47, 0.29, 0.53
about the target space. Since the target space complexity is and 0.22, respectively. While the TRI may be a reliable pre-
the same for all shoppers for any given product, it is sen- dictor of a users readiness to use desktop computer technol-
sible to suggest that the shopper’s individual sensory and ogies, it was not a reliable predictor of how the participants
physical abilities will make the greatest difference in the in our sample performed with RoboCart.
absence of any knowledge of the target space. Using the We conducted exit surveys to assess the participants’
data from all participants after the first shopping Fig. 4: per- subjective impression of safety, smoothness, comfort, infor-
formance versus iteration: learning in target space iteration, mativeness, and overall experience with RoboCart. The ques-
one-way ANOVA was used to test for a statistically signif- tions in our survey focused more on safety and comfort and
icant difference in the target space performance between are less generic than the NASA Task Load Index (NASA-
he participants. To make the test less biased, we removed TLX) questionnaire [15] that attempts to assess the perceived
Participant 10, because she had partial sight sufficient to level of workload using more abstract categories. We obtai-
detect product shapes and even read product labels at a close ned the average values for each quality, in the range of 1–10,
distance. with 1 being the worst and 10 being the best. The averages
Among the other nine participants, four had some resid- were as follows: safety = 8.66; smoothness = 7; comfort of
ual vision and could see color blobs. However, none of them navigation = 8.5; informative feedback from the robot = 6.66;
could read enlarged text. The other five participants were overall experience = 8.33. Low value for smoothness was
completely blind. No significant difference in performance given by one participant who thought that RoboCart made
was found (d f = 8, F = 1.504, P = 0.17). One explana- several sharp turns at the ends of aisles. Two participants gave
tion is that RoboCart minimizes minor sensory differences low scores on robot feedback from the robot and indicated
of the shoppers and enables them to retrieve products in the in their comments that they wanted more feedback from the
absence of any knowledge of the target space. robot during locomotion. We will address their comments in
One-way ANOVA was computed on the data from nine Sect. 5.
participants after the fifth iteration, i.e., after the participants
were exposed to the target spaces. It was found that there 5.2 Product selection interface experiments
was significant difference in performance between partici-
pants after exposure (d f = 8, F = 5.961, P < 0.0001). It was logistically difficult to acquire a large number of blind
Thus, minor differences in sensory abilities appear to make participants for the study. Experiments were conducted with
a difference given some knowledge of the target space. The five blind and five sighted-blindfolded participants. The par-
shopper may be able to utilize his or her sensory abilities ticipants’ age ranged from 17 through 32 years and all par-
optimally after receiving some exposure to the target space, ticipants were males. To avoid the discomfort of wearing a
but not before. blindfold, the keypad was covered with a box to prevent the
Hypothesis 5 is based upon our conjecture that some parts sighted participants from seeing it. The experiment was con-
of the target space are more easily accessible than others. We ducted in a laboratory setting. The primary purpose behind
expected that there might be significant differences in perfor- using sighted-blindfolded participants was to test whether
mance of the shoppers between retrieving products from top, they differed significantly from the blind participants, and
middle and bottom shelves. Using the data collected after the thus decide whether they can be used in further experiments
first iteration, one-way ANOVA was computed for three sam- to replace blind participants. We propose to test the following
ples of runs, each of size 40. It was found that there was signif- planned research hypotheses. Let H1-0, H2-0, H3-0, H4-0
icant difference in performance (d f = 2, F = 4.737, P = and H5-0 be the corresponding null hypotheses.
0.011). We were now interested in finding out if knowledge
of the target space was a factor. One-way ANOVA was com- Hypothesis 1 (H1) Sighted-blindfolded participants
puted on three samples of runs, each of size 40, obtained after perform significantly faster than the blind participants, on
the fifth iteration. No significant difference in performance average.
was found (d f = 2, F = 0.2701, P = 0.76). Some knowl-
edge of the target space appears to make different parts of Hypothesis 2 (H2) The browsing interface is significantly
the space equally accessible. worse than the typing interface.
123
10. 246 Intel Serv Robotics (2008) 1:237–251
Hypothesis 3 (H3) The browsing interface is significantly Table 1 Main effects
worse than the speech interface. Source Main Effects (ANOVA)
Hypothesis 4 (H4) The typing and speech interfaces are sig- Interface F(2, 243) = 42.84, P < 0.0001
nificantly different. Condition F(1, 243) = 9.8, P = 0.002
Participant F(8, 243) = 9.88, P < 0.0001
5.2.1 Procedure
We used the product repository obtained from the house- Model factors were: interface (three levels: browsing, typing,
hold products database website Household Products Data- speech), condition (two levels: blind, sighted-blindfolded),
base [16]. The repository contains 11,147 products organised participant (ten levels: nested within condition, five partici-
into a four level hierarchy. A grocery store typically con- pants per blind/sighted-blindfolded condition), and set (two
tains over 20,000 products. We used the household products levels: set-1 and set-2, each containing ten products). The
database since we were unable to obtain a sufficiently large ten products within each set were replications. Since each
grocery database. participant selected each product in each set, the ten product
The following procedure was followed for each responses for each set were repeated measures for this study.
participant. After arriving at the lab, the participant was first Since the browsing interface was missing for all participants
given an introduction about the purpose of the experiments. for set-2 products, models comparing selection time between
Each participant took on an avergae of 20 min to become sets included only typing and speech interfaces. The depen-
familiar with the interfaces. In that, the participant tried to dent variable was, in all models, the product selection time,
select three product using each interface, under the guidance with the exception of analyses using the NASA-TLX work-
of the experimenter. The session-1 of experiments started load measure. The overall models and all primary effects
after the training session. Each task was to select a product were tested using an α-level of 0.05, whenever these effects
using an interface. A set of ten randomly selected products constituted planned comparisons (see hypotheses). However,
(set-1) was formed. Each participant was thus required to in the absence of a significant overall F test for any given
perform 30 tasks (10 products × 3 interfaces). Because of model, post hoc comparisons among factor levels were con-
his schedule, one of the participants was unable to perform ducted using a Bonferroni adjusted α-level of 0.05/K , where
the ten browsing interface tasks. The product description K is the number of post hoc comparisons within any given
was broken down into four parts: product name, brand, spe- model, to reduce the likelihood of false significance.
cial description (scent/flavor/color), and the text that would
appear in the result. During the course of a task, if the partic- 5.2.3 Results
ipants forgot the product description, they were allowed to
revisit it by pressing a key. This would be avoided if a task For an overall repeated measures model which included the
rises out of the personal needs of the participants. effects of interface, condition, and participant (nested within
For session-2, another ten products (set-2) were randomly condition), and the interaction of interface with each of condi-
selected. After the initial 30 tasks in session-1,20 more tasks tion and participant, using only set-1 data, the overall model
were performed by each participant (10 products × 2 inter- was highly significant, F(26,243) = 7.00, P < 0.0001.
faces). Since the typing and speech interfaces were of more The main effects observed within this model are shown in
interest to us, we skipped the browsing interface tasks in ses- Table 1. All the main effects were significant. Interaction of
sion-2. The purpose of session-2 was to check if, and how interface × condition, F(2,243) = 0.05, P = 0.9558 and
much the participants improved on each of the two inter- interface × participant, F(14,243) = 1.17, P = 0.2976 was
faces, relative to the other. Since the all the tasks were not observed. Thus, mean selection time differed significantly
necessarily of the same complexity, there was no way to among interfaces, but the lack of interactions indicated that
check the learning effect. After both sessions, we conducted the interface differences did not vary significantly between
a subjective evaluation of the interfaces by monitoring the blind and sight-blindfolded groups, nor among individual
NASA Task Load Index (NASA-TLX) to each participant. participants. In the ANOVAs, note that the DoF for the error
The experiments were first conducted with five blind partic- is 243 because one of the participants did not perform the
ipants and then with five sighted, blindfolded participants. browsing tasks.
Mean selection time for the group of blind participants
5.2.2 Statistical methodology was 72.6 s versus a mean of 58.8 s for sighted-blindfolded
participants, and the difference in these means was signifi-
Repeated measures analysis of variance (ANOVA) models cant (t = 3.13, P = 0.0029). As might be expected, partici-
were fitted to the data using the S AS TM statistical system. pants differed on mean selection time. However, the majority
123
11. Intel Serv Robotics (2008) 1:237–251 247
Fig. 6 a Mean selection times for blind and sighted-blindfolded participants against all interfaces. b Change in mean selection times for typing
and speech interfaces from session-1 to session-2
of the differences among participants arose from blind par- Since this difference in typing and speech interfaces con-
ticipant 5, whose mean selection time of 120.9 (s) differed cerned us the most, we decided to compare the interfaces on
significantly from the mean selection time of all others par- the measures obtained from the session-2 of the experiments.
ticipants (whose mean times were in the 53–63 s range) (P < Set-2 was significantly faster than set-1, averaged over the
0.0001 for all comparisons between blind participant 5 and two interfaces and all participants (t = 6.14, P < 0.0001).
all other participants). When blind participant 5 was dropped Since we did not have a metric for the task complexity, we
from the analysis, the main effect of both condition and were unable to deduce if this result reflected the learning
participant(condition) became non-significant [F(1,216) = effect of the participants from session-1 to session-2. How-
0.16, P = 0.6928, and F(6,216) = 0.44, P = 0.8545, ever, a significant interaction of interface × set, F(1,382) =
respectively]. The interactions of interface with each of con- 13.8, P = 0.0002, was observed. The graph of the selection
dition and participant remained non-significant also. It times during sessions 1 and 2, against the interface type is
appears that on an average, when the outlier (participant 5) shown in Fig. 6b. It appears from the graph that the improve-
was removed, blind and sighted-blindfolded participants did ment with the typing interface was much larger than that
not really differ. Thus the data leads us to confirm the null with the speech interface. The reduction in selection times
hypothesis H1-0, and thereby reject H1. from session-1 to session-2 varied significantly for the typ-
A graph of the mean selection times of the blind and the ing and speech interface. This was probably because, since
sighted-blindfolded participants for each interface is shown the participants were already much faster with the speech
in Fig. 6a. The almost parallel lines for the blind and sighted- interface than the typing interface during session-1, they had
blindfolded participants suggest that there is no interaction much less room to improve with the speech interface during
between the interface and the participant type, which is also session-2.
confirmed by the ANOVA result presented earlier. In other A strong Pearson’s product moment correlation was found
words, the result suggests that the interface which is best for between selection time and query length for both typing and
sighted-blindfolded users may be the best for blind users. speech interfaces, with r = 0.92 and r = 0.82, respectively.
The main effect of interface type as shown in Table 1, To calculate the PPM correlation, we averaged the selection
suggests that, on average (over all participants), two or more times over all products having the same query length. This
interfaces differ significantly. Mean selection times for the just confirms the obvious that, on average, selection time
three interfaces were, respectively: 85.5, 74.1, and 37.5 (s). increases with number of characters typed or words spoken.
Post hoc pairwise t tests showed that the typing interface was We used a between-subjects design to study the data
faster than the browsing interface (t = 2.10, P = 0.0364), obtained from the NASA TLX questionnaire. The modality
although statistical significance is questionable if the type was the independent variable and mental demand, frus-
Bonferroni adjusted is used. Consequently, we were unable tration and overall workload were the dependent
to reach a definite conclusion about H2. Each of the brows- variables.
ing and typing interfaces was significantly slower than the A one-way ANOVA indicated that there was a signifi-
speech interface (t = 8.84, P < 0.0001, and t = 6.74, cant difference among the three modalities in terms of the
P < 0.0001, respectively). This led us to reject the null hy- mental demand, frustration, and overall workload, [F(2,27)
pothesess H3-0 and H4-0. = 16.63, P < 0.0001], [F(2,27) = 16.63, P < 0.0001],
123
12. 248 Intel Serv Robotics (2008) 1:237–251
Table 2 Post hoc ttests to study
workload, mental demand and Post hoc ttests
frustration imposed by the Browsing × Typing Browsing × speech Typing × speech
modalities (∗ indicates a
signifcant test) Mental t = 1.075, ∗t = 3.822, ∗t = 4.011,
demand P = 0.2962 P = 0.0012 P = 0.0008
Frustration ∗t = 6.974, t = 1.348, ∗t = 4.428,
P <0.0001 P = 0.1833 P = 0.0004
Overall ∗t = 3.369, ∗t = 4.126, t = 0.9910,
workload P = 0.0034 P = 0.0006 P = 0.3348
Table 3 Mean values of mental demand, frustration and overall shopper acquire individually valid haptic cues not explicitly
workload mentioned by the system. Minor differences in sensory abil-
Browsing Typing Speech ities appear to make a difference only after some knowledge
(0–100) (0–100) (0–100) of the target space is acquired by the shopper.
Mental demand 45.6 35.9 13.4 We have argued that, to guarantee independence, an
Frustration 47.8 1.8 34 assistive shopping device for the visually impaired must pro-
Overall workload 64.4 41.65 35 vide the shopper with interfaces to the haptic and locomotor
spaces in the supermarket. Robotic mobility aids, such as
GuideCane [41], Guido [14], and Harunobu-6 [32] focus on
and [F(2, 27) = 10.07, P = 0.0005] respectively). Post the haptic space and leave the handling of the locomotion
hoc pair-wise t-tests for the three dependent variables with space to the user. Recent proposals to address the visually
Bonferoni adjusted α-level of 0.016 are shown in Table 2. impaired shopping problem with purely wearable means also
The mean values of mental demand, frustration and overall stay in the haptic space [28]. In our opinion, these solutions
workload for the three modalities are shown in Table 3. are unlikely to result in independent shopping for the blind,
either large scale or small scale, because they focus only on
one space (haptic) and do not address the loss of indepen-
6 Discussion dence. We believe that, to remain independent, the visually
impaired shopper must be able not only to retrieve products
The shopper’s efficiency in maneuvering the haptic space in from shelves but to handle the locomotor space in the super-
a given target space appears to improve with experience pro- market. The trichotomous ontology of spaces in the super-
vided that the target space remains the same over time. By market developed in this paper can be used to evaluate the
providing an effective interface to the locomotor space of performance of any assisted shopping device.
the supermarket, the robot ensures the stability of the target Are robots necessary to overcome this barrier for the
space with respect to each product over time, which gives the visually impaired? For large scale independent shopping the
shopper an opportunity to learn how to maneuver the haptic answer appears to be affirmative, because locomotion with a
space inside the target space. shopping cart in front may not be possible in the absence of
Ideally, the target space coincides with the haptic space. vision. Whether or not wearable small scale shopping solu-
In practice, the best the robot can do is to bring the size of the tions will be found for the visually impaired may well depend
target space as close to the size of the haptic space as pos- on the ability of those solutions to provide effective interfaces
sible. A low correlation coefficient was found between the to the locomotor spaces of supermarkets.
target space area and the shopper’s performance in our sam-
ple, which may indicate that the target space sizes ensured
by RoboCart and the egocentric verbal instructions inside the
7 User comments
target spaces contribute to faster product retrievals and min-
imize the effect of minor differences in visual abilities. The
User comments provide valuable insights into the limitations
shopper’s performance in the target space is likely to become
of our system. In this section, several user comments are ana-
optimal with respect to the shopper’s sensory, cognitive, and
lyzed.
physical abilities. It remains an open question what is the
optimal size of the target space for a given shopper and a
given product. 7.1 Comments on RoboCart
Our results suggest that the paucity of verbal instructions
in target spaces may be desirable for some shoppers, because Comment 1 Instead of just following the robot, doing noth-
it does not raise the shopper’s cognitive load and enables the ing, I would like to know what products I am passing by.
123
13. Intel Serv Robotics (2008) 1:237–251 249
The comment suggests that some participant may want to Comment 6 How will this thing help me with produce?
get more out of their shopping experience than just buying
the required products. It is an open research question how It will not. RoboCart can take the shopper to the target
much information is enough. More broadly, this question is space of a given item in the produce section and instruct the
related to the question of understanding and representing the shopper on how to find the target item. However, RoboCart
users context to provide more useful computational services cannot help the shopper recognize the freshness of fruits and
dynamically [4,17,39]. In the context of a supermarket, it vegetables. We consider this important problem to be outside
is not practical to tell the user about each and every prod- of the scope of our project.
uct that the robot passes by. Perhaps, informing the shopper
about sales and higher product categories, e.g., cereal, canned 7.2 Comments on product selection interfaces
products, etc., is what some participants would like RoboCart
to inform them about. Comment 1 Though the speech-based interface works fast,
the speech recognition errors are really frustrating.
Comment 2 It would really help me if it can tell me in which
direction to go after I scan every barcode. That way I can The comment conforms with the high frustration level
know if I miss a barcode. indicated by the participants in the NASA TLX survey for
the speech based interface. Often times some words could
During the experiments it was observed that some partici- not be recognized and the participants would just sit there
pants would often overestimate the distance from the robot to wondering what to do next. Such situations would be greatly
the product. In such cases it was hard for them to realize that discomforting for the shopper in a real supermarket.
they should scan the next barcode in the opposite direction.
What is needed is a detailed spatial topology of barcodes in Comment 2 It would be nice to have some combination of
target spaces. Once such a topology is available, RoboCart speech and typing. That way I can speak a word that I can’t
can give the user step-wise instructions to find the product in spell.
the target space: scan left, scan right, move one shelf up, etc.
A hybrid interface which allows the user to switch between
Comment 3 It would be really helpful if the robot could typing and speech could be advantageous. Though there is
stop exactly in front of the product. been significant research in multi-modal interfaces, none of
these interfaces are targeted specifically toward rapid selec-
RoboCart cannot stop exactly in front of the product. Even tion in large hierarchies.
if the robot knew the x and y coordinates of every product
in the store, it would still be unable to stop exactly in front Comment 3 Browsing is useful when you are not sure what
of each product due to the presence of other shoppers in the exactly you want to buy. It helps you to kind of browse through
store. When this was explained to the participant, the partic- different kinds of products out there.
ipant gave the following comment.
We believe that the browsing interface is not appropri-
Comment 4 Okay, I understand that. However, it is easier ate for rapid product selection. However, the shopper can
to stay behind the robot than go around it. So it would be be allowed to switch to the browsing mode when the exact
nice if the robot could stop after passing the product. product is not known. Also the shopper might be able to
select a product category fairly rapidly if the product hierar-
This is a fair comment that gives us a valuable insight on chy is trimmed to contain only the high level categories. The
how we can improve the robot’s interface to the target space browsing efficiency with a trimmed hierarchy would however
in the future. depend on the size and organization of the trimmed hierarchy.
Comment 5 It would be better if the robot can emit some
beeps instead of the sonar clicking. The clicking sound gets
irritating real soon and is difficult to trace in presence of 8 Conclusions
background noise.
An assistive shopping device for the visually impaired that
This comment suggests that such seemingly minor issues ensures independence must provide the shopper with inter-
like the sonar clicking can affect the shopper’s experience. faces to the haptic and locomotor spaces in the supermar-
Since the shopper must return to RoboCart after retrieving ket. These interfaces should either eliminate the necessity of
the product from the shelf, RoboCart should emit adequate frame alignment or enable the visually impaired to reliably
sonification beacons in the target space. What beacons are align the frames. Robots can function as interfaces to the hap-
adequate is a research question that we hope to address in tic and locomotor spaces in supermarkets by eliminating the
the future. need of frame alignment in the locomotor space and cuing the
123
14. 250 Intel Serv Robotics (2008) 1:237–251
shopper to the salient features of the target space sufficient 6. Fox D (1998) Markov localization: a probabilistic framework for
for product retrieval. mobile robot localization and navigation. Ph.D. thesis, University
of Bonn, Germany
The paucity of verbal instructions in target spaces may 7. Freundschuh S, Egenhofer M (1997) Human conceptions of
be desirable for some shoppers, because it does not raise spaces: implications for geographic information systems. Trans
the shopper’s cognitive load and enables the shopper acquire GIS 2(4):361–365
individually valid haptic cues not explicitly mentioned by 8. Gaver W (1989) The sonicfinder: An interface that uses auditory
icons. Hum Comput Interact 4(1):57–94
the system. Minor differences in sensory abilities appear to 9. Gharpure C (2004) Orientation-free rfid based navigation in a
make a difference only after some knowledge of the target robotic guide for the visually impaired. Master’s thesis, Utah State
space has been acquired. In other words, the shopper may University, USA
be able to put to personally optimal use his or her sensory 10. Gharpure C, Kulyukin V, Kutiyanawala A, Jiang M (2006) Passive
radio frequency exteroception in robot assisted shopping for the
abilities after receiving some exposure to the target space.
blind. In: Proceedings of the international conference on ubiqui-
An interface which enables rapid selection of products is tous intelligence and computing (UIC), Wuhan and Three Gorges,
essential. Browsing interfaces which work well for smaller China
menu driven hierarchies, performs poorly for large object 11. Gockley R, Mataric MJ (2006) Encouraging physical therapy com-
pliance with a hands-off mobile robot. In: Proceedings of human
hierarchies. A hybrid interface combining speech and typing robot interaction (HRI) conference, Salt Lake City, USA
would be helpful. 12. Golledge R, Klatzky R, , Loomis J (1998) Cognitive mapping and
To remain independent, the visually impaired shopper wayfinding by adults without vision. In: Portugali J(ed) The con-
must be able not only to retrieve products from shelves but to struction of cognitive maps, pp 215–246
13. Hahnel D, Burgard W, Fox D, Fishkin K, Philipose M (2003) Map-
handle the locomotor space in the supermarket. Thus, robot ping and localization with rfid technology. Technical report, IRS-
assistants may be necessary to for the visually impaired in TR-03-014, Intel Research Institute, Seattle, WA
large scale independent shopping, because locomotion with 14. Haptica C (2001) Guido (c). http://www.haptica.com
a shopping cart in front may not be possible in the absence of 15. Hart S, Staveland L (1988) Development of nasa-tlx: results
of empirical and theoretical research. In: Hancock P, Meshkati N
vision. Whether or not wearable small scale shopping solu- (eds) Human mental overload. North Holland, Amsterdam,
tions will be found for the visually impaired may well depend pp 139–183
on the ability of those solutions to provide effective interfaces 16. Household-Products-Database (2004) http://www.householdprod
to the locomotor spaces of supermarkets. ucts.nlm.nih.gov
17. Jacquet C, Bellik Y, Bourda Y (2004) A context-aware locomo-
tion assistance device for the blind. In: Fincher S, Markopoulos P,
Acknowledgments V. A. Kulyukin would like to acknowledge that ´
Moore D, Ruddle R (eds) People and computers XVIII fb design
this research has been supported, in part, through NSF grant (IIS-
for life. Springer, London
0346880), the National Eye Institute of the National Institutes of Health
18. Kantor G, Singh S (2002) Priliminary results in range-only locali-
under Grant (1 R41 EY017516-01A1), and three Community University
zation and mapping. In: IEEE conference on robotics and automa-
Research Initiative (CURI) grants (CURI-04, CURI-05, and CURI-06)
tion, Washington, DC
from the State of Utah. We would like to thank Mr. Lee Badger, the
19. Kay L (1974) A sonar aid to enhance spatial perception of the blind:
owner of Lee’s MarketPlace in Logan, UT, for allowing us to use his
engineering design and evaluation. Radio Electron Eng 44:40–62
supermarket as a research site. We are grateful to Mr. Ron Gardner, Pres-
20. Klante P (2004) Auditory interaction objects for mobile applica-
ident of the Utah NFB Chapter, for his help with recruiting participants.
tions. In: Proceedings of 7th international conference on work with
We would like to thank the participants for their time and feedback.
computing systems, WWCS2004, Kuala Lumpur
21. Kulyukin V, Gharpure C, Nicholson J (2005) Robocart: toward
robot-assisted navigation of grocery stores by the visually
impaired. In: Proceedings of the IEEE/RSJ international confer-
ence on intelligent robots and systems (IROS)
References 22. Kulyukin V, Gharpure C, Nicholson J, Osborne G (2006) Robot-
assisted wayfinding for the visually impaired in structured indoor
1. Bradley N, Dunlop M (2005) An experimental investigation into environments. Auton Robots 21(1):29–41. http://dx.doi.org/10.
wayfinding directions for visually impaired people. Ubiquitous 1007/s10514-006-7223-8
Comput 9:395–403 23. Kulyukin V, Gharpure C (2006) Ergonomics-for-one in a robotic
2. Brent J, Modi N (2000) Shopping aid for the visually impaired. shopping cart for the blind. In: Proceedings of the 2006 ACM con-
In: Proceedings of conference on rehabilitation engineering and ference on human–robot interaction, Salt Lake City, UT
assistive technology society of North America 24. Kulyukin V, Gharpure C, Pentico C(2007) Robots as interfaces to
3. Brewster SA (1998) Using nonspeech sounds to provide naviga- haptic and locomotor spaces. In: Proceedings of the 2007 ACM
tion cues. ACM Trans Comput Hum Interact 5(3):224–259 conference on human–robot interaction, Arlington, VA
4. Dey A, Abowd G (2000) Towards a better understanding of context 25. Kupiers B (2000) The spatial semantic hierarchy. Artif Intell
and context-awareness. In: Proceedings of the CHI workshop on 119:191–233
the what, who, where, and how of context-awareness, The Hague, 26. Lahav O, Mioduser D (2003) A blind persons cognitive mapping
The Netherlands of new spaces using a haptic virtual environment. J Res Spec Educ
5. Divi V, Forlines C, Gemert J, Raj B, Schmidt-Nielsen B, Witten- Needs 3(3):172–177
burg K, Woelfel P, Zhang F (2004) A speech-in list-out approach 27. Lalatendu S, Pierce N, Anijo M (2006) Cat eye: an assistance sys-
to spoken user interfaces. In: Proceedings of human language tech- tem for independent shopping. In: Proceedings of international con-
nologies, Boston, MA ference on aging, disability and independence, St Petersburg, FL
123
15. Intel Serv Robotics (2008) 1:237–251 251
28. Lanigan P, Paulos A, Williams A, Narasimhan P (2006) Trinetra: 38. Smith A, Francioni J, Anwar M, Cook J, Hossain A, Rahman
assistive technologies for the blind. In: CyLab technical report M (2004) Non-visual tool for navigating hierarchical structures.
CMU-CyLab-06-006, Carnegie Mellon University In: Proceedings of international ACM SIGACCESS conference on
29. Millar S (1982) The problem of imagery and spatial development computers and accessibility (ASSETS), Atlanta, GA
in the blind. In: de Gelder B(ed) Knowledge and representation, 39. Tversky B and Lee P (1998) How space structures language. In:Fre-
pp 111–120 ksa C, Habel C, Wender KF(eds) Spatial cognition an interdisci-
30. Millar S (1995) Understanding and representing spatial informa- plinary approach to representing and processing spatial knowledge,
tion. Br J Vis Impairment 13:8–11 pp 157–175
31. Millar S (1997) Theory, experiment and practical application 40. Tversky B, Morrison J, Franklin N, Bryant D (1999) Three spaces
in research on visual impairment. Eur J Psychol Educ 12:415– of spatial cognition. Prof Geogr 51:516–524
430 41. Ulrich I, Borenstein J (2001) The guidecane: applying mobile
32. Mori H, Kotani S (1998) Robotic travel aid for the blind: Harunobu- robot technologies to assist the visually impaired. IEEE Trans Syst,
6. In: Second European conference on disability, virtual reality, and Man Cybern A Syst Hum 31:131–136
assistive technology, Sovde, Sweden 42. Ungar S (2000) Cognitive mapping without visual experience. In:
33. Parasuraman A (2000) Technology reacdiness index(tri): a Cognitive mapping: past present and future.Routledge,London
multiple-item scale to measure readiness to embrace new tech- 43. W3C (2000) Web content accessibility guidelines 1.0. In: Web
nologies. J Serv Res, p 307 accessibility Initiative
34. Passini R, Proulx G (1988) Wayfinding without vision An exper- 44. Walker B, Nance A, Lindsay J (2006) Spearcons: speech-based
iment with congenitally totally blind people. Environ Behav earcons improve navigation performance in auditory menus. In:
22:227–252 Proceedings of the 12th international conference on auditory dis-
35. Raman TV (1997) Concrete implementation of an audio desktop. play, London, UK
In: Auditory user interfaces toward the speaking computer 45. Wolf P, Woelfel J, Gemert J, Raj B, Wong D (2004) Spokenquery:
36. Scooter S, Helal S (2004) A passive rfid information grid for loca- an alternate approach to choosing items with speech. In: Mitsubishi
tion and proximity sensing for the blind user. University of Florida Electric Research Laboratory, TR-TR2004-121. Cambridge, MA,
Technical Report number TR04-009 USA
37. Sidner C, Forlines C (2002) Subset language for conversing with
collaborative interface agents. In: Mitsubishi Electric Research
Laboratory, TR-TR2002-36, Cambridge, MA, USA
123