PDR Challenge in Warehouse Picking and Virtual Mapping Party

国立研究開発法人
PDR Challenge in
Warehouse Picking and
Virtual Mapping Party
Takeshi Kurata12
1AIST, 2Univ. of Tsukuba

PDR（Pedestrian Dead-Reckoning）
Estimates velocity vector, relative altitude, and action
type by measurements from a wearable sensor module.
 Wearing a sensor module on waist (2D SHS (Steps and Heading Systems) PDR)
 Easy to wear and maintain
 Easy to measure data for action recognition
 Relatively easily apply for handheld setting compared to shoe-mounted PDR
(3D-INS (Inertial Navigation System) PDR)
2
Handheld PDR From PDR to PDRplus
10-axis sensors
• Accelerometers
• Magnetic sensors
• Gyro sensors
• Barometer
Shoe-mounted PDR
Waist-worn PDR

ISMAR2003 Image registration + PDR
PLANS2010 PDRplus (PDR + Action recognition)
PLANS2014 Handheld PDR (Walking Direction Estimation)
2015- 2015-
PDR Module
ICAT2006 PDR + GPS + RFID
G-spatial EXPO2010 PDR demo on iPhone 4
ICServ2013 Best Paper: Computer-Supported QC Circle (CSQCC) with PDR
IPIN2015, 2016 10-axes PDR, Time-series sub-optimization in SDF
2011-
Docomo map navi (As of Sep, 2016, 480
areas including subways and underground
shopping arcades in Japan)
2014-
産総研技術移転ベンチャー
PDR Benchmark
Standardization Committee
Established in 2014 (Chair: T. Kurata, 33 organizations
in Japan as of April, 2016)
Overview: History of PDR in AIST
AIST Start-up
AcademiaIndustry
PDR: Pedestrian Dead Reckoning
SDF: Sensor Data Fusion (Hybrid Positioning)
RFID: Radio Frequency Identifier
GPS: Global Positioning System

AR by PDR + Image registration (1999-2003)
Panorama-based Annotation:
IWAR1999, ISWC2001,
ISMAR2003
G
Environmental map
A
B C D
E
A
B
C
F
Input frames
Position at which
a panorama is taken
Position
Direction
235 [deg]
5 [deg]
From the user’s
camera
Located Orientated
4

In the year of 2010
• iPhone 4: the first popular consumer mobile device equipped
with 9-axis sensors including accelerometers, magnetic
sensors, and gyro sensors
6
G-spatial EXPO 2010:
Handheld PDR on iPhone 4
(Worldʼs first-ever live demo)
PLANS2010, PLANS2014

Frontier of PDR:
Walking direction estimation
8
• Tutorial: Personal Navigation with Handheld Devices by Valerie Renaudin, IPIN 2015.

Frontier of PDR:
Walking direction estimation
9
• Tutorial: Personal Navigation with Handheld Devices by Valerie Renaudin, IPIN 2015.
• Long Paper: Christophe Combettes, Valerie Renaudin, Comparison of Misalignment
Estimation Techniques Between Handheld Device and Walking Directions, IPIN 2015.
• FIS was proposed by Kourogi and Kurata in PLANS 2014.
“Globally, the FIS method provides better results than
the other two methods.” by IFSTTAR
Frequency analysis of Inertial Signals
Forward and Lateral Acc. Modeling
Principal Component Analysis

Global Trend on PDR
PDR R&D players have rapidly indicated their
presence all over the world on and after 2010.
Movea (France)
Sensor Platforms (USA)
CSR (UK)
TRX (USA)
Trusted Positioning (Canada)
11
Acquired by QualcommAcquired by InvenSenseAcquired by InvenSense
Acquired by Audience
Indoo.rs (USA)
SFO
Acquired by TDK?

Standardization on PDR Benchmarking
• PDR related R&D is highly active worldwide: Necessity for sharing
common measures.
• Description of the performance should be unified in spec sheets
and scientific papers.
• Different measures from absolute positioning methods such as
GNSS, Wi-Fi, and BLE are required for PDR, which is a method of
relative positioning.
• PDR Benchmark Standardization Committee was established in
2014 as a platform of the grassroots activity.
13https://www.facebook.com/pdr.bms

Scene in data collection
15

Multi-Algorithm On-Site Evaluation System
• Evaluates the accuracy of each PDR algorithm automatically as often
as sensor data is uploaded to the server
• Provides trajectory images so that participants can compare their PDR
• algorithms in real time.
16
http://pdrsv.hasc.jp
K. Kaji, K. Kanagu, K. Murao, N. Nishio, K. Urano, H. Iida, N. Kawaguch, Multi-Algorithm On-Site Evaluation
System for PDR Challenge, ICMU2016.

UbiComp/ISWC 2015 PDR Challenge Corpus
• Is now open to the public. (http://hub.hasc.jp/)
17
Routes 5
Devices 7
Subjects 93
# of pedestrian sensing data 241
# of pedestrian sensing data with
calibration data
230
# of pedestrian sensing data with
LIDAR data
10
Avg. of walking time [sec] 101
Avg. of moving distance [m] 115
Avg. of angular change [°] 606
K. Kaji, M. Abe, W. Wang, K. Hiroi, and N. Kawaguchi, UbiComp/ISWC 2015 PDR challenge corpus, HASCA2016
(UbiComp2016 Proceedings: Adjunct), pp.696-704
Statistics of the corpus
Detailed route statistics of pedestrian
sensing data with calibration data

Open Data Contest in Logistics &
PDR Challenge in Warehouse Picking
• Open data contest in
logistics by Frameworx
– Submission: 2016/4/18-
2016/7/18
– Award ceremony: 2016/9/12
• PDR Challenge in Warehouse
Picking
– Will be held as an international
contest in IPIN 2017
18

PDR Challenge Series
• Ubicomp/ISWC 2015 PDR Challenge
– Scenario: Indoor Navigation
– On-site
– Continuous walking while keeping watching the navigation
screen by holding the smartphone
– Several minutes per trial
• 2017 PDR Challenge in Warehouse Picking
– Scenario: Picking work in a warehouse
– Off-site
– Not only walking but various actions including picking and
carrying
– Several hours per trial
– Will be held in IPIN 2017
19

Examples of picking workersʼ trajectories estimated by
PDR + WMS (Warehouse Management System)
20

How to design benchmark Indicators?
• Other aspects to be considered
– Reliability: Different measures from absolute positioning
methods are required for PDR
– Efficiency: Power consumption
– Repeatability: Temperature Hysteresis, Magnetic field, etc.
– Representativeness: How to hold, Route shape, etc.
21
Benchmark indicators of vision-based spatial registration and tracking for MAR
(ISO/IEC WD 18520) (PEVO: Projection Error of Virtual Objects)

How to compare and visualize?
22
Easy Difficult
Method 1
Easy Difficult
Method 2

How to compare and visualize?
23
Easy Difficult
Method1Method2

Competitions: IPIN and the others
(cf. EvAAL presentation in IPIN 2105 etc.)
24
IPIN year EvAAL, IPSN, UbiComp/ISWC
Zurich, Switzerland 2010 universAAL is launched
Guimaraes, Portugal 2011 EvAAL: indoor localization
Sidney, Australia 2012 EvAAL: + activity recognition
Montbeliard, France 2013 EvAAL: same as 2012
Busan, Korea
1st IPIN competition
2014
EvAAL: 3 floors, smartphone
IPSN: infrastruc. based + free
Banff, Canada
EvAAL-ETRI comp.
2015
EvAAL-ETRI: 6 floors, on/off-site
IPSN: infrastruc. based + free
UbiComp/ISWC: 2 floors, smartphone PDR,
90 subjects
Madrid, Spain
Indoor Localization
Competition
2016
IPIN: smartphone (on/off-site), PDR, Robot
IPSN: infrastruc. based + free, 2D/3D

IPIN2017
25

Virtual Mapping Party
26
which enables the participants to contribute to the accessible information collection
for visually impaired people from anywhere and at any time.

Characteristics of each mapping work
27
Type of activities location Time Remarks
Conventional
mapping party
On-site Sync.
Face to face communication
Deep understanding of real conditions
Mandatory skill for organizing events
Up to weather
Mapping party
utilizing
smartphones
app.
On-site
Any time
(Async.)
Mapping while commuting
Easy to contribute
Deep understanding of real conditions
Position of contents depending on localization
methods
Mapping party
utilizing
crowdsourcing
image sharing
service
Anywhere
(Off-site)
Any time
(Async.)
Crowdsourcing
Remote mapping
Easy to contribute anytime and anywhere
Depend on shared data
Limited understanding of real conditions
Virtual mapping
party
Anywhere
(Off-site)
Any time
(Async.)
Crowdsourcing
Remote mapping
Easy to contribute anytime and anywhere
Easy to measure contentsʼ position
Easy to verify registered contents
Mandatory pre-recording

How to decide POI/POR position
28
Using intersection of
line of sight and
the ground
Triangulating
with plural
panoramas.
POI: Point Of Interest (Landmark such as Store, restaurant, hospital, facilities, etc.)
POR: Point Of Reference (specific point location the existence of which is easily recognized for
confirming routes such as characteristic shape and material of ground (steps, stairs, sloop, door),
sound/noise, and scent/odor.)

Screenshots for the virtual
mapping interface.
29

Desktop vs. Smartphone VR
30
• The number of registered POI/PORs in WSs held at Miraikan : 598 (42 participant,
6 one-hour Workshops)
• Sense of Immersion: Desktop << Cardboard HMD
• Registration efficiency: Desktop/Smartphone=1.43
ＰＯＩ/POR/Request on OSM

Feedback from WS participants
31
Categories of
feedback
Positive feedback Negative feedback/ Suggestions
About VR experience
with omnidirectional
images/movies and
3D sound
I like the function for 3D sounds.
3D sounds seem to be very useful, since visually impaired
people can confirm amount of traffic on roads.
I could more realistically experience the VR scene by
omnidirectional movies than still omnidirectional images.
Estimating direction of sound sources was
difficult.
The quality of the images/movies was not
perfect.
About devices for VR
experience
I like instant HMD, since we can experience VR with what
I have.
I like VR experience with Oculus VR HMD since I can
realistically experience by movies.
I like Samsung's Gear VR HMD, because the image quality
looked good and it was confortable for wearing.
It took a while to get used to HMD, and I got
tired when I wore HMD.
Mapping with PC is better in terms of degree
of fatigue.
An instant HMD was not so comfortable for
wearing.
I thought wearing HMD on glass was difficult.
About user interfaces
I like the function for pointing in first personʼs view not
map view.
PC is the easiest platform for inputting
POR/POI.
About AR Tactile map
I like the function for sending request by visually impaired
people.
The accuracy of gesture recognition for AR
tactile map needs to be improved.
About POR/POIs
There are so many POR/POIs in the display. I think it becomes more clear if the displayed contents are
limited to nearby contents.
I found empty POR/POIs without detailed information. I think filtering of the registered contents are
required.
Other suggestions
I wondered if the system could support communication between participants.
I would like to regularly contribute virtual mapping parties from my home.
It was the most beneficial application of VR I have ever experienced.

Usage of the AR tactile map for
virtual mapping party
• Allowing the visually-impaired people to join
the mapping party by gesture
– Search: Confirming POR/POI on the tactile map with
sound for telling existence of POR/POI where user
touches
– Tap: Confirming POR/POI
with Text-to-Sound when
user taps the specific point
– Double Tap: Requesting
the POR/POI registration for
specific points of the map

AR tactile maps
with HP Sprout

Automatic identification and
tracking of tactile maps
34
• ORB Feature point detector/local feature descriptor is used for identifying
tactile map with RGB image
• Estimating homography matrix between rectified image templates and
input image

PDR Challenge in Warehouse Picking and Virtual Mapping Party

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (20)

Similar to PDR Challenge in Warehouse Picking and Virtual Mapping Party

Similar to PDR Challenge in Warehouse Picking and Virtual Mapping Party (20)

More from Kurata Takeshi

More from Kurata Takeshi (20)

Recently uploaded

Recently uploaded (20)

PDR Challenge in Warehouse Picking and Virtual Mapping Party