Review of PDR Challenge in Warehouse Picking and Advancing to xDR Challenge

National Institute of Advanced Industrial Science and Technology
Review of PDR Challenge in
Warehouse Picking and Advancing to
xDR Challenge (Regular Paper)
Ryosuke Ichikari1, Ryo Shimomura12, Masakatsu Kourogi1,
Takashi Okuma1, and Takeshi Kurata123
AIST1, University of Tsukuba2,
3Sumitomo Electric Industries, Ltd
1
IPIN 2018@Nantes
SS A Survey on Indoor Localization Competitions
10:20 - 12:20 Sept. 26th, 2018

PDR Benchmarking
• The number of research and development
for PDR is increasing all over the world
• PDR is relative tracking, therefore, the
evaluation method should be different from
one for absolute tracking methods such as
GPS, WiFi based localization
• The format of specification/evaluation in
the papers should be unified for far
comparison
2
Benchmark
Indicators ＋
Benchmarking
Process
Trial Set
(Dataset)＋

Type of Benchmarking/Competitions
• Off-site benchmarking/competitions
– Organizer prepare shared dataset and evaluate
submitted result by compared with ground truth
e.g.) EvAAL/IPIN competitions (Off-site）
ISMAR tracking competition(Off-site）
KITTI Vision Benchmarking
• On-site benchmarking/competitions
– Organizer prepare controlled environment, participants
gather at the site and run their method.
– Check points are measured by organizer in advance.
e.g.) IPIN competition(On-site), ISMAR tracking
competition(On-site), UbiComp/ISWC PDR Challenge
3

A Short Survey of Indoor
Localization Competitions
4

National Institute of Advanced Industrial Science and Technology 5
Comparison of Indoor Localization Competitions
PerfLoc by NIST EvAAL/IPIN Competitions
Microsoft
Competition@IPSN
Scenario
30 Scenarios
(Emergency scenario)
Smart House/Assisted Living
Competing maximum
accuracy in 2D or 3D
Walking
/motion
walking/running/
backwards/sidestep/
crawling/pushcart/ elevators
(walked by actors on planed
path with CPs)
Walking/Stairs/Lift/Phoning
/Lateral movement (walked by
Actors on planed path with
CPs)
Depends on operators
(developers can operate
their devices by themselves
On-site or
Off-site
Off-site competition and
Live demo
Separated On-site and Off-
site tracks
On-site
Target
Methods
Arm-mounted smartphone
based localization method
(IMU, WiFi, GPS, Cellular)
Off-site: Smartphone base
On-site : Smartphone base/
any body-mounted device
(separated tracks)
2D:Infra-free methods
3D:Allowed to arrange Infra.
(# of anchor and type of devices
are limited on 2018)
# of
people
and trial
1 person × 4 devices
(at the same time)
× 30 scenarios
Depends on year and track
(e.g. 9 trials, 2016T3)
N/A
Time
per trial
Total 16 hours
Depends on year and track
(e.g. 15 mins (2016T1,T2), 2
hours (2016T3))
N/A
Evaluation
metric
SE95
(95% Spherical Error)
75 Percentile Error Mean error
History 1 time (2017-2018)
7 times (2011,2012,2013,
(EvAAL),2014,2015(+ETRI),20
16,2017(EvAAL/IPIN))
5 times
(2014,2015,2016,2017,
2018)

Comparison of PDR Challenges
Ubicomp/ISWC 2015 PDR
Challenge
PDR Challenge in Warehouse
Picking in IPIN 2017
xDR Challenge for
Warehouse Operations
2018
Scenario
Indoor pedestrian
navigation
Picking work inside a
logistics warehouse
(Specific Industrial Scenario)
General warehouse
operations including picking,
shipping and driving forklift
Walking
/motion
Continuous walking while
holding smartphone and
looking at navigation screen
Includes many motions
involved in picking work, not
only walking
involved in picking, shipping
operations and, not only
walking. Some workers
may drive forklift
On-site or
off-site
Data collection: on-site
Evaluation: off-site
Off-site Off-site
Number
of people
and trial
90 people, 229 trials 8 people, 8 trials
34 people + 6 forklifts,
170 trials (PDR) +
30 trials (VDR)
Time
per trial
A few minutes About 3 hours About 8 hours
Evaluation
metric
Mean Error, SD of Error
Integrated Evaluation
(integrated by accuracy,
naturalness, warehouse
dedicated metrics)
dedicated metrics)
Remark
Collection of data of
participants walking. The
data are available at HASC
(http://hub.hasc.jp/) as
corpus data
Competition over integrated
position using not only PDR,
but also correction information
such as BLE beacon signal,
picking log (WMS), and maps
Consists of PDR and VDR
tracks．Referential motion
captured by MoCap. also
shared for introducing
typical motions.

（Original）PDR Challenge
• Held in Ubicomp/ISWC2015 as a competition for
PDR
• Purposes：
– Evaluating various PDR engines
– Evaluating evaluation metrics
– Collecting dataset by participants
• The participants submit
source code of algorithm
based on app skeleton.
7

• Held “PDR Challenge in warehouse picking” as the official
competition track (Track 4) of IPIN (Int. Conf. on Indoor Positioning
& Indoor Navigation) in Sapporo, Japan 2017
• IPIN 2017 competitions: total 4 tracks, 20 teams joined (CN5, KR4,
JP3, TW2, GE2, AU1, FR1, CL1, PT)
• Track4: Off-site PDR competition for tracking workers during picking
operations in actual warehouse
• Chairs: Ichikari & Kourogi from AIST WMS: warehouse management system
PDR Challenge in Warehouse Picking

How to collect raw sensor data by smartphone
 Device：Android Nexus 5
 Collected raw sensor data required for
PDR
（Acceleration，Angular velocity，
magnetism，Atmospheric pressure, BLE
signal log from beacons）～100Hz
 Sensor data for Calibration
 Measured while fixing for a whole
 8 shaped movement for magnetic
calibration

Shared data for PDR Challenge 2017
10
P1 P2
P3
WMS picking log
(WMS:Warehouse Management System)
Sensor raw data
(gyro. ,acceleration, magnetism,
air pressure, BLE log)
Specification of the target
warehouse
(Map, Shelves, Obstacle)

Ground truth is extracted from WMS picking log
11
X Y time1
X Y time2
X Y time3
WMS data
X Y time n-2
X Y time n-1
Share with
participants
for error correction
Hided for
participants and
used for evaluation
(as GT)
X Y time n-3
X Y time0
X Y time 1
X Y time 2
X Y time 3
X Y time 0
X Y time n-2
X Y time n-1
X Y time n
X Y time n-3
X Y time n
Original data

12
Metrics related to accuracy
- Metric related to integrated positioning error
(Ed)
- Metric related to PDR error based on EAG
(Es)
Metrics related to the trajectory naturalness
- Metric related to the naturalness of travel
speed (Ev)
- Metric related to position measurement
output frequency (Ef)
Specific metrics for warehouse picking scenario
- Metric related to collision with obstacles (Eo)
- Metric related to motions during picking
work (Ep)
Evaluation Metric
Comprehensive evolutions（C.E.）
Evaluation Metric

Metric related to integrated positioning error
(Ed)
13
Pos Err: Vector consists of all positional
error at the ground truth available points
Calculating positional error as
Euclid distance between
ground truth by WMS and
points in submitted trajectory

Metric related to PDR accumulating error (Es)
14
EAG
EAG
EAG
EAG
Es is based on speed for error
accumration called
Error Accumulation Gradient (EAG)
calculated by elapsed time from
correcting points and error
Ref.：

Example of error plot for obtaining EAG
15
In PDR Challenge2017, we adopt simple linear regression whose
intersection equal to 0 for calculating representative EAG
EAG[m/sec]

Metric related to motions during picking work
(Ep)
16
・Evaluating naturalness during picking operations
・Assumed that worker should stop in front of selves during
the picking
・We check the total length of the movement measured from
1.5 sec before the picking to 1.5 sec after the picking is less
than 1 meter

Metric related to the naturalness of travel speed
(Ev)
17
・We assumed natural human travel speed is less than 1.5m/sec.
・We check whether local speed is less than 1.5m/sec or not

Metric related to collision with obstacles (Eo)
18
・We assume that ideal trajectories do not enter the area
where employees cannot walk inside or pass over
・This metric quantify the degree of incursion of the trajectories
into the forbidden area
・We defined a tolerance area with 0.17 m width around
borders of the forbidden area for ignoring small amount of the
incursion.

Metric related to output frequency (Ef)
19
・Ideal trajectory should frequently submitted with less than
certain length of interval.
⇒ Frequency is less than 1Hz should be deducted
・Local frequency can be calculated by elapsed time from the
previous submitted points in the trajectory

Example of submitted trajectories
• Terminal 2
20

Results of the evaluation metrics and final C.E.
Result of Metric Eo
Result of EAG for Metric Es
PDR Challenge in Warehouse Picking
Example：EAG:0.12m/sec. ⇒ 7.2m/min,
Target accuracy of the integrated localization: 4.0 m
Guideline of absolute positioning method: every 30 sec. & 0.4 m or less error (3.6+0.4=4.0m)
Team Ed Es Ep Ev Eo Ef Median
of Error
[m]
Median
of EAG
[m/s]
C.E.
Team1 66.876 93.692 97.195 99.998 51.821 11.323 10.606 0.173 68.652
Team2 71.524 94.872 43.545 100 100 9.258 0.150 90.419
Team3 76.459 95.333 72.719 87.835 93.549 99.271 7.827 0.141 89.161
Team4 51.934 90.769 84.965 95.657 59.623 99.239 14.939 0.230 74.948
Team5 78.386 96.308 97.484 99.093 45.530 100 7.268 0.122 78.336
AIST 80.272 96.718 81.057 98.711 89.968 95.879 6.721 0.114 90.836
99.876

Findings from the PDR Challenge 2017
• We successfully encouraged the competitors to
develop practical localization methods for warehouse.
• Are the weights for evaluation metrics appropriate?
– The weighs for obstacle interference seems to be too high.
– Weights for error evaluation should be added.
• There are not much difference in PDR error
evaluation metric Es
– Is linear regression for calculating EAG is appropriate?
• Workers who drive forklift cannot be tracked.
• The competitors (researchers) are not familiar with
the warehouse operations.
• Scale should be extended.

Re-consideration of the weights (1)
• Results with original weights at the competition
Ed(median_error):Es(EAG):Ep(picking):Ev(velocity):Eo(obstacle):Ef(frequency)
=20%:20%:5%:15%:30%:10%
23
Team Ed Es Ep Ev Eo Ef Median
Error
[m]
Median
EAG
[m/s]
C.E.
Team1 66.876 93.692 97.195 99.998 51.821 11.323 10.606 0.173 68.652
Team2 71.524 94.872 43.545 100 99.876 100 9.258 0.150 90.419
Team3 76.459 95.333 72.719 87.835 93.549 99.271 7.827 0.141 89.161
Team4 51.934 90.769 84.965 95.657 59.623 99.239 14.939 0.230 74.948
Team5 78.386 96.308 97.484 99.093 45.530 100 7.268 0.122 78.336

Re-consideration of the weights (2)
• Trial by adding weight for error metrics (Es(median_error),
Ed(EAG)) and reducing weight of Eo(obstacle).
• The weights will be replaced with updated one.
24
Ed(median_error):Es(EAG):Ep(picking):Ev(velocity):Eo(obstacle):Ef(frequency)
=30%：30%：5%:10%：15%：10%
Team Ed Es Ep Ev Eo Ef 50%
eCDF
[m]
50%
eCDF
[m/s]
C.E.
Team1 66.876 87.053 97.195 99.998 51.821 11.323 10.606 0.173 69.944
Team2 71.524 89.474 43.545 100 99.876 100 9.258 0.150 85.458
Team3 76.459 90.421 72.719 87.835 93.549 99.271 7.827 0.141 86.443
Team4 51.934 81.053 84.965 95.657 59.623 99.239 14.939 0.230 72.577
Team5 78.386 96.308 97.484 99.093 45.530 100 7.268 0.122 84.021

xDR Challenge for Warehouse Operations
• xDR Challenge for Warehouse Operations 2018 was held as an
sequel competition to "PDR Challenge in Warehouse Picking“
• Host：PDR Benchmark Standardization Committee
• Competition of Dead-Reckoning for Pedestrian and Vehicle
– xDR=PDR+VDR
• Important dates
– Testing period: Mid May to Mid Aug., 2018
– Results submission due: 18th Sept, 2018
– Announcement of Winners:
Now (This Special Session)
• Sponsors:
25

Two competition tracks
• Individually determining winners in each tracks
• PDR-Track
– Tracking workers who move by foot during warehouse
operations
– Shared data: Smartphone sensor data for PDR, BLE tag’s
signal, warehouse’s spec, Partial WMS log. reference movie
for typical picking
• VDR-Track
– Tracking forklift driven by employee during warehouse
operations
– Smartphone sensor data measured by attaching
smartphone onto the forklifts
– Shared data: Smartphone sensor data for VDR, BLE tag’s
signal, warehouse’s spec, forklift spec., partial WMS log.
– Sample data with known path are prepared. (For beginners)
26

Ubicomp/ISWC 2015 PDR
Challenge
PDR Challenge in Warehouse
Picking in IPIN 2017
xDR Challenge for
Warehouse Operations
2018
Scenario
Indoor pedestrian
navigation
Picking work inside a
logistics warehouse
(Specific Industrial Scenario)
General warehouse
operations including picking,
shipping and driving forklift
Walking
/motion
Continuous walking while
holding smartphone and
looking at navigation screen
involved in picking work, not
only walking
involved in picking, shipping
operations and, not only
walking. Some workers
may drive forklift
On-site or
off-site
Data collection: on-site
Evaluation: off-site
Off-site Off-site
Number
of people
and trial
90 people, 229 trials 8 people, 8 trials
34 people + 6 forklifts,
170 trials (PDR) +
30 trials (VDR)
Time
per trial
A few minutes About 3 hours About 8 hours
Evaluation
metric
Mean Error, SD of Error
dedicated metrics)
dedicated metrics)
Remark
Collection of data of
participants walking. The
data are available at HASC
(http://hub.hasc.jp/) as
corpus data
Competition over integrated
position using not only PDR,
but also correction information
such as BLE beacon signal,
picking log (WMS), and maps
Consists of PDR and VDR
tracks．Referential motion
captured by MoCap. also
shared for introducing
typical motions.
Comparison of PDR Challenges

Prizes
• VDR Track: (a) { VDR module (SSEI, Eq. to 200,000) +
Android IoT device BL-02 (BIGLOBE) + 150,000 cash}
or (b) {200,000 cash +BL-02}
• PDR Track: (a) {TECCO (Eq. to 100,000) + BL-02 +
150,000 cash}
or (b) {200,000 cash + BL-02}
• Runner-Up：BL-02 + 100,000 cash
28
VDR module TECCO BL-02

VDR Module (SUC-VDR100)
• Relative vehicle tracking module by VDR
• Manufactured by Sugihara SEI, and its vibration-
based VDR algorithm is licensed by AIST
• Spec
– Battery life:
12 hours
29

Tecco (TC-A01)
• Wearable RFID-tag reader for picking operation
• Manufactured by GOV
• Spec
– Interface: Bluetooth
– Battery life: 40 hours
30

Android IoT device (BL-02)
• Android IoT device sold by BIGLOBE
• Ideal characteristics for industrial use
– LTE capable
– No camera (for security/confidential point of view)
– 10-axis sensors for PDR
– Android version (6.0) is fixed.
31

Rigorous evaluation of error accumulation by BUP
(BLE Unreachable Period)
• Intentionally deleting partial BLE signal logs from the
test data for evaluating PDR accumulated error
Period when BLE signals are deleted: BLE unreachable period (BUP)
• WMS Reference points provided before and after BUP
BUPBUP BUPRSSI
of BLE tag.
Evaluation Points by WMS
⇒ Position data are hided
Correction Points by WMS
⇒ Position data are provided
t
Evaluating positional errors of integrated localization system with BLE beacon
Evaluating accumulated errors caused by only PDR
Emedian_error
Eaccum_error

Conclusions
• PDR Challenge Advancing to xDR Challenge.
• We promoted and encouraged, but the starting development of VDR
is too challenging.
• The dataset of PDR Challenge & xDR Challenge are very realistic and
good for evaluating practical performance
– Real data
– Huge data including dirty data.
• Survey of the exiting competitions
• Future Works
– Competition whose targets seamlessly changing walking and driving
forklifts (Universal Dead-reckoning: UDR)
– Dead-reckoning for other targets (e.g. Drone Dead-reckoning: DDR)
– Utilizing whole body motion
– Competition of action recognition
33

Thank you!
• Contact Info.
– Ryosuke Ichikari, Ph.D.（r.ichikari@aist.go.jp）
34

Review of PDR Challenge in Warehouse Picking and Advancing to xDR Challenge

Recommended

Recommended

More Related Content

Similar to Review of PDR Challenge in Warehouse Picking and Advancing to xDR Challenge

Similar to Review of PDR Challenge in Warehouse Picking and Advancing to xDR Challenge (20)

Recently uploaded

Recently uploaded (20)

Review of PDR Challenge in Warehouse Picking and Advancing to xDR Challenge

Editor's Notes