Download to read offline
![National Institute of Advanced Industrial Science and Technology
Example of error plot for obtaining EAG
15
In PDR Challenge2017, we adopt simple linear regression whose
intersection equal to 0 for calculating representative EAG
EAG[m/sec]](https://image.slidesharecdn.com/ipin2018sspresentationpublishrp-181005072447/85/Review-of-PDR-Challenge-in-Warehouse-Picking-and-Advancing-to-xDR-Challenge-15-320.jpg)
![National Institute of Advanced Industrial Science and Technology 21
Results of the evaluation metrics and final C.E.
Result of Metric Eo
Result of EAG for Metric Es
PDR Challenge in Warehouse Picking
Example：EAG:0.12m/sec. ⇒ 7.2m/min,
Target accuracy of the integrated localization: 4.0 m
Guideline of absolute positioning method: every 30 sec. & 0.4 m or less error (3.6+0.4=4.0m)
Team Ed Es Ep Ev Eo Ef Median
of Error
[m]
Median
of EAG
[m/s]
C.E.
Team1 66.876 93.692 97.195 99.998 51.821 11.323 10.606 0.173 68.652
Team2 71.524 94.872 43.545 100 100 9.258 0.150 90.419
Team3 76.459 95.333 72.719 87.835 93.549 99.271 7.827 0.141 89.161
Team4 51.934 90.769 84.965 95.657 59.623 99.239 14.939 0.230 74.948
Team5 78.386 96.308 97.484 99.093 45.530 100 7.268 0.122 78.336
AIST 80.272 96.718 81.057 98.711 89.968 95.879 6.721 0.114 90.836
99.876](https://image.slidesharecdn.com/ipin2018sspresentationpublishrp-181005072447/85/Review-of-PDR-Challenge-in-Warehouse-Picking-and-Advancing-to-xDR-Challenge-21-320.jpg)
![National Institute of Advanced Industrial Science and Technology
Re-consideration of the weights (1)
• Results with original weights at the competition
Ed(median_error):Es(EAG):Ep(picking):Ev(velocity):Eo(obstacle):Ef(frequency)
=20%:20%:5%:15%:30%:10%
23
Team Ed Es Ep Ev Eo Ef Median
Error
[m]
Median
EAG
[m/s]
C.E.
Team1 66.876 93.692 97.195 99.998 51.821 11.323 10.606 0.173 68.652
Team2 71.524 94.872 43.545 100 99.876 100 9.258 0.150 90.419
Team3 76.459 95.333 72.719 87.835 93.549 99.271 7.827 0.141 89.161
Team4 51.934 90.769 84.965 95.657 59.623 99.239 14.939 0.230 74.948
Team5 78.386 96.308 97.484 99.093 45.530 100 7.268 0.122 78.336](https://image.slidesharecdn.com/ipin2018sspresentationpublishrp-181005072447/85/Review-of-PDR-Challenge-in-Warehouse-Picking-and-Advancing-to-xDR-Challenge-23-320.jpg)
![National Institute of Advanced Industrial Science and Technology
Re-consideration of the weights (2)
• Trial by adding weight for error metrics (Es(median_error),
Ed(EAG)) and reducing weight of Eo(obstacle).
• The weights will be replaced with updated one.
24
Ed(median_error):Es(EAG):Ep(picking):Ev(velocity):Eo(obstacle):Ef(frequency)
=30%：30%：5%:10%：15%：10%
Team Ed Es Ep Ev Eo Ef 50%
eCDF
[m]
50%
eCDF
[m/s]
C.E.
Team1 66.876 87.053 97.195 99.998 51.821 11.323 10.606 0.173 69.944
Team2 71.524 89.474 43.545 100 99.876 100 9.258 0.150 85.458
Team3 76.459 90.421 72.719 87.835 93.549 99.271 7.827 0.141 86.443
Team4 51.934 81.053 84.965 95.657 59.623 99.239 14.939 0.230 72.577
Team5 78.386 96.308 97.484 99.093 45.530 100 7.268 0.122 84.021](https://image.slidesharecdn.com/ipin2018sspresentationpublishrp-181005072447/85/Review-of-PDR-Challenge-in-Warehouse-Picking-and-Advancing-to-xDR-Challenge-24-320.jpg)
![Sensor Fusion Study - Real World 2: GPS & INS Fusion [Stella Seoyeon Yang]](https://cdn.slidesharecdn.com/ss_thumbnails/gpsins-200817095309-thumbnail.jpg?width=640&height=640&fit=bounds)
Review of PDR Challenge in Warehouse Picking and Advancing to xDR Challenge
- 1. National Institute ofAdvanced Industrial Science and Technology Review of PDR Challenge in Warehouse Picking and Advancing to xDR Challenge (Regular Paper) Ryosuke Ichikari1, Ryo Shimomura12, Masakatsu Kourogi1, Takashi Okuma1, and Takeshi Kurata123 AIST1, University of Tsukuba2, 3Sumitomo Electric Industries, Ltd 1 IPIN 2018@Nantes SS A Survey on Indoor Localization Competitions 10:20 - 12:20 Sept. 26th, 2018
- 2. National Institute ofAdvanced Industrial Science and Technology PDR Benchmarking • The number of research and development for PDR is increasing all over the world • PDR is relative tracking, therefore, the evaluation method should be different from one for absolute tracking methods such as GPS, WiFi based localization • The format of specification/evaluation in the papers should be unified for far comparison 2 Benchmark Indicators ＋ Benchmarking Process Trial Set (Dataset)＋
- 3. National Institute ofAdvanced Industrial Science and Technology Type of Benchmarking/Competitions • Off-site benchmarking/competitions – Organizer prepare shared dataset and evaluate submitted result by compared with ground truth e.g.) EvAAL/IPIN competitions (Off-site） ISMAR tracking competition(Off-site） KITTI Vision Benchmarking • On-site benchmarking/competitions – Organizer prepare controlled environment, participants gather at the site and run their method. – Check points are measured by organizer in advance. e.g.) IPIN competition(On-site), ISMAR tracking competition(On-site), UbiComp/ISWC PDR Challenge 3
- 4. National Institute ofAdvanced Industrial Science and Technology A Short Survey of Indoor Localization Competitions 4
- 5. National Institute ofAdvanced Industrial Science and Technology 5 Comparison of Indoor Localization Competitions PerfLoc by NIST EvAAL/IPIN Competitions Microsoft Competition@IPSN Scenario 30 Scenarios (Emergency scenario) Smart House/Assisted Living Competing maximum accuracy in 2D or 3D Walking /motion walking/running/ backwards/sidestep/ crawling/pushcart/ elevators (walked by actors on planed path with CPs) Walking/Stairs/Lift/Phoning /Lateral movement (walked by Actors on planed path with CPs) Depends on operators (developers can operate their devices by themselves On-site or Off-site Off-site competition and Live demo Separated On-site and Off- site tracks On-site Target Methods Arm-mounted smartphone based localization method (IMU, WiFi, GPS, Cellular) Off-site: Smartphone base On-site : Smartphone base/ any body-mounted device (separated tracks) 2D:Infra-free methods 3D:Allowed to arrange Infra. (# of anchor and type of devices are limited on 2018) # of people and trial 1 person × 4 devices (at the same time) × 30 scenarios Depends on year and track (e.g. 9 trials, 2016T3) N/A Time per trial Total 16 hours Depends on year and track (e.g. 15 mins (2016T1,T2), 2 hours (2016T3)) N/A Evaluation metric SE95 (95% Spherical Error) 75 Percentile Error Mean error History 1 time (2017-2018) 7 times (2011,2012,2013, (EvAAL),2014,2015(+ETRI),20 16,2017(EvAAL/IPIN)) 5 times (2014,2015,2016,2017, 2018)
- 6. National Institute ofAdvanced Industrial Science and Technology 6 Comparison of PDR Challenges Ubicomp/ISWC 2015 PDR Challenge PDR Challenge in Warehouse Picking in IPIN 2017 xDR Challenge for Warehouse Operations 2018 Scenario Indoor pedestrian navigation Picking work inside a logistics warehouse (Specific Industrial Scenario) General warehouse operations including picking, shipping and driving forklift Walking /motion Continuous walking while holding smartphone and looking at navigation screen Includes many motions involved in picking work, not only walking Includes many motions involved in picking, shipping operations and, not only walking. Some workers may drive forklift On-site or off-site Data collection: on-site Evaluation: off-site Off-site Off-site Number of people and trial 90 people, 229 trials 8 people, 8 trials 34 people + 6 forklifts, 170 trials (PDR) + 30 trials (VDR) Time per trial A few minutes About 3 hours About 8 hours Evaluation metric Mean Error, SD of Error Integrated Evaluation (integrated by accuracy, naturalness, warehouse dedicated metrics) Integrated Evaluation (integrated by accuracy, naturalness, warehouse dedicated metrics) Remark Collection of data of participants walking. The data are available at HASC (http://hub.hasc.jp/) as corpus data Competition over integrated position using not only PDR, but also correction information such as BLE beacon signal, picking log (WMS), and maps Consists of PDR and VDR tracks．Referential motion captured by MoCap. also shared for introducing typical motions.
- 7. National Institute ofAdvanced Industrial Science and Technology （Original）PDR Challenge • Held in Ubicomp/ISWC2015 as a competition for PDR • Purposes： – Evaluating various PDR engines – Evaluating evaluation metrics – Collecting dataset by participants • The participants submit source code of algorithm based on app skeleton. 7
- 8. National Institute ofAdvanced Industrial Science and Technology 8 • Held “PDR Challenge in warehouse picking” as the official competition track (Track 4) of IPIN (Int. Conf. on Indoor Positioning & Indoor Navigation) in Sapporo, Japan 2017 • IPIN 2017 competitions: total 4 tracks, 20 teams joined (CN5, KR4, JP3, TW2, GE2, AU1, FR1, CL1, PT) • Track4: Off-site PDR competition for tracking workers during picking operations in actual warehouse • Chairs: Ichikari & Kourogi from AIST WMS: warehouse management system PDR Challenge in Warehouse Picking
- 9. National Institute ofAdvanced Industrial Science and Technology How to collect raw sensor data by smartphone Device：Android Nexus 5 Collected raw sensor data required for PDR （Acceleration，Angular velocity， magnetism，Atmospheric pressure, BLE signal log from beacons） ～100Hz Sensor data for Calibration Measured while fixing for a whole 8 shaped movement for magnetic calibration
- 10. National Institute ofAdvanced Industrial Science and Technology Shared data for PDR Challenge 2017 10 P1 P2 P3 WMS picking log (WMS:Warehouse Management System) Sensor raw data (gyro. ,acceleration, magnetism, air pressure, BLE log) Specification of the target warehouse (Map, Shelves, Obstacle)
- 11. National Institute ofAdvanced Industrial Science and Technology Ground truth is extracted from WMS picking log 11 X Y time1 X Y time2 X Y time3 WMS data X Y time n-2 X Y time n-1 Share with participants for error correction Hided for participants and used for evaluation (as GT) X Y time n-3 X Y time0 X Y time 1 X Y time 2 X Y time 3 X Y time 0 X Y time n-2 X Y time n-1 X Y time n X Y time n-3 X Y time n Original data
- 12. National Institute ofAdvanced Industrial Science and Technology 12 Metrics related to accuracy - Metric related to integrated positioning error (Ed) - Metric related to PDR error based on EAG (Es) Metrics related to the trajectory naturalness - Metric related to the naturalness of travel speed (Ev) - Metric related to position measurement output frequency (Ef) Specific metrics for warehouse picking scenario - Metric related to collision with obstacles (Eo) - Metric related to motions during picking work (Ep) Evaluation Metric Comprehensive evolutions（C.E.） Evaluation Metric
- 13. National Institute ofAdvanced Industrial Science and Technology Metric related to integrated positioning error (Ed) 13 Pos Err: Vector consists of all positional error at the ground truth available points Calculating positional error as Euclid distance between ground truth by WMS and points in submitted trajectory
- 14. National Institute ofAdvanced Industrial Science and Technology Metric related to PDR accumulating error (Es) 14 EAG EAG EAG EAG Es is based on speed for error accumration called Error Accumulation Gradient (EAG) calculated by elapsed time from correcting points and error Ref.：
- 15. National Institute ofAdvanced Industrial Science and Technology Example of error plot for obtaining EAG 15 In PDR Challenge2017, we adopt simple linear regression whose intersection equal to 0 for calculating representative EAG EAG[m/sec]
- 16. National Institute ofAdvanced Industrial Science and Technology Metric related to motions during picking work (Ep) 16 ・Evaluating naturalness during picking operations ・Assumed that worker should stop in front of selves during the picking ・We check the total length of the movement measured from 1.5 sec before the picking to 1.5 sec after the picking is less than 1 meter
- 17. National Institute ofAdvanced Industrial Science and Technology Metric related to the naturalness of travel speed (Ev) 17 ・We assumed natural human travel speed is less than 1.5m/sec. ・We check whether local speed is less than 1.5m/sec or not
- 18. National Institute ofAdvanced Industrial Science and Technology Metric related to collision with obstacles (Eo) 18 ・We assume that ideal trajectories do not enter the area where employees cannot walk inside or pass over ・This metric quantify the degree of incursion of the trajectories into the forbidden area ・We defined a tolerance area with 0.17 m width around borders of the forbidden area for ignoring small amount of the incursion.
- 19. National Institute ofAdvanced Industrial Science and Technology Metric related to output frequency (Ef) 19 ・Ideal trajectory should frequently submitted with less than certain length of interval. ⇒ Frequency is less than 1Hz should be deducted ・Local frequency can be calculated by elapsed time from the previous submitted points in the trajectory
- 20. National Institute ofAdvanced Industrial Science and Technology Example of submitted trajectories • Terminal 2 20
- 21. National Institute ofAdvanced Industrial Science and Technology 21 Results of the evaluation metrics and final C.E. Result of Metric Eo Result of EAG for Metric Es PDR Challenge in Warehouse Picking Example：EAG:0.12m/sec. ⇒ 7.2m/min, Target accuracy of the integrated localization: 4.0 m Guideline of absolute positioning method: every 30 sec. & 0.4 m or less error (3.6+0.4=4.0m) Team Ed Es Ep Ev Eo Ef Median of Error [m] Median of EAG [m/s] C.E. Team1 66.876 93.692 97.195 99.998 51.821 11.323 10.606 0.173 68.652 Team2 71.524 94.872 43.545 100 100 9.258 0.150 90.419 Team3 76.459 95.333 72.719 87.835 93.549 99.271 7.827 0.141 89.161 Team4 51.934 90.769 84.965 95.657 59.623 99.239 14.939 0.230 74.948 Team5 78.386 96.308 97.484 99.093 45.530 100 7.268 0.122 78.336 AIST 80.272 96.718 81.057 98.711 89.968 95.879 6.721 0.114 90.836 99.876
- 22. National Institute ofAdvanced Industrial Science and Technology Findings from the PDR Challenge 2017 • We successfully encouraged the competitors to develop practical localization methods for warehouse. • Are the weights for evaluation metrics appropriate? – The weighs for obstacle interference seems to be too high. – Weights for error evaluation should be added. • There are not much difference in PDR error evaluation metric Es – Is linear regression for calculating EAG is appropriate? • Workers who drive forklift cannot be tracked. • The competitors (researchers) are not familiar with the warehouse operations. • Scale should be extended.
- 23. National Institute ofAdvanced Industrial Science and Technology Re-consideration of the weights (1) • Results with original weights at the competition Ed(median_error):Es(EAG):Ep(picking):Ev(velocity):Eo(obstacle):Ef(frequency) =20%:20%:5%:15%:30%:10% 23 Team Ed Es Ep Ev Eo Ef Median Error [m] Median EAG [m/s] C.E. Team1 66.876 93.692 97.195 99.998 51.821 11.323 10.606 0.173 68.652 Team2 71.524 94.872 43.545 100 99.876 100 9.258 0.150 90.419 Team3 76.459 95.333 72.719 87.835 93.549 99.271 7.827 0.141 89.161 Team4 51.934 90.769 84.965 95.657 59.623 99.239 14.939 0.230 74.948 Team5 78.386 96.308 97.484 99.093 45.530 100 7.268 0.122 78.336
- 24. National Institute ofAdvanced Industrial Science and Technology Re-consideration of the weights (2) • Trial by adding weight for error metrics (Es(median_error), Ed(EAG)) and reducing weight of Eo(obstacle). • The weights will be replaced with updated one. 24 Ed(median_error):Es(EAG):Ep(picking):Ev(velocity):Eo(obstacle):Ef(frequency) =30%：30%：5%:10%：15%：10% Team Ed Es Ep Ev Eo Ef 50% eCDF [m] 50% eCDF [m/s] C.E. Team1 66.876 87.053 97.195 99.998 51.821 11.323 10.606 0.173 69.944 Team2 71.524 89.474 43.545 100 99.876 100 9.258 0.150 85.458 Team3 76.459 90.421 72.719 87.835 93.549 99.271 7.827 0.141 86.443 Team4 51.934 81.053 84.965 95.657 59.623 99.239 14.939 0.230 72.577 Team5 78.386 96.308 97.484 99.093 45.530 100 7.268 0.122 84.021
- 25. National Institute ofAdvanced Industrial Science and Technology xDR Challenge for Warehouse Operations • xDR Challenge for Warehouse Operations 2018 was held as an sequel competition to "PDR Challenge in Warehouse Picking“ • Host：PDR Benchmark Standardization Committee • Competition of Dead-Reckoning for Pedestrian and Vehicle – xDR=PDR+VDR • Important dates – Testing period: Mid May to Mid Aug., 2018 – Results submission due: 18th Sept, 2018 – Announcement of Winners: Now (This Special Session) • Sponsors: 25
- 26. National Institute ofAdvanced Industrial Science and Technology Two competition tracks • Individually determining winners in each tracks • PDR-Track – Tracking workers who move by foot during warehouse operations – Shared data: Smartphone sensor data for PDR, BLE tag’s signal, warehouse’s spec, Partial WMS log. reference movie for typical picking • VDR-Track – Tracking forklift driven by employee during warehouse operations – Smartphone sensor data measured by attaching smartphone onto the forklifts – Shared data: Smartphone sensor data for VDR, BLE tag’s signal, warehouse’s spec, forklift spec., partial WMS log. – Sample data with known path are prepared. (For beginners) 26
- 27. National Institute ofAdvanced Industrial Science and Technology 2727 Ubicomp/ISWC 2015 PDR Challenge PDR Challenge in Warehouse Picking in IPIN 2017 xDR Challenge for Warehouse Operations 2018 Scenario Indoor pedestrian navigation Picking work inside a logistics warehouse (Specific Industrial Scenario) General warehouse operations including picking, shipping and driving forklift Walking /motion Continuous walking while holding smartphone and looking at navigation screen Includes many motions involved in picking work, not only walking Includes many motions involved in picking, shipping operations and, not only walking. Some workers may drive forklift On-site or off-site Data collection: on-site Evaluation: off-site Off-site Off-site Number of people and trial 90 people, 229 trials 8 people, 8 trials 34 people + 6 forklifts, 170 trials (PDR) + 30 trials (VDR) Time per trial A few minutes About 3 hours About 8 hours Evaluation metric Mean Error, SD of Error Integrated Evaluation (integrated by accuracy, naturalness, warehouse dedicated metrics) Integrated Evaluation (integrated by accuracy, naturalness, warehouse dedicated metrics) Remark Collection of data of participants walking. The data are available at HASC (http://hub.hasc.jp/) as corpus data Competition over integrated position using not only PDR, but also correction information such as BLE beacon signal, picking log (WMS), and maps Consists of PDR and VDR tracks．Referential motion captured by MoCap. also shared for introducing typical motions. Comparison of PDR Challenges
- 28. National Institute ofAdvanced Industrial Science and Technology Prizes • VDR Track: (a) { VDR module (SSEI, Eq. to 200,000) + Android IoT device BL-02 (BIGLOBE) + 150,000 cash} or (b) {200,000 cash +BL-02} • PDR Track: (a) {TECCO (Eq. to 100,000) + BL-02 + 150,000 cash} or (b) {200,000 cash + BL-02} • Runner-Up：BL-02 + 100,000 cash 28 VDR module TECCO BL-02
- 29. National Institute ofAdvanced Industrial Science and Technology VDR Module (SUC-VDR100) • Relative vehicle tracking module by VDR • Manufactured by Sugihara SEI, and its vibration- based VDR algorithm is licensed by AIST • Spec – Battery life: 12 hours 29
- 30. National Institute ofAdvanced Industrial Science and Technology Tecco (TC-A01) • Wearable RFID-tag reader for picking operation • Manufactured by GOV • Spec – Interface: Bluetooth – Battery life: 40 hours 30
- 31. National Institute ofAdvanced Industrial Science and Technology Android IoT device (BL-02) • Android IoT device sold by BIGLOBE • Ideal characteristics for industrial use – LTE capable – No camera (for security/confidential point of view) – 10-axis sensors for PDR – Android version (6.0) is fixed. 31
- 32. National Institute ofAdvanced Industrial Science and Technology Rigorous evaluation of error accumulation by BUP (BLE Unreachable Period) • Intentionally deleting partial BLE signal logs from the test data for evaluating PDR accumulated error Period when BLE signals are deleted: BLE unreachable period (BUP) • WMS Reference points provided before and after BUP BUPBUP BUPRSSI of BLE tag. Evaluation Points by WMS ⇒ Position data are hided Correction Points by WMS ⇒ Position data are provided t Evaluating positional errors of integrated localization system with BLE beacon Evaluating accumulated errors caused by only PDR Emedian_error Eaccum_error
- 33. National Institute ofAdvanced Industrial Science and Technology Conclusions • PDR Challenge Advancing to xDR Challenge. • We promoted and encouraged, but the starting development of VDR is too challenging. • The dataset of PDR Challenge & xDR Challenge are very realistic and good for evaluating practical performance – Real data – Huge data including dirty data. • Survey of the exiting competitions • Future Works – Competition whose targets seamlessly changing walking and driving forklifts (Universal Dead-reckoning: UDR) – Dead-reckoning for other targets (e.g. Drone Dead-reckoning: DDR) – Utilizing whole body motion – Competition of action recognition 33
- 34. National Institute ofAdvanced Industrial Science and Technology Thank you! • Contact Info. – Ryosuke Ichikari, Ph.D.（r.ichikari@aist.go.jp） 34
Editor's Notes
- #2 This presentation contain the presentation about the regular paper and survey of the exiting competitions and the announcement of Winer of our new competition xDR challenge.
- #3 As you all know the number of R&D about PDR is increasing all over the world. So you have many choices when you start services. We need special evaluating method for fair comparison. We introduce our activity based on benchmarking and competitions. Basically the element of the benchmarking are these there. In this presentation, we introduce our benchmarking process and indicators, and dataset.
- #4 Benchmarking and competitions have been held in various research area, such as computer vision and indoor localization. The competitions can be separated into two types: One type is Off-site competition: In the off-site competition, … Off-site tracks of the EvAAL/IPIN competition, and ISMAR competitions are categorized as type. The other type is on-site competitions. In this type of competitions … On-site tracks of the EvAAL/IPIN competition, and ISMAR competitions, and the original PDR Challenge are categorized as type.
- #5 Before introducing the our PDR Challenge and xDR challenge, Let me summarize an existing competitions as a short survey of indoor localization competitions.
- #6 In this table, we picked up PerfLoc competitions and EvAAL/IPIN competitions, As well as the Microsoft competition at IPSN conference, Which is the other famous competition in the indoor localization, We summarize competitions in terms of competition’s scenario, including walking style and motion, competitions category, , target methods, # of people and trials are measured for sharing dataset, and length of the dataset, ,main evaluation metric and its history. I marked remarkable characteristics as red. PerfLoc’s summaries are: There are 30 scenarios for dataset, and its main scenario is localization for emergency scenario In the variety of scenario, they include various type of walking and motion such as … The main caricaturists is that they mounted smartphone onto arm, 4 devices at the same time. Maybe this is good for emergency scenario, but not be good for the general daily scenario. As the evaluation metric, they adopt 95 % spherical error. Summery of EvAAL/IPIN competitions are follows: EvAAL competitions have been held since 2011 It is an one of the oldest competitions, Their main scenario is smart house and assistive living. Their competition is not single track, off-site tracks shared smartphone data base, One on-site track only allow the participant to use smartphone, the other on-site track they allow the participants to any body-mounted devices. Their main evaluation metrics are 75 percentile error. Microsoft Competition also has long history and gathered many participants. Their competition site is relatively small area So it think they are not based on the realistic scenario. Therefore, I think Microsoft competition is a competition where the participants compete maximum accuracy on site. Walking and movement are depending on operators, because thy can operate by themselves In the Microsoft competition, there are two type of competition track, 2D track, the competitor cannot arrange any infra-structure, 3D track they are allowed to arrange Infra-structure such as UWB, infra-red.
- #7 Here is a competition of PDR Challenge. All PDR challenges are designed with PDR benchmark standardization committee. PDR challenges are held under specific scenarios. Original PDR challenge was held in 2015, the its scenario was Indoor pedestrian navigation. PDR challenge in warehouse picking follow the concepts. Its’ scenario is industries scenario, especially warehouse. We started with picking operation first year, and We added general warehouse operation including driving forklift this year.
- #8 This slide shows the introduction of original PDR challenge littlie bit in detail. The original PDR challenge was held in UbiComp/ISWC2015 as a completions for PDR. The purpose of the competitions are: - - Actually there was a separated track for presenting the proposal of the evaluation method. - The participant are required to submit source code of the algorithm. They wrote based on the app skeleton provided by the organizer.
- #9 We held a tracking competition called PDR challenge in warehouse picking as an official competition track of IPIN last year. Our track was track 4, it was an off-site competition track In our, the competitors compete performance of the indoor localization regarding various aspect using the data measured during actual picking operation in real warehouse. The chairs of the track are my self and dr. Kourogi from AIST.
- #10 This slide shows how to collect the raw sensor data by smartphone. We asked to the worker to wear the pouch with smartphone putted in it during the work. We adopt Nexus 5, and measure acceleration, angler velocity, magnetism, atmospheric pressure, and BLE signal log from beacon. We captured some calibration data for them.
- #11 Here are all data shared with competitors in PDR Challenge 2017 In addition to the sensor data, We shared picking log from the warehouse management system log It is used for managing items in the selves for daily operation. Importance thing is the picking log can be converted into record of the workers position at The specific moment. We shared detailed specification of the target warehouse as well.
- #12 In our competitions, the ground truth used for evaluating submitted results is extracted from the WMS picking log. Some part of the ground truth data are shared for competitors for error correction. And others are hided for participants, they are used for error evolution
- #13 Evaluation metric is one of important part for our PDR Challenge. Most of the competition only evaluate the error compared with ground truth. In addition to the error, we evaluate practical performance of the method with multi-faceted evaluation metrics For the metrics related accuracy (省略) We evaluated integrated positioning error Ed We evaluated PDR accumulating error based on EAG as Es For metrics to evaluating naturalness of the trajectory We evaluate the naturalness of speed as Ev We evaluate the frequency points in the trajectory as Ef For specific evaluation metric for warehouse scenario. - We evaluate collision with the obstacle in the warehouse.
- #14 Let me introduce detail of each metric For calculating the integrated position error Ed, We adopt following formula. In this, we calculate positional error as Euclid distance between ground truth by WMS And submitted trajectory.
- #15 Next metric for accuracy is PDR accumulation error metric Es We assume that the participants can cancel the error to 0 at the reference points and error is increased as time passes Es is based on ・・・
- #16 Here is an example of error plot for EAG The EAG can be calculated every reference points. Therefore we can get the eCDF (Empirical cumulative distribution function) of EAG. We can determine the repesentativ EAG by eCDF such as median or 75th percentile EAG. In PDR challenge ・・・
- #17 Next metric is related to motions during the picking work. (Ep) With this metric we evaluate ・・・ We assume. We check.
- #18 Here is the detailed calculation for the metric related to naturalness of the speed. We assume We check --- with following formula. …
- #19 Here is the detailed calculation for the metric related to to collision with obstacle This is belong to the warehouse dedicated scenario, but can be applied for other cased as well. We assume This metric We define … …
- #20 Here is the detailed calculation for the metric related to frequency Ideal trajectory ・・・ This metric encourage the method to submit as many as possible, But just points with high confidence.
- #21 Finally, I will introduce the result of the PDR Challenge 2017. Here are submitted trajectory, As you can see, the trajectories from each competitors look different.
- #22 Here is final results evaluated by our evaluation metrics. In our track we got five competitors from 4 country. And added our result just for reference The team 2 won the competition, and their algorithm is tuned very well for our regulation. One of most interesting finding is error accumulation gradient as introduced in previous slide. For example, given, the EAG is 0.12m/sec and target accuracy of the localization 4.0 m. We can get guideline for the configuration of the system from the EAG If the PDR can be corrected by high quality absolute positioning method with less than 0.4m error every 30 second. Then we can achieved target accuracy. According to the guideline, we can arrange beacon or other positioning method Consequently, we can get many participants and findings. We believed we have successfully organized the competition.
- #23 Here is a summery of findings and review of the PDR Challenge 2017. We successfully ・・・ by adopting multi-faceted evaluation metric. But The detail of the evaluation metrics are open to question. First question is Are the weights for evaluation metrics appropriate? The weighs for obstacle interference seems to be high. A team tuned for this metric very well, and trajectory is not natural as the PDR trajectory Therefore, the Weights for error evaluation should be added. Another unexpected fact is there are not much difference in PDR error evaluation metric Es. We need to further investigate linear regression is appropriate for representative EAG or not. Other reviews for improving the competitions are Many forklift are operated for carrying big and heavy items or picking up from high selves. Workers who drive forklift cannot be tracked only from the PDR The competitors (researchers) are not familiar with the warehouse operations. We need to have a way to introduce what the picking operation is Scale of the measurement was not so big.
- #24 We reconsider about wights as the internal trial Here are original weight used in the competitions. Again, the weight for obstacle interference is 30%, which is very high. But the some competitors might focus on this one, while discounting the error metrics This is a kind good strategy for the last year’s competition. But not always good in general.
- #25 This slide show the trial for adding weights for error related metrics Es, Ed, And reducing the weight of Eo. The winner is changed, but not so big changeover. We guess this weights are more suitable for general cases. In the xDR Challenge, the weight are replaced with these one.
- #26 This years competition xDR challenge in warehouse operations 2018 was held as an sequent competition to the PDR challenge. In the new competition, we added dead reckoning for forklift as the tracking target. We renamed competitions name xDR challenge: xDR means PDR plus VDR (Vehicle dead-reckoning) Maybe we are going to add other types of dead-rekoning in the future Here are important date for the competition, it was very tight schedule for the competitors. In this presentation we will announce the winner of the competitions. Our sponsors are: BIGLOBE, Sugihara software & electron industry, GOV, sumitomo electric Industries, and PDR benchmark Standardization committee.
- #27 As I mentioned, In this year, There are two competition tracks And they individually determine winner in each track. In the PDR, as the same as last year’s competition, The competitors’ PDR algorithm is suppose to track worker who move by foot during warehouse operations This is a off-site competition, We share the smartphone and warehouse data as same as last year In the VDR track, which is a new track this year, The VDR algorithm is supposed to track forklift driven by employee during the operation. The shared data are saved in same format and contends are almost same with PDR track, The smartphone data for VDR are measured by attaching smartphone onto the forklift. For the VDR beginners, we provide sample easy test data with know path. Also we plan to give some points for those who can only estimate partial element such as speed in operation or not.
- #28 Finally, we introduced xDR Challenge parts The scenario of the xDR Challenge is warehouse work, It become more general including tracking the forklift. The added the scale of the data measument, We measured the warehouse work for 5 Full-business days. Total 34 people, 6 forklift equals 170 (PDR data ) and 30 VDR data. About 8 hours / data.
- #29 Here are Prizes (読まない) We awarded winner and runner-up for each tracks. The winners can cases and extra prize shown in the figure.
- #30 One extra prize is VDR module. As you can see, winner can get the devices which can track the forklift.
- #33 As we documented in the regulation documents, This year, we adopt rigorous evaluation of error accumulation. Last year, we only evaluation the submitted result which potentially include effort with PDR, BLE-beacon, WMS, and MAP Map matching. It is hard to extract the effect purely from the PDR. This year, we Intentionally delete partial BLE signal logs from the test data for evaluating PDR accumulated error. We call these period the BLE signal are deleted as BLE unreachable period: BUP in short. With BUP, we evaluation absolute median error of integrated localization only outsize of BUP. And we evaluate the PDR accumulating error in BUP