Download to read offline
Announcement of Results of xDR Challenge 2018
- 1. National Institute ofAdvanced Industrial Science and Technology The announcement of the result of xDR Challenge 2018 Ryosuke Ichikari1, Ryo Shimomura12, Masakatsu Kourogi1, Takashi Okuma1, and Takeshi Kurata123 AIST1, University of Tsukuba2, 3Sumitomo Electric Industries, Ltd 1 IPIN 2018@Nantes SS A Survey on Indoor Localization Competitions 10:20 - 12:20 Sept. 26th, 2018
- 2. National Institute ofAdvanced Industrial Science and Technology xDR Challenge for Warehouse Operations • xDR Challenge for Warehouse Operations 2018 was held as an sequel competition to "PDR Challenge in Warehouse Picking“ • Host：PDR Benchmark Standardization Committee • Competition of Dead-Reckoning for Pedestrian and Vehicle – xDR=PDR+VDR • Important dates – Testing period: Mid May to Mid Aug., 2018 – Results submission due: 18th Sept, 2018 – Announcement of Winners: Now (This Special Session) • Sponsors: 2
- 3. National Institute ofAdvanced Industrial Science and Technology Two competition tracks • Individually determining winners in each tracks • PDR-Track – Tracking workers who move by foot during warehouse operations – Shared data: Smartphone sensor data for PDR, BLE tag’s signal, warehouse’s spec, Partial WMS log. reference movie for typical picking • VDR-Track – Tracking forklift driven by employee during warehouse operations – Smartphone sensor data measured by attaching smartphone onto the forklifts – Shared data: Smartphone sensor data for VDR, BLE tag’s signal, warehouse’s spec, forklift spec., partial WMS log. – Sample data with known path are prepared. (For beginners) 3
- 4. National Institute ofAdvanced Industrial Science and Technology 44 Ubicomp/ISWC 2015 PDR Challenge PDR Challenge in Warehouse Picking in IPIN 2017 xDR Challenge for Warehouse Operations 2018 Scenario Indoor pedestrian navigation Picking work inside a logistics warehouse (Specific Industrial Scenario) General warehouse operations including picking, shipping and driving forklift Walking /motion Continuous walking while holding smartphone and looking at navigation screen Includes many motions involved in picking work, not only walking Includes many motions involved in picking, shipping operations and, not only walking. Some workers may drive forklift On-site or off-site Data collection: on-site Evaluation: off-site Off-site Off-site Number of people and trial 90 people, 229 trials 8 people, 8 trials 34 people + 6 forklifts, 170 trials (PDR) + 30 trials (VDR) Time per trial A few minutes About 3 hours About 8 hours Evaluation metric Mean Error, SD of Error Integrated Evaluation (integrated by accuracy, naturalness, warehouse dedicated metrics) Integrated Evaluation (integrated by accuracy, naturalness, warehouse dedicated metrics) Remark Collection of data of participants walking. The data are available at HASC (http://hub.hasc.jp/) as corpus data Competition over integrated position using not only PDR, but also correction information such as BLE beacon signal, picking log (WMS), and maps Consists of PDR and VDR tracks．Referential motion captured by MoCap. also shared for introducing typical motions. Comparison of PDR Challenges
- 5. National Institute ofAdvanced Industrial Science and Technology Prizes • VDR Track: (a) { VDR module (SSEI, Eq. to 200,000) + Android IoT device BL-02 (BIGLOBE) + 150,000 cash} or (b) {200,000 cash +BL-02} • PDR Track: (a) {TECCO (Eq. to 100,000) + BL-02 + 150,000 cash} or (b) {200,000 cash + BL-02} • Runner-Up：BL-02 + 100,000 cash 5 VDR module TECCO BL-02
- 6. National Institute ofAdvanced Industrial Science and Technology VDR Module (SUC-VDR100) • Relative vehicle tracking module by VDR • Manufactured by Sugihara SEI, and its vibration- based VDR algorithm is licensed by AIST • Spec – Battery life: 12 hours 6
- 7. National Institute ofAdvanced Industrial Science and Technology Tecco (TC-A01) • Wearable RFID-tag reader for picking operation • Manufactured by GOV • Spec – Interface: Bluetooth – Battery life: 40 hours 7
- 8. National Institute ofAdvanced Industrial Science and Technology Android IoT device (BL-02) • Android IoT device sold by BIGLOBE • Ideal characteristics for industrial use – LTE capable – No camera (for security/confidential point of view) – 10-axis sensors for PDR – Android version (6.0) is fixed. 8
- 9. National Institute ofAdvanced Industrial Science and Technology Rigorous evaluation of error accumulation by BUP (BLE Unreachable Period) • Intentionally deleting partial BLE signal logs from the test data for evaluating PDR accumulated error Period when BLE signals are deleted: BLE unreachable period (BUP) • WMS Reference points provided before and after BUP BUPBUP BUPRSSI of BLE tag. Evaluation Points by WMS ⇒ Position data are hided Correction Points by WMS ⇒ Position data are provided t Evaluating positional errors of integrated localization system with BLE beacon Evaluating accumulated errors caused by only PDR Emedian_error Eaccum_error
- 10. National Institute ofAdvanced Industrial Science and Technology Results of xDR Challenge 2018 10
- 11. National Institute ofAdvanced Industrial Science and Technology List of Participants We allow participants to use team name for admission • # of preadmission: 7 • PDR Track – No PDR, No future – HBSM – KisekioL – Team:SL_MCL – Xiamen University • VDR Track – HBSM – Team:SL_MCL 11
- 12. National Institute ofAdvanced Industrial Science and Technology List of test data used for competition 12
- 13. National Institute ofAdvanced Industrial Science and Technology Statistics of test data (Added) • PDR test data: – Total # of trajectory: 15 – Total time length of sensor data: 176 h. 58min. 54sec. – Total # of WMS points shared: 271 – Total # of WMS points used for evaluation: 4877 • VDR test data: – Total # of trajectory: 8 – Total time length of sensor data: 84 h. 15min. 43sec. – Total # of WMS points shared: 125 – Total # of WMS points used for evaluation: 1027 13
- 14. National Institute ofAdvanced Industrial Science and Technology Submitted trajectories (3/14 PDR#10) 14 Submitted trajectories (3/14 PDR#13)
- 15. National Institute ofAdvanced Industrial Science and Technology Example of VDR trajectories 15 Submitted trajectories (3/19 VDR#57)Submitted trajectories (3/15 VDR#57)
- 16. National Institute ofAdvanced Industrial Science and Technology Final Results (modified) 16 PDR Track: Winner : HBSM Runner-Up : No PDR, No Future! VDR Track: Winner :HBSM PDR E_median_ error CE50(m) E_accum_ error EAG50 (m/sec.) E_velocity E_ frequency E_obstacle E_picking C.E HBSM 69.46 11.18 98.40 0.0655 99.00 99.87 100.00 98.73 90.18 KisekioL 26.50 26.27 93.63 0.1640 99.00 79.06 99.93 99.00 73.78 Xiamen University 50.42 3637.97 78.04 0.7652 98.40 40.10 86.69 98.27 70.31 No PDR, No future 70.45 10.85 98.53 0.0660 96.93 100.00 97.60 95.80 89.82 VDR E_median_ error CE50(m) E_accum_er ror EAG50 (m/sec.) E_velocity E_ frequency E_obstacle E_picking C.E HBSM 54.65 16.12 98.19 0.0706 99.75 99.25 100.00 97.38 85.62 Note that scores are calculated for individual trajectories and calculating average for filling this table. CE50 means Circular Error 50%, EAG50 means 50 percentile of EAG.
- 17. National Institute ofAdvanced Industrial Science and Technology Thank you! • Contact Info. – Ryosuke Ichikari, Ph.D.（r.ichikari@aist.go.jp） 17
Editor's Notes
- #2 This presentation contain the presentation about the regular paper and survey of the exiting competitions and the announcement of Winer of our new competition xDR challenge.
- #3 This years competition xDR challenge in warehouse operations 2018 was held as an sequent competition to the PDR challenge. In the new competition, we added dead reckoning for forklift as the tracking target. We renamed competitions name xDR challenge: xDR means PDR plus VDR (Vehicle dead-reckoning) Maybe we are going to add other types of dead-rekoning in the future Here are important date for the competition, it was very tight schedule for the competitors. In this presentation we will announce the winner of the competitions. Our sponsors are: BIGLOBE, Sugihara software & electron industry, GOV, sumitomo electric Industries, and PDR benchmark Standardization committee.
- #4 As I mentioned, In this year, There are two competition tracks And they individually determine winner in each track. In the PDR, as the same as last year’s competition, The competitors’ PDR algorithm is suppose to track worker who move by foot during warehouse operations This is a off-site competition, We share the smartphone and warehouse data as same as last year In the VDR track, which is a new track this year, The VDR algorithm is supposed to track forklift driven by employee during the operation. The shared data are saved in same format and contends are almost same with PDR track, The smartphone data for VDR are measured by attaching smartphone onto the forklift. For the VDR beginners, we provide sample easy test data with know path. Also we plan to give some points for those who can only estimate partial element such as speed in operation or not.
- #5 Finally, we introduced xDR Challenge parts The scenario of the xDR Challenge is warehouse work, It become more general including tracking the forklift. The added the scale of the data measument, We measured the warehouse work for 5 Full-business days. Total 34 people, 6 forklift equals 170 (PDR data ) and 30 VDR data. About 8 hours / data.
- #6 Here are Prizes (読まない) We awarded winner and runner-up for each tracks. The winners can cases and extra prize shown in the figure.
- #7 One extra prize is VDR module. As you can see, winner can get the devices which can track the forklift.
- #10 As we documented in the regulation documents, This year, we adopt rigorous evaluation of error accumulation. Last year, we only evaluation the submitted result which potentially include effort with PDR, BLE-beacon, WMS, and MAP Map matching. It is hard to extract the effect purely from the PDR. This year, we Intentionally delete partial BLE signal logs from the test data for evaluating PDR accumulated error. We call these period the BLE signal are deleted as BLE unreachable period: BUP in short. With BUP, we evaluation absolute median error of integrated localization only outsize of BUP. And we evaluate the PDR accumulating error in BUP
- #11 Finally I’m going to announce the result of this years xDR Challenge.
- #12 The xDR Challenge is successfully gather the participants from three contrites # number of pre-admission was 7. 5 teams registered final registration for PDR track, And 2 teams registers in VDR track Unfortunately 1 team for registered both tracks withdrew at the last moment.
- #13 Here is a list of the sensor data provided for the competitors. Despite, We measured much more data in the measuring we selected the data. The amount data is still big. The competitors submit 16 trajectories for PDR track (130 hours) And 8 (60 hours) trajectories for VDR track. We think this huge amount of data works for avoiding fine-tuning.
- #17 OK, here is a final result I guess the what makes a difference between winner and runner-up is that High level balance of error related metric and other metrics. In other words, 1th ranked team and 2nd ranked team almost get similar score for Emedian_error nad E_accumu_error The perfectness of the other mtrics makes the difference. I think this is very good result for evaluating metric designer, because many metrics are contribute The final results.