Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Benchmarking framework of vision-based spatial registration and tracking methods for MAR (ISO/IEC CD 18520)

195 views

Published on

The update report on Benchmarking framework of vision-based spatial registration and tracking methods for MAR (ISO/IEC CD 18520) was presented in ISO IEC/JTC 1/SC 24 meetings (2017/8/7-8).

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

Benchmarking framework of vision-based spatial registration and tracking methods for MAR (ISO/IEC CD 18520)

  1. 1. Benchmarking framework of vision-based spatial registration and tracking methods for MAR (ISO/IEC CD 18520) Takeshi Kurata AIST, Japan ISO IEC/JTC 1/SC 24 (2017/8/7-8)
  2. 2. Contents before CD ballot • Main Body – Terms and Definitions – Benchmarking processes – Benchmark indicators – Trial set for benchmarking – Conformance 2 • Annex A: Benchmarking organizations and activities • Annex B: Tracking competitions in ISMAR • Annex C: Conceptual relationship between this document and other benchmarking standards
  3. 3. Contents after CD ballot • Main Body (Remove proper/individual names, from academic paper style to specification style, include self- benchmarking more) – Terms and Definitions – Benchmarking processes (More narrative descriptions) – Benchmark indicators – Trial set for benchmarking – Conformance (More specific: conformance example with check sheets) 3 • Annex A: Use case examples (More compact) • Annex B: Conceptual relationship between this document and other benchmarking standards (More narrative descriptions)
  4. 4. Benchmarking framework vSRT: Vision-based spatial registration and tracking
  5. 5. Example of stakeholders and their roles
  6. 6. Benchmark indicators vSRT: Vision-based spatial registration and tracking
  7. 7. Benchmark indicators PEVO: Projection error of virtual objects, which is the most direct and intuitive indicator for vSRT methods for MAR NOTE: Indicators for off-site benchmarking can also be used for on-site benchmarking.
  8. 8. Benchmark indicators PEVO: Projection error of virtual objects, which is the most direct and intuitive indicator for vSRT methods for MAR NOTE: Indicators for off-site benchmarking can also be used for on-site benchmarking. ISMAR 2015 Tracking competition
  9. 9. Trial set for benchmarking vSRT: Vision-based spatial registration and tracking
  10. 10. Trial set for benchmarking
  11. 11. Trial set for benchmarking TrakMark
  12. 12. Trial set for benchmarking Metaio
  13. 13. Trial set for benchmarking The City of Sights: An Augmented Reality Stage Set
  14. 14. Trial set for benchmarking ISMAR 2015 Tracking competition
  15. 15. Trial set for benchmarking ISMAR 2014 Tracking competition
  16. 16. Trial set for benchmarking ISMAR 2015 Tracking competition
  17. 17. Meetings after SC 24 meetings in Beijing • Editing meetings in WG 9 – 2017/01/18-19 (WG 9 in Seoul), 2017/02/09 • Drafting meetings in WG 9 Japanese subcommittee – 2016/10/18, 2016/11/16, 2016/12/26, 2017/03/08, 2017/06/20, 2017/07/19 – Members • T. Kurata (AIST/Univ. of Tsukuba) • M. Aono (Toyohashi Univ. of Tech.) • T. Kondo (The Open Univ. of Japan) • F. Shibata (Ritsumeikan Univ.) • T. Taketomi (NAIST) • H. Uchiyama (Kyushu Univ.) • S. Mori (Keio Univ./Graz University of Technology) • K. Makita (Canon/AIST) (Expert) 17
  18. 18. Next Step: 40.00: DIS registered 18
  19. 19. TODO for DIS registration • More narrative • Make conformance check sheets • Compact annexes • Target dates • Due dates 19 DIS FDIS IS 9/17 1/18 7/18 DIS FDIS IS 12/17 6/18 12/18
  20. 20. Conformance check sheet (tentative) Process Target (T)/ Input (I)/ Output (O)/ Organized storage (S) Reliability Temporality Variety Contents [ ] Image sequences: ____________________________ [ ] Intrinsic/extrinsic camera parameters: ___________ [ ] Challenge points: _____________________________ [ ] Optional contents: ____________________________ Metadata [ ] Scenario: ____________________________________ [ ] Camera motion type: __________________________ [ ] Camera configuration: _________________________ [ ] Image quality: ________________________________ Contents [ ] Physical objects: ______________________________ Metadata [ ] How to find the physical objects: ________________ Trial set format Dataset Physical object Process flow [ ] develop vSRT methods and/or MAR systems: ______________________ [ ] gather vSRT methods and/or MAR systems: _______________________ [ ] prepare and conduct benchmarking: ______________________________ [ ] provide and maintain benchmarking instruments: ___________________ [ ] provide and maintain benchmarking repositories: ___________________ [ ] share benchmarking results: _____________________________________ [ ] vSRT method: _________________________________________________ [ ] MAR system: __________________________________________________ [ ] trial sets and physical objects: ___________________________________ [ ] benchmarking instruments: ______________________________________ [ ] benchmarking results: __________________________________________ [ ] benchmarking surveys: _________________________________________ [ ] benchmarking repository: ________________________________________ [ ] external repositories: ____________________________________________ Indicator formura [ ] PEVO: ________________________________________________________ [ ] Reprojection error of image features: _____________________________ [ ] Position and posture errors of a camera: __________________________ [ ] Completeness of a trial: _________________________________________ [ ] Throughput: ___________________________________________________ [ ] Latency: ______________________________________________________ [ ] Time for trial completion: ________________________________________ [ ] Number of datasets/trials: ________________________________________ [ ] Variety on properties of datasets/trials: _____________________________

×