![VACE-I (2000 to 2002) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Performance Evaluation of Video Analysis and Content Extraction (VACE) Algorithms VACE V ideo A nalysis C ontent E xtraction](https://image.slidesharecdn.com/vaceobjectdetectiontrackingoverview-1257708627184-phpapp02/85/Performance-Evaluation-1-320.jpg)
Recommended
More Related Content
Viewers also liked
Viewers also liked (10)
Similar to Performance Evaluation
Similar to Performance Evaluation (20)
Performance Evaluation
- 2. Evaluation Support Researcher’s Input Converge on Common Ideas Progress Protocols, Metrics, Annotation guidelines, Scoring tool specs & I/O Formatting VACE-II Evaluation Process Performance Evaluation of Video Analysis and Content Extraction (VACE) Algorithms VACE V ideo A nalysis C ontent E xtraction Eva l Release Training Data Dry-run Process Participants USF/VM/NIST Sponsor Needs
- 3. Design Deploy Disseminate VACE-II Evaluation Framework Participants Scoring tool (USF-DATE) Ground truth data (training & testing) Quality control Annotation Results / Analyses Workshop presentation and Technical reports Formal evaluation Task definitions Protocols / Metrics Annotation guidelines Scoring tool specs Schedule Dry-run (Micro-corpus) Data identification Performance Evaluation of Video Analysis and Content Extraction (VACE) Algorithms VACE V ideo A nalysis C ontent E xtraction
- 4. Participants Object Task Sites Data Cycle I Text Detection CMU, COLIB, SRI BN Tracking SRI Face Detection PPATT, UIUC, UMD-Y, TAUCF BN/MR Tracking PPATT, UIUC, UMD-Y Hands Detection VT MR Tracking VT Person Detection USC, UMD-Y MR Tracking USC, UMD-Y Detection USC UAV Tracking USC Vehicle Detection USC, DCU UAV Tracking USC, DCU Eng Text Recognition BBN/SRI BN Cycle II Face Detection PPATT, QMUL MultiSiteMR Tracking PPATT, QMUL Person Detection AIT MultiSiteMR Tracking AIT Detection USC, QMUL, UMD-L Surveillance Tracking USC, QMUL, UMD-L Moving Vehicle Tracking USC, UCF, QMUL, UMD-L Surveillance Performance Evaluation of Video Analysis and Content Extraction (VACE) Algorithms VACE V ideo A nalysis C ontent E xtraction
- 9. Cycle-I Results (Summer 2005) Performance Evaluation of Video Analysis and Content Extraction (VACE) Algorithms VACE V ideo A nalysis C ontent E xtraction
- 10. Cycle-II Results (Spring 2006) Performance Evaluation of Video Analysis and Content Extraction (VACE) Algorithms VACE V ideo A nalysis C ontent E xtraction
- 11. Results: Text Recognition in BNews (Spring 2006) Participant: SRI/BBN Performance Evaluation of Video Analysis and Content Extraction (VACE) Algorithms VACE V ideo A nalysis C ontent E xtraction
- 13. Face Detection: MRoom (SFDA Score distribution) 2 levels of Ambiguity Algo1 Algo1-Amb Algo2 Algo2-Amb Algo3 Algo3-Amb Performance Evaluation of Video Analysis and Content Extraction (VACE) Algorithms VACE V ideo A nalysis C ontent E xtraction
Editor's Notes
- What is the most accurate task definition? How to come up with consistent and reliable reference annotations that facilitate meaningful evaluation? Though there were more questions than answers, the first step towards a large scale empirical evaluation of object detection and tracking was finally taken
- i-Lids logo on one of the slides
- Give source distribution agency (i-lids, LDC)
- VACE metrics are normalized in a way that misses, false alarms, and track splits/merges are penalized; Before the metrics are computed, a one-to-one mapping between the reference and system output objects are established through a Bi-partite graph matching algorithm
- Mean performance scores using the VACE metrics
- Besides generating a set of numbers, we did a bunch of analyses on the results that would both aid developers in their debugging and also help us in providing deeper insights into the evaluation results
- Mention that results are likely to improve; Realize our ultimate goal of providing long lasting resources to the computer vision for many years to come;