Download to read offline
Fcv dataset zisserman_2
- 1. The PASCAL VisualObject Classes (VOC) Dataset and Challenge Andrew Zisserman Visual Geometry Group University of Oxford
- 2. The PASCAL VOCChallenge • Challenge in visual object recognition funded by PASCAL network of excellence • Publicly available dataset of annotated images • Main competitions in classification (is there an X in this image), detection (where are the X’s), and segmentation (which pixels belong to X) • Competition has run each year since 2006. • Organized by Mark Everingham, Luc Van Gool, Chris Williams, John Winn and Andrew Zisserman
- 3. Dataset Content • 20classes: aeroplane, bicycle, boat, bottle, bus, car, cat, chair, cow, dining table, dog, horse, motorbike, person, potted plant, sheep, train, TV • Real images downloaded from flickr, not filtered for “quality” • Complex scenes, scale, pose, lighting, occlusion, truncation ...
- 4. Annotation • Complete annotationof all objects • Annotated in one session with written guidelines Occluded Difficult Object is Not scored significantly in evaluation occluded within BB Truncated Pose Object extends Facing left beyond BB
- 5. Examples Aeroplane Bicycle Bird Boat Bottle Bus Car Cat Chair Cow
- 6. Examples Dining Table Dog Horse Motorbike Person Potted Plant Sheep Sofa Train TV/Monitor
- 7. Dataset Statistics –VOC 2010 Training Testing Images 10,103 9,637 Objects 23,374 22,992 • Minimum ~500 training objects per category – ~1700 cars, 1500 dogs, 7000 people • Approximately equal distribution across training and test sets • Data augmented each year
- 8. Design choices: Thingswe got right … 1. Standard method of assessment • Train/validation/test splits given • Standard evaluation protocol – AP per class • Software supplied – Includes baseline classifier/detector/segmenter – Runs from training to generating PR curve and AP on validation or test data out of the box • Has meant that results on VOC can be consistently compared in publications • “Best Practice” webpage on how the training/validation/test data should be used for the challenge
- 9. Design choices: Thingswe got right … 2. Evaluation of test data Three possibilities 1. Release test data and annotation (most liberal) and participants can assess performance – Cons: open to abuse 2. Release test data, but test annotation withheld - participants submit results and organizers assess performance (use an evaluation server) 3. No release of test data - participants have to submit software and organizers run this and assess performance
- 10. Design choices: Thingswe got right … 3. Augmentation of dataset each year • VOC2010 around 40% increase in size over VOC2009 Training Testing Images 10,103 (7,054) 9,637 (6,650) Objects 23,374 (17,218) 22,992 (16,829) VOC2009 counts shown in brackets • Has prevented over fitting on data • 2008/9 datasets retained as subset of 2010 – Assignments to training/test sets maintained – So can measure progress from 2008 to 2010
- 11. Detection Challenge: Progress2008-2010 60 50 Max AP (%) 40 2008 30 2009 2010 20 10 0 tvmonitor pottedplant bottle motorbike diningtable aeroplane horse sofa person train sheep bicycle cow cat boat bus bird dog car chair • Results on 2008 data improve for best 2009 and 2010 methods for all categories, by over 100% for some categories – Caveat: Better methods or more training data?
- 12. Design choices: Thingswe got right … 4. Equivalence bands • Uses Friedman/Nemenyi ANOVA significance test • Makes explicit that many methods are just slight perturbations of each other • Need more research on this area for obtaining equivalence classes for ranking (rather than classification) • Plan to include banner/header results on evaluation server (to aid comparisons for reviewing etc)
- 13. Statistical Significance • Friedman/Nemenyianalysis – Compare differences in mean rank of methods over classes using non- parametric version of ANOVA – Mean rank must differ by at least 5.4 to be considered significant (p=0.05) Mean rank 45 40 35 30 25 20 15 10 5 (chance) NUSPSL_KERNELREGFUSING HIT_PROTOLEARN_2 NUSPSL_MFDETSVM NII_SVMSIFT NUSPSL_EXCLASSIFIER BUPT_LPBETA_MULTFEAT NEC_V1_HOGLBP_NONLIN_SVMDET NTHU_LINSPARSE_2 NLPR_VSTAR_CLS_DICTLEARN BUPT_SVM_MULTFEAT NEC_V1_HOGLBP_NONLIN_SVM LIG_MSVM_FUSE_CONCEPT UVA_BW_NEWCOLOURSIFT WLU_SPM_EMDIST UVA_BW_NEWCOLOURSIFT_SRKDA BUPT_SPM_SC_HOG CVC_PLUSDET LIP6UPMC_RANKING SURREY_MK_KDA LIP6UPMC_KSVM_BASELINE CVC_PLUS LIP6UPMC_MKL_L1 BUT_FU_SVM_SIFT UC3M_GENDISC XRCE_IFV NUDT_SVM_WHGO_SIFT_CENTRIST_LLM CVC_FLAT RITSU_CBVR_WKF BONN_FGT_SEGM TIT_SIFT_GMM_MKL LIRIS_MKL_TRAINVAL NUDT_SVM_LDP_SIFT_PMK_SPMK
- 14. The future … ActionClassification Taster Challenge – 2010 on • Given the bounding box of a person, predict whether they are performing a given action Playing Instrument? Reading? • Developed with Ivan Laptev • Strong participation from the start (11 methods, 8 groups)
- 15. Ten Action Classes Phoning Playing Instrument Reading Riding Bike Riding Horse Running Taking Photo Using Computer Walking Jumping • 2011: added jumping + `other’ class
- 16. Person Layout TasterChallenge – 2009 on • Given the bounding box of a person, predict the positions of head, hands and feet. • aim: to encourage research on more detailed image interpretation hand hand head head head hand hand hand foot foot foot • 2009: no participation • 2010: relaxed success criteria, 2 participants • Assessment method still not satisfactory?
- 17. Futures 2012 • Last officialyear of PASCAL2 • May introduce a new taster competition, e.g – Materials – Actions in video • Open to suggestions