Riccardo Sattariccardo.satta@diee.unica.itDissimilarity-basedpeople re-identification and searchfor intelligent video surv...
OutlineUniversityOf CagliariDepartment of Electricaland Electronic Engineering2• General contextIntelligent Video-Surveill...
Intelligent Video SurveillanceUniversityOf CagliariDepartment of Electricaland Electronic Engineering3Machine LearningBiom...
UniversityOf CagliariDepartment of Electricaland Electronic EngineeringPerson re-identificationPerson Re-Identification is...
UniversityOf CagliariDepartment of Electricaland Electronic EngineeringPerson re-identificationFace recognition cannot be ...
UniversityOf CagliariDepartment of Electricaland Electronic EngineeringClothing appearance descriptors6Blob detectionand t...
Appearance-based people searchUniversityOf CagliariDepartment of Electricaland Electronic Engineering7Clothing appearance ...
UniversityOf CagliariDepartment of Electricaland Electronic Engineering8THE MULTIPLE COMPONENTDISSIMILARITY FRAMEWORK
Dissimilarity representationsUniversityOf CagliariDepartment of Electricaland Electronic Engineering9An alternative way [1...
The Multiple Component Dissimilarity frameworkUniversityOf CagliariDepartment of Electricaland Electronic Engineering10Ext...
The Multiple Component Dissimilarity frameworkUniversityOf CagliariDepartment of Electricaland Electronic Engineering11Pro...
The Multiple Component Dissimilarity frameworkUniversityOf CagliariDepartment of Electricaland Electronic Engineering12MCD...
UniversityOf CagliariDepartment of Electricaland Electronic Engineering13MCD FORPERSON RE-IDENTIFICATION
MCD and person re-identificationUniversityOf CagliariDepartment of Electricaland Electronic Engineering14Person re-identif...
MCD-based matchingUniversityOf CagliariDepartment of Electricaland Electronic Engineering15A novel weighted Euclidean dist...
UniversityOf CagliariDepartment of Electricaland Electronic Engineering16USING MCD TO SPEED UPEXISTING METHODS
MCD to speed up existing methodsUniversityOf CagliariDepartment of Electricaland Electronic Engineering17MCD has been appl...
Experimental evaluationUniversityOf CagliariDepartment of Electricaland Electronic Engineering18
Experimental evaluationUniversityOf CagliariDepartment of Electricaland Electronic Engineering19Trade-off between accuracy...
Experimental evaluationUniversityOf CagliariDepartment of Electricaland Electronic Engineering20Impact of the number and s...
UniversityOf CagliariDepartment of Electricaland Electronic Engineering21USING MCD TO COMBINEFEATURE SETS
Fusion of different feature sets by MCDUniversityOf CagliariDepartment of Electricaland Electronic Engineering22Prototypes...
Fusion of different feature sets by MCDUniversityOf CagliariDepartment of Electricaland Electronic Engineering23This techn...
Performance of the single feature setsUniversityOf CagliariDepartment of Electricaland Electronic Engineering24I-LIDS: 119...
Comparison with the state-of-the-artUniversityOf CagliariDepartment of Electricaland Electronic Engineering25Comparison wi...
UniversityOf CagliariDepartment of Electricaland Electronic Engineering26USING MCD TO PERFORMMULTI-MODAL PERSONRE-IDENTIFI...
Multi-modal person re-identificationUniversityOf CagliariDepartment of Electricaland Electronic Engineering27• Appearance ...
Multi-modal person re-identificationUniversityOf CagliariDepartment of Electricaland Electronic Engineering28
Multi-modal person re-identificationUniversityOf CagliariDepartment of Electricaland Electronic Engineering29A proper fusi...
Multi-modal person re-identificationUniversityOf CagliariDepartment of Electricaland Electronic Engineering30MCD provides ...
Multi-modal person re-identificationUniversityOf CagliariDepartment of Electricaland Electronic Engineering31This MCD-base...
Experimental evaluationUniversityOf CagliariDepartment of Electricaland Electronic Engineering32Experiments have been carr...
Experimental evaluationUniversityOf CagliariDepartment of Electricaland Electronic Engineering33Experiments:one video-sequ...
Experimental evaluationUniversityOf CagliariDepartment of Electricaland Electronic Engineering34Comparison of MCD-based fu...
UniversityOf CagliariDepartment of Electricaland Electronic Engineering35USING MCD TO PERFORMAPPEARANCE-BASEDPEOPLE SEARCH
MCD for people searchUniversityOf CagliariDepartment of Electricaland Electronic Engineering36Implementation by MCD: high-...
MCD for people searchUniversityOf CagliariDepartment of Electricaland Electronic Engineering37How to implement people sear...
Experimental evaluationUniversityOf CagliariDepartment of Electricaland Electronic Engineering38Dataseta subset of 512 ima...
Experimental evaluationUniversityOf CagliariDepartment of Electricaland Electronic Engineering39For each basic query:(i) t...
Experimental evaluationUniversityOf CagliariDepartment of Electricaland Electronic Engineering40Red shirtBlacktrousersShor...
ConclusionsUniversityOf CagliariDepartment of Electricaland Electronic Engineering41What has been done(i) MCD, a novel dis...
ConclusionsUniversityOf CagliariDepartment of Electricaland Electronic Engineering42What to do next (long list…!)THE FRAME...
UniversityOf CagliariDepartment of Electricaland Electronic Engineering43QUESTION TIME!
Upcoming SlideShare
Loading in …5
×

Dissimilarity-based people re-identification and search for intelligent video surveillance

1,035 views
929 views

Published on

Slides of my PhD thesis, available at:
http://pralab.diee.unica.it/satta_phdthesis2013
http://pralab.diee.unica.it/en/AmbientIntelligence

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,035
On SlideShare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
40
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Dissimilarity-based people re-identification and search for intelligent video surveillance

  1. 1. Riccardo Sattariccardo.satta@diee.unica.itDissimilarity-basedpeople re-identification and searchfor intelligent video surveillancePhD final dissertationPhD School on Information EngineeringApril 2013UniversityOf CagliariDepartment of Electricaland ElectronicEngineeringPattern Recognitionand Applications Lab1
  2. 2. OutlineUniversityOf CagliariDepartment of Electricaland Electronic Engineering2• General contextIntelligent Video-Surveillance, and in particular– Person Re-identification– Appearance-based People Search• A framework for constructing descriptors of people– dissimilarity-based representations and their advantages– the Multiple Component Dissimilarity (MCD) framework• MCD and person re-identification• MCD and people search• Discussion and conclusions
  3. 3. Intelligent Video SurveillanceUniversityOf CagliariDepartment of Electricaland Electronic Engineering3Machine LearningBiometrics and patternrecognitionNovel sensortechnologiesUseful tools for operators and forensicinvestigators• person identification• on-line tracking of persons and objects• detection of events of interest• detection of suspicious actions• summarisation of long video footages…IntelligentVideo Surveillance
  4. 4. UniversityOf CagliariDepartment of Electricaland Electronic EngineeringPerson re-identificationPerson Re-Identification is the ability to determine if anindividual has already been observed over a network of video-surveillance cameras4ABScenarios- on-line (e.g. peopletracking among differentcameras)- off-line (e.g. retrieve all theframes showing an individualof interest)
  5. 5. UniversityOf CagliariDepartment of Electricaland Electronic EngineeringPerson re-identificationFace recognition cannot be used- bad quality images (low resolution, blur, …)- unconstrained poseOther cues must be used clothing appearance(easy to extract, good uniqueness in limited time spans) other ones (e.g. gait) are impractical in real-worldscenarios5
  6. 6. UniversityOf CagliariDepartment of Electricaland Electronic EngineeringClothing appearance descriptors6Blob detectionand trackingBG/FGsegmentationDescriptorcomputationDescriptor = body part subdivision + appearancefeaturesEach body part is automatically detected and describedseparately by e.g.- colour (e.g., histograms)- texture (e.g., DCT, LBP)- local/global features
  7. 7. Appearance-based people searchUniversityOf CagliariDepartment of Electricaland Electronic Engineering7Clothing appearance descriptors can enable another usefultask, appearance-based people search (a novelty in the literature) Retrieve images of people via a query expressed as a high-level description oftheclothes (es. “people with red shirt and blue trousers”), instead of as an image
  8. 8. UniversityOf CagliariDepartment of Electricaland Electronic Engineering8THE MULTIPLE COMPONENTDISSIMILARITY FRAMEWORK
  9. 9. Dissimilarity representationsUniversityOf CagliariDepartment of Electricaland Electronic Engineering9An alternative way [1] to represent objects in patternrecognition, useful when it is unclear how to choose a features it is difficult to find a good feature setfeature-based representationdissimilarity-based representationObjectfeatureextraction[ x1 x2 … xn ]feature vectorprototypes[1] Pekalska and Duin. The Dissimilarity Representation for Pattern Recognition: Foundations andApplications. World Scientific Publishing, 2005[ d1 d2 … dn ]dissimilarity vectorObjectdissimilaritiescomputationP1 P2 Pn
  10. 10. The Multiple Component Dissimilarity frameworkUniversityOf CagliariDepartment of Electricaland Electronic Engineering10Extension of the dissimilarity-based approach to objects represented by- multiple parts- multiple local features (components)Prototypesfor bodypart #1Prototypesfor bodypart #2Dissimilarity vectors(one for each bodypart)LocalappearanceGlobalappearance
  11. 11. The Multiple Component Dissimilarity frameworkUniversityOf CagliariDepartment of Electricaland Electronic Engineering11Prototype construction From a design set of images of people various possible approaches, e.g. clusteringClustering-based prototype creation example (two body parts)Design setCreate a set of all thecomponents of body part 1Create a set of all thecomponents of body part 2Clusterthe setTake centroids asprototypesClusterthe setTake centroids asprototypes
  12. 12. The Multiple Component Dissimilarity frameworkUniversityOf CagliariDepartment of Electricaland Electronic Engineering12MCD representations will be exploited for person re-identification appearance-based people search[d1,1 d1,2 d1,3 d1,4 d2,1 d2,2 d2,3 ] [d1,1 d1,2 d1,3 d1,4 d2,1 d2,2 d2,3 ][d1,1 d1,2 d1,3 d1,4 d2,1 d2,2 d2,3 ] [d1,1 d1,2 d1,3 d1,4 d2,1 d2,2 d2,3 ]
  13. 13. UniversityOf CagliariDepartment of Electricaland Electronic Engineering13MCD FORPERSON RE-IDENTIFICATION
  14. 14. MCD and person re-identificationUniversityOf CagliariDepartment of Electricaland Electronic Engineering14Person re-identificationMCD salient features for person re-identification: a very compact representationdescriptors are small real vectors (low storage requirements, fastmatching) dissimilarity vectors are representation-independentthey can be used to combine different features and modalitiesApplications: 1) Speed up person re-identification methods2) Feature combination for person re-identification3) Multimodal person re-identificationmatchingranked list of templates(w.r.t. the degree of similarity)template galleryprobe0.03 0.28 0.33 0.34
  15. 15. MCD-based matchingUniversityOf CagliariDepartment of Electricaland Electronic Engineering15A novel weighted Euclidean distance for dissimilarity spacesRATIONALE: - each dissimilarity is a degree of relevance of the correspondingprototype;- lower dissimilarity values carry more information; in fact, theyencode themost relevant characteristics of the sample.Weights: where (xi, yi in the range [0,1])The weighting rule f() is a monotonically increasingfunction; its choice governs the difference betweenrelevant and non-relevant prototypesx and y: dissimilarity vectors;W such that
  16. 16. UniversityOf CagliariDepartment of Electricaland Electronic Engineering16USING MCD TO SPEED UPEXISTING METHODS
  17. 17. MCD to speed up existing methodsUniversityOf CagliariDepartment of Electricaland Electronic Engineering17MCD has been applied to an existing method, MCMimpl [2]MCMimpl in short:part subdivision:torso – legs exploiting symmetry andanti-symmetry properties, discarding headmultiple component representation:for each part, randomly taken and partlyoverlapping patchesFour data sets of increasing size:i-LIDS (119 pedestrians) VIPeR-316 (316 pedestrians)VIPeR-474 (474 pedestrians) VIPeR-632 (632 pedestrians)[2] Satta, Fumera, Roli, Cristani, and Murino. A Multiple Component Matching Framework for Person Re-Identification. In: ICIAP, 2011
  18. 18. Experimental evaluationUniversityOf CagliariDepartment of Electricaland Electronic Engineering18
  19. 19. Experimental evaluationUniversityOf CagliariDepartment of Electricaland Electronic Engineering19Trade-off between accuracy and computational timeIt can be shown that the overall re-identification time* in a practical searchscenario is much lower when using MCD* sum of processing time plus the averagesearch time spent by the operator
  20. 20. Experimental evaluationUniversityOf CagliariDepartment of Electricaland Electronic Engineering20Impact of the number and source of prototypes
  21. 21. UniversityOf CagliariDepartment of Electricaland Electronic Engineering21USING MCD TO COMBINEFEATURE SETS
  22. 22. Fusion of different feature sets by MCDUniversityOf CagliariDepartment of Electricaland Electronic Engineering22Prototypes in MCD are representation-independent MCD dissimilarity vectors can be used to combine together different kinds offeatures, either global or local each feature set will be responsible for a different sub-set of prototypes
  23. 23. Fusion of different feature sets by MCDUniversityOf CagliariDepartment of Electricaland Electronic Engineering23This technique has been used to combine five different feature sets• RandPatchesHSV• RandPatchesLBP• FCTH [3]• EdgeHistogram [4]• SCD [4]exploiting a 4-body-parts subdivisionFirst two feature sets:200 prototypes per feature set per body partLast three feature sets:100 prototypes per feature set per body part3200 prototypes in total[3] Chatzichristofis and Boutalis. FCTH: Fuzzy Color and Texture Histogram – a Low Level Feature forAccurate Image Retrieval. In: WIAMIS, 2008[4] Sikora. The MPEG-7 Visual Standard for Content Description – an Overview. IEEE Transactions onCircuits and Systems for Video Technology, 2001
  24. 24. Performance of the single feature setsUniversityOf CagliariDepartment of Electricaland Electronic Engineering24I-LIDS: 119 individuals
  25. 25. Comparison with the state-of-the-artUniversityOf CagliariDepartment of Electricaland Electronic Engineering25Comparison with two state-of-the-art methods- SDALF [5]- CPS [6][5] Farenzena, Bazzani, Perina, Murino, and Cristani. Person Re-Identification by Symmetry-DrivenAccumulation of Local Features. In: CVPR, 2010[6] Cheng, Cristani, Stoppa, Bazzani, and Murino. Custom Pictorial Structures for Re-Identification. In:BMVC, 2011
  26. 26. UniversityOf CagliariDepartment of Electricaland Electronic Engineering26USING MCD TO PERFORMMULTI-MODAL PERSONRE-IDENTIFICATION
  27. 27. Multi-modal person re-identificationUniversityOf CagliariDepartment of Electricaland Electronic Engineering27• Appearance is a widely used cue for person re-identification other cues (e.g., gait) pose constraints that limit their applicabilityin real world scenarios• However, the recent introduction of RGB-D sensors makes itpossible to extract anthropometric measures that can becombined with appearanceExample  MS Kinect™!By processing RGB-D data, it is possible to estimate a 3D model of a person in real-time [7]From this model, one can extract various anthropometric measures (e.g., height, armlength)[7] Shotton, Fitzgibbon, Cook, Sharp, Finocchio, Moore, Kipman, and Blake. Real-time Pose Recognition inParts from Single Depth Images. In: CVPR, 2011
  28. 28. Multi-modal person re-identificationUniversityOf CagliariDepartment of Electricaland Electronic Engineering28
  29. 29. Multi-modal person re-identificationUniversityOf CagliariDepartment of Electricaland Electronic Engineering29A proper fusion strategy must be used to combine different modalities.Score-level fusion Feature-level fusion- Performance of score-level fusion is affected by the choice of the fusionrule (e.g.,mean, min); a suitable choice for re-id is not trivial- Feature-level fusion requires homogeneous featuresFusionModality 1 Matching scoreModality 2 Matching scoreModality n Matching scoreFusion scoreModality 1Modality 2Modality nMatching
  30. 30. Multi-modal person re-identificationUniversityOf CagliariDepartment of Electricaland Electronic Engineering30MCD provides a way to combine non-homogeneous modalities at featurelevel, by exploiting its representation-independency
  31. 31. Multi-modal person re-identificationUniversityOf CagliariDepartment of Electricaland Electronic Engineering31This MCD-based approach has been used to combine appearance with anthropometryAppearance:two descriptors, MCMimpl v2 and SDALFAnthropometry:three measures from the skeleton:- normalised height- normalised average arm length- normalised average leg lengthMCMimpl v2 SDALF
  32. 32. Experimental evaluationUniversityOf CagliariDepartment of Electricaland Electronic Engineering32Experiments have been carried out on a novel dataset acquired using Kinectcameras, Kinect4REID video sequences of 80 individuals taken at different locations different lighting conditions and view points 2 to 7 different video sequences per person many persons are carrying bags or accessories
  33. 33. Experimental evaluationUniversityOf CagliariDepartment of Electricaland Electronic Engineering33Experiments:one video-sequence per person taken as template, the remaining ones as probe20 repetitions
  34. 34. Experimental evaluationUniversityOf CagliariDepartment of Electricaland Electronic Engineering34Comparison of MCD-based fusion with other fusion rulesSimilar results have been obtained with SDALF + Anthropometry
  35. 35. UniversityOf CagliariDepartment of Electricaland Electronic Engineering35USING MCD TO PERFORMAPPEARANCE-BASEDPEOPLE SEARCH
  36. 36. MCD for people searchUniversityOf CagliariDepartment of Electricaland Electronic Engineering36Implementation by MCD: high-level concepts that describe certain clothingcharacteristics (e.g., “red shirt”) may be encoded by one or more visualprototypes, according to the low-level features and part subdivision usedPrototypes (rectangular patches) extracted from a set of24 people (upper body part)Correlation with the presence of the concept “red shirt”
  37. 37. MCD for people searchUniversityOf CagliariDepartment of Electricaland Electronic Engineering37How to implement people search(i) define a set of basic queries(ii) construct a detector for each basic query, using dissimilarity values as inputComplex queries can be built by connecting basic ones through Booleanoperators,e.g., “red shirt AND (blue trousers OR black trousers)”Detector[ d1 d2 … dn ] SCORE
  38. 38. Experimental evaluationUniversityOf CagliariDepartment of Electricaland Electronic Engineering38Dataseta subset of 512 images taken from the VIPeR data-set, tagged with respect to 14different basic queriesExamples:Three descriptors:i) MCMimplii) SDALFiii) MCMimpl-PS, which uses a pictorial structure [8] to subdivide the body into ninepartsbody subdivision,MCMimpl and SDALFbody subdivision,MCMimpl-PS[8] Andriluka, Roth, and Schiele. Pictorial Structures Revisited: People Detection and Articulated PoseEstimation. In: CVPR 2009
  39. 39. Experimental evaluationUniversityOf CagliariDepartment of Electricaland Electronic Engineering39For each basic query:(i) the VIPeR-Tagged is subdivided into a training and a testing sets of equal size(ii) a linear SVM is trained on training images to implement a detector(iii) the P-R curve is evaluated on testing images, by varying the SVM decision thresholdThis procedure is repeated ten timesBreak-even points for all classes:
  40. 40. Experimental evaluationUniversityOf CagliariDepartment of Electricaland Electronic Engineering40Red shirtBlacktrousersShortsleeves
  41. 41. ConclusionsUniversityOf CagliariDepartment of Electricaland Electronic Engineering41What has been done(i) MCD, a novel dissimilarity-based framework for describing individuals(ii) an approach based on MCD to speed up any existing person re-identification method(iii) a state-of-the-art re-identification method, that combines differentfeatures obtained through the use of MCD(iv) a method to perform multi-modal person re-identification based onMCD and using RGB-D cameras, and a novel data set to assessperformance of multi-modal re-identification systems(v) a method that uses MCD to perform the novel task of “appearance-based people search”
  42. 42. ConclusionsUniversityOf CagliariDepartment of Electricaland Electronic Engineering42What to do next (long list…!)THE FRAMEWORK(i) explore the commonalities between MCD and Visual Words and FisherVectors(ii) extend MCD to other domainsMULTIMODAL RE-ID(i) explore the use of other cues (other anthropometries, skeleton-basedgait…)(ii) extend the approach to support missing cuesPEOPLE SEARCH(i) address the problem of ambiguity of concepts(ii) add semantic interpretation (Natural Language Processing) to supportqueries in natural language
  43. 43. UniversityOf CagliariDepartment of Electricaland Electronic Engineering43QUESTION TIME!

×