This document presents a sketch-to-photo matching system that can automatically match sketches to mugshot photos. It uses two algorithms - a holistic algorithm that extracts features from patches across the whole face, and a component-based algorithm that extracts features from individual facial components. The system detects eyes in the sketch and photo, aligns them, and extracts SIFT and MLBP features. It then calculates matching scores between the sketch and photo features using L2 norm. The scores from the two algorithms are combined to find the best match in a mugshot database. The goal is to develop a more effective and automated way to identify criminal suspects based on eyewitness sketches.

1. MET’S School of Engineering
Department of Computer Science and Engineering
Academic Year 2015-2016
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 1
SKETCH TO PHOTO MATCHING SYSTEM
Submitted by
Chippy Thomas (GWAOCCS004)
S4-M.Tech CSE (2014-16 Batch)
19-03-2016

2. May 24, 2016 2
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE
Introduction
System that match sketch with mugshot
For the identification of suspects
Use when eyewitness are only evidence
Sketch/composites
-hand-drawn
-software generated
-surveillance composites
•mugshot
19-03-2016 2

3. May 24, 2016 3
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE
Introduction – contd…
So matching sketch and photo
Direct matching is not possible
Ie..We don’t match them pixel by pixel
Match by finding high-level features from both the sketch
and the photo
Features such as the structural distribution and the shape
of the eyes, nose and chin etc.
19-03-2016 3

4. May 24, 2016 4
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE
Introduction – contd…
19-03-2016 3

5. May 24, 2016 5
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE
Hand-drawn composite
19-03-2016 4
•Drawn by forensic artist
•Based on description provided by one or more witness
•Artist need proper training

6. May 24, 2016 6
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE
Software-generated composites
19-03-2016 5
•Drawn by operator using software
•Most used software's are identikit and FACES
•Operator need training

7. May 24, 2016 7
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE
Surveillance composites
19-03-2016 6
•Drawn by forensic artist
•Based on poor quality surveillance images

8. May 24, 2016 8
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE
Example of Verbal Description
The man was about 5ft. 6in. in
height, and 34 or 35 years of
age, with dark complexion and
dark mustache turned up at the
ends . . . He had a heavy
mustache curled up, and dark
eyes and bushy eyebrows. He
had no side whiskers, and his
chin was clean shaven. He
looked like a foreigner."
19-03-2016 7

9. Objective
SKETCH TO PHOTO MATCHING SYSTEM
Develop a system that can automatically match sketch with
mugshots
System that manage every types of sketch such as
-forensic sketches
-composite sketches
-surveillance sketches
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 19-03-2016 8

10. • Law enforcement agencies disseminate a sketch to media
outlets
• Hope that someone will recognise the suspect
• And provide information leading to an arrest.
19-02-2016
2
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE
Existing system
19-03-2016 9

11.
May 24, 2016 11
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 19-03-2016 10
Existing system – contd…

12. Problem description
SKETCH TO PHOTO MATCHING SYSTEM
•Slow and tedious process
•Fails to leverage resources available to law enforcement agencies
(mugshot databases)
•So there is a need for a method that can automatically match facial
composites to large police mugshot databases.
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 19-03-2016 11

13. • Automatically match facial composites against mugshots
• For this 2 algorithms are used
• Holistic representation (developed by Klare and Jain)
• Component representation (developed by Han et al.)
19-02-2016
2
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE
Proposed system
19-03-2016 12

14. May 24, 2016 14
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 19-03-2016 13
Proposed system – contd…

15. SKETCH TO PHOTO MATCHING SYSTEM
Working
•Sketches that is given as input to the system
•From sketch features are extracted based on 2 algorithms
•There will be a database to hold the features of large mugshot
•By applying 2 algorithms it matches with the mugshot
•2 algorithms produce matching scores
•Combining matching scores of 2 algorithm
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 19-03-2016 14

16. SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE
Figure
19-03-2016 15

17. Holistic algorithm
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 16
•First step is detection of eye
•Alignment and CSDN
•Feature extraction
-SIFT
-MLBP
•LDA- learn optimal subspaces & projecting into subspaces
•PCA
19-03-2016

18. 18
SKETCH
EYE
DETECTION
ALIGNMENT
AND CSDN
PATCHWISE
SEGMENTATION
DENSE GRID
OF
KEYPOINTS
MLBP
SIFT
LDA Patch-wise
concatenation
& PCA
Sketch
Feature
Vector
Mugshot
Feature
Vector
Repeated for all patches
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE
L2
norm
19-03-2016 17
Holistic Algorithm – contd…

19. Holistic algorithm – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 18
•The same steps are performed for sketch and photo
•Sketch feature vector and mugshot feature vector
•These 2 features are comparing for similarity using L2
norm
•Finding score of matching
19-03-2016

20. Component-based algorithm
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 19
•In holistic algorithm, features are extracted from each patch / grid
from images
•But component-based algorithm, features are extracted from each
component
•Per component, features are extracting
19-03-2016

21. May 24, 2016 21
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 19-03-2016 20
Component-based algorithm – contd…

22. Modules
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 21
Eye Detection
•Accurate and reliable
•Eyes are protected and remain reliably unchanged for decades
•Eyes are relatively stable feature
19-03-2016

23. Modules – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 2219-02-2016
Alignment and CSDN
•composite is normalized to a fixed height and width
•transformed such that right and left eyes are at the same
position for every composite.
•The center-surround divisive normalization (CSDN)
•filter is then applied to the composite to compensate change
in modality between composite and mugshot.
19-03-2016

24. Modules – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 23
•SIFT and MLBP Feature Extraction
•SIFT features are extracted from dense grid of keypoints
•Multi-scale local binary pattern (MLBP) features are extracted
from a uniform patches across the face,
19-03-2016

25. Modules – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 24
LDA
•For both SIFT and MLBP features, optimal subspaces are learned
for each patch using linear discriminant analysis (LDA).
•projecting the patch-wise features into their respective subspaces
PCA
PCA is applied to the feature vector to reduce template size
19-03-2016

26. Modules – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 25
L2
norm
•It is used as classifier
•For finding similarity between mugshot and sketch features
Sum of Score Fusion
•Scores from 2 algorithms are fused using sum of score fusion
19-03-2016

27. Innovative idea
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 26
•Modifying existing systems with Enhanced Holistic method and
Enhanced Composite method.
•Replacing feature extraction methods and classifier
19-03-2016

28. Innovative idea
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 2719-03-2016

29. Innovative idea – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 28
•In the proposed scheme it is done by replacing the existing
systems algorithms such as SIFT and MLBP with SURF and
harlick algorithms.
•Also adding Neural network classifier instead of L
2
norm for
better accuracy.
19-03-2016

30. screenshots – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 2919-03-2016
•Input sketch

31. screenshots – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 3019-03-2016
•Eye detection from sketch

32. screenshots – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 3119-03-2016
•Aligned sketch

33. screenshots – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 3219-03-2016
•Features extracted from sketch

34. screenshots – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 3319-03-2016
•Eye detection from photo

35. screenshots – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 3419-03-2016
•Feature extraction from photo

36. Conclusion
SKETCH TO PHOTO MATCHING SYSTEM
3619-02-2016
•Facial composites are used by law enforcement agencies for the
identification of suspects involved in criminal activities
•Sketch to photo system effectively make the use of facial
composites
•Automatic
•Can be used with every type of composites
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 19-03-2016 35

37. Number: 01 Guide Name: Mr. JOBI V DAS, AP/CSE 36
SKETCH TO PHOTO MATCHING SYSTEM
•For that 2 algorithms are using here
•Holistic algorithm and component based algorithm
•Match scores from 2 algorithms are combining
•Also filtering based on demographic information
•Along with the matched image some details such as name, age can
be displayed.
19-3-2016
Conclusion – contd…

38. References
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 3819-02-2016
•Bonnen K. , B. F. Klare, and A. K. Jain, “Component-based
representation in automated face recognition,” IEEE Trans. Inf.
Forensics Security, vol. 8, no. 1, pp. 239–253, Jan. 2012.
•Best-Rowden L. , H. Han, C. Otto, B. Klare, and A. Jain,
“Unconstrained face recognition: Identifying a person of interest
from a media collection,” IEEE Trans. Inf. Forensics Security,Dec.
2014
•Han H. , B. F. Klare, K. Bonnen, and A. K. Jain, “Matching
composite sketches to face photos: A component-based approach,”
IEEE Trans. Inf. Forensics Security, vol. 8, no. 1, pp. 191–204,
Jan. 2013
19-03-2016

39. References – Contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 39
•Scott J. Klum, Hu Han, Brendan F. Klare Anil K. Jain, ”The
FaceSketchID System: Matching Facial Composites to Mugshots”
IEEE Trans. Inf. Forensics Security,Dec. 2014 ,Vol. 9, no. 12, Dec
2014
19-03-2016

40. Quires
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 40
SKETCH TO PHOTO MATCHING SYSTEM
May 24, 2016
?

41. Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 41
SKETCH TO PHOTO MATCHING SYSTEM
May 24, 2016
Thank you