Prof. Lior Wolf, Tel-Aviv University He is a faculty member at the School of Computer Science at Tel-Aviv University. Previously, he was a post-doctoral associate in Prof. Poggio's lab at MIT. He graduated from the Hebrew University, Jerusalem, where he worked under the supervision of Prof. Shashua. He was awarded the 2008 Sackler Career Development Chair, the Colton Excellence Fellowship for new faculty (2006-2008), the Max Shlumiuk award for 2004, and the Rothchild fellowship for 2004. His joint work with Prof. Shashua in ECCV 2000 received the best paper award, and their work in ICCV 2001 received the Marr prize honorable mention. He was also awarded the best paper award at the post ICCV workshop on eHeritage 2009. In addition, Lior has held several development, consulting and advisory positions in computer vision companies including face.com and superfish, and is a co-founder of FDNA. Presentation topic: Using classifiers to compute similarities between images of faces. Key points: The One-Shot-Similarity (OSS) is a framework for classifier-based similarity functions. It is based on the use of background samples and was shown to excel in tasks ranging from face recognition to document analysis. In this talk we will present the framework as well as the following results: (1) when using a version of LDA as the underlying classifier, this score is a Conditionally Positive Definite kernel and may be used within kernel-methods (e.g., SVM), (2) OSS can be efficiently computed, and (3) a metric learning technique that is geared toward improved OSS performance.

1. Learning visual similarity using classifiers Lior Wolf, The Blavatnik School of Computer Science, Tel-Aviv University Collaborators: Students: YanivTaigman Tal Hassner Orit Klipper-Gross Itay Maoz face.com Open U Weizmann inst. Tel-Aviv U 1

2. The Blavatnik School of Computer ScienceTel-Aviv University An example of higher education in Israel 2

3. A school in the Faculty of Exact Sciences that also includes: Mathematics, Physics, Chemistry, Geophysics and Planetary Sciences Originated in the 1970’s as part of the School of Math, since 2000 a separate School 39 Faculty Members ~1000 undergrads ~200 MSc students ~70 PhD students Post Docs and other research personnel 3

4. School Ranking in the world TAU/CS Ranked #29 in number of citations - Thompson Scientific, (for the years 2000-2010).[Technion #33 , Weizmann #72, HebrewU #105] TAU/CS Ranked #28 by the Shanghai Academic Ranking of World Universities in Computer Science – 2011[Weizmann #12, Technion #15, HebrewU #21] TAU/CS Ranked #14 in the world in CS impact – Scientometrics, Vol. 76, No. 2, 2008. 12 TAU/CS faculty in positions 1-100 in “list of most central computer Scientists in Theory of Computer Science” (Kuhn – Wattenhofer, Sigact news, Dec ’07) 4

5. Computer vision in search Query Raw data: images,video, audio Information:objects,tags, IDs,context Searchresults Preprocessing 5

6. The pain: too many images On : Over 1,000,000,000 photos uploaded each month shared by 200,000,000+ users 10’s of billions served/week No tags  No Photos… “can I see all my photos?” “tagging takes hours, can you do that for me?” 6

7. The evolution of perceptual search Reranking bysimilarity Text-basedimage search Specializationin face identification Low-level vision Mid-level vision No vision With basicproperties Gist-based Image similarity Catalog basedsearch High-level vision: scene understanding 7

8. Photo Finder for facebook 8

9. THE 1st MOBILE APP TO FIND 3D ITEMS 9

10. WHAT MAKES IT SO HARD? 10

11. High-level vision: what is where? A happy couple walks in a fieldWhat kind of field? Where? Which season? How old are they? Gender? How attractive? What are they wearing? High-level vision: scene understanding 11

12. YaC, Moscow, September 19, 2011 Learning visual similarity using classifiers Lior Wolf, The Blavatnik School of Computer Science, Tel-Aviv University Collaborators: Students: YanivTaigman Tal Hassner Orit Klipper-Gross Itay Maoz face.com Open U Weizmann inst. Tel-Aviv U 12

13. The Pair-Matching Problem Training: 13

14. Modeling never before seen objects Natural setup for image retrieval with no categories Training: The Pair-Matching Problem 14

15. Instances Document Analysis Video Action Recogniti0n Face Recogniti0n Video Face Recogniti0n 15

16. The Pair-Matching Problem Training: 16

17. Labeled Faces in the Wild (LFW) 13,000 labeled images of faces collected from the web 5,749 individuals 1-150 images per individual Training: 17

18. Restricted Protocol 10-fold cross validation tests on randomly generated splits, each with: 300 same pairs 300 not same pairs 18

19. Pipeline (take 1)* Training. Note: no use of labels! =1 Sim ( , ) same =2 Sim( , ) Classifier (e.g.SVM) =i Sim ( , ) not same =i+1 Sim( , ) Threshold * “Descriptor Based Methods in the Wild,” ECCVw’08 19

20. Pipeline (take 1)* Training – multiple descriptors similarities (1,1,1,2,…,1,n) same (2,1,2,2,…,2,n) Classifier (e.g.SVM) (i,1,i,2,…,i,n) not same (i+1,1,i+1,2,…,i+1,n) * “Descriptor Based Methods in the Wild,” ECCVw’08 20

21. Some Questions How to represent the images? Which similarity to use? Grayscales, Edge responces [Brunelli & Poggio’93], C1-Gabor [e.g., Riesenhuber & Poggio’99], SIFT [Lowe’04], LBP [e.g., Ojala & Pietikainen & Harwood’96],… Later on: How can subject IDs help improve pair-matching performance? L2, Correlation, Learned metrics [e.g., Bilenko etal.’04, Cristianini etal.’02, Hertz etal. 04, …], “hand-crafted” metrics [e.g., Belongie etal.’01] 21

22. One-Shot Similarity (OSS) Score* What: A measure of the similarity between two vectors Input: The two vectors A set of “Background samples” How: Use “One-Shot Learning” (classification with one positive example) * “Descriptor Based Methods in the Wild,” ECCVw’08 “The One-Shot Similarity Kernel”, ICCV’09 22

23. Computing the “One-Shot” Similarity Step a: Model1 = train (p, A) Step b: Score1 = classify(q, Model1) Step c: Model2 = train (q, A) Set “A” of background examples Step d: Score2 = classify(p, Model1) q One-Shot-Sim = (score1 + score2) /2 Similarity  p 23

24. Euclidean vs. One-Shot Visualized One-Shot Euclidean 24

25. Euclidean vs. One-Shot Visualized One-Shot Euclidean 25

26. Computing the “One-Shot” Similarity Using LDA as the underlying classifier : Where is the mean of set A, and is the pseudo-inverse of the intra-class cov. matrix. * “The One-Shot Similarity Kernel”, ICCV’09 26

27. Computing the “One-Shot” Similarity Using Free-Scale LDA as the underlying classifier : Where is the mean of set A, and is the pseudo-inverse of the intra-class cov. matrix. * “The One-Shot Similarity Kernel”, ICCV’09 27

28. Some Properties of the OSS* Uses unlabeled training data OSS based on Free-Scale LDA is a CPD Kernel May be efficiently computed Complexity: is independent of the two vectors compared, and so computed only once. Also, repeated comparisons of a vector xi to different xj may be performed in O(n) * “The One-Shot Similarity Kernel”, ICCV’09 28

29. Some Properties of the OSS* * “The One-Shot Similarity Kernel”, ICCV’09 29

30. Some Properties of the OSS* OSS based on Free-Scale LDA is a CPD Kernel * “The One-Shot Similarity Kernel”, ICCV’09 30

31. Metric learning for OSS* Instead of examples xi use Txi for some “optimal” T The transformation T is obtained by a gradient decent procedure that optimizes the score: *“One Shot Similarity Metric Learning for Action Recognition”, In submission. 31

32. The Unrestricted Protocol 10-fold cross validation tests on randomly generated splits, each with: 300 same pairs 300 not same pairs Training now includes subject labels 32

33. Multiple One-Shots* We now have IDs. How do we use them? Compute multiple OSS, each time using examples from a single class * “Multiple One-Shots for Utilizing Class Label Information,” BMVC’09 33

34. Multiple One-Shots ID-based OSS 34

35. Multiple One-Shots We now have IDs. How do we use them? Compute multiple OSS, each time using examples from a single class Discrimination based on different sources of variation: Subject ID, Pose, etc. 35

36. The Pose Issue Most confident wrong results* * “Descriptor Based Methods in the Wild,” ECCVw’08 36

37. Getting Poses To compute Pose based OSS, you need sets of images in the same pose… 7 fiducial points (eyes, mouth, nose) 14 x,y coordinates 14D vector of alignment errors (similarity trnsf.) Project to first Principal Component Bin to 10 classes 37

38. Multiple One-Shots Pose-based OSS 38

39. Multiple One-Shots - Examples 5 Id-based OSS and 5 Pose-based OSS scores Identity Pose 39

40. Multiple One-Shots - Examples Identity Pose 40

41. Multiple One-Shots - Examples Identity Pose 41

42. Pipeline* Input image pair Image alignment Commercial alignment software by * “Multiple One-Shots for Utilizing Class Label Information,” BMVC’09 42

44. LBP [Ojala etal.’96, 01,02]

45. TPLBP, FPLBP [Wolf etal.’08]* “Multiple One-Shots for Utilizing Class Label Information,” BMVC’09 43

46. Pipeline* Input image pair Image alignment Feature vectors PCA+ITML Information Theoretic Metric Learning [Davis etal.’07] * “Multiple One-Shots for Utilizing Class Label Information,” BMVC’09 44

47. Pipeline* Input image pair Image alignment Feature vectors Multiple OSS scores PCA+ITML 20 Subjects 10 Poses * “Multiple One-Shots for Utilizing Class Label Information,” BMVC’09 45

48. Pipeline* Input image pair Image alignment Feature vectors Multiple OSS scores SVM classifier Output PCA+ITML Same Not-same * “Multiple One-Shots for Utilizing Class Label Information,” BMVC’09 46

49. Pipeline – Multiple Descriptors* Feature vectors SIFT Multiple OSS scores PCA+ITML Image alignment SVM classifier Output Same Not-same Feature vectors LBP Multiple OSS scores PCA+ITML * “Multiple One-Shots for Utilizing Class Label Information,” BMVC’09 47

50. 0.7847 ± 0.0051 [WHT’08] 0.8398 ± 0.0035 [WHT’08 + alignment] Results 0.8517 ± 0.0061 [this work, only LBP] 0.8950 ± 0.0051 [this work, multi-desc.] 0.9753 [Kumar etal. 09 - HUMAN] 48

51. Pair-Matching of Sets * Face Recognition in Unconstrained Videos with Matched B/G Similarity. CVPR 2011. 49

52. Pair-Matching of Sets Training: 50

54. pose based methods, comparing the two most frontal faces in each video or the two faces with the most similar pose.

55. algebraic methods set-to-set methods, such as max correlation, projection and Procrustes.

56. non algebraic methods such as PMK and LLC. 51

58. B: background set of faces.Similarity = MBGS(X1, X2, B) B1 = Find_Nearest_Neighbors(X1,B) Model1 = train(X1, B1) Confidences1 = classify(X2, Model1) Sim1 = mean(Confidences1) X2 Similarity  X1 52

60. B: background set of faces.Similarity = MBGS(X1, X2, B) B1 = Find_Nearest_Neighbors(X1,B) Model1 = train(X1, B1) Confidences1 = classify(X2, Model1) Sim1 = mean(Confidences1) B2 = Find_Nearest_Neighbors(X2, B) Model2 = train(X2, B2) Confidences2 = classify(X1, Model2) Sim2 = mean(Confidences2) Similarity = (Sim1+Sim2)/2 53

61. Thank you! Software available: http://www.cs.tau.ac.il/~wolf 54

63. Easy to benchmark

64. We show that whatever works for pair matching works for identification

65. L. Wolf, T. Hassner, and Y. Taigman Effective Face Recognition by Combining Multiple Descriptors and Learned Background Statistic. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 2011.55

66. Papers L. Wolf, T. Hassner, and Y. Taigman Effective Face Recognition by Combining Multiple Descriptors and Learned Background Statistic. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 2011. L. Wolf, T. Hassner, and Y. Taigman The One-Shot Similarity Kernel IEEE International Conference on Computer Vision (ICCV), 2009. L. Wolf, Y. Taigman and T. Hassner Similarity Scores based on Background Samples Asian Conference on Computer Vision (ACCV), 2009. Y. Taigman, L. Wolf, and T. Hassner Multiple One-Shots for Utilizing Class Label Information The British Machine Vision Conference (BMVC) , 2009. L. Wolf, T. Hassner, and Y. Taigman Descriptor Based Methods in the Wild Post ECCV workshop on Faces in Real-Life Images: Detection, Alignment, and Recognition , 2008 56

67. Papers L. Wolf, T. Hassner, and I. Maoz. Face Recognition in Unconstrained Videos with Matched Background Similarity. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR),2011. L. Wolf, R. Littman, N. Mayer, T. German, N. Dershowitz, R. Shweka, and Y. Choueka. Identifying Join Candidates in the Cairo Genizah. International Journal of Computer Vision (IJCV), 2011. O. Klipper-Gross, T. Hassner, and L. Wolf. One Shot Similarity Metric Learning for Action Recognition. Submitted, 2011. 57