Face Recognition for Personal Photos using Online Social Network Context and Collaboration Guest Lecture at KAIST 14 Decem...
Professional Background <ul><li>Licentiate degree in computer science (June 2002) </li></ul><ul><ul><li>at Ghent Universit...
Outline <ul><li>Introduction </li></ul><ul><li>Face recognition 101 </li></ul><ul><li>Face recognition using online social...
Outline <ul><li>Introduction </li></ul><ul><li>Face recognition 101 </li></ul><ul><li>Face recognition using online social...
Introduction (1/2) <ul><li>The number of personal photos shared online keeps growing </li></ul><ul><ul><li>thanks to easy-...
Introduction (2/2) <ul><li>Problem: digital information overload </li></ul><ul><ul><li>our ability to automatically organi...
Face Detection, Recognition, and Annotation /55 face detection Identity tags: Barack Obama, Joe Biden face annotation retr...
Lecture Goals and Main Sources <ul><li>To provide an answer to the following questions </li></ul><ul><ul><li>what is face ...
Outline <ul><li>Introduction </li></ul><ul><li>Face recognition 101 </li></ul><ul><li>Face recognition using online social...
Application Areas of Face Recognition <ul><li>Identity verificiation </li></ul><ul><ul><li>face recognition is used to con...
Conceptual Design of a Face Recognition System /55 preprocessing (e.g., scaling and rotation to put eyes on fixed location...
<ul><li>Face feature vector </li></ul><ul><ul><li>a  d -dimensional vector of feature values (e.g., grayscale pixel values...
Possible Outcomes of Face Recognition True negative (system correctly decides that the gallery does  not contain the ident...
Effectiveness of Face Recognition (1/2) <ul><li>Automatic appearance-based face recognition for personal photos is a hard ...
Effectiveness of Face Recognition (2/2) <ul><li>Appearance-based face recognition for Facebook photos </li></ul><ul><ul><l...
Room for Improvement... <ul><li>Gartner hype cycle </li></ul><ul><ul><li>describes the adoption of new technology </li></u...
Outline <ul><li>Introduction </li></ul><ul><li>Face recognition 101 </li></ul><ul><li>Face recognition using online social...
Online Sharing of Personal Photos <ul><li>Photos on online social networks do not exist in isolation </li></ul><ul><ul><li...
Online Social Network Context (1/2) <ul><li>Research question </li></ul><ul><ul><li>how to use online social network conte...
Online Social Network Context (2/2) <ul><li>Contact list of the photographer </li></ul><ul><ul><li>used for reducing the n...
Empirical Study for Facebook <ul><li>Collected manually labeled face images from 50 college-aged volunteers and their frie...
Observations (1/2) <ul><li>Most people can be associated with at least one identity tag </li></ul><ul><ul><li>out of the 2...
Observations (2/2) <ul><li>About 30% of the tagged faces in a photographer’s albums belong on average to the photographer ...
Summary Observations <ul><li>Empirical study for Facebook </li></ul><ul><ul><li>suggests that it may be useful to rely on ...
Mathematical Modeling (1/2) <ul><li>Need for a mathematical tool that supports labeling of face images by combining two he...
Mathematical Modeling (2/2) <ul><li>Use of a probability model known as a Markov Random Field </li></ul><ul><ul><li>allows...
Experimental Results (1/2) Probability  – proportion of face images with a correct label in the top R  suggested labels /5...
Experimental Results (2/2) <ul><li>Additional observations and discussion </li></ul><ul><ul><li>the combined use of pixel ...
Outline <ul><li>Introduction </li></ul><ul><li>Face recognition 101 </li></ul><ul><li>Face recognition using online social...
Centralized and Decentralized Online Social Networks <ul><li>Centralized online social networks </li></ul><ul><ul><li>are ...
Architecture of Centralized and Decentralized Online Social Networks <ul><li>Centralized </li></ul><ul><ul><li>central ser...
Collaborative Face Recognition (1/2) <ul><li>Decentralized online social networks </li></ul><ul><ul><li>each user will hav...
Collaborative Face Recognition (2/2) /55 personal server photographer photos FR engine personal server contact 1 (family m...
Proposed Framework for Collaborative FR <ul><li>Research challenges </li></ul><ul><ul><li>how to select expert face recogn...
Selection of Expert FR Engines using Online Social Network Context the thicker the line, the stronger the social tie, the ...
Experimental Data Collected for Cyworld <ul><li>Retrieval of 547,991 personal photos from four volunteers and their contac...
Observations <ul><li>A non-trivial number of face images belong to the photographer </li></ul><ul><ul><li>numbers range fr...
Distribution of FR Engine Relevance Values Relevance FR engine FR engine index (in decreasing order of relevance) experime...
FR Effectiveness of Selected FR Engines Number of FR engines used Number of correctly recognized face images the collabora...
Experimental Results (1/2) Collaborative FR (Bayesian) Collaborative FR (Voting) Non-collaborative FR (Avg.) Rank ( R ) Pr...
Experimental Results (2/2) <ul><li>Explanatory notes </li></ul><ul><ul><li>non-collaborative FR </li></ul></ul><ul><ul><ul...
Outline <ul><li>Introduction </li></ul><ul><li>Face recognition 101 </li></ul><ul><li>Face recognition using online social...
Microsoft OneAlbum <ul><li>OneAlbum project </li></ul><ul><ul><li>allows users to find relevant photos across a social net...
Augmented Identity <ul><li>Augmented reality </li></ul><ul><ul><li>superimposes virtual objects and info on top of the rea...
Socially-Aware Advertisement Billboards <ul><li>Quotes </li></ul><ul><ul><li>Ray Ozzie (ex-Microsoft) </li></ul></ul><ul><...
Socially-Aware Video Surveillance (1/2) <ul><li>Video surveillance </li></ul><ul><ul><li>used to prevent and detect crime ...
Socially-Aware Video Surveillance (2/2) <ul><li>Research challenges </li></ul><ul><ul><li>robust and large-scale face reco...
Socially-Aware Robots <ul><li>Humanoid robots </li></ul><ul><ul><li>overall appearance is based on that of the human body,...
Outline <ul><li>Introduction </li></ul><ul><li>Face recognition 101 </li></ul><ul><li>Face recognition using online social...
Conclusions <ul><li>Online social networks </li></ul><ul><ul><li>contain vast amounts of collective knowledge (‘human comp...
<ul><li>Thank you! Any questions or comments? </li></ul><ul><ul><ul><li>Contact information   e-mail:  [email_address] </l...
Video Demos <ul><li>Microsoft OneAlbum </li></ul><ul><ul><li>[online]  http://www.youtube.com/watch?v=BXv9Kk8y7xg </li></u...
References (1/2) <ul><li>[1] Z. Stone, T. Zickler, T. Darrell, “Autotagging Facebook: Social Network Context Improves Phot...
References (2/2) <ul><li>[6] N. Mavridis, W. Kazmi, P. Toulis, “Friends with Faces: How Social Networks Can Enhance Face R...
Picture Credits <ul><li>Flickr: Barack Obama's Photostream </li></ul><ul><ul><li>[online]  http://www.flickr.com/photos/ba...
Upcoming SlideShare
Loading in …5
×

Face Recognition for Personal Photos using Online Social Network Context and Collaboration

2,421 views

Published on

Thanks to easy-to-use multimedia devices and cheap storage and bandwidth, present-day social media applications host staggering numbers of personal photos. As the number of personal photos shared on social media applications continues to accelerate, the problem of organizing and retrieving relevant photos becomes more apparent for consumers. Automatic face recognition assists in bringing order to collections of personal photos. However, personal photos pose a plethora of challenges for automatic face recognition. Face images may widely differ in terms of lighting, expressions, and pose. As a result, the accuracy of appearance-based techniques for automatic face recognition in collections of personal photos cannot be considered satisfactory.

This talk aims at providing insight into timely developments in the area of socially-aware face recognition. We first discuss how online social network context can be used to substantially improve the effectiveness of appearance-based techniques for automatic face recognition, as recently demonstrated by researchers of Harvard University. Next, we pay attention to collaborative face recognition in decentralized online social networks, as studied at KAIST. For both of the aforementioned topics, we present experimental results obtained for real-world collections of personal photos, contributed by volunteers who are members of online social networks such as Facebook and Cyworld. Finally, we conclude our talk with an outline of future applications of socially-aware face recognition, including augmented identity and socially-aware robots.

Published in: Technology, News & Politics
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,421
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
0
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide
  • http://www.gartner.com/it/page.jsp?id=1447613
  • http://www.economist.com/node/10880936?story_id=10880936 https://joindiaspora.com/ http://www.thimbl.net/
  • http://ilabs.microsoft.com/Project/Pages/Project.aspx?ProjectId=4
  • http://www.youtube.com/watch?v=tb0pMeg1UN0
  • Screenshot from “Minority Report”.
  • Pictures of “Albert Einstein Hubo”.
  • Face Recognition for Personal Photos using Online Social Network Context and Collaboration

    1. 1. Face Recognition for Personal Photos using Online Social Network Context and Collaboration Guest Lecture at KAIST 14 December, 2010 Wesley De Neve, Jaeyoung Choi, Yong Man Ro Image and Video Systems Lab Department of Electrical Engineering Korea Advanced Institute of Science and Technology (KAIST) Yuseong-gu, Daejeon, Republic of Korea e-mail: [email_address] web: http://ivylab.kaist.ac.kr
    2. 2. Professional Background <ul><li>Licentiate degree in computer science (June 2002) </li></ul><ul><ul><li>at Ghent University, Ghent, Belgium </li></ul></ul><ul><li>Ph.D. degree in computer science engineering (February 2007) </li></ul><ul><ul><li>at Ghent University, Ghent, Belgium </li></ul></ul><ul><ul><li>dissertation on “Description-driven media resource adaptation” </li></ul></ul><ul><li>Postdoctoral researcher (September 2007) </li></ul><ul><ul><li>at Ghent University ‐ IBBT, Ghent, Belgium </li></ul></ul><ul><ul><li>at Information and Communications University (ICU), Daejeon, Korea </li></ul></ul><ul><li>Senior researcher (March 2009) </li></ul><ul><ul><li>at KAIST, Daejeon, Korea </li></ul></ul><ul><ul><li>research focus on combining visual content analysis and collective knowledge in social media applications </li></ul></ul>/55
    3. 3. Outline <ul><li>Introduction </li></ul><ul><li>Face recognition 101 </li></ul><ul><li>Face recognition using online social network context </li></ul><ul><li>Collaborative face recognition in online social networks </li></ul><ul><li>Future applications </li></ul><ul><li>Conclusions </li></ul>/55
    4. 4. Outline <ul><li>Introduction </li></ul><ul><li>Face recognition 101 </li></ul><ul><li>Face recognition using online social network context </li></ul><ul><li>Collaborative face recognition in online social networks </li></ul><ul><li>Future applications </li></ul><ul><li>Conclusions </li></ul>/55
    5. 5. Introduction (1/2) <ul><li>The number of personal photos shared online keeps growing </li></ul><ul><ul><li>thanks to easy-to-use multimedia devices and online services </li></ul></ul><ul><ul><li>thanks to cheap storage and bandwidth </li></ul></ul><ul><ul><li>thanks to an increasing number of people going online </li></ul></ul><ul><li>Statistics </li></ul><ul><ul><li>Flickr (as of September 2010) </li></ul></ul><ul><ul><ul><li>hosts 5 billion images, with 3,000 new images uploaded every minute </li></ul></ul></ul><ul><ul><ul><li>more than 40 million users </li></ul></ul></ul><ul><ul><li>Facebook (as of January 2010) </li></ul></ul><ul><ul><ul><li>more than 2.5 billion photos are uploaded each month </li></ul></ul></ul><ul><ul><ul><li>more than 500 million active users </li></ul></ul></ul>/55
    6. 6. Introduction (2/2) <ul><li>Problem: digital information overload </li></ul><ul><ul><li>our ability to automatically organize photos does not keep up with our ability to create and store photos </li></ul></ul><ul><li>Promising solution </li></ul><ul><ul><li>automatic face detection, face recognition, and face annotation </li></ul></ul><ul><ul><li>allows identity-based organization and retrieval of photos </li></ul></ul>/55
    7. 7. Face Detection, Recognition, and Annotation /55 face detection Identity tags: Barack Obama, Joe Biden face annotation retrieval of photos based on identity tags Barack Obama Joe Biden face recognition
    8. 8. Lecture Goals and Main Sources <ul><li>To provide an answer to the following questions </li></ul><ul><ul><li>what is face recognition and why is it relevant? </li></ul></ul><ul><ul><li>what is the value of </li></ul></ul><ul><ul><ul><li>face recognition using online social network context? </li></ul></ul></ul><ul><ul><ul><li>collaborative face recognition? </li></ul></ul></ul><ul><ul><li>what are future applications of socially-aware face recognition? </li></ul></ul><ul><li>Main academic sources </li></ul><ul><ul><li>Z. Stone, T. Zickler, T. Darrell, “Toward Large-Scale Face Recognition using Social Network Context”, Proceedings of the IEEE , 2010 [ DOI ] </li></ul></ul><ul><ul><li>J.Y. Choi, W. De Neve, K. N. Plataniotis, Y.M. Ro, “Collaborative Face Recognition for Improved Face Annotation in Personal Photo Collections Shared on Online Social Networks”, IEEE Transactions on Multimedia , 2011 [ DOI ] </li></ul></ul>/55
    9. 9. Outline <ul><li>Introduction </li></ul><ul><li>Face recognition 101 </li></ul><ul><li>Face recognition using online social network context </li></ul><ul><li>Collaborative face recognition in online social networks </li></ul><ul><li>Future applications </li></ul><ul><li>Conclusions </li></ul>/55
    10. 10. Application Areas of Face Recognition <ul><li>Identity verificiation </li></ul><ul><ul><li>face recognition is used to confirm the identity claim of a given person </li></ul></ul><ul><ul><li>relevant to applications such as </li></ul></ul><ul><ul><ul><li>controlling access to buildings and computer terminals (e.g., Kinect) </li></ul></ul></ul><ul><ul><ul><li>identity verification of passport holders (immigration) </li></ul></ul></ul><ul><li>Identity recognition </li></ul><ul><ul><li>face recognition is used to identify an unknown person, by matching his/her face image against a gallery of known face images </li></ul></ul><ul><ul><li>relevant to applications such as </li></ul></ul><ul><ul><ul><li>video surveillance </li></ul></ul></ul><ul><ul><ul><li>face annotation in personal photo collections </li></ul></ul></ul>/55
    11. 11. Conceptual Design of a Face Recognition System /55 preprocessing (e.g., scaling and rotation to put eyes on fixed locations) face detection gallery of known face images Hillary Joe Barack Robert matching unknown probe face images input photo ~ rank 1 rank 2 rank 3 rank 4 ranked list of candidate identities
    12. 12. <ul><li>Face feature vector </li></ul><ul><ul><li>a d -dimensional vector of feature values (e.g., grayscale pixel values) </li></ul></ul><ul><ul><ul><li>appearance-based face recognition </li></ul></ul></ul>Matching Face Images /55 g a l l e r y p r o b e x = [ x 1 , ..., x 72 ] feature extraction y = [ y 1 , ..., y 72 ] feature extraction |x - y| n
    13. 13. Possible Outcomes of Face Recognition True negative (system correctly decides that the gallery does not contain the identity of the probe face image) False negative (system incorrectly decides that the gallery does not contain the identity of the probe face image) False positive (system incorrectly matches the probe face image with one of the gallery face images) True positive (system correctly matches the probe face image with one of the gallery face images) /55 = X = V = ? V = ? X min ( |x - y| n )
    14. 14. Effectiveness of Face Recognition (1/2) <ul><li>Automatic appearance-based face recognition for personal photos is a hard problem </li></ul><ul><ul><li>uncontrolled variations in expression, pose, lighting, and spatial resolution </li></ul></ul><ul><ul><li>presence of heavy makeup, eye glasses, facial hair, and occlusions </li></ul></ul><ul><li>Automatic appearance-based face recognition is even more difficult in large online photo collections </li></ul><ul><ul><li>may contain hundreds of millions of individuals </li></ul></ul>/55
    15. 15. Effectiveness of Face Recognition (2/2) <ul><li>Appearance-based face recognition for Facebook photos </li></ul><ul><ul><li>is only able to deal with a limited number of gallery face images </li></ul></ul><ul><ul><li>the difference in appearance between individuals becomes very small relative to the difference in appearance of any particular individual </li></ul></ul>probe face image first hit at rank 12 rank 1 rank 2 ... /55 image from [2]
    16. 16. Room for Improvement... <ul><li>Gartner hype cycle </li></ul><ul><ul><li>describes the adoption of new technology </li></ul></ul>/55 (1) appearance-based face recognition for personal photos (1) (2) appearance-based face recognition for personal photos using online social network context (2)
    17. 17. Outline <ul><li>Introduction </li></ul><ul><li>Face recognition 101 </li></ul><ul><li>Face recognition using online social network context </li></ul><ul><li>Collaborative face recognition in online social networks </li></ul><ul><li>Future applications </li></ul><ul><li>Conclusions </li></ul>/55
    18. 18. Online Sharing of Personal Photos <ul><li>Photos on online social networks do not exist in isolation </li></ul><ul><ul><li>arrive in a batch of related photos from a trip or event </li></ul></ul><ul><ul><li>are associated with their photographer </li></ul></ul><ul><ul><li>are broadcasted out to the online contacts of the photographer </li></ul></ul><ul><ul><li>join a collection of billions of other photos </li></ul></ul>/55 event photographer sharing
    19. 19. Online Social Network Context (1/2) <ul><li>Research question </li></ul><ul><ul><li>how to use online social network context for improving the accuracy of appearance-based face recognition for personal photos on Facebook? </li></ul></ul>/55 manually labeled face images contact list of the photographer (social network structure)
    20. 20. Online Social Network Context (2/2) <ul><li>Contact list of the photographer </li></ul><ul><ul><li>used for reducing the number of known face images in the gallery </li></ul></ul><ul><ul><ul><li>from millions to hundreds of candidate identities </li></ul></ul></ul><ul><li>Manually labeled face images </li></ul><ul><ul><li>used for learning subject popularity and the strength of social ties </li></ul></ul><ul><ul><ul><li>e.g., occurrences and co-occurrences of individuals </li></ul></ul></ul><ul><ul><li>available thanks to the provision of social incentives </li></ul></ul><ul><ul><ul><li>different from general image tagging </li></ul></ul></ul><ul><li>Use of a contact list and manually labeled face images motivated by an empirical study for Facebook </li></ul>/55
    21. 21. Empirical Study for Facebook <ul><li>Collected manually labeled face images from 50 college-aged volunteers and their friends </li></ul><ul><ul><li>by means of a Facebook platform application </li></ul></ul><ul><li>After pre-processing, a collection was obtained of 2.5 million reliably labeled face images of 385,624 individuals </li></ul>(*) using an open-source frontal face detector, it was found that 32% of the 8.1 million manually attached identity tags could be reliably associated with a machine-detectable frontal face /55 July 2009 Friends per volunteer (avg.) 645 Volunteers and friends (total #individuals) 22,108 Photos 7.7 million Identity tags (*) 8.1 million
    22. 22. Observations (1/2) <ul><li>Most people can be associated with at least one identity tag </li></ul><ul><ul><li>out of the 22,108 volunteers and friends, 67% could be associated with at least one labeled face image </li></ul></ul>significant amount of labeled face images that can be used to train and test face recognition algorithms /55 image from [2] Number of tags ( N ) Fraction of individuals tagged N times
    23. 23. Observations (2/2) <ul><li>About 30% of the tagged faces in a photographer’s albums belong on average to the photographer him or herself </li></ul><ul><ul><li>a face recognition system can draw upon social context surrounding the photographer to reduce the set of possible identity labels </li></ul></ul><ul><li>People appear in photos with fewer people than they count among their Facebook friends (with less than 13%) </li></ul><ul><ul><li>photo co-occurrence defines a subgraph of an individual’s friend graph that may be more relevant for predicting co-occurrence in new photos </li></ul></ul>/55
    24. 24. Summary Observations <ul><li>Empirical study for Facebook </li></ul><ul><ul><li>suggests that it may be useful to rely on online social network context for improving the accuracy of appearance-based face recognition for personal photos on Facebook </li></ul></ul><ul><ul><ul><li>use of the contact list of the photographer </li></ul></ul></ul><ul><ul><ul><ul><li>for recuding the number of candidate identity labels </li></ul></ul></ul></ul><ul><ul><ul><li>use of manually labeled face images </li></ul></ul></ul><ul><ul><ul><ul><li>for learning the strength of social ties </li></ul></ul></ul></ul><ul><li>Question </li></ul><ul><ul><li>how to integrate online social network context into appearance-based face recognition? </li></ul></ul>/55
    25. 25. Mathematical Modeling (1/2) <ul><li>Need for a mathematical tool that supports labeling of face images by combining two heterogeneous sources of information </li></ul><ul><ul><li>the appearance of each face (i.e., pixel data) </li></ul></ul><ul><ul><ul><li>as used by conventional face recognition techniques </li></ul></ul></ul><ul><ul><li>the strength of social ties (i.e., social network structure) </li></ul></ul><ul><ul><ul><li>learned from the manually labeled face images </li></ul></ul></ul>/55 image adopted from [2]
    26. 26. Mathematical Modeling (2/2) <ul><li>Use of a probability model known as a Markov Random Field </li></ul><ul><ul><li>allows inferring an identity label y i for each face by combining </li></ul></ul><ul><ul><ul><li>node features φ i (appearance-based face recognition scores) </li></ul></ul></ul><ul><ul><ul><li>edge features φ i,j (pairwise co-occurrences of individuals) </li></ul></ul></ul>Markov Random Field (MRF) /55 y 1 y 2 y 3 φ 2 φ 1 φ 3 φ 1,2 φ 2,3 φ 1,3
    27. 27. Experimental Results (1/2) Probability – proportion of face images with a correct label in the top R suggested labels /55 image from [2] Rank ( R ) – number of suggested identity labels
    28. 28. Experimental Results (2/2) <ul><li>Additional observations and discussion </li></ul><ul><ul><li>the combined use of pixel data and social context yields higher face recognition rates than the use of either information source alone </li></ul></ul><ul><ul><li>the probability of having a correctly suggested label at rank one is low </li></ul></ul><ul><ul><ul><li>from a practical point of view, socially-aware face recognition could be used to create a short list of R candidate identity tags </li></ul></ul></ul><ul><ul><li>room for use of other social signals </li></ul></ul><ul><ul><ul><li>social interaction (e.g., message exchanges and comments) </li></ul></ul></ul><ul><ul><ul><li>gender and age </li></ul></ul></ul><ul><ul><li>how about relationships changing over time? </li></ul></ul>/55
    29. 29. Outline <ul><li>Introduction </li></ul><ul><li>Face recognition 101 </li></ul><ul><li>Face recognition using online social network context </li></ul><ul><li>Collaborative face recognition in online social networks </li></ul><ul><li>Future applications </li></ul><ul><li>Conclusions </li></ul>/55
    30. 30. Centralized and Decentralized Online Social Networks <ul><li>Centralized online social networks </li></ul><ul><ul><li>are highly popular (e.g., Facebook) </li></ul></ul><ul><ul><li>present several problems </li></ul></ul><ul><ul><ul><li>information silos </li></ul></ul></ul><ul><ul><ul><li>ownership of data </li></ul></ul></ul><ul><ul><ul><li>privacy issues </li></ul></ul></ul><ul><li>Decentralized online social networks </li></ul><ul><ul><li>are attracting more and more interest (e.g., Diaspora and Thimbl) </li></ul></ul><ul><ul><li>present several advantages </li></ul></ul><ul><ul><ul><li>do not bind users to a particular online social network </li></ul></ul></ul><ul><ul><ul><li>give users more control over data ownership and privacy </li></ul></ul></ul>/55 image from The Economist
    31. 31. Architecture of Centralized and Decentralized Online Social Networks <ul><li>Centralized </li></ul><ul><ul><li>central server with all user data (e.g., contact lists, photos) </li></ul></ul><ul><li>Decentralized </li></ul><ul><ul><li>each user has a personal server containing his/her user data </li></ul></ul>/55
    32. 32. Collaborative Face Recognition (1/2) <ul><li>Decentralized online social networks </li></ul><ul><ul><li>each user will have a personalized face recognition engine , optimized for recognizing face images of the user in question </li></ul></ul><ul><li>Research question </li></ul><ul><ul><li>given a user, how about using the face recognition engines of other users for face annotation in the personal photos of the given user? </li></ul></ul><ul><ul><ul><li>i.e., how about using the personalized face recognition engines of users that often occur in the personal photos of the given user? </li></ul></ul></ul>/55
    33. 33. Collaborative Face Recognition (2/2) /55 personal server photographer photos FR engine personal server contact 1 (family member) photos FR engine personal server contact 3 (friend) photos FR engine personal server contact 2 (family member) photos FR engine personal server contact 4 (co-worker) photos FR engine
    34. 34. Proposed Framework for Collaborative FR <ul><li>Research challenges </li></ul><ul><ul><li>how to select expert face recognition (FR) engines? </li></ul></ul><ul><ul><ul><li>by means of online social network context </li></ul></ul></ul><ul><ul><li>how to merge multiple face recognition scores into a single decision? </li></ul></ul><ul><ul><ul><li>by means of a Bayesian decision rule or majority voting </li></ul></ul></ul>/55 input photo face detection FR engine 2 FR engine 1 FR engine K ... nametagged photo Mark Zuckerberg Jet Li fusion
    35. 35. Selection of Expert FR Engines using Online Social Network Context the thicker the line, the stronger the social tie, the more important the personalized FR engine of the corresponding contact /55 Weighted social graph model for the photographer Contact list contact 1 contact 2 contact 3 contact 4 contact 5 contact 6 Social graph model for the photographer occurrence probabilities co-occurrence probabilities Labeled face images
    36. 36. Experimental Data Collected for Cyworld <ul><li>Retrieval of 547,991 personal photos from four volunteers and their contacts on Cyworld, a Korean online social network </li></ul>/55 ID Age Gender Contacts Years active Volunteer 1 28 Female 165 7 Volunteer 2 29 Male 118 4 Volunteer 3 30 Female 170 6 Volunteer 4 27 Male 84 8 ID Photos Photos with tagged individuals Individuals tagged Detected face images Volunteer 1 251,211 188,422 2,510 213,363 Volunteer 2 109,021 81,211 1,834 94,452 Volunteer 3 117,772 94,297 2,607 104,408 Volunteer 4 69,987 59,753 1,302 64,412
    37. 37. Observations <ul><li>A non-trivial number of face images belong to the photographer </li></ul><ul><ul><li>numbers range from 3.4% for ‘Volunteer 2’ to 14.7% for ‘Volunteer 1’ </li></ul></ul><ul><li>Most face images belong to contacts of the photographer </li></ul><ul><ul><li>numbers range from 73% for ‘Volunteer 3’ to 93% for ‘Volunteer 4’ </li></ul></ul><ul><ul><li>the identity of probe face images not belonging to individuals enrolled in the contact list of a volunteer needs to be asked to the volunteer </li></ul></ul><ul><li>Most of the face images only belong to a small number of contacts of the photographer </li></ul><ul><ul><li>e.g., 91% of the probe face images of ‘Volunteer 1’ belong to 28 contacts </li></ul></ul>/55
    38. 38. Distribution of FR Engine Relevance Values Relevance FR engine FR engine index (in decreasing order of relevance) experimental results for Volunteer 1, having 165 Cyworld contacts /55 28 out of 166 appearance-based FR engines come with high relevance values (‘inner social circle’) head of the distribution
    39. 39. FR Effectiveness of Selected FR Engines Number of FR engines used Number of correctly recognized face images the collaborative use of 28 out of 166 appearance-based FR engines results in a maximum number of correctly recognized face images experimental results for Volunteer 1, having 165 Cyworld contacts number of correctly recognized face images when 28 FR engines are used /55 x
    40. 40. Experimental Results (1/2) Collaborative FR (Bayesian) Collaborative FR (Voting) Non-collaborative FR (Avg.) Rank ( R ) Probability (proportion of face images with a correct label in the top R suggested labels) /55 experimental results for the 28 FR engines selected for Volunteer 1, for both collaborative and non-collaborative FR collaborative vs. non-collaborative FR
    41. 41. Experimental Results (2/2) <ul><li>Explanatory notes </li></ul><ul><ul><li>non-collaborative FR </li></ul></ul><ul><ul><ul><li>accuracy is measured by averaging the face annotation accuracy of all FR engines used to perform collaborative FR </li></ul></ul></ul><ul><ul><li>collaborative FR </li></ul></ul><ul><ul><ul><li>weighting of the different FR scores is either done using a Bayesian decision rule or majority voting </li></ul></ul></ul><ul><ul><ul><ul><li>take into account the relevance of a FR engine </li></ul></ul></ul></ul><ul><li>Collaborative FR is more effective than non-collaborative FR </li></ul><ul><ul><li>by virtue of a complementary effect caused by fusion of multiple face recognition scores </li></ul></ul>/55
    42. 42. Outline <ul><li>Introduction </li></ul><ul><li>Face recognition 101 </li></ul><ul><li>Face recognition using online social network context </li></ul><ul><li>Collaborative face recognition in online social networks </li></ul><ul><li>Future applications </li></ul><ul><li>Conclusions </li></ul>/55
    43. 43. Microsoft OneAlbum <ul><li>OneAlbum project </li></ul><ul><ul><li>allows users to find relevant photos across a social network (e.g., all photos taken by friends at a birthday party and shared on Facebook) </li></ul></ul><ul><ul><li>makes use of unsupervised </li></ul></ul><ul><ul><ul><li>event recognition </li></ul></ul></ul><ul><ul><ul><ul><li>time stamps & visual content </li></ul></ul></ul></ul><ul><ul><ul><li>socially-aware face recognition </li></ul></ul></ul><ul><ul><ul><ul><li>the social graph & occurrence stats </li></ul></ul></ul></ul><ul><li>Related research of IVY Lab </li></ul><ul><ul><li>J.Y. Choi, W. De Neve, Y.M. Ro, K. N. Plataniotis, “Automatic Face Annotation in Personal Photo Collections Using Context-Based Unsupervised Clustering and Face Information Fusion,” IEEE Transactions on Circuits and Systems for Video Technology , 2010 [ DOI ] </li></ul></ul>/55
    44. 44. Augmented Identity <ul><li>Augmented reality </li></ul><ul><ul><li>superimposes virtual objects and info on top of the real world, facilitating interaction between virtual and real objects </li></ul></ul><ul><li>Augmented identity </li></ul><ul><ul><li>user points a smart phone at a person </li></ul></ul><ul><ul><li>software extracts a face feature vector and sends the feature vector to a server </li></ul></ul><ul><ul><li>server matches the feature vector with a pre-registered identity in a database </li></ul></ul><ul><ul><li>server sends back the identity of the subject, as well as contact information </li></ul></ul>/55
    45. 45. Socially-Aware Advertisement Billboards <ul><li>Quotes </li></ul><ul><ul><li>Ray Ozzie (ex-Microsoft) </li></ul></ul><ul><ul><ul><li>“ [We will see] service-connected devices going far beyond just the ‘screen, keyboard and mouse’:  humanly-natural ‘conscious’ devices that’ll see, recognize, hear & listen to you and what’s around you, that’ll feel your touch and gestures and movement, that’ll detect your proximity to others; that’ll sense your location, direction, altitude, temperature, heartbeat & health.” </li></ul></ul></ul><ul><ul><li>Nicholas Negroponte (MIT Media Lab) </li></ul></ul><ul><ul><ul><li>“ Every surface will be a display. Everything will be linked to every other thing. Things will know where they are and some may know who they are.” </li></ul></ul></ul><ul><li>Face recognition </li></ul><ul><ul><li>facilitates customized advertising </li></ul></ul>/55
    46. 46. Socially-Aware Video Surveillance (1/2) <ul><li>Video surveillance </li></ul><ul><ul><li>used to prevent and detect crime </li></ul></ul><ul><ul><li>used to identify terrorists </li></ul></ul><ul><li>Socially-aware video surveillance </li></ul><ul><ul><li>identification by means of social network knowledge </li></ul></ul><ul><ul><ul><li>cf. augmented identity </li></ul></ul></ul>/55
    47. 47. Socially-Aware Video Surveillance (2/2) <ul><li>Research challenges </li></ul><ul><ul><li>robust and large-scale face recognition </li></ul></ul><ul><ul><ul><li>using the entire social graph as a gallery </li></ul></ul></ul><ul><ul><li>gathering of representative data, including access to a social graph </li></ul></ul><ul><ul><ul><li>core asset of online social networks </li></ul></ul></ul><ul><ul><li>privacy issues </li></ul></ul><ul><ul><ul><li>cf. Google Goggles </li></ul></ul></ul><ul><li>Research of IVY Lab on scrambling of face images in surveillance video content </li></ul><ul><ul><li>H. Sohn, W. De Neve, Y. M. Ro, “Privacy Protection in Video Surveillance Systems: Analysis of Subband-Adaptive Scrambling in JPEG XR”, IEEE Transactions on Circuits and Systems for Video Technology , 2011 </li></ul></ul>/55
    48. 48. Socially-Aware Robots <ul><li>Humanoid robots </li></ul><ul><ul><li>overall appearance is based on that of the human body, allowing interaction with made-for-human tools or environments </li></ul></ul><ul><ul><li>need to have the ability to recognize and remember people they interact with </li></ul></ul><ul><ul><ul><li>will be able to learn about characteristics of each individual and treat them uniquely as individuals </li></ul></ul></ul><ul><ul><ul><li>leads to complex social behavior, such as cooperation, dislike, loyalty, and affection </li></ul></ul></ul><ul><ul><li>prototypes are already accessing Facebook </li></ul></ul>/55 Albert Einstein Hugo
    49. 49. Outline <ul><li>Introduction </li></ul><ul><li>Face recognition 101 </li></ul><ul><li>Face recognition using online social network context </li></ul><ul><li>Collaborative face recognition in online social networks </li></ul><ul><li>Future applications </li></ul><ul><li>Conclusions </li></ul>/55
    50. 50. Conclusions <ul><li>Online social networks </li></ul><ul><ul><li>contain vast amounts of collective knowledge (‘human computation’) </li></ul></ul><ul><ul><li>allow researchers to test algorithms in realistic conditions without exceptional data collection effort </li></ul></ul><ul><li>Online social network context and collaboration </li></ul><ul><ul><li>allow for a substantial increase in the effectiveness of appearance-based face recognition for personal photos shared online </li></ul></ul><ul><li>Socially-aware face recognition </li></ul><ul><ul><li>will enable applications in the (near) future that may have a tremendous impact on our daily lifes </li></ul></ul>/55
    51. 51. <ul><li>Thank you! Any questions or comments? </li></ul><ul><ul><ul><li>Contact information e-mail: [email_address] </li></ul></ul></ul><ul><ul><ul><li>web: http://ivylab.kaist.ac.kr </li></ul></ul></ul>/55
    52. 52. Video Demos <ul><li>Microsoft OneAlbum </li></ul><ul><ul><li>[online] http://www.youtube.com/watch?v=BXv9Kk8y7xg </li></ul></ul><ul><li>Augmented identity </li></ul><ul><ul><li>Fraunhofer Augmented Identity </li></ul></ul><ul><ul><ul><li>[online] http://www.scivee.tv/node/17965 </li></ul></ul></ul><ul><ul><li>TAT Augmented Identity </li></ul></ul><ul><ul><ul><li>[online] http://www.youtube.com/watch?v=tb0pMeg1UN0 </li></ul></ul></ul>/55
    53. 53. References (1/2) <ul><li>[1] Z. Stone, T. Zickler, T. Darrell, “Autotagging Facebook: Social Network Context Improves Photo Annotation”,   Proc. of the IEEE Computer Vision and Pattern Recognition Workshops , 2008 </li></ul><ul><li>[2] Z. Stone, T. Zickler, T. Darrell, “Toward Large-Scale Face Recognition using Social Network Context”, Proceedings of the IEEE , 2010 </li></ul><ul><li>[3] J.Y. Choi, W. De Neve, K. N. Plataniotis, Y.M. Ro, “Collaborative Face Recognition for Improved Face Annotation in Personal Photo Collections Shared on Online Social Networks”, IEEE Transactions on Multimedia , 2011 </li></ul><ul><li>[4] J.Y. Choi, W. De Neve, Y.M. Ro, K. N. Plataniotis, “ Automatic Face Annotation in Personal Photo Collections Using Context-Based Unsupervised Clustering and Face Information Fusion ”, IEEE Transactions on Circuits and Systems for Video Technology , 2010 </li></ul><ul><li>[5] K. W. Bowyer, “Face Recognition Technology: Security versus Privacy”, IEEE Technology and Society Magazine , 2004 </li></ul>/55
    54. 54. References (2/2) <ul><li>[6] N. Mavridis, W. Kazmi, P. Toulis, “Friends with Faces: How Social Networks Can Enhance Face Recognition and Vice Versa”, Computational Social Network Analysis , 2010 </li></ul><ul><li>[7] L. Aryananda, “Online and Unsupervised Face Recognition for Humanoid Robot: Toward Relationship with People”, Proc. of the 2001 IEEE-RAS International Conference on Humanoid Robots , 2001 </li></ul><ul><li>[8] C. Au Yeung, I. Liccardi, K. Lu, O. Seneviratne, T. Berners-Lee, “ Decentralization: The Future of Online Social Networking”, W3C Workshop on the Future of Social Networking , 2009 </li></ul><ul><li>[9] H. Sohn, W. De Neve, Y. M. Ro, “Privacy Protection in Video Surveillance Systems: Analysis of Subband-Adaptive Scrambling in JPEG XR”, IEEE Transactions on Circuits and Systems for Video Technology , 2011 </li></ul><ul><li>[10] P. Levy (Author), R. Bononno (Translator), “Collective Intelligence: Mankind's Emerging World in Cyberspace”, Perseus Books, 1999 </li></ul>/55
    55. 55. Picture Credits <ul><li>Flickr: Barack Obama's Photostream </li></ul><ul><ul><li>[online] http://www.flickr.com/photos/barackobamadotcom/ </li></ul></ul><ul><li>Flickr: Mynameisharsha's Photostream </li></ul><ul><ul><li>[online] http://www.flickr.com/photos/mynameisharsha/ </li></ul></ul>/55

    ×