Relationship plays an important role, and a picture is worth a thousand words. In this work, we Infer relationship in social network from images.

1. Match?

2. Infer Relationship in Social
Network from Images
Huan-Cheng Hsu

3. Social network

4. Relationship in Social Network
• Relationship plays an important role
10 Mutual
Friends

5. What Type of Person Are You?
• A picture is worth a thousand words
Am I handsome?
15, Oct. 2015
Good Job! Baby
28, Sep. 2015
Morning!
2, Sep. 2015

6. Do We Match Each Other?
• Inferring from Images
Match?

7. Framework
Training Data
Sentiment Feature Extraction
Learning from Label Proportion
Negative Model
Positive Model
Event Detection

8. Sentiment Feature Extraction
• Nowadays the Internet, as a major platform for
communication and information exchange,
provides a rich repository of people’s opinion and
sentiment about a vast spectrum of topics
• The analysis of such information, such as
comments, tags, browsing actions, as well as
shared media objects, plays an important role in
behavior sciences, which aims to understand and
predict human decision making
Borth, Damian, et al. "Large-scale visual sentiment ontology and detectors using adjective noun
pairs." Proceedings of the 21st ACM international conference on Multimedia. ACM, 2013.

9. Sentiment Feature Extraction
• The computational analysis of sentiment
mostly concentrates on the textual content.
Limited efforts have been conducted to
analyze sentiments from visual content
• Visual content conveying valuable sentiment
information primarily visually

10. Sentiment Feature Extraction
• Sentiment analysis research is the need of
visual content analysis

11. Framework for Constructing SentiBank
mid-level
visual
concept

12. Experiment
• cute_smile, sweet_chocolate, magnificent_flowers

13. Experiment
• magnificent_sea, crazy_car, scenic_ocean

14. Experiment
• mad_hair, sweet_chocolate, outdoor_party

15. Experiment
• sexy_body, funny_glasses, fancy_dress

16. Video Event Detection
Am I handsome?
15, Oct. 2015
Good Job! Baby
28, Sep. 2015
Morning!
2, Sep. 2015
Continuous!
Key Frame Key Frame Key Frame

17. Plutchik’s Wheel of Emotions
24 Events
Negative Emotion
Positive Emotion
2 Events

18. Video Event Detection
Negative
Positive
100% positive
100% negative

19. Learning with Label Proportions(LLP)
Yu, F. X., et al. "SVM for learning with label proportions." Proceedings of the 30rd
International Conference on Machine learning. 2013.
Negative
Positive
33% positive
75% positive

20. Video Detection by LLP
Lai, Kuan-Ting, et al. "Video event detection by inferring temporal instance labels." Computer
Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on. IEEE, 2014.

21. Experiment Setting
• Dataset
– This dataset contains 2 events
– 5816 people
– Every person has 100 images
– Split 2/3 of people as training set and 1/3 as test set
• Features
– Frame features are quantized into 1 and -1 to represent
positive and negative emotion respectively
• Label
– Bags are labeled positive when having 50% more positive
images and negative if less than 50%

22. Experiment Result
• Precision of Event1(Positive)
– 99.92%
• Precision of Event2(Negative)
– 99.28%
• Average Precision
– 99.6%

23. Do We Match Each Other?
• 99.92%
Match?

24. Thank you for your attention!