The talk at the Huang Symposium to celebrate Tom Huang's 80th Birthday

1. Forever Young
A Tribute to the Grandmaster
& A Recount of Personal Journey
Jiebo Luo
University of Rochester

2. 2
Why Do We Need to Do Research?

3. 3
Why Do We Need to Do Research?

4. 4
Why Do We Need to Do Research?

5. 5
Learned from the Grandmaster
Curiosity
Open minded
New problems
New techniques
Passion
Love, focus - sustained
Scholarship
Unaffected by noise, hype
Uncompromised integrity
Never too old to learn

6. 6
PhD Thesis
Image Processing
Image and Video Coding
wavelet transform
Scene-adaptive Coding
primitive/budding
scene understanding

7. 7
Kodak Years
• Intelligent Image Processing and Semantic Image Understanding
– Automatic Redeye Removal
– Automatic Image Orientation Detection
– Image Classification
– Image Annotation
– Geotagging
– Context + Content

8. From Classification to Description
Recognizing Realistic Actions from Videos "in the Wild"
UCF-11 to UCF-101
(CVPR 2009)
Similarity btw Videos Cross-Domain Learning
Visual Event Recognition in Videos by
Learning from Web Data
(CVPR2010 Best Student Paper)
Heterogeneous Feature Machine For Visual Recognition
(ICCV 2009)

9. Image Captioning with Semantic Attention
• Motivations
– Real-world Usability
• Help visually impaired people, learning-impaired
– Improving Image Understanding
• Classification, Objection detection
– Image Retrieval
1. a young girl inhales with the intent of blowing
out a candle
2. girl blowing out the candle on an ice cream
1. A shot from behind home
plate of children playing
baseball
2. A group of children playing
baseball in the rain
3. Group of baseball players
playing on a wet field

10. Key Elements
• Additional textual information
– Leverage noisy titles, tags or captions (Web)

11. Key Elements
• Additional textual information
– Leverage noisy titles, tags or captions (Web)
– Leverage visually similar nearest neighbor images

12. Key Elements
• Additional textual information
– Leverage noisy titles, tags or captions (Web)
– Leverage visually similar nearest neighbor images
– Incorporate success of low-level tasks
• Visual attribute detection

13. Attention Model on Attributes
• Instead of using the same set of attributes at every
step
• At each step, select the attributes (attention)
 m mtmt kKwatt ),(
)softmax VK(wT
tt 
))],,(;([),( 11   tttttt hKwattxfhxfh

14. Overall Framework
• Training with a bilinear/bilateral attention model
ht
pt
xt
v
{Ai}
Yt~
RNN
Image
CNN
AttrDet 1
AttrDet 2
AttrDet 3
AttrDet N

t = 0
Word

15. Performance
• MS-COCO Image Captioning Challenge

16. TGIF: A New Video Dataset and Benchmark

17. Examples
a skate boarder is
doing trick on his skate
board.
a gloved hand opens to
reveal a golden ring.
a sport car is swinging on
the race playground
the vehicle is moving fast
into the tunnel

18. Machine Generated Sentence Examples

19. 19
Two Most Important Social Signals (IMO)
• User
• Sentiment

20. • Cross‐modality Consistent Regression
Joint Visual-Textual Sentiment Analysis

21. 21
Infer User Demographics & Interests

22. • Social interactions and social activities
• Public health surveillance
• Web sentiment analysis and trend prediction
• Cyber terrorism, extremism, and activism
• Fads and infectious ideas
• Marketing intelligence analytics
• Traffic and human mobility patterns
• Human and environment
• Social unrest, protest and riot
Understanding the Pulse of Society

23. 23
•Correctly classified flattering expressions
•Correctly classified unflattering expressions
•Correctly classified neutral expressions
• Acts like a prism to reveal the spectrum of opinions
• Competitive Vector Autoregressive Model
2012: Calling the Swing States

24. 2016: Fine-Grained Campaign Analysis

25. 2016: Shifting Tide? Too Close to Call
25
August 26, 2016
September 15, 2016

26. Social
Multimedia
Visual
Data
Textual
Data
Computer
Vision
NLP
User
Demographics
User
Activities
Behavior
Patterns
Using Social Multimedia to Study Social Problems

27. Time Pattens of Underage Alcohol Use
Temporal Patterns of Underage Drinking

28. Drug Use Pattern from Instagram

29. 29
When We Were Young
29

30. 30
Forever Young
30
Forever young,
I want to be forever young.
Do you really want to live forever?
Forever, and ever
Forever young,
I want to be forever young.
Do you really want to research forever?
Forever, and ever
Image Processing, Computer Vision, Multimedia, Social Media, Big Data, …
(A younger version of an old song)
……
Let’s Celebrate the Forever Young Huang Academic Tree!