User intentions in multimedia, implications, collection of example projects and outlook.

1. Why and how?
User Intentions and
their Relation to
Multimedia Applications
Mathias Lux

2. Why and how?
User Intentions and
their Relation to
Multimedia Applications
Mathias Lux

3. The „intention gap“
[...] Between the user’s search intention and what he submits as the
query, he is caught in the “intention gap”, due to the difficulty of
converting the intention into a search-engine-friendly query. The user
intention gap remains a major obstacle to meeting the music
information needs of users [...]
Src. Zhu, Shenggao, et al. "Bridging the User Intention Gap: an Intelligent and
Interactive Multidimensional Music Search Engine." Proceedings of the First
International Workshop on Internet-Scale Multimedia Management. ACM, 2014.

4. The „intention gap“
Is it like ... ???
„users are not able to query a retrieval
system the right way“

5. *RTFM*

6. The „intention gap“
• Common idea is to improve either
– the system or the
– user interface
• By query re-formulation, query expansion, and query
suggestion

7. Example: QBE
• User has particular information need
• Need reflected by example image
• Query is expressed visually

8. PHOG & Flowers

9. ColorLayout & Sunsets

10. EdgeHistogram & Portraits

11. JCD & Portraits

12. ColorLayout & Landscapes

13. PHOG & Birds on the Water

14. We know that ...
• Some features work better than others
• Features have different characteristics
• Some features work out well for some
domains, while others don’t

15. Which one is right?
• How to determine the right feature?
• What are the necessary characteristics?
• How do I define visual similarity
within the domain?
• What is visual similarity
for the user?

16. Approaches ...
• Show all features, but with a clever UI

17. Approaches ...
• Educate users?

18. Approaches ...
• Create a clever system.
• BUT: How?

19. OK, back again ...
• We need systems that infer / know what
users want.
• Are there working examples?

20. Amazon?
• Well, ... we all know the drawbacks of
that one.

21. Why is there a different
ranking?

22. Users in Context

23. Definition: Context
“Context is any information that can be used
to characterize the situation of an entity.
An entity is a person, place, or object that
is considered relevant to the interaction
between a user and an application, including
the user and applications themselves.”
Ref. G. Abowd et al., “Towards a better understanding
of context and context-awareness. In Handheld and
Ubiquitous Computing, vol. 1707 LNCS, 1999.

24. Definition: Intention
noun
(1) thing intended; an aim or plan
(2) Medicine the healing process of a wound
(3) (intentions) Logic conceptions formed
by directing the mind towards an object

25. Context vs. Intention?
Context is any
information that can
be used to
characterize the
situation of an
entity. An entity is a
person, […]
noun
(1) thing intended; an
aim or plan
[…]

26. A User’s Intention is
• part of a user’s context
• of manageable size (verb & frame)
• related to the information need in search
Examples
• I want to download a new background for my mobile.
• I want to share the first step of my kid.
• I want to see what the Mars surface looks like.

27. User Intentions in the Web
Underlying goals of web searches
• Informational
– to learn / know something
• Navigational
– to go to a specific place (on the web)
• Transactional
– to go somewhere to ultimately buy sth.
Ref. Broder: A Taxonomy of Web Search. SIGIR Forum,
2002

28. User Intentions in the Web
Ref. Rose & Levinson: Understanding user goals in web
search, WWW 2004
Broder Survey Broder Log
Analysis
R&L St udy 1 R&L St udy 2 R&L St udy 3
24,5 20
14,7 11,7 13,5
39 48 60,9
61,3 61,5
36
30
24,3 27 25
Navigat ional
Inf ormat ional
Transact ional

29. User Intentions in
Multimedia
Why do people
• search
• produce
• share
• archive
For which type
• image
• video
• audio
• other

30. Hand-picked Examples
Right now there is no all-in-one publication
on user intentions in multimedia …

31. User Intentions in
Image Search
Ref. Lux, Kofler & Marques: A classification scheme for user
intentions in image search, CHI 2010
beautiful sunset for the
background of my mobile
old vs. new
Starbucks logo
my friend’s new
car on flickr.com
Latest “explore”
photos

32. Do queries help with the
search intention?
User information need vs. query formulation in
video search.
• How to support users with video indexing and
search methods?
• Search goal failure is (partially) predictable
– Based on keywords and
– Based on natural language
Ref. Kofler, Larson & Hanjalic: To Seek, Perchance to
Fail: Expressions of User Needs in Internet Video
Search, ECIR 2011

33. Asking for the “Why?”
behind the “What?”
Ref. Hanjalic, Kofler & Larson: Intent and its
Discontents: The User at the Wheel of the Online Video
Search Engine, ACM MM 2012
Hear others singing …
Learn to sing …

34. Online Survey: Why did
you watch this video?
• 270 participants
• Online questionnaire
– Why do you watch videos?
– Which category for which reason?
• Results in a intention 2 category table
Ref. Lagger, Lux & Marques: An Adaptive Video Retrieval
System Based On Recent Studies On User Intentions
While Watching Videos Online. ACM CIE, online.

35. Helping with clever Uis?
Ref. Lagger, Lux & Marques: An Adaptive Video Retrieval
System Based On Recent Studies On User Intentions
While Watching Videos Online. ACM CIE, online.

36. Who are the Users in a
Video Search System?
• Study on users of
– YouTube
– BBC iPlayer
– Uitzending Gemist
Ref. Kemman, Kleppe & Beunders: Who are the users of
a video search system? Classifying a heterogeneous
group with a profile matrix, WIAMIS 2012

37. Who are the Users in a Video
Search System?
Ref. Kemman, Kleppe & Beunders: Who are the users of
a video search system? Classifying a heterogeneous
group with a profile matrix, WIAMIS 2012

38. Why do People make Videos?
• Study on four main goals:
– Affection, Function, Sharing & Preservation.
Ref. Lux & Huber: Why did you record this video?
WIAMIS 2012

39. Correlation of Clusters
• ρ-Coefficient
– Showing minimum,
actual & maximum
Sharing Af f ect ion Funct ion
- 0,59 - 0,7 8 - 0,36
- 0,05 - 0,26 - 0,36
0,39 0,84 0,55
- 0,50 - 0,93
0,25 - 0,07
0,46 0,21
- 0,43
- 0,21
0,47
Preservat ion
Sharing
Af f ect ion

40. Implications?
• 81% of the instances were assigned to more than one category.
– Hypothesis: a discrete mapping to exactly one category is insufficient
• “Preservation” and “Sharing” seem to be independent
– Hypothesis: “Personal reflection” is not the opposite of “publishing”
(cp. Kindberg)
• „Preservation“ and „Function“ are neg. correlated
– Hypothesis: Videos are not archived if they are taken for a functional
reason.

41. Intentional Framing
Def. The framing of a picture is the
Deutungsrahmen (interpretation frame of
the image) which image producers have in
mind during the creation moment of the
picture.

42. Intentional Framing

43. Intentional Framing

44. Intentional Framing and ..

45. Examples for Intentional Framing

46. Intentional Framing
Assumption: Similar intentions for taking
the pictures will lead to similar framings
of the images.
Hypothesis: Certain global image features
can be used to differentiate framings.
Ref. Riegler, Lux, Larson, Kofler (2014) How `how' reflects what's what:
Content-based classification exploiting how users frame images, ACM MM 2014

47. Uploader Intent
• Typology based on
– text mining, crowd sourcing, literature
1. Explaining
2. Sharing
3. Promoting
4. Communicating
5. Entertaining
Kofler et al. (2015) “Uploader Intent for Online Video:
Typology, Inference and Applications”, ACM TOMM, accepted

48. Uploader Intent
Kofler et al. (2015) “Uploader Intent for Online Video:
Typology, Inference and Applications”, ACM TOMM, accepted

49. Finding User Intentions &
Goals is a hard task ….

50. Demand Media – The
Answer Factory
• Demand mined from search queries
• Requests for content put on auction
• Contractors create content
• Crowd does quality control
see i.e. eHow.com
The Answer Factory: Demand Media and the Fast,
Disposable, and Profitable as Hell Media Model,
http://www.wired.com/magazine/2009/10/ff_demandmedia/

51. Human Computation

52. Human Computation

53. Human Computation
• Is crowd-sourcing of any help here?

54. Crowd-Sourcing
• It’s hard to judge intentions of others
– That makes it error prone
“a reminder of the beautiful Island
were [sic] my father came from”
o Recall situation
o Preserve good feelings
o Publish online
o Show to family & friends
o Support task of mine
o Preserve bad feeling

55. turkers
Recall situation 2 2 0 1 0 2
Preserve good feeling 2 -2 1 0 0 1
Publish online 2 0 0 0 1 2
Show to family & friends 2 1 2 1 1 0
Support task of mine 0 -2 1 1 1 -2
Preserve bad feeling -2 0 -2 0 -2 -2
“a reminder of the beautiful Island
were [sic] my father came from”
o Recall situation
o Preserve good feelings
o Publish online
o Show to family & friends
o Support task of mine
o Preserve bad feeling

56. Crowd-Sourcing
• Turkers disagreed with original publishers.
• But pretests had better inter-rater
agreements.
Ref. Lux, Taschwer & Marques: A Closer Look at
Photographers’ Intentions: a Test Dataset, CrowdMM WS
at ACM MM 2012
Int ent ions Ot her
t urkers 0,147 0,232
pret est s 0,57 1 0,510
Ref. Lux, Xhura & Kopper: User Intentions in Digital
Photo Production: a Test Data Set, accepted, MMM 2014

57. Where did we go?
• Intention gap
• User Intentions
– Search
– Production
– Sharing
• Crowdsourcing as a
Method

58. What did we learn?
• It‘s hard for people to judge the intent
of others
• It‘s hard for people to express
their own intent

59. A Word of Caution …
• People can only use the tools that are
available to them.

60. What is left?
• Lots of loose ends & open grounds for
research …

61. Short Term Goals:
Inter-relate CBIR & Intentions
We did some work on the relation of BoVW
& the degree of intentionality ...
• New or adapted features
• Adaptive retrieval systems
• New approach to ground truth

62. Medium Term Goals:
Interdisciplinary Approach
• Group intentions vs. individual intentions
• Empirical and qualitative research on user
context & intentions

63. Medium Term:
Is there an over model?
• Search
• Production
• Sharing
• Archiving
• Image
• Video
• Audio
• Other modalities

64. Long term goals
• Find a way to integrate user context in
multimedia information systems through
intentions
• Find out intentions of users with a high
degree of accuracy

65. Thanks for listening …
• Mathias Lux
– Assoc. Prof @ Klagenfurt University
• mlux@itec.uni-klu.ac.at