Musings at the Crossroads of Digital Libraries, Information Retrieval, and Scientometrics

Musings at the Crossroads ofMusings at the Crossroads of
Digital Libraries, Information Retrieval,Digital Libraries, Information Retrieval,
and Scientometricsand Scientometrics
http://bit.ly/rguCabanac2012http://bit.ly/rguCabanac2012
Guillaume Cabanac
guillaume.cabanac@univ-tlse3.fr
March 28th, 2012

Outline of these Musings
2
Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac
Digital LibrariesDigital Libraries
• Collective annotations
• Social validation of discussion threads
• Organization-based document similarity
Information RetrievalInformation Retrieval
• The tie-breaking bias in IR evaluation
• Geographic IR
• Effectiveness of query operators
ScientometricsScientometrics
• Recommendation based on topics and social clues
• Landscape of research in Information Systems
• The submission-date bias in peer-reviewed conferences

3
• Geographic IR

4
Question DL-1
How to transpose paper-based
annotations into digital documents?
IRIRDLDL
SCIMSCIM
Guillaume Cabanac, Max Chevalier, Claude Chrisment, Christine Julien. “Collective annotation: Perspectives for
information retrieval improvement.” RIAO’07 : Proceedings of the 8th conference on Information Retrieval and its
Applications, pages 529–548. CID, may 2007.

5
 Characteristics of paper annotation
 Secular activity: older than 4 centuries
 Numerous applicative contexts: theology, science, literature …
 Personal use: “active reading” (Adler & van Doren, 1972)
 Collective use: review process, opinion exchange …
From Individual Paper-based Annotation …
US students
(Marshall, 1998)
1541
Annotated bible
(Lortsch, 1910)
Fermat’s last
theorem
(Kleiner, 2000)
Annotations from
Blake, Keats…
(Jackson, 2001)
Les Misérables
Victor Hugo
1630 1790 1830 1881 1998

6
… to Collective Digital Annotations
author
87%
reader
13%
1993 2005
ComMentor … iMarkup … Yawas … Amaya …
> 20 annotation systems
(Cabanac et al., 2005)
Web servers (Ovsiannikov et al., 1999)
Annotation
server
a discussion thread
Hard to share ⇒ ‘lost’
hardcopy

7
 W3C Annotea / Amaya (Kahan et al., 2002)
Digital Document Annotation: Examples
a reader’s comment
discussion
thread
 Arakne, featuring “fluid annotations” (Bouvin et al., 2002)

8
Collective Annotations
 Reviewed 64 systems designed during 1989–2008
 Collective Annotation
 Objective data
 Owner, creation date
 Anchoring point within the document. Granularity: all doc, words…
 Subjective information
 Comments, various marks: stars, underlined text…
 Annotation types: support/refutation, question…
 Visibility: public, private, group…
 Purpose-oriented annotation categories
Annotation remark
Annotation reminder
Annotation argumentation
Personal Annotation Space

9
Question DL-2
How to measure the social validity of
a statement according to the
argumentative discussion it sparked off?
IRIRDLDL
SCIMSCIM
Guillaume Cabanac, Max Chevalier, Claude Chrisment, Christine Julien. “Social validation of collective
annotations : Definition and experiment.” Journal of the American Society for Information Science and
Technology, 61(2):271–287, feb. 2010, Wiley. DOI:10.1002/asi.21255

10
 Scalability issue 
 Which annotations
should I read?
 Social validation = degree of consensus of the group
Social Validation
Social Validation of Argumentative Debates

11
Social Validation of Argumentative Debates
Before
Annotation magma
After
Filtered display
 Informing readers about how validated each annotation is

12
 Overview
 Two proposed algorithms
 Empirical Recursive Scoring Algorithm (Cabanac et al., 2005)
 Bipolar Argumentation Framework Extension
 based on Artificial Intelligence research works (Cayrol & Lagasquie-Schiex, 2005)
Social Validation Algorithms
validity
0
socially neutral
– 1
socially refuted
1
socially confirmed
case 1case 2case 3 case 4
A
B
A
B

13
 Example
 Computing the social validity of a debated annotation
Social Validation Algorithm

14
Validation with a User-study
 Design
 Corpus: 13 discussion threads
= 222 annotations + answers
 Task of a participant
 Label opinion type
 Infer overall opinion
 Volunteer subjects
53
119
 Aim: social validation vs human perception of consensus

15
 Q1 Do people agree when labeling opinions?
 Kappa coefficient (Fleiss, 1971; Fleiss et al., 2003)
Inter-rater agreement among n > 2 raters
 Weak agreement, with variability ⇒ subjective task
Experimenting the Social Validation of Debates
Debate Id
Fair to good
Poor
ValueofKappa
agreement

16
 Q2 How well SV approximates HP?
 HP = Human Perception of consensus
 SV = Social Validation algorithm
1. Test whether PH and VS are different (p < 0.05)
⇒ Student’s paired t-test: (p = 0,20) > (α = 0,05)
2. Correlate HP et SV
⇒ Pearson’s coefficient of correlation r
r(HP, SV) = 0.48 shows a weak correlation
Experimenting the Social Validation of Debates
HP – SV
Density y = p(HP – SV)
example: HP = SV for 24 % of all cases
Density

17
Question DL-3
How to harness a quiescent capital
present in any community:
its documents?
IRIRDLDL
SCIMSCIM
Guillaume Cabanac, Max Chevalier, Claude Chrisment, Christine Julien. “Organization of digital resources as an
original facet for exploring the quiescent information capital of a community.” International Journal on Digital
Libraries, 11(4):239–261, dec. 2010, Springer. DOI:10.1007/s00799-011-0076-6

18
 Personal Documents
 Filtered, validated, organized information…
 … relevant to activities in the organization
 Paradox: profitable, but under-exploited
 Reason 1 –  folders and files are private
 Reason 2 –  manual sharing
 Reason 3 –  automated sharing
 Consequences
 People resort to resources available outside of the community
 Weak ROI ⇒ why would we have to look outside when it’s already there?
Documents as a Quiescent Wealth

19
 Mapping the documents of the community
 SOM [Kohonen, 2001] Umap [Triviumsoft] TreeMap [Fekete & Plaisant, 2001]…
 Limitations
 Find the documents with same topicssame topics as D
 Find documents that colleagues useuse with D
→ concept of usage: grouping documentsgrouping documents ⇆ keeping stuff in commonkeeping stuff in common
How to Benefit from Documents in a Community?

20
 Organization-based similarities
 inter-folder
 inter-document
 inter-user
How to Benefit from Documents in a Community?

21
 Purpose: Offering a global view of
 … people and their documents
 Based on document contents
 Based on document usage/organization
 Requirement: non-intrusiveness and confidentiality
 OperationalOperational needs
 Find documents
 With related materials
 With complementary materials
 Seeking people ⇆ seeking documents
 ManagerialManagerial needs
 Visualize the global/individual activity
 Work position → required documents
How to Help People to Discover/Find/Use Documents?
community

22
4 views = {documents, people} × {group, unit}
1. Group of documents
 Main topics
 Usage groups
2. A single document
 Who to liaise with?
 What to read?
3. Group of people
 Community of interest
 Community of use
4. A single people
 Interests
 Similar users (potential help)
Proposed System: Static Aspect

23
• Geographic IR

24
Question IR-1
Is document tie-breaking
affecting the evaluation of
Information Retrieval systems?
IRIRDLDL
SCIMSCIM
• Geographic IR
Guillaume Cabanac, Gilles Hubert, Mohand Boughanem, Claude Chrisment. “Tie-breaking Bias : Effect of an
Uncontrolled Parameter on Information Retrieval Evaluation.” M. Agosti, N. Ferro, C. Peters, M. de Rijke, and A. F.
Smeaton (Eds.) CLEF’10 : Proceedings of the 1st Conference on Multilingual and Multimodal Information Access
Evaluation, volume 6360 de LNCS, pages 112–123. Springer, sep. 2010. DOI:10.1007/978-3-642-15998-5_13

25
Measuring the Effectiveness of IR systems
 User-centered vs. System-focused [Spärck Jones & Willett, 1997]
 Evaluation campaigns
 1958 Cranfield, UK
 1992 TREC (Text Retrieval Conference), USA
 1999 NTCIR (NII Test Collection for IR Systems), Japan
 2001 CLEF (Cross-Language Evaluation Forum), Europe
 …
 “Cranfield” methodology
 Task
 Test collection
 Corpus
 Topics
 Qrels
 Measures : MAP, P@X ...
using trec_eval [Voorhees, 2007]

26
Runs are Reordered Prior to Their Evaluation
Qrels = 〈qid, iter, docno, rel〉 Run = 〈qid, iter, docno, rank, sim, run_id〉
Reordering by trec_eval
qid asc, sim desc, docno desc
Effectiveness measure = f (intrinsic_quality, )
MAP, P@X, MRR…

27
Consequences of Run Reordering
 Measures of effectiveness for an IRS s
 RR(s,t) 1/rank of the 1st
relevant
document, for topic t
 P(s,t,d) precision at document d, for
topic t
 AP(s,t) average precision for topic t
 MAP(s) mean average precision
 Tie-breaking bias
 Is the Wall Street Journal collection more relevant than Associated Press?
ChrisChris
EllenEllen

Sensitive to
document
rank

28
What we Learnt: Beware of Tie-breaking for AP
 Poor effect on MAP, larger effect on AP
 Measure bounds APRealistic ≤ APConventionnal ≤ APOptimistic
padre1, adhoc’94

29
Question IR-2
How to retrieve documents
matching keywords and
spatiotemporal constraints?
IRIRDLDL
SCIMSCIM
• Geographic IR
Damien Palacio, Guillaume Cabanac, Christian Sallaberry, Gilles Hubert. “On the evaluation of geographic
information retrieval systems: Evaluation framework and case study.” International Journal on Digital Libraries,
11(2):91–109, june 2010, Springer. DOI:10.1007/s00799-011-0070-z

30
Geographic Information Retrieval
 Query = “Road trip around Aberdeen summer 1982”
 Search engines
 Topic term ∈ {road, trip, Aberdeen, summer}
spatial ∈ {AberdeenCity, AberdeenCounty…}
 Geographic temporal ∈ [21-JUN-1982 .. 22-SEP-1982]
term ∈ {road, trip, Aberdeen, summer}
 ≈ 1/6 queries = geographic queries
 Excite (Sanderson et al., 2004)
 AOL (Gan et al., 2008)
 Yahoo! (Jones et al., 2008)
⇒ Current issue worth studying

31
The Internals of a Geographic IR System
 3 dimensions to process
 Topical, spatial, temporal
 1 index per dimension
 Topic bag of words, stemming, weighting, comparing with VSM…
 Spatial spatial entity detection, spatial relation resolution…
 Temporal temporal entity detection…
 Query processing with sequential filtering
 e.g., priority to theme, then filtering according to other dimensions
 Issue: effectiveness of GIRSs vs state-of-the-art IRSs?
 Hypothesis: GIRSs better than state-of-the-art IRSs

32
Case Study: the PIV GIR System
 Indexing: one index per dimension
 Topical = Terrier IRS Spatial = tiling Temporal = tiling
 Retrieval
 Identification of the 3 dimensions in the query
 Routing towards each index
 Combination of results with CombMNZ [Fox & Shaw, 1993; Lee 1997]

33
 Principle of CombMNZ and Borda Count

34
 Gain in effectiveness

35
Question IR-3
Do operators in search queries improve
the effectiveness of search results?
IRIRDLDL
SCIMSCIM
• Geographic IR
Gilles Hubert, Guillaume Cabanac, Christian Sallaberry, Damien Palacio. “Query Operators Shown Beneficial for
Improving Search Results.” S. Gradmann, F. Borri, C. Meghini, H. Schuldt (Eds.) TPDL’11 : Proceedings of the 1st
International Conference on Theory and Practice of Digital Libraries, volume 6966 de LNCS, pages 118–129.
Springer, sep. 2011. DOI:10.1007/978-3-642-24469-8_14.

 Various Operators
 Quotation marks, Must appear (+), boosting operator (^),
Boolean operators, proximity operators…
36
Information need
“I’m looking for research projects funded in the DL domain”
Regular query Query with operators
Search Engines Offer Query Operators

Our Research Questions
37

38
Our Methodology in a Nutshell
Regular query V1: Query variant with operators
  
 


<
V3
V2
V4
VN. . .

39
Effectiveness of Query Operators
 TREC-7 per Topic Analysis: Boxplots
 ‘+’ and ‘^’

40
 Per Topic Analysis: Box plot
AP of TREC’s regular query
Query variant highest AP
32
Topics
AP(AveragePrecision)
0.2
0.1
0.3
0.4
Query variant lowest AP

41
 TREC-7 Per Topic Analysis
 ‘+’ and ‘^’
MAP  = 0.1554
MAP ┬ = 0.2099
+35.1%

42
• Geographic IR

43
Question SCIM-1
How to recommend researchers
according to their research topics
and social clues?
IRIRDLDL
SCIMSCIM
Guillaume Cabanac. “Accuracy of inter-researcher similarity measures based on topical and social clues.”
Scientometrics, 87(3):597–620, june 2011, Springer. DOI:10.1007/s11192-011-0358-1

44
Recommendation of Literature (McNee et al., 2006)
 Collaborative filtering
 Principle: mining the preferencespreferences of researchers
→ those who liked this paper also liked…
 Snowball effect / fad
 Innovation?
 Relevance of theme?
 Cognitive filtering
 Principle: mining the contentscontents of articles
→ profile of resources (researcher, articles)
→ citation graph
 Hybrid approach





????

45
Foundations: Similarity Measures Under Study
 Model
 Coauthors graph authors ↔ auteurs
 Venues graph authors ↔ conferences / journals
 Social similarities
 Inverse degree of separation length of the shortest path
 Strength of the tie number of shortest paths
 Shared conferences number of shared conference editions
 Thematic similarity
 Cosine on Vector Space Model di = (wi
1
, … , wi
n
)
built on titles (doc / researcher)

46
Computing Similarities with Social Clues
 Task of literature review
 Requirement topical relevance
 Preference social proximity (meetings, project…)
⇒ re-rank topical results with social clues
 Combination with CombMNZ (Fox & Shaw, 1993)
 Final result: list of recommended researchers
CombMNZ
Degree of separation
Strength of ties
Shared conferences
Social list
Topical list
∩
CombMNZ TS listTS list

47
Evaluation Design
 Comparison of recommendations and researchers’ perception
 Q1 : Effectiveness of topical (only) recommendations?
 Q2 : Gain due to integrating social clues?
 IR experiments: Cranfield paradigm (TREC…)
 Does the search engine retrieve relevant documents?
Doc relevant?
assessor
relevance judgments
{0, 1} binary
[0, N] gradual
qrels
trec_eval
Effectiveness measures
Mean Average Precision
Normalized Discounted Cumulative Gain
topic S1 S2
1 0.5687 0.6521
… … …
50 0.7124 0.7512
avg 0.6421 0.7215
improvement +12.3 %
significativity p < 0.05 (paired t-test)
search engine x
input
topic
corpus

48
Evaluating Recommendations
doc relevant ?
assessor
relevance judgments
{0, 1} binary
[0, N] gradual
qrels
trec_eval
Effectiveness measures
Mean Average Precision
Normalized Discounted Cumulative Gain
topic S1 S2
1 0.5687 0.6521
… … …
50 0.7124 0.7512
avg 0.6421 0.7215
improvement +12.3 %
significativity p < 0.05 (paired t-test)
search engine x
input
topic
corpus
name of a
researcher
researcher
« With whom would you like to chat for
improving your research? »
recommender system
topical
topical +
social
#subjects
Top 25

49
Experiment
 Features
 Data dblp.xml (713 MB = 1.3M publications for 811,787 researchers)
 Subjects 90 researchers-contacts contacted by mail
74 researchers began to fill the questionnaire. 71 completed it
 Interface for assessing recommendations




50
Experiments: Profile of the Participants
 Experience of the 71 subjects Mdn = 13 years
74
 Productivity of the 71 subjects Mdn = 15 publications
NumberofparticipantsNumberofparticipants
Seniority
Number of publications

51
Empirical Validation of our Hypothesis
 Strong baseline ⇒ effective approach based on VSM
 +8.49 % = significant improvement (p < 0.05 ; n = 70)
of topical recommendations by social clues
0,5
0,6
0,7
0,8
0,9
1
global < 15 publis >= 15 publis < 13 ans >= 13 ans
Thématique Thématique + Social
productivity experience
+8,49 %+8,49 % +10,39 %+10,39 % +7,03 %+7,03 % +6,50 %+6,50 % +10,22 %+10,22 %
NDCG
Topical Topical + social
yearsyears

52
Question SCIM-2
What is the landscape of research in
Information Systems from the
perspective of gatekeepers?
IRIRDLDL
SCIMSCIM
Guillaume Cabanac. “Shaping the landscape of research in Information Systems from the perspective of editorial
boards : A scientometric study of 77 leading journals.” Journal of the American Society for Information Science
and Technology, 63, to appear in 2012, Wiley. DOI:10.1002/asi.22609

53
Landscape of Research in Information Systems
 The gatekeepers of science

54
 The 77 core peer-reviewed IS journals in the WoS

55
 Exploratory data analysis

56
 Exploratory data analysis

57
 Topical map of the IS field

58
 Most influential
gatekeepers

59
 Number of gatekeepers per country

60
 Geographic and gender diversity

61
Question SCIM-3
What if submission date influenced the
acceptance of conference papers?
IRIRDLDL
SCIMSCIM
Guillaume Cabanac. “What if submission date influenced the acceptance of conference papers?” Submitted to
the Journal of the American Society for Information Science and Technology, Wiley.

62
Conferences Affected by a Submission-Date bias?
 Peer-review

63
The Submission-Date bias
 Dataset from the ConfMaster conference management system

64
 Influence of submission date on bids

65
 Influence of submission date on average marks

Conclusion
66
• Geographic IR

Thank you
http://www.irit.fr/~Guillaume.Cabanachttp://www.irit.fr/~Guillaume.Cabanac
Twitter: @tafanorTwitter: @tafanor

Musings at the Crossroads of Digital Libraries, Information Retrieval, and Scientometrics

Recommended

Recommended

More Related Content

Similar to Musings at the Crossroads of Digital Libraries, Information Retrieval, and Scientometrics

Similar to Musings at the Crossroads of Digital Libraries, Information Retrieval, and Scientometrics (20)

More from Guillaume Cabanac

More from Guillaume Cabanac (20)

Recently uploaded

Recently uploaded (20)

Musings at the Crossroads of Digital Libraries, Information Retrieval, and Scientometrics

Editor's Notes