Recommended
Recommended
More Related Content
What's hot
What's hot (12)
Similar to Scholars’ Perceptions of Relevance in Bibliography-Based People Recommender System
Similar to Scholars’ Perceptions of Relevance in Bibliography-Based People Recommender System (20)
Recently uploaded
Recently uploaded (20)
Scholars’ Perceptions of Relevance in Bibliography-Based People Recommender System
- 1. Scholars’ Perceptions of Relevance in Bibliography-based People Recommender System Ekaterina Olshannikova, Thomas Olsson, Jukka Huhtamäki, Peng Yao Tampere University, Tampere, Finland ekaterina.olshannikova@tuni.fi thomas.olsson@tuni.fi jukka.huhtamaki@tuni.fi peng.yao@tuni.fi ECSCW 2019, Salzburg, Austria, 8 – 12 June 2019
- 2. Contents 1. Motivation and Research Gap 2. Experiment Design 3. Findings 4. Discussion and Conclusions 5. FutureWork
- 3. 1.Motivation and Research Gap
- 4. Academic Partnering ACCESS TO EXPERTISE ACCESS TO FUNDING IDEA CREATION KNOWLEDGE EXHANGE 1. Motivation and Research Gap 4
- 5. Social Matching Systems • People Recommender Systems • Opportunistic Matching Applications • Expert Finding System • Event-based Mobile Applications • Bibliography-based Systems for Researchers 1. Motivation and Research Gap 5
- 6. Behavioral Biases HOMOPHILY BIAS ECHO CHAMBERS 1. Motivation and Research Gap 6
- 7. Research Gap • Lack of research and design for diversifying people recommendations • Little focus on user perceptions in evaluation of people recommendations 1. Motivation and Research Gap 7
- 8. Goal and Objectives Goal: Contribute to the development of matching methods that not only identify maximal similarity but help to explore the difference dimension Objectives: • Design bibliography-based people recommender system for researchers • Evaluate perceived relevance of recommendations • Observe the homophily bias amongst scholars 1. Motivation and Research Gap 8
- 9. Research Questions and Hypothesis RQ1: What level of measured similarity of publication history is preferred in recommendations of potential collaborators? RQ2: What specific needs and expectations scholars have in regard to seeking professional collaboration? H: The more similar recommendation is (i.e., the similarity of publishing history), the more relevant and similar the recommendation would be perceived. 1. Motivation and Research Gap 9
- 11. 1. Data source and Input Dataset (DBLP) (Author, Title, Year, Conference) Record 1 Record 2 Record n … INPUT Participant Name Participant 1 Participant 20 Conference Name … Conference 1 Conference n … … … Conference 1 Conference n Low distance recom. (R1, R2, R3) Medium distance recom. (R4, R5, R6) High distance recom. (R7, R8, R9) 8.Output Recommendations 7. Co-Authors Network Filter out co-authors 2. Extract target conference(s) records for Participants Conference 1 (Author (s), Title) Record 1 Record 2 Record m … Conference n (Author (s), Title) … Record 1 Record 2 Record m … 3. Clean and aggregate each authour’s records … Conference 1 (Author, Content) Record 1 Record m … Conference n (Author, Content) Record 1 Record m … 4. Run TF-IDF TF-IDF Feature Vector Record 1 Record k … 5. Sorting Cosine Distances for Participants … Distances for Participant 1 Distance 1 Distance i … Distances for Participant n Distance 1 Distance i … 6. OTSU Filter to identify recommendations of low, medium and high distances 0.0 0 2000 4000 6000 8000 10000 0.2 0.4 0.6 0.8 1.0 End of low distance Mid Point Start of high distance System Design 2. Experiment Design 11
- 12. User Interface Research topics Recommendation 1 Full name Social Recommendations for The Participant 1 Co-author 1 Co-author 2 Co-author 3 Co-author 4 Co-author 5 Co-author 6 Topic 1 Topic 6 Topic 2 Topic 7 Topic 3 Topic 8 Topic 4 Topic 5 Co-authors 8 Recent publications Authors. Publication 1 title. Year, Venue Authors. Publication 2 title. Year, Venue Authors. Publication 3 title. Year, Venue 8 2. Experiment Design 12
- 13. Participants 7 SENIOR RESEARCHERS 6 POSTDOCTORAL RESEARCHERS 4 FULL PROFESSOR 1 ASSOCIATE PROFESSOR 13 MALE AND 5 FEMALE AGES FROM 32 TO 66, MDN 42 EXPERIENCE FROM 10 TO 46, MDN 19 2. Experiment Design 13
- 14. Participants’ Expertise • HCI (10) • gaze technologies and interactions (7) • wireless technologies (6) • interaction design and techniques (6) • interfaces and information systems (5) • usability/user experience and user-centered design (5) • telecommunications and networking (5) • virtual reality (3) • wellness/health technologies (3) 2. Experiment Design 14
- 15. Evaluation Criteria • Perceived relevance of the recommendation • Perceived similarity • Levels of perceived familiarity • Expected willingness to interact with a recommended person 2. Experiment Design 15
- 16. 3. Findings
- 18. Perceived Relevance and Similarity 3. Findings 18
- 20. Willingness to Interact 3. Findings 20
- 21. Needs and Important Factors in Research Collaboration FUNDING (12 answers) NEEDS FACTORS KNOWLEDGE SHARING (12 answers) CONFLICT OF INTEREST (5 answers) RESEARCH MOBILITY (5 answers) AFFILIATION (14 answers) QUALITY OF RESEARCH (8 answers) SENIORITY (7 answers) PERSONAL CHEMISTRY (7 answers) 3. Findings 21
- 22. 4. Discussion and Conclusions
- 23. Criteria of Relevance • Similarity of background, attitudes, values, beliefs, goals and intentions (e.g., research aims) • Complementarity of professional roles, skills, knowledge, and social capital • Compatibility for direct cooperation in terms of being mentally, socially, morally or emotionally close to each other • Approachability/logistics – the availability of a person for direct or indirect interaction in terms of physical proximity as well as social and organizational distance 4. Discussion and Conclusions 23
- 24. Limitations • Limited content – only publications titles are available for analysis • DBLP data set provides limited understanding to the social ties between researchers • The evaluation was largely based on participants’ first impression • Participants’ were lacking a timely need for collaboration 4. Discussion and Conclusions 24
- 25. Conclusions • We provided empirical findings on subjective perceptions of people recommendation relevance • Presented academics' needs in collaboration and factors that might affect decision-making in choosing partners • The homophily bias is evident also in scholars' intuitive assessments of relevance and willingness to interact • There is a mismatch between people's intuitive choices and the deliberate intentions in decision-making on potential collaborators • We operationalized measures for subjective opinions on people recommendations 4. Discussion and Conclusions 25
- 26. 5. Future Work Exploring Structural Positions onTwitter Networks for Followee Recommendations: • Dormant tie – someone that the target user is already but has never mentioned; • Mentions-of-Mentions -- a friend-of-a-friend type of a connection in mention network; • A user identified to belong to the same community cluster Future work 26
Editor's Notes
- In scholarly work, collaboration has become a normative form of knowledge production. In academic research, collaboration takes place on a dyadic level between individuals, amongst research teams, as well as within international consortia. However, identifying new suitable candidates for academic collaboration requires high investment in social networking, and the disciplinary structures can prevent unexpected combinations of individuals.
- Following this trend of the increasing importance of collaboration, supporting social networking and encouraging new social encounters have become central design goals in the HCI & CSCW communities.
- However, the majority of professional matching systems tend to utilize a similarity-maximizing approach, providing recommendations of like-minded others with similar interests. Homogeneity produces both positive and negative effects on interpersonal communication, community formation, and knowledge work – homophily not only unifies, it also divides a network. Collaboration within a group of people with shared interests can contribute to a safe and trustworthy work environment. It has been found that researching and cooperating with diverse individuals is essential in tasks that aim to create new knowledge.
- While research on diversifying item recommendations is gaining interest, few attempts have been made to match people based on diversity. The evaluation of people recommender systems and matching applications is geared toward the assessment of algorithm effectiveness, with little focus on user perceptions. Although there is well-established research on user-centered evaluation of content recommender systems, the choice of potential collaborators is significantly different and, therefore, requires contextually operationalized evaluation metrics.
- To enable the gathering of such data about recommendations’ perceptions, we developed a simple DBLP-based people recommender system that provides the user with recommendations of other scholars from three different levels of similarity regarding their publication history—low, moderate and high.
- With a user study that combines a controlled experiment and semi-structured interviewing, we address the following research questions: (RQ1) What level of measured similarity of publication history is preferred in recommendations of potential collaborators? (RQ2) What specific needs and expectations scholars have in regard to seeking professional collaboration?
- The people recommender system runs on the subset of the parsed almost 6 mln records and depends on the input of publication venues of a given user (participant of the study). For the topic modeling we apply TF-IDF. For identifying similarity between authors -- we compute the cosine distances. Accordingly, the closest, or the most similar author to a participant will have the smallest cosine distance. Otsu filter is applied to trace three groups groups of similarity – low (very similar), medium (somewhat similar), high (very low similarity). Three recommendations that have no co-authorship with the given participant and published in the same venues are picked from each distance group as the final output, thus delivering nine recommendations in total.
- A single-page web application was deployed in Firebase development platform. Personalized recommendations visualized in the form of a carousel-based list. The UI visualizes all information about authors, which DBLP data set allows to extract. The publication list represents only works from conferences where both the recommended person and the participant of the experiment have published in. Accordingly, the list of co-authors is taken from those publications only. The topics were generated through bigram analysis on the corpus profiles of each recommended person.
- We recruited 18 English-speaking senior researchers who work at two university campuses in Tampere, Finland. Following the assumption that senior researchers often have more needs for finding collaborators, we limited our scope to postdoctoral researchers, professors, or otherwise senior academic positions. Overall, we had 13 male and five female participants, all based in either of the two universities in the same city. The ages vary from 32 to 66 (Median: 42, Mean: 45).
- Before starting the numerical evaluation, participants were given time to explore all the recommendations and get a general overview of the alternatives. The evaluation was constructed according to four variables. The variables were operationalized based on the authors' personal experiences and qualitative research insights on academic collaboration and user experience evaluation. The interview questions were designed to obtain participants' rationale behind the scoring of recommendations as well as to reveal needs and factors that affect decision-making in academic networking practices.
- Although the participants were unaware of the three different similarity distances, in their feedback they distinguish between different degrees of perceived relevance by using phrases like `very/most relevant,' `somewhat relevant/not exact match' and `irrelevant/totally irrelevant.’ Decision-making on perceived relevance was found to be influenced by the participants' estimation about how topics of recommended people match with their own -- whether they are similar, very different or complementary.
- Considering our first research question about which level of system-defined similarity is preferred in participants' evaluations, the findings demonstrate the highest ratings for most similar people. Methodologically, the subjective evaluations of different similarity levels seem to consolidate the system design. So subjective perception match well with objectively defined similarity levels. So even the relatively simple analytics procedure with scarce data seemed to work sufficiently, and the publication data represented the participants' topics accurately enough.
- Familiarity variable was evaluated from three perspectives. In the interview, participants addressed that already known people are less exciting recommendations (quotes 6 and 7). Even though we intentionally filtered out all the co-authors, the bibliographic data prevents understanding of the actual social relationships between researchers. Thus, in some cases (9 recommendations out of 54), the system recommended people from very close social circles.
- Evaluation of the willingness to interact with recommended people reflects six predefined scenarios of face-to-face interaction or follow-up collaboration at the context of a conference. In general, participants have diversified opinions regarding the estimation of any interactions and follow-up activities. The majority of participants consider such social activities to happen naturally at conferences and do not necessarily require high investments of time in collaboration. For some, it was hard to envision willingness to interact with unfamiliar people based on papers titles and topics of the research. Some emphasize that even in a real context of visiting the conference, it might be challenging to find relevant people in a crowd and contextualized use of such recommender systems might simplify the process and encourage social interaction with unfamiliar people.
- Regarding the second research question about academics' needs and expectations in professional collaboration, the results demonstrate that the optimal area on the similarity-difference continuum highly depends on the type and context of collaboration. For instance, crucial factors in direct cooperation, such as personal compatibility and similarity of attitudes and beliefs, are not as emphasized in short-term and indirect professional interactions (e.g., consultancy type of cooperation) as in long-term collaboration. Furthermore, the nature of the collaboration task might influence the perceived relevance of potential candidates, for example, regarding the complementarity of professional roles, skills, and knowledge.
- Following the participants' rationale about important factors in collaboration, we propose that the diversity of recommendations in professional social matching could be enhanced through several dimensions or criteria of relevance: Previous research addressed that similarity of such qualities can raise cohesion or so-called `affinity' in interpersonal relationships, and can even reduce adverse effects of individual dissimilarities in collaborative work; identifying beneficial intersections between individuals' qualities. Collaboration is a process of knowledge production, in which compatibility of such qualities can establish trustful, joyful and personally valued cooperation.
- Dorm as a recommendation mechanism, this could re-introduce users with a possibly ignored or forgotten tie; MoM in contrast to typical followership networks, the mention network is based on more explicit interactions; Com -- this could introduce new people in a programmatically identified network cluster, however not necessarily having explicit ties or interactions