1HUMANE INFORMATION SEEKING:GOING BEYOND THE IRWAY       JIN YOUNG KIM @ SNU DCC
2Jin Young Kim• Graduate of SNU EE / Business• 5th Year Ph.D Student in UMass Computer Science• Starting as a Applied Rese...
3Today’s Agenda• A brief introduction of IR as a research area• An example of how we design a retrieval model• Other resea...
4BACKGROUNDAn Information Retrieval Primer
5Information Retrieval?• The study of how an automated system can enable its users to access, interact with, and make sens...
6IR Research in Context• Situated between human interface and system / analytics research • Aims at satisfying user’s info...
7Major Problems in IR• Matching  • (Keyword) Search : query – document  • Personalized Search : (user+query) – document  •...
8HUMANE INFORMATIONRETRIEVALGoing Beyond the IR Way
Information seeking requires a                                        9communication.  You need the freedom of expression....
Information Seeking circa 2012                                     10  Search engine accepts keywords only.  Search engine...
11Toward Humane Information Seeking   Rich User        Rich User   Modeling         Interactions   Profile          Search...
12  The HCIR Way:from User Modeling      IR Way:  Rich Query to Session                         Interaction               ...
The Rest of Talk…                                                               13       Personal Search       Improving s...
14PERSONAL SEARCHRetrieval And Evaluation Techniquesfor Personal Information [Thesis]
15Example: Desktop Search Media         Search over Social                       Evaluating Search in                     ...
16Structured Document Retrieval: Background • Field Operator / Advanced Search Interface • User’s search terms are found i...
17Structured Document Retrieval: Models• Document-based Retrieval Model                           f1  • Score each documen...
18Improved Matching for Email Search                      Structured Documents                                            ...
19Estimating the Field Relevance• If User Provides Feedback   • Relevant document provides sufficient information• If No F...
20  Retrieval Using the Field Relevance  • Comparison with Previous Work      q1 q2 ... qm                     q1 q2 ... q...
21Evaluating the Field Relevance Model• Retrieval Effectiveness          (Metric: Mean Reciprocal Rank)                DQL...
22Evaluation Challenges for PersonalSearch                     [CIKM09,SIGIR10,CIKM11]• Evaluation of Personal Search   • ...
23DocTrack Game   Target Item                 Find It!
24Summary so far…• Query Modeling for Structured Documents  • Using the estimated field relevance improves the retrieval  ...
25WEB SEARCHCharacterizing Web Content, User Interests, andSearch Behavior by Reading Level and Topic                     ...
Reading level distribution varies acrossmajor topical categories
User Modeling by Reading Level andTopic• Reading Level and Topic   • Reading Level: proficiency (comprehensibility)   • To...
Profile matching can predict user’spreference over search results• Metric  • % of user’s preferences predicted by profile ...
Comparing Expert vs. Non-expert URLs• Expert vs. Non-expert URLs taken from [White’09]       Lower Topic Diversity        ...
30Enabling Browsing for Web Search                                   [Work-in-progress]• SurfCanyon®  • Recommend results ...
31BOOK SEARCHUnderstanding Book Search Behavior on the Web                             [Submitted to SIGIR12]
32Understanding Book Search on theWeb• OpenLibrary  • User-contributed online digital library  • DataSet: 8M records from ...
33Comparison of Navigational Behavior• Users entering directly show different behaviors from users entering via web search...
34Comparison of Search Behavior      Rich interaction reduces the query lengths  Filtering induces more interactions than ...
35LOOKING ONWARD
36Where’s the Future? – Social Search• The New Bing Sidebar makes search a social activity.
37Where’s the Future? – Semantic Search• The New Google serves ‘knowledge’ as well as docs.
38Where’s the Future? – Siri-like Agent• The New Google serves ‘knowledge’ as well as docs.
39Exciting Future is Awaiting US!• Recommended Readings in IR:  • http://www.cs.rmit.edu.au/swirl12                       ...
40 Selected Publications                                More at @lifidea, or                                              ...
41My Self-tracking Efforts• Life-optimization Project (2002~2006)• LiFiDeA Project (2011-2012)
42OPTIONAL SLIDES
43The Great Divide: IR vs. HCI            IR                         HCI• Query / Document          • User / System• Relev...
44The Great Divide: IR vs. RecSys           IR                      RecSys• Query / Document         • User / Item• Reacti...
45The Great Divide: IR in CS vs. LIS       IR in CS                   IR in LIS• Focus on ranking &      • Focus on behavi...
46Problems & Techniques in IR• WhatData           Format (documents, records and linked data) /               Size / Dynam...
47More about the Matching Problem• Finding Representations   • Term vector vs. Term distribution   • Topical category, Rea...
Upcoming SlideShare
Loading in …5
×

랭킹 최적화를 넘어 인간적인 검색으로 - 서울대 융합기술원 발표

5,872 views

Published on

Published in: Technology, Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
5,872
On SlideShare
0
From Embeds
0
Number of Embeds
4,530
Actions
Shares
0
Downloads
14
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • Hello, everyone. I'm Jin Kim from UMass Amherst. A fifth year Ph.D student graduating this summer. The title of this talk is 'Humane Info-seeking …' namedafter JefRaskin's book. As it says on the title, although I'm from IR research background, I believe that there are opportunities to significantly improve info-seeking by going beyond traditional IR way. Feel free to interrupt me for questions.
  • More about me: I’m 5th year ph.d student. I came with the vision of becoming a superhuman by enhancing personal information access.I did internship at MSR twice, where I had fun with web search personalization.
  • More about me: I’m 5th year ph.d student. I came with the vision of becoming a superhuman by enhancing personal information access.I did internship at MSR twice, where I had fun with web search personalization.
  • Now I'll talk about personal search, which has been the domain of my thesis.
  • Now I'll talk about personal search, which has been the domain of my thesis.
  • I'll start my talk by thoughts of what makes an info-seeking humane. In a sense, we can think of it as a communication activity, as you can see from this picture of a librarian and a visitor. 1>Two important criteria for humane info-seeking. It's important that you can express what you need with no restriction, and have someone who understands. 
  • However, where are we now? If you think of info-seeking today, in a word, it's quite frustrating, far from being humane. 1> Typical search engine doesn't accept anything but keyword, 2> and you don't feel that it understands you and treat you as such.
  • How can we achievehumane info seeking? Today, I'll talk about two ways of reaching it. 1> First, to allow the freedom of expression, we need rich interaction methods beyond typing keywords, such as browsing and filtering. 2> Second, to implement the system that really 'understand' the user and personalize the interaction, we need a rich user model of profile, context and behavior. 
  • To put my key points in the context of IR research, I'm arguing we need to go beyond IR way, which focuses on query-response modeling, to what I can call HCIR way.?> How many of you have heard of term ‘HCIR”?2> To present several key departures, it tries to go from query to session.Secondly, looking at the whole sequence of interaction, as opposed to current query, naturally enables rich user modeling. Finally, it provides rich interactions including…3> Finally to discuss the relationship between user modeling / rich interaction
  • For the rest of this talk, I'll talk about how my work addresses the challenges above, based on three info-seeking scenarios, each with unique characteristics. I first talk about searching personal information (for half of time), then searching book information on the web. Then finally on generic web search. Note that I used the term ‘search’ to refer to generic information seeking, as opposed to a particular form of info-sccess.
  • Now I'll talk about personal search, which has been the domain of my thesis.
  • An traditional example of PS is desktop search. User types in keyword …. My previous work at SIGIRpresented a retrieval framework where results from each collection is merged into single final results.2> Recently, personal information is getting even more diverse, including contents from social media as well. Recently we found that the retrieval framework is equally valid for this scenario.
  • But how generalizable is this case? Here we provide the evidence that … For one thing, here’s an interface from popular DL, where the existence of … exemplify this. Also, we know from recent studies that user's search terms span across multiple terms.
  • B> Assume queries with m words, documents with n fields.F> Why will this work?Terms from shorter fields are better representedEach field can be weighted differently
  • Matching is an important problem for IR. Now I'll talk about how we can improve the matching for SDR2> I firstintroduce this notion of field relevance, which means … 3> Field operator or advanced UI is an option, yet… Let’s assume users would type in keyword queries4> If you recall earlier example of email search, you can see that each query-term is found in different part of the document.5> For the first query-term, …
  • Estimation is the next big question. 1> Field relevance can be estimated easily if we observe any relevant document based on user’s feedback. For this example of email, you can …2> Otherwise, we try to approximate
  • When we apply this notion of field relevance for retrieval, it's natural to use it as field weights. There has been previous …, which sums up field-level score for each query-term, and then …, yet they’re limited …So, difference from the previous work is that, now, field weights vary for different query-term, which is more realistic model of user’s 2> The resulting retrieval model, FRM, now has per-term, per-field weights, by which we can combine field-level scores.
  • I’ll briefly talk about evaluation.1> Our evaluation shows that this per-term field weighting improves performance over fixed weighting baselines in three standard collections 2> Also, if you compare the performance of .., you can see that performance improves as we use better source of estimation. Here, FRM-C uses … / FRM-R assumes that user’s relevance feedback is available, in which case retrieval performance …
  • Now, I'll switch gears and talk about contributions in evaluating personal search. Previous work mostly employed diary study based on a closed group of users, which made data sharing,…1> In our approach, we replaced actual user data with simulated data. For instance, we used crawl of …2> Two approaches in getting interaction data. In DocTrack, we used simulated collection and task. In PUM, we used simulated user interaction.
  • I’ll describe how we collected logs in this simulated known-item finding scenario. 1> In DocTrack game, the system randomly chooses two candidate documents, and present it to the user after being given some time to skim through each document2> The user is asked to find one of these documents. Now, since user may have fuzzy memory, we simulate known-item finding. We collected queries and interaction logs this way.3> DEMO
  • In summary, we talked about …, where we showed that …Connection to the user modeling is that, if user picked the example documents, user will see more document whose field-level term distribution would be similar, Yet it’ll not be the case for other users.Now I switch gears, and talk about an alternative to keyword search, as well as the evaluation.
  • Firstly, I’ll show you how reading level an topic distribution look like on the web. The heat-map shows the reading level distribution across different topic area, where we can see that topics like …. It’s clear that reading level varies a lot for different topic, which shows why we need a joint modeling.
  • Firstly, I my last intern project at MSR was to build a rich user model based on the joint space of reading level (comprehension level) and topical category.  Here, reading level means …1> We had a prob. Classifier for RLT, and aggregated documents seen by user to build user’s profile.2> We also explored several applications,
  • As an attempt to evaluate the value of  profile matching for  search personalization, we measured the degree to which the profile matching predict user’s content preference, where we used the % of user preference accurately predicted in each clicked document over the skipped document above in the SERP page. When we broke down into the focus in user group, measured in the entropy of RLT joint distribution, and the distance metric between
  • Finally, we thought that combining reading level and topic can help characterize the “expertness” of an entity. We found that we can separate expert-oriented URLs from novice-oriented URLs in the 4 domains studied in a previous work on domain expertise. The graph shows the expected RL (X axis), and topic diversity (Y axis) for each URL group, where we found that…. We went onto classify expert vs. non-expert contents, and found …, which can be useful for expert site recommendation to expert users.
  • Let’s switch gears and talk about rich interaction on web, based on my recent collaboration with the startup company SurfCanyon. 2>As you can see on this,…For instance, given an ambiguous query 'dolphins', it recommends results based on your click, allowing…3> Each user can get their own results based on which document they expend or skip, drilling down into a branch of tree which representsall possible rankings. We’re studying various questions including when it’ll help, and … Our initial results indicate that …
  • Now I’ll talk about search books on the web, which is based on my recent SIGIR submission.
  • For our study, we analyzed user's search behavior on this digital library called OL, which is based on …The following study is based on 8M records extracted from web server logs, 1> It is is interesting because of richinteraction methods in provides.And the fact that most users use web search engine to get to the website, which we’ll analyze in following
  • We first focused on characterizing the difference between … where we found that …For our analysis, we built a probabilistic model of user's behavior similarly to the PUM I presented earlier.2> Let’s look at user’s navigation pattern when they entered from Homepage. Here, numbers represent …3> How about when they entered via web search engine?
  • We also analyzed user’s search behavior. 1> When we compared query length between external and internal queries, meaning …Since we also found that query length is negatively correlated with …2> Also, when we built the model of user’s search behavior similarly to previous analysis, we found that filtering is used almost …
  • For most of my Ph.D periods, I've worked on IR research, especially in the area of structured document retrieval (SDR), personal information retrieval (PIR) and web search. So, what's the story behind? Why did I enter seemingly unrelated business of ranking documents? You'll find answers for the rest of my talk.
  • 랭킹 최적화를 넘어 인간적인 검색으로 - 서울대 융합기술원 발표

    1. 1. 1HUMANE INFORMATION SEEKING:GOING BEYOND THE IRWAY JIN YOUNG KIM @ SNU DCC
    2. 2. 2Jin Young Kim• Graduate of SNU EE / Business• 5th Year Ph.D Student in UMass Computer Science• Starting as a Applied Researcher at Microsoft Bing
    3. 3. 3Today’s Agenda• A brief introduction of IR as a research area• An example of how we design a retrieval model• Other research projects and recent trends in IR
    4. 4. 4BACKGROUNDAn Information Retrieval Primer
    5. 5. 5Information Retrieval?• The study of how an automated system can enable its users to access, interact with, and make sense of information. Query Surface Issue Document User Visit
    6. 6. 6IR Research in Context• Situated between human interface and system / analytics research • Aims at satisfying user’s information needs • Based on large-scale system infrastructure & analytics• Need for convergence research! End-user Interface (UX / HCI / InfoViz) Information Retrieval Large-scale Large-scale (Text)Analytic System Infra. s
    7. 7. 7Major Problems in IR• Matching • (Keyword) Search : query – document • Personalized Search : (user+query) – document • Contextual Advertising : (user+context) – advertisement• Quality Query • Authority/ Spam / Freshness Surface Issue • Various ways to capture them Document User Visit• Relevance Scoring • Combination of matching and quality features • Evaluation is critical for optimal performance
    8. 8. 8HUMANE INFORMATIONRETRIEVALGoing Beyond the IR Way
    9. 9. Information seeking requires a 9communication. You need the freedom of expression. You need someone who understands.
    10. 10. Information Seeking circa 2012 10 Search engine accepts keywords only. Search engine doesn’t understand you.
    11. 11. 11Toward Humane Information Seeking Rich User Rich User Modeling Interactions Profile Search Context Browsing Behavior Filtering
    12. 12. 12 The HCIR Way:from User Modeling IR Way: Rich Query to Session Interaction USER SYSTEM Action Response UserInteraction Model Action Response History Profile Context Action Response Behavior Filtering / Browsing Filtering Conditions Relevance Feedback Related Items … … HCIR = HCI + IR
    13. 13. The Rest of Talk… 13 Personal Search Improving search and browsing for known-item finding Evaluating interactions combining search and browsing Web Search User modeling based on reading level and topic Providing non-intrusive recommendations for browsing Book Search Analyzing interactions combining search and filtering
    14. 14. 14PERSONAL SEARCHRetrieval And Evaluation Techniquesfor Personal Information [Thesis]
    15. 15. 15Example: Desktop Search Media Search over Social Evaluating Search in Ranking using Multiple Personal Social Media Document Types for Collections [WSDM12] Desktop Search [SIGIR10]
    16. 16. 16Structured Document Retrieval: Background • Field Operator / Advanced Search Interface • User’s search terms are found in multiple fields Understanding Re-finding Behavior in Naturalistic Email Interaction Logs. Elsweiler, D, Harvey, M, Hacker., M [SIGIR11]
    17. 17. 17Structured Document Retrieval: Models• Document-based Retrieval Model f1 • Score each document as a whole f2 ...• Field-based Retrieval Model fn • Combine evidences from each field q1 q2 ... qm q1 q2 ... qm f1 f1 w1 w1 f2 f2 w2 w2 ... ... fn fn wn wn Document-based Scoring Field-based Scoring
    18. 18. 18Improved Matching for Email Search Structured Documents [CIKM09, ECIR09,12] • Field Relevance • Different field is important for different query-term 2 1 ‘registration’ is relevant when it occurs in <subject> 2 1 2 1 ‘james’ is relevant when it occurs in <to>
    19. 19. 19Estimating the Field Relevance• If User Provides Feedback • Relevant document provides sufficient information• If No Feedback is Available • Combine field-level term statistics from multiple sources from/to from/to from/to title content + title content ≅ title content Collection Top-k Docs Relevant Docs
    20. 20. 20 Retrieval Using the Field Relevance • Comparison with Previous Work q1 q2 ... qm q1 q2 ... qm f1 f1 f1 f1 w1 w1 P(F1|q1) P(F1|qm)sum f2 f2 f2 f2 w2 w2 P(F2|q1) P(F2|qm) ... ... ... ... fn fn fn fn wn wn P(Fn|q1) P(Fn|qm) multiply • Ranking in the Field Relevance Model Per-term Field Score Per-term Field Weight
    21. 21. 21Evaluating the Field Relevance Model• Retrieval Effectiveness (Metric: Mean Reciprocal Rank) DQL BM25F MFLM FRM-C FRM-T FRM-R TREC 54.2% 59.7% 60.1% 62.4% 66.8% 79.4% IMDB 40.8% 52.4% 61.2% 63.7% 65.7% 70.4% Monster 42.9% 27.9% 46.0% 54.2% 55.8% 71.6% Fixed Field Weights Per-term Field Weights 80.0% 75.0% 70.0% 65.0% TREC 60.0% IMDB 55.0% Monster 50.0% 45.0% 40.0% DQL BM25F MFLM FRM-C FRM-T FRM-R
    22. 22. 22Evaluation Challenges for PersonalSearch [CIKM09,SIGIR10,CIKM11]• Evaluation of Personal Search • Each based on its own user study • No comparative evaluation was performed yet• Solution: Simulated Collections • Crawl CS department webpages, docs and calendars • Recruit department people for user study• Collecting User Logs • DocTrack: a human-computation search game • Probabilistic User Model: a method for user simulation
    23. 23. 23DocTrack Game Target Item Find It!
    24. 24. 24Summary so far…• Query Modeling for Structured Documents • Using the estimated field relevance improves the retrieval • User’s feedback can help personalize the field relevance• Evaluation Challenges in Personal Search • Simulation of the search task using game-like structures • Related work : ‘Find It If You Can’ [SIGIR11]
    25. 25. 25WEB SEARCHCharacterizing Web Content, User Interests, andSearch Behavior by Reading Level and Topic [WSDM12]
    26. 26. Reading level distribution varies acrossmajor topical categories
    27. 27. User Modeling by Reading Level andTopic• Reading Level and Topic • Reading Level: proficiency (comprehensibility) • Topic: topical areas of interests• Profile Construction P(R|d1)) P(R|d1 ) P(T|d1)) P(T|d1 ) P(R,T|u) P(R|d1 P(T|d1• Profile Applications • Improving personalized search ranking • Enabling expert content recommendation
    28. 28. Profile matching can predict user’spreference over search results• Metric • % of user’s preferences predicted by profile matching • Profile matching measured in KL-Divergence of RT profiles• Results • By the degree of focus in user profile • By the distance metric between user and website User Group #Clicks KLR(u,s) KLT(u,s) KLRT(u,s) ↑Focused 5,960 59.23% 60.79% 65.27% 147,195 52.25% 54.20% 54.41% ↓Diverse 197,733 52.75% 53.36% 53.63%
    29. 29. Comparing Expert vs. Non-expert URLs• Expert vs. Non-expert URLs taken from [White’09] Lower Topic Diversity Higher Reading Level
    30. 30. 30Enabling Browsing for Web Search [Work-in-progress]• SurfCanyon® • Recommend results based on clicks Initial results indicate that recommendations are useful for shopping domain.
    31. 31. 31BOOK SEARCHUnderstanding Book Search Behavior on the Web [Submitted to SIGIR12]
    32. 32. 32Understanding Book Search on theWeb• OpenLibrary • User-contributed online digital library • DataSet: 8M records from web server log
    33. 33. 33Comparison of Navigational Behavior• Users entering directly show different behaviors from users entering via web search engines Users entering the site directly Users entering via Google
    34. 34. 34Comparison of Search Behavior Rich interaction reduces the query lengths Filtering induces more interactions than search
    35. 35. 35LOOKING ONWARD
    36. 36. 36Where’s the Future? – Social Search• The New Bing Sidebar makes search a social activity.
    37. 37. 37Where’s the Future? – Semantic Search• The New Google serves ‘knowledge’ as well as docs.
    38. 38. 38Where’s the Future? – Siri-like Agent• The New Google serves ‘knowledge’ as well as docs.
    39. 39. 39Exciting Future is Awaiting US!• Recommended Readings in IR: • http://www.cs.rmit.edu.au/swirl12 Any Questions ?
    40. 40. 40 Selected Publications More at @lifidea, or cs.umass.edu/~jykim• Structured Document Retrieval • A Probabilistic Retrieval Model for Semi-structured Data [ECIR09] • A Field Relevance Model for Structured Document Retrieval [ECIR11]• Personal Search • Retrieval Experiments using Pseudo-Desktop Collections [CIKM09] • Ranking using Multiple Document Types in Desktop Search [SIGIR10] • Building a Semantic Representation for Personal Information [CIKM10] • Evaluating an Associative Browsing Model for Personal Info. [CIKM11] • Evaluating Search in Personal Social Media Collections [WSDM12]• Web / Book Search • Characterizing Web Content, User Interests, and Search Behavior by Reading Level and Topic [WSDM12] • Understanding Book Search Behavior on the Web [In submission to SIGIR12]
    41. 41. 41My Self-tracking Efforts• Life-optimization Project (2002~2006)• LiFiDeA Project (2011-2012)
    42. 42. 42OPTIONAL SLIDES
    43. 43. 43The Great Divide: IR vs. HCI IR HCI• Query / Document • User / System• Relevant Results • User Value / Satisfaction• Ranking / Suggestions • Interface / Visualization• Feature Engineering • Human-centered Design• Batch Evaluation (TREC) • User Study• SIGIR / CIKM / WSDM • CHI / UIST / CSCW Can we learn from each other?
    44. 44. 44The Great Divide: IR vs. RecSys IR RecSys• Query / Document • User / Item• Reactive (given query) • Proactive (push item)• SIGIR / CIKM / WSDM • RecSys / KDD / UMAP
    45. 45. 45The Great Divide: IR in CS vs. LIS IR in CS IR in LIS• Focus on ranking & • Focus on behavioral relevance optimization study & understanding• Batch & quantitative • User study & qualitative evaluation evaluation• SIGIR / CIKM / WSDM • ASIS&T / JCDL• UMass / CMU / • UNC / Rutgers / UW Glasgow
    46. 46. 46Problems & Techniques in IR• WhatData Format (documents, records and linked data) / Size / Dynamics (static, dynamic, streaming)User & End User (web and library)Domain Business User (legal, medical and patent) System Component (e.g., IBM Watson)Needs Known-item vs. Exploratory Search Recommendation• HowSystem Indexing and Retrieval (Platforms for Big Data Handling)Analytics Feature Extraction Retrieval Model Tuning & EvaluationPresentation User Interface Information Visualization
    47. 47. 47More about the Matching Problem• Finding Representations • Term vector vs. Term distribution • Topical category, Reading level, … Query• Estimating Representations Surface Issue • By counting terms • Using automatic classifiers Document User Visit• Calculating Matching Scores • Cosine similarity vs. KL-divergence • Combining multiple reps.

    ×