Personalized Search-Building a prototype to infer the user's interest
•
0 likes•2 views
In the world of Search, understanding the intend of the user is often seen as the holy grail. When a user performs multiple search and click actions while having a conversation with the search engine, then this behavior reveals a piece of her/his interest. A search engine that is aware of the user’s interest is able to add a personal layer in its responses and this could add a new dimension of accuracy and value to a search implementation. But what technology does it take to build it? What data is needed? How well does it really work? This presentation describes the journey to find a practical implementation of a recommendation engine. It answers all the questions above and more. We’ll guide you through the lessons learned while creating an engine that generates potentially interesting items for the user based on collaborative filtering and anomaly detection. We’ll demonstrate a prototype where even a minimal set of user actions could lead to a personalized search experience.
Recommended
Recommended
More Related Content
Similar to Personalized Search-Building a prototype to infer the user's interest
Similar to Personalized Search-Building a prototype to infer the user's interest (20)
Recently uploaded
Recently uploaded (20)
Personalized Search-Building a prototype to infer the user's interest
- 1. Personalized Search Building a prototype to infer the user's interest Tom Burgmans Technology Product Owner Search Wolters Kluwer April 28th 2022
- 2. 2
- 3. Make search better through Personalization….a controversial topic If you know the users, you could improve their search There is too little traffic to build user profiles Our competitors claim to have it too… Our users don’t need it, they work on different cases every day Our customers expect our search to learn what they need Our search is already personal…via subscription filters This could destroy the user’s trust 3
- 4. 4 Personalized search is web based search results that are tailored specifically to an individual's interests by incorporating information about the individual beyond the specific query provided. Wikipedia says
- 5. What could make search “personal” Search bonus depreciation Activity • Past queries • Document interactions • Saved preferences • Filter actions Demographics • Nationality • Gender • Age • Profession Context • Location • Date/time • Device • Weather Social • Network connections • Loyalty • Shared information Factors to infer the user’s interest 5
- 6. What could make search “personal” (this PoC) Search bonus depreciation Activity • Past queries • Document interactions • Saved preferences • Filter actions Context Demographics Social • Nationality • Gender • Age • Profession • Location • Date/time • Device • Weather • Network connections • Loyalty • Shared information Factors to infer the user’s interest 6
- 7. Hypothesis The basis for Personalized Search is a Recommendation Engine which is based on an index of user activity that allows to infer the user’s interest via collaborative filtering. likes: Recommended likes: Similar interests: like s: like s: likes: likes: Everybody else: ???? 7
- 8. Collaborative filtering & anomaly detection • Find users with similar interests • Recommend items with an unusually high presence in this group compared to the rest likes: Similar interests: likes: likes: likes: likes: likes: 8
- 9. very similar users not so similar Collaborative filtering & selecting the foreground • Find only users with very similar interests • Balance between similarity and volume 9 foreground background
- 10. 10 What is the ideal time window to infer the user’s interest? year “session” 1 2 3 4 5 last N actions
- 11. 11 1 2 3 4 5 Example extremely temporary personalization
- 12. 12 How to apply the inferred user’s interest? personal recommendations query boost results personal recommendations query filter results personal recommendations query results results personal recommendations alerts as a boost: as a filter: personal recommendations query autocomplete as query suggestions: as a separate result set: as alerts: personal recommendations results as interesting items:
- 13. Goals of this PoC How to build it? What can we do on our existing technology stack? In what cases to apply it? Where is the need for personalization? What form is needed? What data do we need? Define ‘additional’ data sources; cleaning of noise How to measure success? Can we test/tune it offline? How to A/B test online? • Technology • Need • Data • Verification 13
- 14. 14 Finding a technology for anomaly detection JLH score in Elastic = (fg_percentage - bg_percentage) * (fg_percentage / bg_percentage) foreground background Significant Terms
- 15. 15 Finding a technology for anomaly detection Solr's SignificantTerms score = (log(freq_foreground)+1) * (log( (numDocs + 1)/(freq_background + 1) ) + 1) foreground background Significant Terms (SignificantTermsStream.java) Stream source
- 16. 16 Finding a technology for anomaly detection fq_count - fg_size * bg_prob Z-score = --------------------------------------- sqrt(fg_size * bg_prob * (1 - bg_prob)) foreground background Reletedness() JSON Facet function bg_prob = bg_count / bg_size fg_prob = fg_count / fg_size a.k.a. Semantic Knowledge Graph (RelatednessAgg.java)
- 17. Selecting similar users Wrapping mlt into mltplus very similar users not so similar 17 • MLT standard Solr query parser to find all similar documents • MLTPLUS home made query parser to only keep documents that are very similar.
- 18. 18 Design recommendation engine Source: Wolters Kluwer Navigator 2021 customer usage data. Recommendation engine 88.208 users Year’s activity per user • documents • publications • practice areas • document types • queries recommends given a user’s year • userid (encrypted) • Time stamp • behavior: • # of doc views • Viewed documents • Publications • doc types • practice areas • The strongly liked documents • Favorites • Prints • Downloads • email • Queries • The applied filters Recommendation engine 2.582.900 sessions Activity per session • documents • publications • practice areas • document types • queries recommends given a user’s (previous) session given a user’s last N actions DWH usage
- 19. 19 88.208 users Recommendation engine 2.582.900 sessions Personalization Service Demo app user’s (last) year ID user’s (last) session ID user’s current actions Recommended • documents • publications • practice areas • document types • queries Personalization prototype content of personal folder long-term personalization short-term personalization Very short-term personalization
- 20. 20 What is the need for personalization based on a year history ? 32% users > 20 actions / year 68% users < 20 actions / year From the 32% most active users: 66% have viewed (relatively) consistently documents of the same 2 practice areas. So for 21% of all users recommending PAs based on the activity of past year might make sense. How consistent is the user’s interest through the year?
- 21. 21 What is the need for personalization based on the last session ? For 42% of the users the previous session has a 0.9-1.0 consine overlap with the current one w.r.t. PA interest. This rises to 66% when the time between sessions get shorter. How much does the last session predict the interest of the next one?
- 22. 22 How to measure success? • Offline: • Manual tests by a domain expert • Build a model that predicts the likelihood that the previous session predicts the interest of the current one. • Replay user sessions and guestimate how much shorter they could have been in case personalization would have been applied. • Online: • Measure clicks on explicit recommendations • A/B test (in case recommendations applied as boosts/filters)
- 23. 23 Lessons learned The accuracy of the recommendations depends on the purity/completeness of the usage logs. Preferred: front-end usage logging Quality logs needed Improvement opportunities: - Use frond-end logs - Better data cleaning - Query normalization - Add more signals like filters & autocomplete clicks - Testing strategies We’re not done yet Offline testing of personalization is hard, but slightly easier for the short- term variant. Online testing takes time. Challenge: testing Short-term personalization is applicable to more users than long-term personalization (may differ per business case) What works best We can indeed infer the user’s interest via collaborative filtering of user actions, using Solr and using usage data from a large Wolters Kluwer product Yes we can