Metaphor: A system for related searches recommendations

•

0 likes•1,174 views

Search plays an important role in online social networks as it provides an essential mechanism for discovering members and content on the network. Related search recommendation is one of several mechanisms used for improving members’ search experience in finding relevant results to their queries. This paper describes the design, implementation, and deployment of Metaphor, the related search recommendation system on LinkedIn, a professional social networking site with over 175 million members worldwide. Metaphor builds on a number of signals and filters that capture several dimensions of relatedness across member search activity. The system, which has been in live operation for over a year, has gone through multiple iterations and evaluation cycles. This paper makes three contributions. First, we provide a discussion of a large-scale related search recommendation system. Second, we describe a mechanism for effectively combining several signals in building a unified dataset for related search recommendations. Third, we introduce a query length model for capturing bias in recommendation click behavior. We also discuss some of the practical concerns in deploying related search recommendations.

Science

Metaphor: A System for Related
Searches Recommendations
Mitul Tiwari
Joint work with Azarias Reda, Yubin Park, Christian Posse, and
Sam Shah
LinkedIn

Outline
• Related Searches at LinkedIn
• Metaphor: A System for Related Searches
Recommendations
• Design
• Implementation
• Evaluation

LinkedIn by the numbers
• 175M+ members
• 2+ new user registrations per second
• 4.2 Billion people searches in 2011
• 9.3 Billion page views in Q2 2012
• 100+ million monthly active users in Q2 2012

Related Searches at LinkedIn
• Millions of searches everyday
• Goal: Build related searches system at LinkedIn
• To help users to explore and refine their queries
Azarias Reda, Yubin Park, Mitul Tiwari, Christian Posse, and Sam Shah

Metaphor: A Related Searches
System
• Design
• Implementation
• Evaluation

Design
• Signals
• Collaborative Filtering
• Query-Result Click graph
• Overlapping terms
• Length-bias
• Ensemble approach for unified recommendation
• Practical considerations

Design: Collaborative Filtering
• Searches correlated by time
• Searches done in the same session by the same user
• Collaborative filtering: implicit feedback
• TFIDF scoring to take care of popular queries (e.g. `Obama’)
Q1 Q2 Q3
Time

Design: Query-Result Clicks
• Searches correlated by result clicks
Q1
Qn
R1
Rm

Design: Overlapping Terms
• Searches with overlapping terms
• TFIDF scoring to give importance to terms
Software Developer
Software Engineer
Q1
Q2

Design: Length Bias
• Insight: clicks on suggestions one term longer

Design: Length Bias
• Insight: clicks on suggestions one term longer
• Corresponds to refining the initial query
• Statistical biasing model to score a longer query higher

Design: Ensemble Approach
• Need to generate unified recommendation dataset
• Analysis to figure out engagement of each signal
• Attempted ML approach
• Minimal overlap across different signals

Design: Ensemble Approach
• Step-wise unionization
• Importance based on individual signal performance
• First, collaborative filtering
• Second, queries correlated by query-result clicks
• Third, queries overlapping terms

Design: Practical Considerations
• System designed for public consumption
• Strong profanity filters
• Need to deal with misspellings
• Languages
• Remove spammy search queries

Implementation Challenge
• Scale
• 175M+ members
• Billions of searches
• Terabytes of data to process

Implementation
• Kafka: publish-subscribe messaging system
• Hadoop: MapReduce data processing system
• Azkaban: Hadoop workflow management tool
• Voldemort: Key-value store

Evaluation
• Performance of each signal and combination
• How does the system scale?

Evaluation Cont’d
• Offline evaluation
• Precision-Recall
• Online evaluation
• A/B testing to measure engagement
• Performance evaluation

Offline Evaluation
• Correct set: set of searches performed by a user in the
following K minutes, here K=10

Online Evaluation
• Used A/B testing
• Metrics
• Coverage: queries with recommendations
• Impressions: # of recommendations shown
• Clicks: Clicks on recommendations
• Click-through rate (CTR): Clicks per impression

Selected References
• R. Baeza-Yates. Graphs from search engine queries. LNCS, 4362:1–8, 2007.
• P. Boldi, F. Bonchi, C. Castillo, D. Donato, and S. Vigna. Query suggestions using query-
flow graphs. In Proceedings of the WSDM, 2009.
• M. A. Hasan, N. Parikh, B. Singh, and N. Sundaresan. Query suggestion for E-commerce
sites. In Proceedings of the WSDM, 2011.
• Q. Mei, D. Zhou, and K. Church. Query suggestion using hitting time. In Proceedings of
the CIKM, 2008.
• Z. Zhang and O. Nasraoui. Mining search engine query logs for query recommendation.
In Proceedings of the WWW, 2006.

The document summarizes a Chrome extension called Relatable that provides related news articles on a topic to online readers. It collects user data that shows high satisfaction with article relevance, layout, and inclusion of thumbnails. The MVP2 aims to address shortcomings by using additional APIs to improve relevance, displaying articles from more well-known sites first, and allowing user filtering of recommendations by date, site name, and keywords.

Recommender Systems @ Scale - PyData 2019

Sonya Liberman

Serving tens of billions of personalized recommendations a day under a latency of 30 milliseconds is a challenge. In this talk I'll share our algorithmic architecture, including its Spark-based offline layer, and its Elasticsearch-based serving layer, that enable running complex models under difficult scale constrains and shorten the cycle between research and production. Sonya Liberman leads the Personalization team @ Outbrain's Recommendations group, developing large-scale machine learning algorithms for Outbrain's content recommendations platform serving tens of billions real-time recommendations a day. She specializes in Information Retrieval, Machine Learning, and Computational Linguistics. Before joining Outbrain, she led the Research and Algorithms @ ConvertMedia (acquired by Taboola). She holds an MSc in Computer Science and a BSc in Computer Science and Computational Biology. This invited talk was given at PyData Meetup, April 2019 https://www.meetup.com/PyData-Tel-Aviv/

Agnes Molnar - Scoping and Enterprise Search Implementation

Agnes Molnar

This document summarizes the scoping of an enterprise search implementation for a global pharmaceutical company with 100,000 employees and a large and growing collection of documents. The challenges included scaling the search infrastructure to support the current collection as well as anticipated growth, improving performance for remote users, and enhancing search functionality. The proposed solution involved developing a content management strategy, search strategy across multiple platforms, and detailed planning and implementation approach. Key challenges were dealing with changing technologies that were still in development and ongoing content migrations. The benefits once completed would include a unified search experience, global availability, and enhanced performance.

Office 365 SharePoint Search Planning

Joel Oleson

FeedoData

Peter Fedoročko

The document discusses the future of business intelligence and data analytics. It argues that just as social networks have evolved from static profiles to more dynamic experiences (Social Networks 2.0), business intelligence also needs to evolve from static reports (Business Intelligence 1.0) to a more engaging experience where data changes in real-time and users can follow relevant reports and analytics. It then describes a product called FeedoData BI that aims to provide this type of socialized analytics experience.

How Does the USA Today Network Provide Its Readers With Meaningful Content? -...

Lucidworks

This document summarizes Devansh Dhutia's presentation about content delivery at USA Today Network. It discusses how the network reaches over 1 billion monthly viewers through 3000 journalists and leverages both active and passive search. It also outlines initiatives like My Topics for personalized alerts, News Near Me for local content, and different types of backfill content used to supplement news coverage. The presentation concludes by discussing lessons learned and opportunities to improve the distributed platform and content pipeline.

This document provides information about an auction of costumes and memorabilia from Debbie Reynolds' collection that will take place on June 18, 2011. It lists the location and viewing times for the auction, as well as contact information for the auction house, Profiles in History. It also includes 8 specific costume items from famous films that the Victoria and Albert Museum in London has requested to borrow for an upcoming exhibition on Hollywood costume design.

Research recommendations at Mendeley

Marco Rossetti

The document discusses recommender systems at Mendeley, an academic research platform. It describes how Mendeley uses recommender systems to help researchers organize their work, contextualize it within broader research, and connect with other researchers. The key components of Mendeley's recommender system are data sources, algorithms, business logic and analytics, and user interface. Mendeley's recommender system aims to provide personalized recommendations to researchers through different algorithms and lists of recommendations tailored to individual information needs.

Idus euroson 2008

sebikovacs

Intraductal ultrasonography (IDUS) can help diagnose and stage bilio-pancreatic tumors by allowing visualization of tumors and assessment of invasion. IDUS involves passing a miniature probe into the biliary or pancreatic ducts under endoscopic and radiologic guidance. In a study of 17 patients, IDUS helped diagnose and stage various biliary and pancreatic conditions like cholangiocarcinoma, ampullary tumors, and pancreatic neoplasms. IDUS had diagnostic value and did not cause any complications, making it a feasible technique to supplement other imaging methods. However, its cost-effectiveness needs further analysis.

핵 없는 세계를 위한 요코하마 선언보아 이

Neo4j - graph database for recommendations

proksik

This document discusses using Neo4j, a graph database, for recommendations. It describes modeling data as graphs in Neo4j and developing recommendation algorithms and plugins for it, such as for document similarity, movie recommendations, and restricting recommendations to a subgraph. An example application called TeleVido.tv is also mentioned that provides media content recommendations using Neo4j.

RESTful API Design & Implementation with CodeIgniter PHP Framework

Bo-Yi Wu

This document provides an overview and summary of Bo-Yi Wu's presentation on implementing a RESTful API with CodeIgniter. The presentation covers RESTful API basics like HTTP methods, JSON response format, API design best practices, and using the CodeIgniter REST Server and REST Client libraries to implement and test APIs within the CodeIgniter framework. Examples are provided for creating, updating, deleting and reading data via API requests and responses. Folder structure and routing configurations for organizing API controllers are also discussed.

Graph Based Recommendation Systems at eBay

DataStax Academy

Recommendation and personalization systems are an important part of many modern websites. Graphs provide a natural way to represent the behavioral data that is the core input to many recommendation algorithms. Thomas Pinckney and his colleagues at Hunch (recently acquired by eBay) built a large scale recommendation system, and then ported the technology to eBay. Thomas will be discussing how his team uses Cassandra to provide the high I/O storage of their fifty billion edge graphs and how they generate new recommendations in real time as users click around the site.

Recommender Systems

T212

Recommender system introduction

Liang Xiang

This document discusses recommender systems, including: 1. It provides an overview of recommender systems, their history, and common problems like top-N recommendation and rating prediction. 2. It then discusses what makes a good recommender system, including experiment methods like offline, user surveys, and online experiments, as well as evaluation metrics like prediction accuracy, diversity, novelty, and user satisfaction. 3. Key metrics that are important to evaluate recommender systems are discussed, such as user satisfaction, prediction accuracy, coverage, diversity, novelty, serendipity, trust, robustness, and response time. The document emphasizes selecting metrics based on business goals.

Collaborative Filtering Recommendation System

Milind Gokhale

Collaborative filtering is a technique used in recommender systems to predict a user's preferences based on other similar users' preferences. It involves collecting ratings data from users, calculating similarities between users or items, and making recommendations. Common approaches include user-user collaborative filtering, item-item collaborative filtering, and probabilistic matrix factorization. Recommender systems are evaluated both offline using metrics like MAE and RMSE, and through online user testing.

Building a Recommendation Engine - An example of a product recommendation engine

NYC Predictive Analytics

This document provides an example of building a predictive model for product recommendations. It outlines using a k-Nearest Neighbor (kNN) algorithm and singular value decomposition (SVD) for dimensionality reduction on order history and cart event data to create a recommendation engine. It discusses selecting features from the data, normalizing the data, using kNN to find similar items, and reducing the data dimensions with SVD before applying kNN. It also introduces using a synthetic dataset to test and tune the model and compares different experimental setups like random, kNN, and SVD+kNN recommendations. The goal is to increase business metrics like revenue, conversion rate, and average order value through effective product recommendations.

Recommender system algorithm and architecture

Liang Xiang

1) The document discusses recommender system algorithms and architecture. It covers common recommendation techniques like collaborative filtering, content-based filtering, and graph-based recommendations. 2) It also discusses challenges like cold starts for new users and items. For new users, it recommends using demographic data or initial feedback to understand interests. For new items, it suggests using content information or initial user feedback. 3) The document proposes a feature-based recommendation framework that connects users, items, and latent features to address challenges like heterogeneous data and cold starts. This framework provides explanations but does not support user-based methods.

Building a real time, solr-powered recommendation engine

Trey Grainger

Searching text is what Solr is known for, but did you know that many companies receive an equal or greater business impact through implementing a recommendation engine in addition to their text search capabilities? With a few tweaks, Solr (or Lucene) can also serve as a full featured recommendation engine. Machine learning libraries like Apache Mahout provide excellent behavior-based, off-line recommendation algorithms, but what if you want more control? This talk will demonstrate how to effectively utilize Solr to perform collaborative filtering (users who liked this also liked…), categorical classification and subsequent hierarchical-based recommendations, as well as related-concept extraction and concept based recommendations. Sound difficult? It’s not. Come learn step-by-step how to create a powerful real-time recommendation engine using Apache Solr and see some real-world examples of some of these strategies in action.

How to Build a Recommendation Engine on Spark

Caserta

How to Build a Recommendation Engine on Spark was a presentation given by Joe Caserta, CEO and founder of Caserta Concepts, at @AnalyticsWeek in Boston. Boston's Data AnalyticsStreet Conference is a 2 day packed event with thought provoking keynotes, knowledge filled sessions, intense workshops, insightful panels, and real-world case studies - engaging analytics community with latest methodologies and trends. The conference encompasses largest Speaker-to-Attendee ratio for unmatched networking and learning opportunity. For more information on the services and solutions Caserta Concepts offers, visit our website at http://casertaconcepts.com/.

Interactive Recommender Systems with Netflix and Spotify

Chris Johnson

Interactive recommender systems enable the user to steer the received recommendations in the desired direction through explicit interaction with the system. In the larger ecosystem of recommender systems used on a website, it is positioned between a lean-back recommendation experience and an active search for a specific piece of content. Besides this aspect, we will discuss several parts that are especially important for interactive recommender systems, including the following: design of the user interface and its tight integration with the algorithm in the back-end; computational efficiency of the recommender algorithm; as well as choosing the right balance between exploiting the feedback from the user as to provide relevant recommendations, and enabling the user to explore the catalog and steer the recommendations in the desired direction. In particular, we will explore the field of interactive video and music recommendations and their application at Netflix and Spotify. We outline some of the user-experiences built, and discuss the approaches followed to tackle the various aspects of interactive recommendations. We present our insights from user studies and A/B tests. The tutorial targets researchers and practitioners in the field of recommender systems, and will give the participants a unique opportunity to learn about the various aspects of interactive recommender systems in the video and music domain. The tutorial assumes familiarity with the common methods of recommender systems.

Recommendation system

Vikrant Arya

Recommendation systems provide users with information they may be interested in based on their preferences and interests. They help address the problem of information overload by retrieving desired information for the user based on their preferences or those of similar users. The two main types of recommendation systems are personalized and non-personalized systems. Common techniques used include collaborative filtering, which finds users with similar tastes, and content-based filtering, which recommends items similar to those a user has liked based on item attributes.

Recommender Systems (Machine Learning Summer School 2014 @ CMU)

Xavier Amatriain

The document summarizes a presentation on recommender systems given by Xavier Amatriain. It begins with introductions to recommender systems and collaborative filtering. Traditional collaborative filtering approaches include user-based and item-based methods. User-based CF finds similar users to a target user and recommends items they liked. Item-based CF finds similar items to those a target user liked and predicts ratings. Both approaches address sparsity and scalability challenges with dimensionality reduction techniques.

How We Incrementally Improved Search

Ravi Mynampaty

This document outlines the incremental steps taken to improve search at an organization. It began with an out-of-the-box search tool that indexed only a fraction of content and provided many dead ends. The vision was to create a single search box that could search all content in a Google-like interface. A roadmap was created to prioritize integrating the most used systems starting with low hanging fruit. Implementation involved customizing the search interface, adding APIs to other systems, and various approaches like blended, brokered and query resubmit searches. Analytics tracked usage and best search terms. Benefits included a single access point and overcoming limitations of individual tools. Challenges remain around complex permissions, design convergence and integrating

Use of data science in recommendation system

AkashPatil334

This document discusses the use of data science in recommendation systems. It defines recommendation systems as systems that predict a user's preferences for items and recommend top items. It also defines data science as using scientific methods to extract knowledge from structured and unstructured data. The document then describes different types of recommendation systems, including collaborative filtering, content-based filtering, and hybrid systems. It provides examples of how Netflix, Amazon, LinkedIn, and Pandora use recommendation systems.

Building Recommender Systems - Mendeley and Science Direct

Daniel Kershaw

Assessment

Jody DeRidder

Developing a digital library doesn’t end when content goes online. You need to know whether what you are doing is effective; whether you’re reaching your users, whether you’re providing them with what they need in the form they need it, and whether you are doing this in the most cost-effective way that you can. This presentation examines the challenges inherent in assessing three different aspects of digital libraries: costs, user needs, and benefits.

Modern Perspectives on Recommender Systems and their Applications in Mendeley

Kris Jack

This document discusses modern perspectives on recommender systems and their applications at Mendeley. It covers the evolution of recommender problems from rating prediction to context-aware recommendations. It also discusses common recommender algorithms like collaborative filtering, content-based filtering and hybrid approaches. The document concludes by discussing how Mendeley uses recommenders for related research, researchers to follow, and other use cases.

Viewers also liked

Catálogo de Debbie Reynolds

jaugustosma

Research recommendations at Mendeley

Marco Rossetti

Idus euroson 2008

sebikovacs

핵 없는 세계를 위한 요코하마 선언보아 이

Neo4j - graph database for recommendations

proksik

RESTful API Design & Implementation with CodeIgniter PHP Framework

Bo-Yi Wu

Graph Based Recommendation Systems at eBay

DataStax Academy

Recommender Systems

T212

Recommender system introduction

Liang Xiang

Collaborative Filtering Recommendation System

Milind Gokhale

Building a Recommendation Engine - An example of a product recommendation engine

NYC Predictive Analytics

Recommender system algorithm and architecture

Liang Xiang

Building a real time, solr-powered recommendation engine

Trey Grainger

How to Build a Recommendation Engine on Spark

Caserta

Interactive Recommender Systems with Netflix and Spotify

Chris Johnson

Recommendation system

Vikrant Arya

Recommender Systems (Machine Learning Summer School 2014 @ CMU)

Xavier Amatriain

Viewers also liked (17)

Catálogo de Debbie Reynolds

Research recommendations at Mendeley

Idus euroson 2008

핵 없는 세계를 위한 요코하마 선언

Neo4j - graph database for recommendations

RESTful API Design & Implementation with CodeIgniter PHP Framework

Graph Based Recommendation Systems at eBay

Recommender Systems

Recommender system introduction

Collaborative Filtering Recommendation System

Building a Recommendation Engine - An example of a product recommendation engine

Recommender system algorithm and architecture

Building a real time, solr-powered recommendation engine

How to Build a Recommendation Engine on Spark

Interactive Recommender Systems with Netflix and Spotify

Recommendation system

Recommender Systems (Machine Learning Summer School 2014 @ CMU)

Similar to Metaphor: A system for related searches recommendations

How We Incrementally Improved Search

Ravi Mynampaty

Use of data science in recommendation system

AkashPatil334

Building Recommender Systems - Mendeley and Science Direct

Daniel Kershaw

Assessment

Jody DeRidder

Modern Perspectives on Recommender Systems and their Applications in Mendeley

Kris Jack

What's new at Crossref - Ed Pentz - London LIVE 2017

Crossref

Netflix Recommender System : Big Data Case Study

Ketan Patil

Netflix faced the challenge of personalized recommendations at scale for its growing user base and content catalog. It developed a recommender system using 3 computation layers - online, offline, and nearline - to process petabytes of user data from ratings, streams, and other sources. This system saved Netflix $1 billion per year by improving user engagement and satisfaction through more accurate recommendations.

How Google works

Accesstrade Vietnam

Ranking engineers at Google work to optimize search results by developing signals, metrics, and experiments. They: 1. Look for new signals that measure page quality and relevance and combine existing signals in new ways. 2. Optimize search results based on metrics like relevance, quality, and time to load; these metrics are measured through live experiments and ratings from human raters. 3. Address issues like systematically bad ratings or missing metrics by fixing rater guidelines, developing new metrics, or identifying patterns of losses to improve results.

Disrupting Data Discovery

markgrover

Lyft developed Amundsen, an internal metadata and data discovery platform, to help their data scientists and engineers find data more efficiently. Amundsen provides search-based and lineage-based discovery of Lyft's data resources. It uses a graph database and Elasticsearch to index metadata from various sources. While initially built using a pull model with crawlers, Amundsen is moving toward a push model where systems publish metadata to a message queue. The tool has increased data team productivity by over 30% and will soon be open sourced for other organizations to use.

Anatomy of Search Relevance: From Data To Action

Saïd Radhouani

Relevance denotes how well a search result satisfies the user information need. In addition to the search engine components (i.e., indexer and query parser), there are many other components that impact relevance. e.g., user understanding , data optimization, domain knowledge, etc. Improving relevance remains the main and most challenging goal of each search engine. Indeed, relevance can be subjective, therefore hard to measure and to improve. In this talk, Saïd will demystify the concept of relevance by defining its main components. For each component, he will present the technology enablers, the data, and processes that are required in order to measure and improve relevance. In this talk, attendees will learn how to provide a relevant user experience and track it over time.

Anatomy of Relevance - From Data to Action: Presented by Saïd Radhouani, Yell...

Lucidworks

This document discusses relevance and how Yellow Pages leverages data to improve relevance. It outlines the importance of relevance for businesses and consumers in local search. The author then describes Yellow Pages and his role in content, search, and relevance. Next, it details the building blocks of relevance including user data, content, knowledge, search, and presentation. It emphasizes measuring relevance through key performance indicators and using data to identify gaps and drive continuous improvement through action. The goal is to use an evidence-based approach to optimize the user experience and satisfaction.

Taking the Pain out of Data Science - RecSys Machine Learning Framework Over ...

Sonya Liberman

1. The document discusses Outbrain's machine learning framework for personalized recommendations using Spark. It describes challenges in collecting and processing large datasets, building predictive models, and deploying models to production through A/B testing. 2. It outlines Outbrain's distributed machine learning framework for data collection, feature engineering, model training, evaluation, and deployment. Standardized model interfaces allow easy implementation of various algorithms and ensemble modeling. 3. The framework aims to streamline the research cycle and connect research to production through automated data preparation, simple model evaluation and simulation, and fast A/B testing and model updates in production.

Dlf 2012

sherriberger

The document discusses planning and conducting assessments of digital library projects. It outlines an assessment process including defining objectives, developing questions, determining appropriate methods, collecting and analyzing data, and communicating results. The document also provides a case study example where they scope an assessment of the Calisphere digital library by defining objectives, questions, and appropriate data collection methods to answer questions about users, usage, and contributor needs.

Building Search and Personalization at Nordstrom Rack | Hautelook

Lucidworks

Recommender System Using AZURE ML

Dev Raj Gautam

This document discusses building recommender systems using Azure ML. It provides an overview of recommender systems including examples of how Netflix, Amazon, and Best Buy use them. It then outlines an agenda to understand recommender systems, load a movie dataset from GroupLens, clean and prepare the data, train a Matchbox recommender model in Azure ML, test the model, and ways to improve predictive modeling. The presentation aims to provide hands-on experience with recommender systems in Azure ML Studio.

CRC-STC May 2013 Summit Presentation

crcstc

The document summarizes key topics from the 2013 STC Summit conference, including professional development, social media, single-sourcing, content strategy, and business metrics. Over 800 people attended the conference, which featured 137 sessions across various communication modes. Popular sessions focused on cultivating online presence, leveraging social media for feedback, conditional text in Flare, and building business cases through metrics that demonstrate revenue impact. The document provides resources for further exploring each topic.

RecSys 2015 Tutorial - Scalable Recommender Systems: Where Machine Learning m...

Joaquin Delgado PhD.

This tutorial gives an overview of how search engines and machine learning techniques can be tightly coupled to address the need for building scalable recommender or other prediction based systems. Typically, most of them architect retrieval and prediction in two phases. In Phase I, a search engine returns the top-k results based on constraints expressed as a query. In Phase II, the top-k results are re-ranked in another system according to an optimization function that uses a supervised trained model. However this approach presents several issues, such as the possibility of returning sub-optimal results due to the top-k limits during query, as well as the prescence of some inefficiencies in the system due to the decoupling of retrieval and ranking. To address this issue the authors created ML-Scoring, an open source framework that tightly integrates machine learning models into Elasticsearch, a popular search engine. ML-Scoring replaces the default information retrieval ranking function with a custom supervised model that is trained through Spark, Weka, or R that is loaded as a plugin in Elasticsearch. This tutorial will not only review basic methods in information retrieval and machine learning, but it will also walk through practical examples from loading a dataset into Elasticsearch to training a model in Spark, Weka, or R, to creating the ML-Scoring plugin for Elasticsearch. No prior experience is required in any system listed (Elasticsearch, Spark, Weka, R), though some programming experience is recommended.

RecSys 2015 Tutorial – Scalable Recommender Systems: Where Machine Learning...

S. Diana Hu

Search engines have focused on solving the document retrieval problem, so their scoring functions do not handle naturally non-traditional IR data types, such as numerical or categorical. Therefore, on domains beyond traditional search, scores representing strengths of associations or matches may vary widely. As such, the original model doesn’t suffice, so relevance ranking is performed as a two-phase approach with 1) regular search 2) external model to re-rank the filtered items. Metrics such as click-through and conversion rates are associated with the users’ response to items served. The predicted selection rates that arise in real-time can be critical for optimal matching. For example, in recommender systems, predicted performance of a recommended item in a given context, also called response prediction, is often used in determining a set of recommendations to serve in relation to a given serving opportunity. Similar techniques are used in the advertising domain. To address this issue the authors have created ML-Scoring, an open source framework that tightly integrates machine learning models into a popular search engine (SOLR/Elasticsearch), replacing the default IR-based ranking function. A custom model is trained through either Weka or Spark and it is loaded as a plugin used at query time to compute custom scores.

Managing plagiarism: Similarity Check

Crossref

- CrossCheck has rebranded to Crossref Similarity Check to provide clearer messaging and reduce confusion. - The service checks documents against over 53 million papers from over 1200 publishers, as well as 105 million items from other sources and over 60 billion web pages. - Over 1200 Crossref publishers and over 100 Brazilian publishers are using the service, with increasing usage in countries like Japan, South Korea, and Turkey. - Publishers are looking to identify issues like poor references, self-plagiarism, unattributed use of others' works, and submitting others' works as their own through the similarity checking service.

Exploratory Search upon Semantically Described Web Data Sources: Service regi...

Marco Brambilla

This presentation was given at the SSW workshop, collocated with VLDB 2012. Exploratory search applications upon structured Web content is becoming one of the main information seeking paradigms for users. This is mainly due to the move towards mobile and pervasive Web access and to the more and more tight intertwining between everyday life and information seeking. Structured data is typically distributed on the Web and accessible through a service-oriented paradigm. This paper proposes a vision on: (1) a semantically-enabled service registration framework for describing in a Web data services in a convenient way; and (2) a design method for applications that exploit such model using a design pattern -based method.

Similar to Metaphor: A system for related searches recommendations (20)

How We Incrementally Improved Search

Use of data science in recommendation system

Building Recommender Systems - Mendeley and Science Direct

Assessment

Modern Perspectives on Recommender Systems and their Applications in Mendeley

What's new at Crossref - Ed Pentz - London LIVE 2017

Netflix Recommender System : Big Data Case Study

How Google works

Disrupting Data Discovery

Anatomy of Search Relevance: From Data To Action

Anatomy of Relevance - From Data to Action: Presented by Saïd Radhouani, Yell...

Taking the Pain out of Data Science - RecSys Machine Learning Framework Over ...

Dlf 2012

Building Search and Personalization at Nordstrom Rack | Hautelook

Recommender System Using AZURE ML

CRC-STC May 2013 Summit Presentation

RecSys 2015 Tutorial - Scalable Recommender Systems: Where Machine Learning m...

RecSys 2015 Tutorial – Scalable Recommender Systems: Where Machine Learning...

Managing plagiarism: Similarity Check

Exploratory Search upon Semantically Described Web Data Sources: Service regi...

More from Mitul Tiwari

Large scale social recommender systems at LinkedIn

Mitul Tiwari

This document discusses the large-scale social recommender systems used at LinkedIn. It describes how LinkedIn uses recommender systems to suggest news articles, people you may know connections, related searches, and skill endorsements. It also discusses the challenges of scaling these recommender systems given LinkedIn's large user base and social graph. Key recommender systems covered include recommending news items to users, identifying potential connections through people you may know, and suggesting skill endorsements. The document also discusses how social graph analysis and modeling virality helps these recommender systems.

Big Data Ecosystem at LinkedIn. Keynote talk at Big Data Innovators Gathering...

Mitul Tiwari

LinkedIn has a large professional network with 360M members. They build data-driven products using members' rich profile data. To do this, they ingest online data into offline systems using Apache Kafka. The data is then processed using Hadoop, Spark, Samza and Cubert to compute features and train models. Results are moved back online using Voldemort and Kafka. For example, People You May Know recommendations are generated by triangle closing in Hadoop and Cubert to count common connections faster. Site speed is monitored in real-time using Samza to join logs from different services.

Modeling Impression discounting in large-scale recommender systems

Mitul Tiwari

Recommender systems have become very important for many online activities, such as watching movies, shopping for products, and connecting with friends on social networks. User behavioral analysis and user feedback (both explicit and implicit) modeling are crucial for the improvement of any online recommender system. Widely adopted recommender systems at LinkedIn such as “People You May Know” and “Suggested Skills Endorsement” are evolving by analyzing user behaviors on impressed recommendation items. In this paper, we address modeling impression discounting of recommended items, that is, how to model users’ no-action feedback on impressed recommended items. The main contributions of this paper include (1) large-scale analysis of impression data from LinkedIn and KDD Cup; (2) novel anti-noise regression techniques, and its application to learn four different impression discounting functions including linear decay, inverse decay, exponential decay, and quadratic decay; (3) applying these impression discounting functions to LinkedIn’s “People You May Know” and “Suggested Skills Endorsements” recommender systems.

Large scale social recommender systems and their evaluation

Mitul Tiwari

This talk will give an overview of some of the large-scale recommender systems at LinkedIn such as People You May Know (PYMK) and Suggested Skills Endorsements. This talk will also address how we formulate machine learning modeling problems to build these recommender systems and evaluate our models. Modeling for these recommender systems involves careful feature engineering and incorporating user feedback - both explicit and implicit. This talk will describe how we feature engineer through an example of modeling organizational overlap between people for link prediction and community detection over social graph. Also, how we incorporate user feedback through impression discounting ignored recommended results will be described. Careful evaluation of modeling changes both offline and online (A/B testing) is inherent part of measuring effectiveness of our recommender systems. We have built a sophisticated end-to-end A/B testing and evaluation platform called XLNT at LinkedIn and this talk will also cover how we use XLNT for power analysis, A/B testing, and measuring confidence of the results.

Structural Diversity in Social Recommender Systems

Mitul Tiwari

Organizational Overlap on Social Networks and its Applications

Mitul Tiwari

Building Data Driven Products at Linkedin

Mitul Tiwari

This document discusses building data products at LinkedIn using Hadoop. It describes how LinkedIn builds recommendations products like "People You May Know" by processing member connection data with Hadoop tools. The workflow involves using Kafka to transfer data to HDFS, Pig and MapReduce to process the data, Azkaban to manage Hadoop jobs, and Voldemort to store results and serve recommendations to members. Triangle closing algorithms in Pig are used to find common connections between members and predict potential new connections. The results are pushed to production systems to power features like "People You May Know" recommendations.

Social Network Analysis at LinkedIn

Mitul Tiwari

This document discusses social network analysis at LinkedIn. It provides an overview of LinkedIn's analytics products, including People You May Know, Profile Stats, and InMaps. It then describes in more detail how LinkedIn builds features like Connection Strength and Related Searches using tools like Hadoop MapReduce, Kafka, Azkaban, and Voldemort. The document outlines LinkedIn's data processing cycle and workflow management for serving analytics data to users.

More from Mitul Tiwari (8)

Large scale social recommender systems at LinkedIn

Big Data Ecosystem at LinkedIn. Keynote talk at Big Data Innovators Gathering...

Modeling Impression discounting in large-scale recommender systems

Large scale social recommender systems and their evaluation

Structural Diversity in Social Recommender Systems

Organizational Overlap on Social Networks and its Applications

Building Data Driven Products at Linkedin

Social Network Analysis at LinkedIn

Recently uploaded

Chapter 12 - climate change and the energy crisis

tonzsalvador2222

Applied Science: Thermodynamics, Laws & Methodology.pdf

University of Hertfordshire

When I was asked to give a companion lecture in support of ‘The Philosophy of Science’ (https://shorturl.at/4pUXz) I decided not to walk through the detail of the many methodologies in order of use. Instead, I chose to employ a long standing, and ongoing, scientific development as an exemplar. And so, I chose the ever evolving story of Thermodynamics as a scientific investigation at its best. Conducted over a period of >200 years, Thermodynamics R&D, and application, benefitted from the highest levels of professionalism, collaboration, and technical thoroughness. New layers of application, methodology, and practice were made possible by the progressive advance of technology. In turn, this has seen measurement and modelling accuracy continually improved at a micro and macro level. Perhaps most importantly, Thermodynamics rapidly became a primary tool in the advance of applied science/engineering/technology, spanning micro-tech, to aerospace and cosmology. I can think of no better a story to illustrate the breadth of scientific methodologies and applications at their best.

molar-distalization in orthodontics-seminar.pptx

Anagha Prasad

The debris of the ‘last major merger’ is dynamically young

Sérgio Sacani

The Milky Way’s (MW) inner stellar halo contains an [Fe/H]-rich component with highly eccentric orbits, often referred to as the ‘last major merger.’ Hypotheses for the origin of this component include Gaia-Sausage/Enceladus (GSE), where the progenitor collided with the MW proto-disc 8–11 Gyr ago, and the Virgo Radial Merger (VRM), where the progenitor collided with the MW disc within the last 3 Gyr. These two scenarios make different predictions about observable structure in local phase space, because the morphology of debris depends on how long it has had to phase mix. The recently identified phase-space folds in Gaia DR3 have positive caustic velocities, making them fundamentally different than the phase-mixed chevrons found in simulations at late times. Roughly 20 per cent of the stars in the prograde local stellar halo are associated with the observed caustics. Based on a simple phase-mixing model, the observed number of caustics are consistent with a merger that occurred 1–2 Gyr ago. We also compare the observed phase-space distribution to FIRE-2 Latte simulations of GSE-like mergers, using a quantitative measurement of phase mixing (2D causticality). The observed local phase-space distribution best matches the simulated data 1–2 Gyr after collision, and certainly not later than 3 Gyr. This is further evidence that the progenitor of the ‘last major merger’ did not collide with the MW proto-disc at early times, as is thought for the GSE, but instead collided with the MW disc within the last few Gyr, consistent with the body of work surrounding the VRM.

aziz sancar nobel prize winner: from mardin to nobel

İsa Badur

EWOCS-I: The catalog of X-ray sources in Westerlund 1 from the Extended Weste...

Sérgio Sacani

Context. With a mass exceeding several 104 M⊙ and a rich and dense population of massive stars, supermassive young star clusters represent the most massive star-forming environment that is dominated by the feedback from massive stars and gravitational interactions among stars. Aims. In this paper we present the Extended Westerlund 1 and 2 Open Clusters Survey (EWOCS) project, which aims to investigate the influence of the starburst environment on the formation of stars and planets, and on the evolution of both low and high mass stars. The primary targets of this project are Westerlund 1 and 2, the closest supermassive star clusters to the Sun. Methods. The project is based primarily on recent observations conducted with the Chandra and JWST observatories. Specifically, the Chandra survey of Westerlund 1 consists of 36 new ACIS-I observations, nearly co-pointed, for a total exposure time of 1 Msec. Additionally, we included 8 archival Chandra/ACIS-S observations. This paper presents the resulting catalog of X-ray sources within and around Westerlund 1. Sources were detected by combining various existing methods, and photon extraction and source validation were carried out using the ACIS-Extract software. Results. The EWOCS X-ray catalog comprises 5963 validated sources out of the 9420 initially provided to ACIS-Extract, reaching a photon flux threshold of approximately 2 × 10−8 photons cm−2 s −1 . The X-ray sources exhibit a highly concentrated spatial distribution, with 1075 sources located within the central 1 arcmin. We have successfully detected X-ray emissions from 126 out of the 166 known massive stars of the cluster, and we have collected over 71 000 photons from the magnetar CXO J164710.20-455217.

ANAMOLOUS SECONDARY GROWTH IN DICOT ROOTS.pptx

RASHMI M G

bordetella pertussis.................................ppt

kejapriya1

Equivariant neural networks and representation theory

Daniel Tubbenhauer

Or: Beyond linear. Abstract: Equivariant neural networks are neural networks that incorporate symmetries. The nonlinear activation functions in these networks result in interesting nonlinear equivariant maps between simple representations, and motivate the key player of this talk: piecewise linear representation theory. Disclaimer: No one is perfect, so please mind that there might be mistakes and typos. dtubbenhauer@gmail.com Corrected slides: dtubbenhauer.com/talks.html

Nucleic Acid-its structural and functional complexity.

Nistarini College, Purulia (W.B) India

8.Isolation of pure cultures and preservation of cultures.pdf

by6843629

Medical Orthopedic PowerPoint Templates.pptx

terusbelajar5

ESR spectroscopy in liquid food and beverages.pptx

PRIYANKA PATEL

With increasing population, people need to rely on packaged food stuffs. Packaging of food materials requires the preservation of food. There are various methods for the treatment of food to preserve them and irradiation treatment of food is one of them. It is the most common and the most harmless method for the food preservation as it does not alter the necessary micronutrients of food materials. Although irradiated food doesn’t cause any harm to the human health but still the quality assessment of food is required to provide consumers with necessary information about the food. ESR spectroscopy is the most sophisticated way to investigate the quality of the food and the free radicals induced during the processing of the food. ESR spin trapping technique is useful for the detection of highly unstable radicals in the food. The antioxidant capability of liquid food and beverages in mainly performed by spin trapping technique.

如何办理(uvic毕业证书)维多利亚大学毕业证本科学位证书原版一模一样

yqqaatn0

原版纸张【微信：741003700 】【(uvic毕业证书)维多利亚大学毕业证】【微信：741003700 】学位证，留信认证（真实可查，永久存档）offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。本公司拥有海外各大学样板无数，能完美还原海外各大学 Bachelor Diploma degree, Master Degree Diploma 1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700 留信网认证的作用: 1:该专业认证可证明留学生真实身份 2:同时对留学生所学专业登记给予评定 3:国家专业人才认证中心颁发入库证书 4:这个认证书并且可以归档倒地方 5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息 6:个人职称评审加20分 7:个人信誉贷款加10分 8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才

3D Hybrid PIC simulation of the plasma expansion (ISSS-14)

David Osipyan

Cytokines and their role in immune regulation.pptx

Hitesh Sikarwar

原版制作(carleton毕业证书)卡尔顿大学毕业证硕士文凭原版一模一样

yqqaatn0

原版纸张【微信：741003700 】【(carleton毕业证书)卡尔顿大学毕业证】【微信：741003700 】学位证，留信认证（真实可查，永久存档）offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。本公司拥有海外各大学样板无数，能完美还原海外各大学 Bachelor Diploma degree, Master Degree Diploma 1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700 留信网认证的作用: 1:该专业认证可证明留学生真实身份 2:同时对留学生所学专业登记给予评定 3:国家专业人才认证中心颁发入库证书 4:这个认证书并且可以归档倒地方 5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息 6:个人职称评审加20分 7:个人信誉贷款加10分 8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才

NuGOweek 2024 Ghent programme overview flyer

pablovgd

Nucleophilic Addition of carbonyl compounds.pptx

SSR02

Nucleophilic addition is the most important reaction of carbonyls. Not just aldehydes and ketones, but also carboxylic acid derivatives in general. Carbonyls undergo addition reactions with a large range of nucleophiles. Comparing the relative basicity of the nucleophile and the product is extremely helpful in determining how reversible the addition reaction is. Reactions with Grignards and hydrides are irreversible. Reactions with weak bases like halides and carboxylates generally don’t happen. Electronic effects (inductive effects, electron donation) have a large impact on reactivity. Large groups adjacent to the carbonyl will slow the rate of reaction. Neutral nucleophiles can also add to carbonyls, although their additions are generally slower and more reversible. Acid catalysis is sometimes employed to increase the rate of addition.

Oedema_types_causes_pathophysiology.pptx

muralinath2

Recently uploaded (20)

Chapter 12 - climate change and the energy crisis

Applied Science: Thermodynamics, Laws & Methodology.pdf

molar-distalization in orthodontics-seminar.pptx

The debris of the ‘last major merger’ is dynamically young

aziz sancar nobel prize winner: from mardin to nobel

EWOCS-I: The catalog of X-ray sources in Westerlund 1 from the Extended Weste...

ANAMOLOUS SECONDARY GROWTH IN DICOT ROOTS.pptx

bordetella pertussis.................................ppt

Equivariant neural networks and representation theory

Nucleic Acid-its structural and functional complexity.

8.Isolation of pure cultures and preservation of cultures.pdf

Medical Orthopedic PowerPoint Templates.pptx

ESR spectroscopy in liquid food and beverages.pptx

如何办理(uvic毕业证书)维多利亚大学毕业证本科学位证书原版一模一样

3D Hybrid PIC simulation of the plasma expansion (ISSS-14)

Cytokines and their role in immune regulation.pptx

原版制作(carleton毕业证书)卡尔顿大学毕业证硕士文凭原版一模一样

NuGOweek 2024 Ghent programme overview flyer

Nucleophilic Addition of carbonyl compounds.pptx

Oedema_types_causes_pathophysiology.pptx

Metaphor: A system for related searches recommendations

1. Metaphor: A System for Related Searches Recommendations Mitul Tiwari Joint work with Azarias Reda, Yubin Park, Christian Posse, and Sam Shah LinkedIn

2. Who am I

3. Outline • Related Searches at LinkedIn • Metaphor: A System for Related Searches Recommendations • Design • Implementation • Evaluation

4. LinkedIn by the numbers • 175M+ members • 2+ new user registrations per second • 4.2 Billion people searches in 2011 • 9.3 Billion page views in Q2 2012 • 100+ million monthly active users in Q2 2012

5. Related Searches at LinkedIn • Millions of searches everyday • Goal: Build related searches system at LinkedIn • To help users to explore and refine their queries Azarias Reda, Yubin Park, Mitul Tiwari, Christian Posse, and Sam Shah

6. Related Searches at LinkedIn

7. Metaphor: A Related Searches System • Design • Implementation • Evaluation

8. Metaphor: A Related Searches System • Design • Implementation • Evaluation

9. Design • Signals • Collaborative Filtering • Query-Result Click graph • Overlapping terms • Length-bias • Ensemble approach for unified recommendation • Practical considerations

10. Design: Collaborative Filtering • Searches correlated by time • Searches done in the same session by the same user • Collaborative filtering: implicit feedback • TFIDF scoring to take care of popular queries (e.g. `Obama’) Q1 Q2 Q3 Time

11. Design: Query-Result Clicks • Searches correlated by result clicks Q1 Qn R1 Rm

12. Design: Overlapping Terms • Searches with overlapping terms • TFIDF scoring to give importance to terms Software Developer Software Engineer Q1 Q2

13. Design: Length Bias • Insight: clicks on suggestions one term longer

14. Design: Length Bias • Insight: clicks on suggestions one term longer • Corresponds to refining the initial query • Statistical biasing model to score a longer query higher

15. Design: Ensemble Approach • Need to generate unified recommendation dataset • Analysis to figure out engagement of each signal • Attempted ML approach • Minimal overlap across different signals

16. Design: Ensemble Approach • Step-wise unionization • Importance based on individual signal performance • First, collaborative filtering • Second, queries correlated by query-result clicks • Third, queries overlapping terms

17. Design: Practical Considerations • System designed for public consumption • Strong profanity filters • Need to deal with misspellings • Languages • Remove spammy search queries

18. Metaphor: A Related Searches System • Design • Implementation • Evaluation

19. Implementation Challenge • Scale • 175M+ members • Billions of searches • Terabytes of data to process

20. Implementation • Kafka: publish-subscribe messaging system • Hadoop: MapReduce data processing system • Azkaban: Hadoop workflow management tool • Voldemort: Key-value store

21. Implementation: Workflow

22. Metaphor: A Related Searches System • Design • Implementation • Evaluation

23. Evaluation • Performance of each signal and combination • How does the system scale?

24. Evaluation Cont’d • Offline evaluation • Precision-Recall • Online evaluation • A/B testing to measure engagement • Performance evaluation

25. Offline Evaluation • Correct set: set of searches performed by a user in the following K minutes, here K=10

26. Online Evaluation • Used A/B testing • Metrics • Coverage: queries with recommendations • Impressions: # of recommendations shown • Clicks: Clicks on recommendations • Click-through rate (CTR): Clicks per impression

27. Online Evaluation

28. Evaluation: System Runtime

29. Selected References • R. Baeza-Yates. Graphs from search engine queries. LNCS, 4362:1–8, 2007. • P. Boldi, F. Bonchi, C. Castillo, D. Donato, and S. Vigna. Query suggestions using query- flow graphs. In Proceedings of the WSDM, 2009. • M. A. Hasan, N. Parikh, B. Singh, and N. Sundaresan. Query suggestion for E-commerce sites. In Proceedings of the WSDM, 2011. • Q. Mei, D. Zhou, and K. Church. Query suggestion using hitting time. In Proceedings of the CIKM, 2008. • Z. Zhang and O. Nasraoui. Mining search engine query logs for query recommendation. In Proceedings of the WWW, 2006.

30. Questions?

Editor's Notes

first context of related searches at LinkedIn then design, implementation and evaluation of our related searches system
Slow down Searches per second: 130, min: 8000, hour: 480000, day: 11.5M Cut down
discovery, exploration, refine
a screenshot of search result page
explore more candidates and scoring
For example, web developer -> HTML why collaborative filtering elaborate session replace to within a time window
Elaborate - across individual put a real example importance of each click query fanout, popular result
mechanical engineer across individual
highlight - next
show evaluation/analysis result about clicks on queries one term longer skip the second equation
describe signals
mention evaluation later
high level design Kafka, Voldemort citations, url to Azkaban
high level design Kafka, Voldemort citations, url to Azkaban
scientific and easily repeatable fast, iterative way for tuning parameters and performance P-decreases with the size of window, R-increases with time window K P/R low: predicting future searches, conservative measure; judging whether a signal can predict future behavior CF has advantage in this measure top-10 recommendations
why CTR is not the only metric: hadoop->mapreduce
normalized legends: elaborate CF, QRQ, partial break down figures: bigger chart
for all possible queries quadratic why? 80 nodes 2 quad-core cpus: 640 cores
questions, details, hiring

Metaphor: A system for related searches recommendations

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (17)

Similar to Metaphor: A system for related searches recommendations

Similar to Metaphor: A system for related searches recommendations (20)

More from Mitul Tiwari

More from Mitul Tiwari (8)

Recently uploaded

Recently uploaded (20)

Metaphor: A system for related searches recommendations

Editor's Notes