Google Tech Talk: Reconsidering Relevance
•Download as PPT, PDF•
37 likes•19,176 views
Reconsidering Relevance We've become complacent about relevance. The overwhelming success of web search engines has lulled even information retrieval (IR) researchers to expect only incremental improvements in relevance in the near future. And beyond web search, there are still broad search problems where relevance still feels hopelessly like the pre-Google web. But even some of the most basic IR questions about relevance are unresolved. We take for granted the very idea that a computer can determine which documents are relevant to a person's needs. And we still rely on two-word queries (on average) to communicate a user's information need. But this approach is a contrivance; in reality, we need to think of information-seeking as a problem of optimizing the communication between people and machines. We can do better. In fact, there are a variety of ongoing efforts to do so, often under the banners of "interactive information retrieval", "exploratory search", and "human computer information retrieval". In this talk, I'll discuss these initiatives and how they are helping to move "relevance" beyond today's outdated assumptions. About the Speaker Daniel Tunkelang is co-founder and Chief Scientist at Endeca, a leading provider of enterprise information access solutions. He leads Endeca's efforts to develop features and capabilities that emphasize user interaction. Daniel has spearheaded the annual Workshops on Human Computer Information Retrieval (HCIR) and is organizing the Industry Track for SIGIR '09. Daniel also publishes The Noisy Channel, a widely read and cited blog that focuses on how people interact with information. Daniel holds undergraduate degrees in mathematics and computer science from the Massachusetts Institute of Technology, with a minor in psychology. He completed a PhD at Carnegie Mellon University for his work on information visualization. His work previous to Endeca includes stints at the IBM T. J. Watson Research Center and AT&T Labs,
Report
Share
Report
Share
![howdy! ,[object Object],[object Object],[object Object]](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
Project Proposal Topics Modeling (Ir)
This document proposes using Wikipedia concepts to improve topic modeling. It discusses using n-grams like bigrams and phrases from Wikipedia categories rather than only individual words. The goal is to develop a topic model that associates words and Wikipedia concepts with mixtures of topics. Important steps include collecting a dataset, preprocessing it, performing topic modeling using an LDA model that incorporates Wikipedia concepts, and evaluating the results against models using only unigrams or n-grams. Key benefits noted are the ability to represent topics with more representative concepts and reduce ambiguity compared to models using only individual words.
Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn
Daniel Tunkelang discusses three key areas - relevance, recommendations, and reputation - that LinkedIn is focusing on to improve the user experience. Relevance refers to improving search and discovery of profiles by combining different types of signals. Recommendations match members to jobs, groups, and other connections. Reputation involves identifying member expertise in skills and who the experts are in particular skills. Open problems remain around exploratory search, balancing exploration and exploitation of the data, and incentivizing the development of online reputation.
The Relevance Trap
The document discusses the importance of distinction over being right in marketing and advertising. It argues that most companies focus on attention, intent, and persuasion in their ads but that over 90% fail because they don't prioritize getting attention first through distinction. It provides examples of brands like Swatch, Red Bull, and Mini that achieved distinction and success by redefining product categories. The document encourages marketers to conduct multilens research, challenge assumptions, and find real distinction to escape the trap of copying competitors and being like every other brand.
Personalizing Search at LinkedIn
LinkedIn's vision is to create economic opportunity for the global workforce by connecting members to other members, knowledge, and opportunities through their economic graph. Their search functionality powers searching across people, jobs, companies, and schools, and aims to understand user intent to provide personalized and relevant results. They use a learning to rank approach trained on clickstream data to rank results based on inferred searcher interests and other features. Federated page construction combines results from different verticals and ranks them based on predicted click probabilities learned from past user behavior and intents.
Literature reviewguidlines
The document provides guidelines for conducting a literature review. It discusses the purpose of a literature review as the starting point of any research where both primary and secondary sources are examined on the research topic. It identifies three common types of literature reviews: narrative reviews, systematic reviews, and meta-analyses. The guidelines also outline steps for ensuring a thorough literature review, including developing search strategies, tracking relevant papers, and avoiding common pitfalls like subjective analyses.
Recommended
Project Proposal Topics Modeling (Ir)
This document proposes using Wikipedia concepts to improve topic modeling. It discusses using n-grams like bigrams and phrases from Wikipedia categories rather than only individual words. The goal is to develop a topic model that associates words and Wikipedia concepts with mixtures of topics. Important steps include collecting a dataset, preprocessing it, performing topic modeling using an LDA model that incorporates Wikipedia concepts, and evaluating the results against models using only unigrams or n-grams. Key benefits noted are the ability to represent topics with more representative concepts and reduce ambiguity compared to models using only individual words.
Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn
Daniel Tunkelang discusses three key areas - relevance, recommendations, and reputation - that LinkedIn is focusing on to improve the user experience. Relevance refers to improving search and discovery of profiles by combining different types of signals. Recommendations match members to jobs, groups, and other connections. Reputation involves identifying member expertise in skills and who the experts are in particular skills. Open problems remain around exploratory search, balancing exploration and exploitation of the data, and incentivizing the development of online reputation.
The Relevance Trap
The document discusses the importance of distinction over being right in marketing and advertising. It argues that most companies focus on attention, intent, and persuasion in their ads but that over 90% fail because they don't prioritize getting attention first through distinction. It provides examples of brands like Swatch, Red Bull, and Mini that achieved distinction and success by redefining product categories. The document encourages marketers to conduct multilens research, challenge assumptions, and find real distinction to escape the trap of copying competitors and being like every other brand.
Personalizing Search at LinkedIn
LinkedIn's vision is to create economic opportunity for the global workforce by connecting members to other members, knowledge, and opportunities through their economic graph. Their search functionality powers searching across people, jobs, companies, and schools, and aims to understand user intent to provide personalized and relevant results. They use a learning to rank approach trained on clickstream data to rank results based on inferred searcher interests and other features. Federated page construction combines results from different verticals and ranks them based on predicted click probabilities learned from past user behavior and intents.
Literature reviewguidlines
The document provides guidelines for conducting a literature review. It discusses the purpose of a literature review as the starting point of any research where both primary and secondary sources are examined on the research topic. It identifies three common types of literature reviews: narrative reviews, systematic reviews, and meta-analyses. The guidelines also outline steps for ensuring a thorough literature review, including developing search strategies, tracking relevant papers, and avoiding common pitfalls like subjective analyses.
Query Understanding and Ecommerce
The document discusses query understanding in ecommerce search. It explains that query understanding is critical for relevance and desirability, and search applications should guide users to more specific queries they can better understand. It also discusses modeling queries as "bags of documents" and vice versa to understand similarity between queries and documents. The goal is to recognize queries with the same intent to improve recall, and model documents based on the queries intended to find them.
Semantic Equivalence of e-Commerce Queries
Title:
Semantic Equivalence of e-Commerce Queries
Authors:
Aritra Mandal, Daniel Tunkelang, Zhe Wu
Presented at KDD 2023 Workshop on E-Commerce and Natural Language Processing (ECNLP 2023).
Helping Searchers Satisfice through Query Understanding
Behavioral economics transformed how we think about human decision making, rejecting expected utility maximization for the real world of heuristics, biases, and satisficing. In this talk, I'll argue that our thinking about search engines needs a similar transformation. I will compare the Probability Ranking Principle to expected utility maximization and offer ways that AI can help searchers satisfice through query understanding.
This was an invited talk given at the 2023 Walmart AI Summit.
Speaker Bio
Daniel Tunkelang is an independent consultant specializing in search, machine learning / AI, and data science. He completed undergraduate and master's degrees in Computer Science and Math at MIT and a PhD in computer science at CMU. He was a founding employee and chief scientist of Endeca, a search pioneer that Oracle acquired in 2011. He then led engineering and data science teams at Google and LinkedIn. He has written a book on Faceted Search, and he blogs on Medium about search-related topics — particularly query understanding. He has worked with numerous tech companies, retailers, and others, including Algolia, Apple, Canva, Coupang, eBay, Etsy, Flipkart, Home Depot, Oracle, Pinterest, Salesforce, Target, Yelp, and Zoom.
MMM, Search!
MMM, Search!
An opinionated discussion of search metrics, models, and methods. Presented to the Wikimedia Foundation on April 27, 2020.
About the Speaker
Daniel Tunkelang is an independent consultant specializing in search, discovery, machine learning / AI, and data science.
He was a founding employee of Endeca, a search pioneer that Oracle acquired. After 10 years at Endeca, he moved to Google, where he led a local search team. He then served as a director of data science and search at LinkedIn.
After leaving LinkedIn in 2015, he became an independent consultant. His clients have included Apple, eBay, Coupang, Etsy, Flipkart, Gartner, Pinterest, Salesforce, and Yelp; as well as some of the largest traditional retailers.
Daniel completed undergraduate and master's degrees in Computer Science and Math at MIT and a Ph.D. in computer science at CMU. He wrote a book on Faceted Search, published by Morgan & Claypool, and he blogs on Medium about search-related topics -- particularly about query understanding. He is also active on Twitter, LinkedIn, and Quora.
Enterprise Intelligence
Enterprise Intelligence: Putting the Pieces Together
http://enterpriserelevance.com/kdd2016/keynote.html
These slides are for a keynote presentation delivered at the Workshop on Enterprise Intelligence, held in conjunction with the 22nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD 2016).
About the author:
Daniel Tunkelang is a data science and engineering executive who has built and led some of the strongest teams in the software industry. He studied computer science and math at MIT and has a PhD in computer science from CMU. He was a founding employee and chief scientist of Endeca, a search pioneer that Oracle acquired for $1.1B. He led a local search team at Google. He was a director of data science and engineering at LinkedIn, and he established their query understanding team. Daniel is a widely recognized writer and speaker. He is frequently invited to speak at academic and industry conferences, particularly in the areas of information retrieval, web science, and data science. He has written the definitive textbook on faceted search (now a standard for ecommerce sites), established an annual symposium on human-computer interaction and information retrieval, and authored 24 US patents. His social media posts have attracted over a million page views. Daniel advises and consults for companies that can benefit strategically from his expertise. His clients range from early-stage startups to "unicorn" technology companies like Etsy and Pinterest. He helps companies make decisions around algorithms, technology, product strategy, hiring, and organizational structure.
Query Understanding: A Manifesto
Query understanding is about focusing less on the results and more on the query. It’s about figuring out what the searcher wants, rather than scoring and ranking results. Once you’ve established this mindset, your approach to search changes: you focus on query performance rather than ranking.
Presented at QConSF 2016: https://qconsf.com/sf2016/presentation/query-understanding-manifesto
Where should you put your data scientists?
I delivered this keynote at the Fast Forward Labs Data Leadership Conference on April 28, 2016. You can find related materials in the following publications:
https://www.oreilly.com/ideas/where-should-you-put-your-data-scientists
http://firstround.com/review/doing-data-science-right-your-most-common-questions-answered/
Data Science: A Mindset for Productivity
Data Science: A Mindset for Productivity
Keynote at 2015 Ronin Labs West Coast CTO Summit
https://www.eventjoy.com/e/west-coast-cto-summit-2015
Abstract
Data science isn't just about using a collection of technologies and algorithms. Data science requires a mindset that solves problems at a higher level of abstraction. How do we model utility when we think about optimization? How do we decide which hypotheses to test? How do we allocate our scarce resources to make progress?
There are no silver bullets. But I'll share what I've learned from a variety of contexts over the course of my work at Endeca, Google, and LinkedIn; and I hope you'll leave this talk with some practical wisdom you can apply to your next data science project.
My Three Ex’s: A Data Science Approach for Applied Machine Learning
The document discusses machine learning and introduces "three ex's" - Express, Explain, and Experiment - as a framework.
Express involves defining an objective function and collecting training data to understand utility and inputs. Explain focuses on model explainability over accuracy, using linear models or decision trees initially before upgrading models. Experiment emphasizes that experiments are now cheap, but to test variables disciplined and optimize the speed of learning.
Web science - How is it different?
Web Science: How is it different?
Daniel Tunkelang, LinkedIn
Keynote Address at ACM Web Science 2014 Conference
The scientific method of observation, measurement, and experiment may be our greatest achievement as a species. The technological innovation we enjoy today is the product of a culture of systematized scientific experimentation.
But historically scientific experimentation has been expensive. Experiments consumed natural resources, took a long time to conduct, and required even more time and labor to analyze. In order to be productive, scientists have had to factor these costs into their work and to optimize accordingly.
Web science is different. Not, as some have speciously argued, because big data has made the scientific method obsolete. The key difference is that web science has changed the economics of scientific experimentation. Thus, even as web scientists apply the traditional scientific method, they optimize based on very different economics.
In this talk, I'll survey how web science has changed our approach to experimentation, for better and for worse. Specifically, I'll talk about differences in hypothesis generation, offline analysis, and online testing.
Bio
Daniel Tunkelang is Head of Query Understanding at LinkedIn, where he previously formed and led the product data science team. LinkedIn search allows members to find people, companies, jobs, groups and other content. His team aims to provide users with the best possible results that satisfy their information needs and help to get insights from professional data. Tunkelang has BS and MS degrees in computer science and math from MIT, and a PhD in computer science from CMU. He co-founded the annual symposium on human-computer interaction and information retrieval (HCIR) and wrote the first book on Faceted Search (Morgan and Claypool 2009). Prior to joining LinkedIn, Tunkelang was Chief Scientist of Endeca (acquired by Oracle in 2011 for $1.1B) and leader of the local search quality team at Google, mapping local businesses to their home pages. He is the co-inventor of 20 patents.
Better Search Through Query Understanding
This document discusses query understanding in search engines. It describes how query understanding involves identifying entities and tags in queries, predicting the user's intent or topic area, expanding queries using related terms, and incorporating spelling corrections. The key aspects of query understanding covered are tagging queries for entities like names, titles, companies; predicting the user's vertical intent like jobs, people or companies; and expanding queries using name synonyms, job title synonyms or signals from past user queries and clicks. The document also suggests giving users more transparency, guidance and control over the search process.
Social Search in a Professional Context
Keynote at CIKM 2013 Workshop on Data-driven User Behavioral Modelling and Mining from Social Media
Social Search in a Professional Context
Daniel Tunkelang (LinkedIn)
Social networks bring a new dimension to search. Instead of looking for web pages or text documents, LinkedIn members search a world of entities connected by a rich graph of relationships. Search is a fundamental part of the LinkedIn ecosystem, as it helps our members find and be found. Unlike most search applications, LinkedIn's search experience is highly personalized: two LinkedIn members performing the same search query are likely to see completely different results. Delivering the right results to the right person depends on our ability to leverage our each member's unique professional identity and network. In this talk, I'll describe the kinds of search behavior we see on LinkedIn, and some of the approaches we've taken to help our members address their information needs.
Find and be Found: Information Retrieval at LinkedIn
Shakti Sinha and Daniel Tunkelang discuss how LinkedIn's search functionality works. They explain that LinkedIn search is personalized based on a user's profile and network. Query understanding involves tagging queries to determine entity types like people, companies, or skills. Ranking is also personalized using machine learning models trained on search logs to determine relevance for a specific user's query. The system aims to provide both globally and personally relevant results, as about two-thirds of clicks come from out of a user's network.
Search as Communication: Lessons from a Personal Journey
The document discusses lessons learned from the author's personal journey in search engineering. It covers insights from library science about treating search as an information-seeking context and communicating with users. It also discusses the importance of entity detection and how to leverage corpus features to improve extraction. The author realized that queries vary in difficulty and systems need to recognize this and adapt accordingly. The key takeaway is that search should be treated as a communication problem rather than just a ranking task.
Enterprise Search: How do we get there from here?
Enterprise Search: How Do We Get There From Here?
by Daniel Tunkelang (Head of Query Understanding, LinkedIn)
Keynote at 2013 Enterprise Search Summit
We've been tackling the challenges of enterprise and site search for at least 3 decades. We've succeeded to the point that search is the gateway to many of our information repositories. Nonetheless, users of enterprise search systems are frustrated with these systems' shortcomings. We see this frustration in surveys, but, more importantly, most of us experience it personally in our daily work life. We all dream of a world where searching any information repository is as effective as searching the web—perhaps even more so. A world where we find what we're looking for, or quickly determine that it doesn't exist. Is this Utopia possible? If so, how do we get there from here? Or at least somewhere close? In this talk, Tunkelang reviews the track record of enterprise search. He talks about what's worked and what hasn't, especially as compared to web search. Finally, he proposes some paths to bring us closer to our dream.
--
Daniel Tunkelang is Head of Query Understanding at LinkedIn. Educated at MIT and CMU, he has his career working on big data, addressing key challenges in search, data mining, user interfaces, and network analysis. He co-founded enterprise search and business intelligence pioneer Endeca, where he spent a decade as its Chief Scientist. In 2011, Endeca was acquired by Oracle for over $1B. Previous to LinkedIn, he led a team at Google working on local search quality. Daniel has authored fifteen patents, written a textbook on faceted search, and created the annual symposium on human-computer interaction and information retrieval.
Big Data, We Have a Communication Problem
Big Data, We Have a Communication Problem
by Daniel Tunkelang
Presented on April 30, 2013 at the TTI/Vanguard Conference on Ginormous Systems
http://www.ttivanguard.com/conference/2013/ginormous.html
It's a cliché that we live in a world of Big Data. But the bottleneck in understanding data is not computational. Rather, the biggest challenge is designing technical solutions that effectively leverage human cognitive ability. Data analysis systems should augment people's capabilities rather than replace them. This argument is as old as computer science itself: in 1962, Doug Engelbart said that the goal of technology is “the enhancement of human intellect by increasing the capability of a human to approach a complex problem situation.” Algorithms extract signal from raw data, but people fill in the gaps, creating models and evaluating analyses.
Empowering people to understand data is not just a surface problem of building better interfaces and visualizations. We need to interact with data not only after performing computational analysis, but throughout the analysis process in order to improve our models and algorithms. In order to do so, we need tools and processes specifically designed to offer people transparency, guidance, and control.
Human-computer information retrieval has been revolutionizing our approach to information seeking -- no modern search engine limits users to black-box relevance ranking and ten blue links. We need to take similar steps in our analysis of big data, making people the center of the analysis process and developing the technical innovations that enable people to fulfill this role.
How to Interview a Data Scientist
How To Interview a Data Scientist
Daniel Tunkelang
Presented at the O'Reilly Strata 2013 Conference
Video: https://www.youtube.com/watch?v=gUTuESHKbXI
Interviewing data scientists is hard. The tech press sporadically publishes “best” interview questions that are cringe-worthy.
At LinkedIn, we put a heavy emphasis on the ability to think through the problems we work on. For example, if someone claims expertise in machine learning, we ask them to apply it to one of our recommendation problems. And, when we test coding and algorithmic problem solving, we do it with real problems that we’ve faced in the course of our day jobs. In general, we try as hard as possible to make the interview process representative of actual work.
In this session, I’ll offer general principles and concrete examples of how to interview data scientists. I’ll also touch on the challenges of sourcing and closing top candidates.
Information, Attention, and Trust: A Hierarchy of Needs
Presented by Daniel Tunkelang, LinkedIn Director of Data Science, at Stanford's 2nd annual conference on Computational Social Science (CSS), hosted by Institute for Research in the Social Sciences (IRiSS).
Details at https://iriss.stanford.edu/css/conference-agenda-2013
Data By The People, For The People
Data By The People, For The People
Daniel Tunkelang
Director, Data Science at LinkedIn
Invited Talk at the 21st ACM International Conference on Information and Knowledge Management (CIKM 2012)
LinkedIn has a unique data collection: the 175M+ members who use LinkedIn are also the content those same members access using our information retrieval products. LinkedIn members performed over 4 billion professionally-oriented searches in 2011, most of those to find and discover other people. Every LinkedIn search and recommendation is deeply personalized, reflecting the user's current employment, career history, and professional network. In this talk, I will describe some of the challenges and opportunities that arise from working with this unique corpus. I will discuss work we are doing in the areas of relevance, recommendation, and reputation, as well as the ecosystem we have developed to incent people to provide the high-quality semi-structured profiles that make LinkedIn so useful.
Bio:
Daniel Tunkelang leads the data science team at LinkedIn, which analyzes terabytes of data to produce products and insights that serve LinkedIn's members. Prior to LinkedIn, Daniel led a local search quality team at Google. Daniel was a founding employee of faceted search pioneer Endeca (recently acquired by Oracle), where he spent ten years as Chief Scientist. He has authored fourteen patents, written a textbook on faceted search, created the annual workshop on human-computer interaction and information retrieval (HCIR), and participated in the premier research conferences on information retrieval, knowledge management, databases, and data mining (SIGIR, CIKM, SIGMOD, SIAM Data Mining). Daniel holds a PhD in Computer Science from CMU, as well as BS and MS degrees from MIT.
Content, Connections, and Context
Content, Connections, and Context
Daniel Tunkelang, LinkedIn
Keynote at Workshop on Recommender Systems and the Social Web
At 6th ACM International Conference on Recommender Systems (RecSys 2012)
Recommender systems for the social web combine three kinds of signals to relate the subject and object of recommendations: content, connections, and context.
Content comes first - we need to understand what we are recommending and to whom we are recommending it in order to decide whether the recommendation is relevant. Connections supply a social dimension, both as inputs to improve relevance and as social proof to explain the recommendations. Finally, context determines where and when a recommendation is appropriate.
I'll talk about how we use these three kinds of signals in LinkedIn's recommender systems, as well as the challenges we see in delivering social recommendations and measuring their relevance.
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
A Mix Chart displays historical data of numbers in a graphical or tabular form. The Kalyan Rajdhani Mix Chart specifically shows the results of a sequence of numbers over different periods.
More Related Content
More from Daniel Tunkelang
Query Understanding and Ecommerce
The document discusses query understanding in ecommerce search. It explains that query understanding is critical for relevance and desirability, and search applications should guide users to more specific queries they can better understand. It also discusses modeling queries as "bags of documents" and vice versa to understand similarity between queries and documents. The goal is to recognize queries with the same intent to improve recall, and model documents based on the queries intended to find them.
Semantic Equivalence of e-Commerce Queries
Title:
Semantic Equivalence of e-Commerce Queries
Authors:
Aritra Mandal, Daniel Tunkelang, Zhe Wu
Presented at KDD 2023 Workshop on E-Commerce and Natural Language Processing (ECNLP 2023).
Helping Searchers Satisfice through Query Understanding
Behavioral economics transformed how we think about human decision making, rejecting expected utility maximization for the real world of heuristics, biases, and satisficing. In this talk, I'll argue that our thinking about search engines needs a similar transformation. I will compare the Probability Ranking Principle to expected utility maximization and offer ways that AI can help searchers satisfice through query understanding.
This was an invited talk given at the 2023 Walmart AI Summit.
Speaker Bio
Daniel Tunkelang is an independent consultant specializing in search, machine learning / AI, and data science. He completed undergraduate and master's degrees in Computer Science and Math at MIT and a PhD in computer science at CMU. He was a founding employee and chief scientist of Endeca, a search pioneer that Oracle acquired in 2011. He then led engineering and data science teams at Google and LinkedIn. He has written a book on Faceted Search, and he blogs on Medium about search-related topics — particularly query understanding. He has worked with numerous tech companies, retailers, and others, including Algolia, Apple, Canva, Coupang, eBay, Etsy, Flipkart, Home Depot, Oracle, Pinterest, Salesforce, Target, Yelp, and Zoom.
MMM, Search!
MMM, Search!
An opinionated discussion of search metrics, models, and methods. Presented to the Wikimedia Foundation on April 27, 2020.
About the Speaker
Daniel Tunkelang is an independent consultant specializing in search, discovery, machine learning / AI, and data science.
He was a founding employee of Endeca, a search pioneer that Oracle acquired. After 10 years at Endeca, he moved to Google, where he led a local search team. He then served as a director of data science and search at LinkedIn.
After leaving LinkedIn in 2015, he became an independent consultant. His clients have included Apple, eBay, Coupang, Etsy, Flipkart, Gartner, Pinterest, Salesforce, and Yelp; as well as some of the largest traditional retailers.
Daniel completed undergraduate and master's degrees in Computer Science and Math at MIT and a Ph.D. in computer science at CMU. He wrote a book on Faceted Search, published by Morgan & Claypool, and he blogs on Medium about search-related topics -- particularly about query understanding. He is also active on Twitter, LinkedIn, and Quora.
Enterprise Intelligence
Enterprise Intelligence: Putting the Pieces Together
http://enterpriserelevance.com/kdd2016/keynote.html
These slides are for a keynote presentation delivered at the Workshop on Enterprise Intelligence, held in conjunction with the 22nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD 2016).
About the author:
Daniel Tunkelang is a data science and engineering executive who has built and led some of the strongest teams in the software industry. He studied computer science and math at MIT and has a PhD in computer science from CMU. He was a founding employee and chief scientist of Endeca, a search pioneer that Oracle acquired for $1.1B. He led a local search team at Google. He was a director of data science and engineering at LinkedIn, and he established their query understanding team. Daniel is a widely recognized writer and speaker. He is frequently invited to speak at academic and industry conferences, particularly in the areas of information retrieval, web science, and data science. He has written the definitive textbook on faceted search (now a standard for ecommerce sites), established an annual symposium on human-computer interaction and information retrieval, and authored 24 US patents. His social media posts have attracted over a million page views. Daniel advises and consults for companies that can benefit strategically from his expertise. His clients range from early-stage startups to "unicorn" technology companies like Etsy and Pinterest. He helps companies make decisions around algorithms, technology, product strategy, hiring, and organizational structure.
Query Understanding: A Manifesto
Query understanding is about focusing less on the results and more on the query. It’s about figuring out what the searcher wants, rather than scoring and ranking results. Once you’ve established this mindset, your approach to search changes: you focus on query performance rather than ranking.
Presented at QConSF 2016: https://qconsf.com/sf2016/presentation/query-understanding-manifesto
Where should you put your data scientists?
I delivered this keynote at the Fast Forward Labs Data Leadership Conference on April 28, 2016. You can find related materials in the following publications:
https://www.oreilly.com/ideas/where-should-you-put-your-data-scientists
http://firstround.com/review/doing-data-science-right-your-most-common-questions-answered/
Data Science: A Mindset for Productivity
Data Science: A Mindset for Productivity
Keynote at 2015 Ronin Labs West Coast CTO Summit
https://www.eventjoy.com/e/west-coast-cto-summit-2015
Abstract
Data science isn't just about using a collection of technologies and algorithms. Data science requires a mindset that solves problems at a higher level of abstraction. How do we model utility when we think about optimization? How do we decide which hypotheses to test? How do we allocate our scarce resources to make progress?
There are no silver bullets. But I'll share what I've learned from a variety of contexts over the course of my work at Endeca, Google, and LinkedIn; and I hope you'll leave this talk with some practical wisdom you can apply to your next data science project.
My Three Ex’s: A Data Science Approach for Applied Machine Learning
The document discusses machine learning and introduces "three ex's" - Express, Explain, and Experiment - as a framework.
Express involves defining an objective function and collecting training data to understand utility and inputs. Explain focuses on model explainability over accuracy, using linear models or decision trees initially before upgrading models. Experiment emphasizes that experiments are now cheap, but to test variables disciplined and optimize the speed of learning.
Web science - How is it different?
Web Science: How is it different?
Daniel Tunkelang, LinkedIn
Keynote Address at ACM Web Science 2014 Conference
The scientific method of observation, measurement, and experiment may be our greatest achievement as a species. The technological innovation we enjoy today is the product of a culture of systematized scientific experimentation.
But historically scientific experimentation has been expensive. Experiments consumed natural resources, took a long time to conduct, and required even more time and labor to analyze. In order to be productive, scientists have had to factor these costs into their work and to optimize accordingly.
Web science is different. Not, as some have speciously argued, because big data has made the scientific method obsolete. The key difference is that web science has changed the economics of scientific experimentation. Thus, even as web scientists apply the traditional scientific method, they optimize based on very different economics.
In this talk, I'll survey how web science has changed our approach to experimentation, for better and for worse. Specifically, I'll talk about differences in hypothesis generation, offline analysis, and online testing.
Bio
Daniel Tunkelang is Head of Query Understanding at LinkedIn, where he previously formed and led the product data science team. LinkedIn search allows members to find people, companies, jobs, groups and other content. His team aims to provide users with the best possible results that satisfy their information needs and help to get insights from professional data. Tunkelang has BS and MS degrees in computer science and math from MIT, and a PhD in computer science from CMU. He co-founded the annual symposium on human-computer interaction and information retrieval (HCIR) and wrote the first book on Faceted Search (Morgan and Claypool 2009). Prior to joining LinkedIn, Tunkelang was Chief Scientist of Endeca (acquired by Oracle in 2011 for $1.1B) and leader of the local search quality team at Google, mapping local businesses to their home pages. He is the co-inventor of 20 patents.
Better Search Through Query Understanding
This document discusses query understanding in search engines. It describes how query understanding involves identifying entities and tags in queries, predicting the user's intent or topic area, expanding queries using related terms, and incorporating spelling corrections. The key aspects of query understanding covered are tagging queries for entities like names, titles, companies; predicting the user's vertical intent like jobs, people or companies; and expanding queries using name synonyms, job title synonyms or signals from past user queries and clicks. The document also suggests giving users more transparency, guidance and control over the search process.
Social Search in a Professional Context
Keynote at CIKM 2013 Workshop on Data-driven User Behavioral Modelling and Mining from Social Media
Social Search in a Professional Context
Daniel Tunkelang (LinkedIn)
Social networks bring a new dimension to search. Instead of looking for web pages or text documents, LinkedIn members search a world of entities connected by a rich graph of relationships. Search is a fundamental part of the LinkedIn ecosystem, as it helps our members find and be found. Unlike most search applications, LinkedIn's search experience is highly personalized: two LinkedIn members performing the same search query are likely to see completely different results. Delivering the right results to the right person depends on our ability to leverage our each member's unique professional identity and network. In this talk, I'll describe the kinds of search behavior we see on LinkedIn, and some of the approaches we've taken to help our members address their information needs.
Find and be Found: Information Retrieval at LinkedIn
Shakti Sinha and Daniel Tunkelang discuss how LinkedIn's search functionality works. They explain that LinkedIn search is personalized based on a user's profile and network. Query understanding involves tagging queries to determine entity types like people, companies, or skills. Ranking is also personalized using machine learning models trained on search logs to determine relevance for a specific user's query. The system aims to provide both globally and personally relevant results, as about two-thirds of clicks come from out of a user's network.
Search as Communication: Lessons from a Personal Journey
The document discusses lessons learned from the author's personal journey in search engineering. It covers insights from library science about treating search as an information-seeking context and communicating with users. It also discusses the importance of entity detection and how to leverage corpus features to improve extraction. The author realized that queries vary in difficulty and systems need to recognize this and adapt accordingly. The key takeaway is that search should be treated as a communication problem rather than just a ranking task.
Enterprise Search: How do we get there from here?
Enterprise Search: How Do We Get There From Here?
by Daniel Tunkelang (Head of Query Understanding, LinkedIn)
Keynote at 2013 Enterprise Search Summit
We've been tackling the challenges of enterprise and site search for at least 3 decades. We've succeeded to the point that search is the gateway to many of our information repositories. Nonetheless, users of enterprise search systems are frustrated with these systems' shortcomings. We see this frustration in surveys, but, more importantly, most of us experience it personally in our daily work life. We all dream of a world where searching any information repository is as effective as searching the web—perhaps even more so. A world where we find what we're looking for, or quickly determine that it doesn't exist. Is this Utopia possible? If so, how do we get there from here? Or at least somewhere close? In this talk, Tunkelang reviews the track record of enterprise search. He talks about what's worked and what hasn't, especially as compared to web search. Finally, he proposes some paths to bring us closer to our dream.
--
Daniel Tunkelang is Head of Query Understanding at LinkedIn. Educated at MIT and CMU, he has his career working on big data, addressing key challenges in search, data mining, user interfaces, and network analysis. He co-founded enterprise search and business intelligence pioneer Endeca, where he spent a decade as its Chief Scientist. In 2011, Endeca was acquired by Oracle for over $1B. Previous to LinkedIn, he led a team at Google working on local search quality. Daniel has authored fifteen patents, written a textbook on faceted search, and created the annual symposium on human-computer interaction and information retrieval.
Big Data, We Have a Communication Problem
Big Data, We Have a Communication Problem
by Daniel Tunkelang
Presented on April 30, 2013 at the TTI/Vanguard Conference on Ginormous Systems
http://www.ttivanguard.com/conference/2013/ginormous.html
It's a cliché that we live in a world of Big Data. But the bottleneck in understanding data is not computational. Rather, the biggest challenge is designing technical solutions that effectively leverage human cognitive ability. Data analysis systems should augment people's capabilities rather than replace them. This argument is as old as computer science itself: in 1962, Doug Engelbart said that the goal of technology is “the enhancement of human intellect by increasing the capability of a human to approach a complex problem situation.” Algorithms extract signal from raw data, but people fill in the gaps, creating models and evaluating analyses.
Empowering people to understand data is not just a surface problem of building better interfaces and visualizations. We need to interact with data not only after performing computational analysis, but throughout the analysis process in order to improve our models and algorithms. In order to do so, we need tools and processes specifically designed to offer people transparency, guidance, and control.
Human-computer information retrieval has been revolutionizing our approach to information seeking -- no modern search engine limits users to black-box relevance ranking and ten blue links. We need to take similar steps in our analysis of big data, making people the center of the analysis process and developing the technical innovations that enable people to fulfill this role.
How to Interview a Data Scientist
How To Interview a Data Scientist
Daniel Tunkelang
Presented at the O'Reilly Strata 2013 Conference
Video: https://www.youtube.com/watch?v=gUTuESHKbXI
Interviewing data scientists is hard. The tech press sporadically publishes “best” interview questions that are cringe-worthy.
At LinkedIn, we put a heavy emphasis on the ability to think through the problems we work on. For example, if someone claims expertise in machine learning, we ask them to apply it to one of our recommendation problems. And, when we test coding and algorithmic problem solving, we do it with real problems that we’ve faced in the course of our day jobs. In general, we try as hard as possible to make the interview process representative of actual work.
In this session, I’ll offer general principles and concrete examples of how to interview data scientists. I’ll also touch on the challenges of sourcing and closing top candidates.
Information, Attention, and Trust: A Hierarchy of Needs
Presented by Daniel Tunkelang, LinkedIn Director of Data Science, at Stanford's 2nd annual conference on Computational Social Science (CSS), hosted by Institute for Research in the Social Sciences (IRiSS).
Details at https://iriss.stanford.edu/css/conference-agenda-2013
Data By The People, For The People
Data By The People, For The People
Daniel Tunkelang
Director, Data Science at LinkedIn
Invited Talk at the 21st ACM International Conference on Information and Knowledge Management (CIKM 2012)
LinkedIn has a unique data collection: the 175M+ members who use LinkedIn are also the content those same members access using our information retrieval products. LinkedIn members performed over 4 billion professionally-oriented searches in 2011, most of those to find and discover other people. Every LinkedIn search and recommendation is deeply personalized, reflecting the user's current employment, career history, and professional network. In this talk, I will describe some of the challenges and opportunities that arise from working with this unique corpus. I will discuss work we are doing in the areas of relevance, recommendation, and reputation, as well as the ecosystem we have developed to incent people to provide the high-quality semi-structured profiles that make LinkedIn so useful.
Bio:
Daniel Tunkelang leads the data science team at LinkedIn, which analyzes terabytes of data to produce products and insights that serve LinkedIn's members. Prior to LinkedIn, Daniel led a local search quality team at Google. Daniel was a founding employee of faceted search pioneer Endeca (recently acquired by Oracle), where he spent ten years as Chief Scientist. He has authored fourteen patents, written a textbook on faceted search, created the annual workshop on human-computer interaction and information retrieval (HCIR), and participated in the premier research conferences on information retrieval, knowledge management, databases, and data mining (SIGIR, CIKM, SIGMOD, SIAM Data Mining). Daniel holds a PhD in Computer Science from CMU, as well as BS and MS degrees from MIT.
Content, Connections, and Context
Content, Connections, and Context
Daniel Tunkelang, LinkedIn
Keynote at Workshop on Recommender Systems and the Social Web
At 6th ACM International Conference on Recommender Systems (RecSys 2012)
Recommender systems for the social web combine three kinds of signals to relate the subject and object of recommendations: content, connections, and context.
Content comes first - we need to understand what we are recommending and to whom we are recommending it in order to decide whether the recommendation is relevant. Connections supply a social dimension, both as inputs to improve relevance and as social proof to explain the recommendations. Finally, context determines where and when a recommendation is appropriate.
I'll talk about how we use these three kinds of signals in LinkedIn's recommender systems, as well as the challenges we see in delivering social recommendations and measuring their relevance.
More from Daniel Tunkelang (20)
Helping Searchers Satisfice through Query Understanding
Helping Searchers Satisfice through Query Understanding
My Three Ex’s: A Data Science Approach for Applied Machine Learning
My Three Ex’s: A Data Science Approach for Applied Machine Learning
Find and be Found: Information Retrieval at LinkedIn
Find and be Found: Information Retrieval at LinkedIn
Search as Communication: Lessons from a Personal Journey
Search as Communication: Lessons from a Personal Journey
Information, Attention, and Trust: A Hierarchy of Needs
Information, Attention, and Trust: A Hierarchy of Needs
Recently uploaded
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
A Mix Chart displays historical data of numbers in a graphical or tabular form. The Kalyan Rajdhani Mix Chart specifically shows the results of a sequence of numbers over different periods.
Choosing The Best AWS Service For Your Website + API.pptx
Have you ever been confused by the myriad of choices offered by AWS for hosting a website or an API?
Lambda, Elastic Beanstalk, Lightsail, Amplify, S3 (and more!) can each host websites + APIs. But which one should we choose?
Which one is cheapest? Which one is fastest? Which one will scale to meet our needs?
Join me in this session as we dive into each AWS hosting service to determine which one is best for your scenario and explain why!
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers:
* What is Vector Search?
* Importance and benefits of vector search
* Practical use cases across various industries
* Step-by-step implementation guide
* Live demos with code snippets
* Enhancing LLM capabilities with vector search
* Best practices and optimization strategies
Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications.
#MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology
Introduction of Cybersecurity with OSS at Code Europe 2024
I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems.
The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS.
Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application.
I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.
Taking AI to the Next Level in Manufacturing.pdf
Read Taking AI to the Next Level in Manufacturing to gain insights on AI adoption in the manufacturing industry, such as:
1. How quickly AI is being implemented in manufacturing.
2. Which barriers stand in the way of AI adoption.
3. How data quality and governance form the backbone of AI.
4. Organizational processes and structures that may inhibit effective AI adoption.
6. Ideas and approaches to help build your organization's AI strategy.
Project Management Semester Long Project - Acuity
Acuity is an innovative learning app designed to transform the way you engage with knowledge. Powered by AI technology, Acuity takes complex topics and distills them into concise, interactive summaries that are easy to read & understand. Whether you're exploring the depths of quantum mechanics or seeking insight into historical events, Acuity provides the key information you need without the burden of lengthy texts.
National Security Agency - NSA mobile device best practices
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
ABSTRACT: A prima vista, un mattoncino Lego e la backdoor XZ potrebbero avere in comune il fatto di essere entrambi blocchi di costruzione, o dipendenze di progetti creativi e software. La realtà è che un mattoncino Lego e il caso della backdoor XZ hanno molto di più di tutto ciò in comune.
Partecipate alla presentazione per immergervi in una storia di interoperabilità, standard e formati aperti, per poi discutere del ruolo importante che i contributori hanno in una comunità open source sostenibile.
BIO: Sostenitrice del software libero e dei formati standard e aperti. È stata un membro attivo dei progetti Fedora e openSUSE e ha co-fondato l'Associazione LibreItalia dove è stata coinvolta in diversi eventi, migrazioni e formazione relativi a LibreOffice. In precedenza ha lavorato a migrazioni e corsi di formazione su LibreOffice per diverse amministrazioni pubbliche e privati. Da gennaio 2020 lavora in SUSE come Software Release Engineer per Uyuni e SUSE Manager e quando non segue la sua passione per i computer e per Geeko coltiva la sua curiosità per l'astronomia (da cui deriva il suo nickname deneb_alpha).
Digital Marketing Trends in 2024 | Guide for Staying Ahead
https://www.wask.co/ebooks/digital-marketing-trends-in-2024
Feeling lost in the digital marketing whirlwind of 2024? Technology is changing, consumer habits are evolving, and staying ahead of the curve feels like a never-ending pursuit. This e-book is your compass. Dive into actionable insights to handle the complexities of modern marketing. From hyper-personalization to the power of user-generated content, learn how to build long-term relationships with your audience and unlock the secrets to success in the ever-shifting digital landscape.
June Patch Tuesday
Ivanti’s Patch Tuesday breakdown goes beyond patching your applications and brings you the intelligence and guidance needed to prioritize where to focus your attention first. Catch early analysis on our Ivanti blog, then join industry expert Chris Goettl for the Patch Tuesday Webinar Event. There we’ll do a deep dive into each of the bulletins and give guidance on the risks associated with the newly-identified vulnerabilities.
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
20240607 QFM018 Elixir Reading List May 2024
Everything I found interesting about the Elixir programming ecosystem in May 2024
OpenID AuthZEN Interop Read Out - Authorization
During Identiverse 2024 and EIC 2024, members of the OpenID AuthZEN WG got together and demoed their authorization endpoints conforming to the AuthZEN API
Skybuffer SAM4U tool for SAP license adoption
Manage and optimize your license adoption and consumption with SAM4U, an SAP free customer software asset management tool.
SAM4U, an SAP complimentary software asset management tool for customers, delivers a detailed and well-structured overview of license inventory and usage with a user-friendly interface. We offer a hosted, cost-effective, and performance-optimized SAM4U setup in the Skybuffer Cloud environment. You retain ownership of the system and data, while we manage the ABAP 7.58 infrastructure, ensuring fixed Total Cost of Ownership (TCO) and exceptional services through the SAP Fiori interface.
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays, June 2024 with Maria Copot 20
Recently uploaded (20)
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Google Tech Talk: Reconsidering Relevance
- 1. Reconsidering Relevance Daniel Tunkelang Chief Scientist, Endeca
- 4. but first let’s set the stage
- 5. iconic businesses of the 20 th and 21 st centuries I’m Feeling Lucky
- 6. process and scale orchestration
- 7. but there’s a dark side
- 14. we need more definitions
- 18. a relevance-centric approach information Need query select from results rank using IR model USER: SYSTEM: tf-idf PageRank
- 23. queries are misinterpreted Results 1-10 out of about 344,000,000 for ir
- 24. ranked lists are inefficient
- 26. can we do better?
- 29. human computer information retrieval
- 31. google him!
- 33. rexa him?
- 34. getting better
- 36. tags provide summarization and guidance
- 37. my information need evolves as i learn
- 38. hcir – implementing the vision
- 39. scatter/gather: a search for “star”
- 40. faceted search
- 42. showing the right facets: microwaves
- 43. showing the right facets: ceiling fans
- 45. results-driven clarification before refinement Search : storage
- 46. crowd-sourcing to tag documents
- 47. hcir cheats the precision / recall trade-off recall precision
- 51. one more thing …
- 52. “ Google's mission is to organize the world's information and make it universally accessible and useful.”
Editor's Notes
- Friends, Countrymen, Googlers, I come to bury relevance, not to praise it. Well, that’s overstating the case. But I am here today to challenge your approach to information access, and more importantly to tease out and question its philosophical underpinnings. I realize that I’m singling you out as Googlers for holding a belief that is far more widely held, but you are the standard bearers of relevance. And you invited me. Notes: this presentation was delivered at the Google NYC office on 1/7/09. The title is an allusion to Tefko Saracevic’s article, “Relevance Reconsidered”. If you are interested in learning more about the history of relevance I highly recommend his 2007 Lazerow Memorial Lecture on “Relevance in information science” (http://mediabeast.ites.utk.edu/mediasite4/Viewer/?peid=fb8f84cb-9f82-499f-b12c-9a56ab5cf5ba).