This document provides an overview of search quality evaluation for beginners. It introduces the concept of "SearchLand" as a metaphor for the domain of web search engines. The document outlines some basics of how search engines work, including crawling, indexing, and ranking pages. It then discusses challenges in measuring search quality, including evaluating relevance, coverage, diversity, and latency. The document concludes by acknowledging the complexity of search quality and outlining opportunities for continued improvement through metrics and analysis.
SriG Systems is the Best Search Engine Optimization- SEO, Search Engine Marketing & Digital Marketing company. We make sure your website at all times shows up in the right place at the right time using only white hat SEO techniques. Our SEO Consultants are extremely familiar in everything SEO related with thorough knowledge in keyword Research and Density.
A Brief (and Practical) Introduction to Information ArchitectureLouis Rosenfeld
Keynote presentation by Louis Rosenfeld at the Usability and Accessibility for the Web International Seminar; 26 July 2007, Monterrey, Nuevo Leon, Mexico
meta search technology are search engine tools that pass quries on to many other search engines and then summerized all the resultsin one handy interface
SriG Systems is the Best Search Engine Optimization- SEO, Search Engine Marketing & Digital Marketing company. We make sure your website at all times shows up in the right place at the right time using only white hat SEO techniques. Our SEO Consultants are extremely familiar in everything SEO related with thorough knowledge in keyword Research and Density.
A Brief (and Practical) Introduction to Information ArchitectureLouis Rosenfeld
Keynote presentation by Louis Rosenfeld at the Usability and Accessibility for the Web International Seminar; 26 July 2007, Monterrey, Nuevo Leon, Mexico
meta search technology are search engine tools that pass quries on to many other search engines and then summerized all the resultsin one handy interface
Search Engines Demystified. The presentation covers about types of engines, search engine internal, comparative study, indexing, searching, information retrieval, inverted index, clustering, meta search engines, semantic search, search engine optimization, search evaluation, how to do search, search architecture and more.
Learn the essentials to getting your business online. Website Planning includes information on choosing your domain name, choosing a hosting company, tips and tricks for usability, design and measuring success.
Search Analytics: Conversations with Your Customersrichwig
Did you know that the search box on your home page handles half or more of all your visitors requests? What do people search for most often when they visit your Web site? How can you tune your site search -- and your site -- to perform better?
Rich Wiggins presents a talk that he and co-author Lou Rosenfeld prepared, covering the topis of search analytics, Best Bets, and tuning your Web site to match what your customers seek.
Enhance the way people collaborate with documents in SharePoint Haaron Gonzalez
Learn those extra settings we can turn on to enhance the way people collaborate with documents in SharePoint. There is a set of out of the box settings available in a document library that we can configure to provide a friction free experience for document authors and content consumers.
Describes taxonomy and metadata implementation in SharePoint 2007, including other findability topics such as search, navigation and 3rd party add-ons.
How did you find that?! Optimizing your SharePoint content for search Sharon Weaver
This session will provide an overview of how search uses files and the metadata associated with them to create an index and why that matters to users. This session will also provide best practices for architecting and managing content so that they are more easily searchable (and findable) by users.
Users will learn about:
SharePoint search weighting and indexing
SharePoint Metadata management
Content management in SharePoint
Folders in SharePoint (Are they Good or Bad?)
This presentation contains information about the different social features in SharePoint 2010. From the value they provide, to how they can be extended from a development perspective.
Search Engines Demystified. The presentation covers about types of engines, search engine internal, comparative study, indexing, searching, information retrieval, inverted index, clustering, meta search engines, semantic search, search engine optimization, search evaluation, how to do search, search architecture and more.
Learn the essentials to getting your business online. Website Planning includes information on choosing your domain name, choosing a hosting company, tips and tricks for usability, design and measuring success.
Search Analytics: Conversations with Your Customersrichwig
Did you know that the search box on your home page handles half or more of all your visitors requests? What do people search for most often when they visit your Web site? How can you tune your site search -- and your site -- to perform better?
Rich Wiggins presents a talk that he and co-author Lou Rosenfeld prepared, covering the topis of search analytics, Best Bets, and tuning your Web site to match what your customers seek.
Enhance the way people collaborate with documents in SharePoint Haaron Gonzalez
Learn those extra settings we can turn on to enhance the way people collaborate with documents in SharePoint. There is a set of out of the box settings available in a document library that we can configure to provide a friction free experience for document authors and content consumers.
Describes taxonomy and metadata implementation in SharePoint 2007, including other findability topics such as search, navigation and 3rd party add-ons.
How did you find that?! Optimizing your SharePoint content for search Sharon Weaver
This session will provide an overview of how search uses files and the metadata associated with them to create an index and why that matters to users. This session will also provide best practices for architecting and managing content so that they are more easily searchable (and findable) by users.
Users will learn about:
SharePoint search weighting and indexing
SharePoint Metadata management
Content management in SharePoint
Folders in SharePoint (Are they Good or Bad?)
This presentation contains information about the different social features in SharePoint 2010. From the value they provide, to how they can be extended from a development perspective.
International Journal of Computational Engineering Research (IJCER) is dedicated to protecting personal information and will make every reasonable effort to handle collected information appropriately. All information collected, as well as related requests, will be handled as carefully and efficiently as possible in accordance with IJCER standards for integrity and objectivity.
Tin180.com là trang tin tức văn hóa lành mạnh. Các nội dung chính: xã hội, thế giới, thể thao, văn hóa, nghệ thuật, khoa học, thế giới số, đời sống, sức khỏe, cư dân mạng, kinh doanh, ô tô, xe máy, chuyện lạ, giải trí
Welcome to the first live UiPath Community Day Dubai! Join us for this unique occasion to meet our local and global UiPath Community and leaders. You will get a full view of the MEA region's automation landscape and the AI Powered automation technology capabilities of UiPath. Also, hosted by our local partners Marc Ellis, you will enjoy a half-day packed with industry insights and automation peers networking.
📕 Curious on our agenda? Wait no more!
10:00 Welcome note - UiPath Community in Dubai
Lovely Sinha, UiPath Community Chapter Leader, UiPath MVPx3, Hyper-automation Consultant, First Abu Dhabi Bank
10:20 A UiPath cross-region MEA overview
Ashraf El Zarka, VP and Managing Director MEA, UiPath
10:35: Customer Success Journey
Deepthi Deepak, Head of Intelligent Automation CoE, First Abu Dhabi Bank
11:15 The UiPath approach to GenAI with our three principles: improve accuracy, supercharge productivity, and automate more
Boris Krumrey, Global VP, Automation Innovation, UiPath
12:15 To discover how Marc Ellis leverages tech-driven solutions in recruitment and managed services.
Brendan Lingam, Director of Sales and Business Development, Marc Ellis
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
Pushing the limits of ePRTC: 100ns holdover for 100 daysAdtran
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...SOFTTECHHUB
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
Enhancing Performance with Globus and the Science DMZGlobus
ESnet has led the way in helping national facilities—and many other institutions in the research community—configure Science DMZs and troubleshoot network issues to maximize data transfer performance. In this talk we will present a summary of approaches and tips for getting the most out of your network infrastructure using Globus Connect Server.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdfPeter Spielvogel
Building better applications for business users with SAP Fiori.
• What is SAP Fiori and why it matters to you
• How a better user experience drives measurable business benefits
• How to get started with SAP Fiori today
• How SAP Fiori elements accelerates application development
• How SAP Build Code includes SAP Fiori tools and other generative artificial intelligence capabilities
• How SAP Fiori paves the way for using AI in SAP apps
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
The Metaverse and AI: how can decision-makers harness the Metaverse for their...Jen Stirrup
The Metaverse is popularized in science fiction, and now it is becoming closer to being a part of our daily lives through the use of social media and shopping companies. How can businesses survive in a world where Artificial Intelligence is becoming the present as well as the future of technology, and how does the Metaverse fit into business strategy when futurist ideas are developing into reality at accelerated rates? How do we do this when our data isn't up to scratch? How can we move towards success with our data so we are set up for the Metaverse when it arrives?
How can you help your company evolve, adapt, and succeed using Artificial Intelligence and the Metaverse to stay ahead of the competition? What are the potential issues, complications, and benefits that these technologies could bring to us and our organizations? In this session, Jen Stirrup will explain how to start thinking about these technologies as an organisation.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
Removing Uninteresting Bytes in Software FuzzingAftab Hussain
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
2. Charting SearchLand:
search quality for beginners
Valeria de Paiva
Cuil, Inc.
Aug 2010
Check http://www.parc.com/event/934/adventures-in-searchland.html
3. Charting SearchLand
SearchLand? codename for
search in start-up culture
Academic research is a
landscape I understand and
can travel in (if you change the field
you do research, you keep the tools and
the methods)
Searchland feels very foreign
and it needs charting
Even a primitive map is better
than no map at all…
4. Outline
SearchLand
Search engine basics
Measuring IR systems
Measuring search quality
Conclusions...
and Opportunities
5. SearchLand?
“Up until now most search
engine development has
gone on at companies with
little publication of technical
details. This causes search
engine technology to remain
largely a black art and to be
advertising oriented.’’ Brin
and Page, “The anatomy of
a search engine”, 1998
Disclaimer: This talk presents the guesswork of the author.
It does not reflect the views of my employers or practices at work.
Thanks kxdc2007!
6. SearchLand
Twelve years later the
complaint remains…
Gap between
research and practice
widened
Measuring SE quality
is `adversarial
computing’
Many dimensions of
quality: pictures,
snippets, categories,
suggestions, etc
7. SearchLand: Draft Map
Based on slides for Croft,
Metzler and Strohman’s
“Search Engines:
Information Retrieval in
Practice”, 2009 the tutorial
`Web Search Engine
Metrics’ by Dasdan,
Tsioutsiouliklis and
Velipasaoglu for WWW09/10
and Hugh Williams slides!
THANKS GUYS!!…
8. Search engines…
My understanding of search
engines is very basic: I know
when I like results and when I
don’t. (yes, you’ll get the user
perspective here…)
Assuming you're like this too,
some basics, from folks that
know more.
Basic metaphor: search engine
as a librarian in super-hyped
library…
9. Search Engine Basics...
Web search engines don’t
search the web:
They search a copy of the web
They crawl documents from the
web
They index the documents, and
provide a search interface
based on that index
They present short snippets that
allow users to judge relevance
Users click on links to visit the
actual web document
( thanks ACMSIG May 2010)
10. Search Engine Basics
Modules: Crawler
- Indexer
- [content mining]*
- Query analyzer
- Query server
- Index server
- Snippeter
- Ranking
- Webserver, etc…
Why writing your own search engine is hard, Patterson, ACM Q, 2004
Search Engines: Information Retrieval in Practice, Croft, Metzler and
Strohman, Addison Wesley, 2009 picture: Trevor S.
11. Search Engines
Basic architecture
From Web Search Engine Metrics for Measuring User Satisfaction
Tutorial at WWW conference, by Dasdan et al 2009,2010
12. Crawlers
First one needs to crawl the pages.
“you don’t need a lot of thinking to do crawling; you
need bandwidth, so any old CPU will do”.
Patterson, 2004
BUT Writing a crawler isn’t straightforward…
13. Crawlers
Writing/running a crawler
isn’t straightforward…
Must respect robots.txt exclusion standard to limit
which pages should be retrieved
Crawler shouldn’t overload or overvisit sites
Many URLs exist for the same resource
URLs redirect to other resources (often)
Dynamic pages can generate loops, unending lists,
and other traps
URLs are difficult to harvest: some are embedded in
JavaScript scripts, hidden behind forms, and so on
14. Crawlers
Crawlers actually need to
Fetch new resources from new domains or pages
Fetch new resources from existing domains or
pages
Re-fetch existing resources that have changed
Crawl prioritization is essential:
There are far more URLs than available fetching
bandwidth
For large sites, it’s difficult to fetch all resources
Essential to balance re-fetch and discovery
Essential to balance new site exploration with old
site exploration
Snapshot or incremental? How broad? How
seeded? Brin and Page again..
15. Crawler Challenges
Not Found pages often return ok HTTP
codes
DNS failures
A pages can redirect to itself, or into a
cycle
Pages can look different to end-user
browsers and crawlers
Pages can require JavaScript
processing
Pages can require cookies
Pages can be built in non-HTML
environments
Open source solutions exist: Heretrix,
Nutch, UbiCrawler, etc..
16. Processing pages…
Second one needs to convert documents into
index terms… to create an index.
“For indexing, you are doing a lot of I/O and a lot of
thinking/analyzing the page, so the bigger (CPU)
the better.” Patterson, 2004
17. To index or not to index…
Writing an indexer isn’t easy…
You need to decide how much of the page you
index and which `features or signals' you care
about.
Also which kinds of file to index? Text, sure.
Pdfs, jpegs, audio, torrents, videos, Flash???
18. Indexing…
There are hundreds of billions of web pages
It is neither practical nor desirable to index all:
Should remove spam pages
Should remove illegal pages
Should remove malware
Should remove repetitive or duplicate pages
Should remove crawler traps
Should remove pages that no longer exist
Should remove pages that have
substantially changed
Most search engines index in the range of 20
to 50 billion documents, says Williams
How many pages each engine indexes, and how many
pages are on the web are hard research problems
19. Indexing: which are the right pages?
pages that users want (duh…)
pages that are popular in the web link
graph
pages that match queries
pages from popular sites
pages that are clicked on in search
results
pages shown by competitors
pages in the language or market of the
users
pages that are distinct from other pages
pages that change at a moderate rate
The head is stable, The tail consists of billions of
candidate pages with similar scores
20. Indexing…
Store (multiple copies of?) the web in a
document store
Iterate over the document store to choose
documents
Create an index, and ship it to the index
serving nodes
Repeat…
Sounds easy? It isn’t!
[..] build an index. This is tricky. Just don’t do
anything wrong, as the saying goes. One
false step and those billions of pages are
going to take too long to process…
Patterson, 2004
Three words: scale, parallelism, time
21. Indexing: features
How to index pages so that we
can find them?
A feature (signal) is an attribute of a
document that a computer can
detect, and that we think may
indicate relevance.
Some features are obvious, some are
secret sauce and how to use them
is definitely a trade secret for each
engine.
27. Indexing: web features
URL text
Prefer documents that contain query words in the
URL.
www.whitehouse.gov
www.linkedin.com/in/valeriadepaiva
28. Indexing: which features?
Prefer documents that are
authoritative and popular.
HOW?
The random surfer is a
hypothetical user that clicks on
web links at random.
31. PageRank: how can we leverage
the connections?
Think of a gigantic
graph and its
transition matrix
Make the matrix
probabilistic
Those correspond to
the random surfer
choices
Find its principal eigen-
vector= pagerank
PageRank is the proportion of time that a random
surfer would jump to a particular page.
32. Selection and Ranking
Whatever features you will want for RANKING and
SELECTION you should consider when indexing…
Patterson’s advice “The hallowed `index format’ is
not the end of the search engine, it is just the
beginning. It is a tool to see results, so change it and
change it often.” ACM Q, 2004
33. Quality: what do we want to do?
Measure user satisfaction
Optimize for user satisfaction in
each component of the pipeline
Automate all metrics
Discover anomalies
Visualize, mine, and summarize
metrics data
Debug problems automatically
Compare different search
engines?
34. User satisfaction?
User issues a query to
Search Engine: Receives a
list of results
How relevant are these
results? How do we
measure relevance of
results? How effectively
was the information need of
the user met?
How many of the results
retrieved were useful?
Were there useful pages
not retrieved?
35. Evaluating Relevance…
is HARD!
Evaluation is key to effective
and efficient search engines
Effectiveness, efficiency and
cost are related - difficult trade-off
Two main kinds of approach:
IR traditional evaluations,
click data evaluation & log
analysis
Many books on IR, many
patents/trade secrets from
search engines…
36. Traditional IR evaluation
TREC competitions (NIST and
U.S. Department of Defense),
since 1992
Goal: provide the infrastructure
necessary for large-scale
evaluation of text retrieval
methodologies
several tracks, including a Web
track, using ClueWeb09, one
billion webpages
TREC results are baseline, do
they work for search engines?
38. Evaluating Relevance in IR…
Many other measures in
literature, e.g precision-recall
curves, MAP (mean of average
precision over all queries), etc..
For search engines the first
results are much more
important than later ones
A natural alternative is to report
at top 5, etc
Instead of binary judgments
(relevant or not) graded: very
relevant, somewhat relevant,
not relevant…
39. Evaluating Relevance: DCG
Very used measure: DCG
(discounted cumulative gain)
Very good=3, somewhat good=1
and not good=0 penalize good
results in bad positions by
using the discounted measure.
Discounts geometric or log..
If url1=3, url2=1 and url3=0 then
DCG= 3/ 1 + 1/2 + 0/4= 3.5
same urls in opposite order
DCG=0/1+ ½ +3/4= 1.25
40. Evaluating Relevance in IR
Other measures:
Kendall tau coefficient (counts
of preferences..)
Bpref (Buckley and Voorhees,
2004)
Rpref (De Beer and Moens,
2006 graded bpref)
Q-measure (Sakai, 2007)
….
42. Evaluating User Satisfaction…
Salient aspects of user
satisfaction are hard for IR
metrics
Relevance metrics not based on
users experiences or tasks
Some attempts:
Huffman and Hochster (2007)
SIGIR 2009 Workshop on The
Future of IR Evaluation (2009)
Markov models of user frustration
and satisfaction (Hassan et al
2010)
43. Evaluating the whole system…
Relevance metrics not based on users
experiences or tasks
SCoCveCoverCome attempts: Co
Coverage metrics
Query by URL
Query by content (strong queries)
Compute coverage ratio
Issues: URL normalization/ Page template
Problems:
Web growth in general and in dimensions???
Improve copy detection methods quality and performance
Bar-Yossef and Gurevich: Efficient Search Engine Measurements WWW2007
Olston and Najork: Web Crawling, 2010
44. Evaluating the whole system…
Relevance metrics not based on users
experiences or tasks
SCoCveCoverCome attempts: Co
Diversity metrics
Some queries have a single news result, reported by many
Some queries are ambiguous e.g [Stanford]:
Do you want to optimize for the dominant concept or all facets: [jaguar] ?
Implications for interfaces/visualization…
Exclusion of near-duplicates
Problems: measure and summarize diversity
Measure tradeoffs between diversity and relevance [beata keller]
Best presentation of diversity?
Agrawal et al 2009
45. Evaluating the whole system…
Relevance metrics not based on users
experiences or tasks
SCoCveCoverCome attempts: Co
Latency and discovery metrics
Given a set of new pages, how long does it take for them to be served? How many of
them become part of the results?
46. Evaluating the whole system…
Relevance metrics not based on users
experiences or tasks
SCoCveCoverCome attempts: Co
Freshness metrics
Freshness of snippets?
Measuring change of all internet?
47. Evaluating the system: presentation metrics
Relevance metrics not based on users
experiences or tasks
SCoCveCoverCome attempts: Co
User studies. Eye-tracking studies?
Presentation modules: suggestions, spelling
corrections, snippets, tabs, categories, definitions,
images, videos, timelines, maplines, streaming
results, social results, …
How to measure success? How to optimize
relevance? How much is too much? International
differences? How to change interfaces?
48. Conclusions
Relevance is elusive, but
essential.
Improvement requires metrics
and analysis, continuously
Gone over a rough map of the
issues and some proposed
solutions
Many thanks to Dasdan,
Tsioutsiouliklis and
Velipasaoglu, Croft, Metzler
and Strohman, (especially
Strohman!) and Hugh
Williams for slides/pictures.
49. Coda
There are many search engines.
Their results tend to be very
similar. (how similar?)
Are we seeing everything?
Reports estimate we can see
only 15% of the existing web.
Probing the web is mostly
popularity based. You're likely to
see what others have seen
before. But your seeing
increases the popularity of what
you saw, thereby reducing the
pool of available stuff. Vicious or
virtuous circle? How to measure?
50. Summing up
Life in Searchland is very
different. And lots of fun!
“[...] once the search bug gets
you, you'll be back. The problem
isn't getting any easier, and it
needs all the experience anyone
can muster.” Patterson, 2004
Thank You!
51. References
Croft, Metzler and Strohman’s “Search
Engines: Information Retrieval in Practice”,
2009, Addison-Wesley
the tutorial `Web Search Engine Metrics’ by
Dasdan, Tsioutsiouliklis and Velipasaoglu for
WWW09/10, available from http://
www2010.org/www/program/tutorials/
Hugh Williams slides for ACM Data Mining
May 2010
Anna Patterson:
Why Writing Your Own Search Engine Is Hard
ACM Q, 2004
52.
53. Evaluating Relevance in IR:
implicit judgments
Explicit judgments are expensive
A search engine has lots of user interaction
data, like which results were viewed for a
query, which received clicks..
How can we use this data?
Basic statistics from raw observations:
abandonment rate, reformulation rate,
number of queries per session, clicks per
query, mean reciprocal rank of clicked
results, time to first (last) click, etc….
(Joachims 2002, etc)
54. Evaluating Relevance in IR:
direct and indirect clicks
Direct and indirect evaluation by clicks
(Radlinski and Joachims, Carterette and
Jones, 2007)
Model based evaluation (Dupret et al, 2007,
Dupret (2009)
ERR( Expected Reciprocal Rank) unlike
DCG has non-zero probability that user
stops browsing, better correlation with click
metrics, etc…
So far evaluation method rather than USER
satisfaction