The New York Times has had search for a long time but 2018 was the year in which the company engaged with relevance in a deep way. The aim of this talk is to share what we've learned as we've increased our search sophistication and some of the challenges we still face.
Some of the techniques we've adopted in this past year include offline metrics testing, reflective testing, and user engagement metrics. We now have a process in place to quickly get mappings changes out to production. As a team we now also have a vocabulary for talking about relevance and can use it to discuss trade-offs and goals in conjunction with our metrics.
We hope this talk is of use to those who've put off working on search relevance due to fear, uncertainty, or ambivalence. We will talk about how we went from working on everything but search relevance to finally pulling back the curtain on the search system. We hope what we've learned can help others get started.
12. The Final Straw
(headline:bruce OR body:bruce OR byline:bruce)
AND
(headline:lambert OR body:lambert OR byline:lambert)
AND
(headline:hot OR body:hot OR byline:hot)
AND
(headline:dogs OR body:dogs OR byline:dogs)
13. The Final Straw
Museum and Gallery Listings
An autumnal landscape called "Straw Dogs" exudes a stirring
romance, part Sam Peckinpah, part Thomas Hardy.
The lines dividing art, music and film were blurry enough to allow
Laurie Anderson, Bruce Nauman and Yoko Ono, among others, to shi
from object making to performance and back again.
Yvon Lambert, 550 West 21st Street, Chelsea.
The work is tightly installed, text-intensive and hot with
information.
15. Document Scoresuseconomy
The March Job Numbers Tell Us the Economy Is (Still) Doing Fin
What Booming Markets Are Telling Us About the Global Econom
The Terrible, Wonderful, Inscrutable Economy
36.2
32.7
31.8
30. Offline Metrics
Apple Loses $900,000 To Fake iPhone Swaps
appleiphone
2019-04-06
It’s Your iPhone. Why Can’t You Fix It Yourself?
2019-04-06
Photo of iPhone is a fake, but buyers seem real
2006-09-12
An Apple Pie That Lasts for Days
2017-10-20
5
5
2
1
36. OCR
A rEw I.ITEnAttYV181TOR: Zi'elteartllatbir. $gRT , the well- , is to make a
tour in this . Mr. Smith, .has been eminently successful also as a
lecturer; .treat us to descriptions of his recent travels in ~'. In England orer.
ing houses."
46. Summary
• Understand search: You have to build a deep understanding of how search works and what all
these systems are doing when you issue a query.
• Make it possible to iterate: Improving search is about making lots of little improvements. You want
to remove as much friction in this process as possible and get the infrastructure out of your way.
• Know where you're going: You have to know whether your changes will improve things. There is
where offline metrics like precision, NDCG, and ERR can help.
• Learn about your users: Know how your users are using search. Without this, you can never really
know what to focus on.
• Leverage your company: You have colleagues with deep domain expertise. Engage them so you
don't have to reinvent everything.