Test Driven Search   DevelopmentAndrew Harvey - rorosyd - 12 July 2011
Introductions•You might remember me telling you  which search engine to use.•Tonight, it hopefully won’t matter what  your...
The ProblemUnscientific development and the lack oftesting lead to unexpected and unseenregressions which reach production.
The Problem    We wanted to    improve search    while still    addressing our    (very) long tail
The Solution?          Tests!
ConceptsRelevanceRecallPrecisionPrecision at K
RelevanceFor the point of this exerciserelevance is a binary value ofwhether a user wants to see thisparticular result for...
RecallHow many of the possible  relevant results for this    query did we return?
PrecisionHow many of the results that we returned were relevant?
Precision at k  How many of the first k  results that we returned            were relevant?
Search Test Harness Helps us ensure that our search changes     don’t cause unexpected regressions.        Presents us wit...
Attribution Ideas shamelessly stolen frompeople much smarter than me.
Basic MethodApplication                            Search Server           Query             Filters Results              ...
Basic Method BadResults             Search    Known Good             Results   Good   Results
The Technology                                        Search Server          Test Harness       ResqueBrowser            R...
Demo
Next Steps•Name it!•Open Source it!•Adaptors!•???•Profit!
ResourcesManning, Raghaven, Schütze:  Introduction to Information Retrieval
Questions?   Andrew Harvey  mootpointer.com   @mootpointer
Upcoming SlideShare
Loading in …5
×

rorosyd - Test Driven Search Development

745 views

Published on

Published in: Technology, Design
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
745
On SlideShare
0
From Embeds
0
Number of Embeds
24
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • rorosyd - Test Driven Search Development

    1. 1. Test Driven Search DevelopmentAndrew Harvey - rorosyd - 12 July 2011
    2. 2. Introductions•You might remember me telling you which search engine to use.•Tonight, it hopefully won’t matter what your search system looks like
    3. 3. The ProblemUnscientific development and the lack oftesting lead to unexpected and unseenregressions which reach production.
    4. 4. The Problem We wanted to improve search while still addressing our (very) long tail
    5. 5. The Solution? Tests!
    6. 6. ConceptsRelevanceRecallPrecisionPrecision at K
    7. 7. RelevanceFor the point of this exerciserelevance is a binary value ofwhether a user wants to see thisparticular result for this query
    8. 8. RecallHow many of the possible relevant results for this query did we return?
    9. 9. PrecisionHow many of the results that we returned were relevant?
    10. 10. Precision at k How many of the first k results that we returned were relevant?
    11. 11. Search Test Harness Helps us ensure that our search changes don’t cause unexpected regressions. Presents us with the challenge of manually tagging relevance for hundreds of queries. Leverages existing data used for faceted browsing to help with this process.
    12. 12. Attribution Ideas shamelessly stolen frompeople much smarter than me.
    13. 13. Basic MethodApplication Search Server Query Filters Results Known Good Test Harness
    14. 14. Basic Method BadResults Search Known Good Results Good Results
    15. 15. The Technology Search Server Test Harness ResqueBrowser Rails App Worker(s) Victim Test Subject App Postgres
    16. 16. Demo
    17. 17. Next Steps•Name it!•Open Source it!•Adaptors!•???•Profit!
    18. 18. ResourcesManning, Raghaven, Schütze: Introduction to Information Retrieval
    19. 19. Questions? Andrew Harvey mootpointer.com @mootpointer

    ×