Better search engine testing for solr

3,441 views

Published on

A refactoring of my earlier presentation targeted more towards Solr, and less on testing. I'm basing my presentation for Lucene EuroCon on this. Would love some more feedback!

Published in: Technology, Design
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
3,441
On SlideShare
0
From Embeds
0
Number of Embeds
11
Actions
Shares
0
Downloads
16
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Better search engine testing for solr

  1. 1. BETTER SEARCH ENGINE TESTING FOR SOLRCVILLE MEETUP | EPUGH@O19S.COM | @DEP4B 1
  2. 2. WHY AM I QUALIFIED TO BE UP HERE?• Principal of OpenSource Connections - Solr Search Consultancy• Member of Apache Software Foundation• Presenter at conferences (OSCON, ApacheCON, UberConf, ExpoQA, STPcon 2009/2011!)• Fascinated by the art of software development 2
  3. 3. 3. X Co So m on ingAUTHOR 3
  4. 4. FATHER 4
  5. 5. AGENDA Why Test Search?Two General Approaches Test Tools for Solr Wrap Up 5
  6. 6. REDUCE HUMAN ERROR 6
  7. 7. photo credit to Jay Grieves @ www.bigswingingdeveloper.com 7
  8. 8. TWO APPROACHES TO TEST Humans Automated Tests 8
  9. 9. HUMAN TESTING• Better deals with ambiguity 9
  10. 10. RELEVANCY IS LIKEPORNOGRAPHY.... 10
  11. 11. I KNOW IT WHEN I SEE IT! http://en.wikipedia.org/wiki/Les_Amants 11
  12. 12. AUTOMATED TESTING 12
  13. 13. SWEET SPOT• Repeat Testing for Relevancy Tuning• Performance Testing: hitting SLA?• Regression Testing (Typical use case for automated testing) 13
  14. 14. “ONE TRUE ANSWER”• Absolute Truth / Matrix / Grid / TREC / Relevancy Assertions • The correct answers for each search are known ahead of time • Humans judges often decide these correct answers, stored as Relevancy Assertions • Can be labor intensive to setup• A “Numerical Grade” is produced for comparision 14
  15. 15. SO WHAT IS A JUDGEMENT LIST? 15
  16. 16. 16
  17. 17. COMPARABLE RESULTS Original AjaxSolr WebSolrgata 4 40.0 80.0 80.0HIF 1 alpha 90.0 100.0 30HIF-1 alpha 100.0 100.0 30HIF1 alpha 70.0 100.0 100.0HIF1A 100.0 100.0 100.0iNOS 60.0 60.0 60.0IRE-1 alpha 30 90.0 90.0 17
  18. 18. WRAP UP 18
  19. 19. Search  Stack User  Interface Search  Engine Informa3on 19
  20. 20. 20
  21. 21. 21
  22. 22. A HYBRID APPROACH? Humans Automated Tests 22
  23. 23. A HYBRID APPROACH? Tool Supported Testing Humans Automated Tests Judgements 23
  24. 24. TESTING IS A HUMAN ACTIVITY James Bach, satisfice.com 24
  25. 25. THANK YOU!• twitter: dep4b• speakerrate: http://www.speakerrate.com/epugh/• email: epugh@opensourceconnections.com 25 25

×