Solr - introducere

1. SOLR – ce & cum
Oana Brezai

2. 2
ce?
● motor de cautare “full text”
● cautare pe categorii (facets)
● sugestii / corectii
● recomandari
● join multicore-uri
● arhitectura cloud
● moduri de lucru in Java

3. motor de cautare “full text”

4. 4
motor de cautare “full text”
● indexare pe campuri
• schema.xml
• campuri + tipuri
• titlu
• autor
• data
• cuvinte cheie
• continut
• solrconfig.xml
● prelucrare limbaj natural
● sinonime
● stopwords
● extragere radacina

5. 5
motor de cautare “full text”
● indexare pe campuri
● prelucrare limbaj natural
● sinonime
● stopwords
● extragere radacina

6. demo...

7. 7
destinatii de calatorie
http://46.101.251.147:8080
● cautari in site:
● moscova
● calatorie
● pe
● cautare
● ou / oua
● (sinonime perfecte)
ranking!!!ranking!!!

8. intrebare: importanta campurilor

9. 9
motor de cautare “full text”
● cautare pe campuri cu “boosting”
name^80 category^70
categories^60 brand^50
keywords^40 name^30
description^10 all_fields^5 boosting!!!boosting!!!

10. cum?

11. 11
motor de cautare “full text”
● indexare pe campuri
● prelucrare limbaj natural

12. index invers

13. 13
indexare
doc1: cuvant1 cuvant2 cuvant3
doc2: cuvant2 cuvant4 cuvant5
doc3: cuvant5 cuvant6 cuvant1

14. 14
indexare
doc1: cuvant1 cuvant2 cuvant3
doc2: cuvant2 cuvant4 cuvant5
doc3: cuvant5 cuvant6 cuvant1
Index invers:
cuvant1 doc1, doc3→
cuvant2 doc1, doc2→
cuvant3 doc1→
...

15. 15
motor de cautare “full text”
● indexare pe campuri
● prelucrare limbaj natural
● sinonime
● stopwords
● extragere radacina

16. cum?

17. analizoare

18. 18
tokenizatoare si filtre
<analyzer >
<tokenizer
class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory"
words="lang/stopwords_ro.txt" />
<filter class="solr.StemmerOverrideFilterFactory"
dictionary="lang/stemdict_ro.txt" />
<filter class="solr.SnowballPorterFilterFactory"
language="Romanian" />
</analyzer>

19. cum masuram calitatea?

20. 20
precision vs recall
rezultate relevante
total corect
rezultate relevante
total rezultate

21. 21
ce?
● motor de cautare “full text”
● cautare pe categorii (facets)
● sugestii / corectii
● recomandari
● join multicore-uri
● arhitectura cloud
● moduri de lucru in Java

22. 22
cautare pe categorii: “facets”
● simple
● cuvinte: marca, culoare
● numere: marime
● intervale: pret
● filtrare multipla
● pivot
● stats

23. 23
cautare pe categorii: “facets”
● simple
● cuvinte: marca, culoare
● numere: marime
● intervale: pret
● filtrare multipla
● pivot
● stats

24. 24
cautare pe categorii: “facets”
● simple
● cuvinte: marca, culoare
● numere: marime
● intervale: pret
● filtrare multipla
● pivot
● stats

25. 25
facets - pivot
trips:
facet=true
facet.pivot=continent,tara,regiune_oras

26. 26
cautare pe categorii: “facets”
● simple
● cuvinte: marca, culoare
● numere: marime
● intervale: pret
● filtrare multipla
● pivot
● stats

27. 27
facets - stats
http://localhost:8983/solr/eshop/select?
q=ruj
&rows=0
&wt=json&indent=true
&stats=true
&stats.field={!tag=piv1}price
&facet=true
&facet.pivot={!stats=piv1}brand

28. 28
ce?
● motor de cautare “full text”
● cautare pe categorii (facets)
● sugestii / corectii
● recomandari
● join multicore-uri
● arhitectura cloud
● moduri de lucru in Java

29. demo...

30. 30
sugestii - demo
● carrefour: lapt
● carturesti: cartarascu (in loc de cartarescu)
● amazon: cartarascu did you mean…?→

31. cum?

32. 32
sugestii – orientat pe cuvinte
<lst name="suggester">
<str name="name">suggestDictionar</str>
<str name="classname">
org.apache.solr.spelling.suggest.Suggester</str>
<str name="lookupImpl">AnalyzingInfixLookupFactory</str>
<str name="indexPath">suggester_infix_dir_</str>
<str name="field">name_suggest</str>
<str name="suggestAnalyzerFieldType">suggestType</str>
<str name="buildOnCommit">false</str>
<str name="buildOnStartup">false</str>
<str name="highlight">false</str>
</lst>

33. 33
sugestii – orientat pe produse
<lst name="suggester">
<str name="name">suggestDictionar</str>
<str name="classname">org.apache.solr.spelling.suggest.Suggester</str>
<str name="dictionaryImpl">DocumentDictionaryFactory</str>
<str name="lookupImpl">AnalyzingInfixLookupFactory</str>
<str name="indexPath">suggester_infix_dir_name</str>
<str name="field">name</str>
<str name="weightField">price</str>
<str name="payloadField">id</str>
<str name="contextField">code</str>
<str name="suggestAnalyzerFieldType">suggestTypeLc</str>
<str name="buildOnCommit">false</str>
<str name="highlight">false</str>
</lst>

34. 34
corectii
<lst name="spellchecker">
<str name="classname">
solr.DirectSolrSpellChecker</str>
<str name="spellcheckIndexDir">
./spellcheckerDirect</str>
<str name="field">name</str>
<str name="buildOnCommit">false</str>
</lst>

35. 35
ce?
● motor de cautare “full text”
● cautare pe categorii (facets)
● sugestii / corectii
● recomandari
● join multicore-uri
● arhitectura cloud
● moduri de lucru in Java

36. demo...

37. 37
recomandari – demo
● emag: laptop
● produse similare
● produse complementare
● altii au mai cumparat si...

38. 38
recomandari – teorie
● content based
● produse similare (aceeasi categorie)
● collaborative filtering
● cine a cumparat asta, a mai cumparat si…. (alte categorii)
● hybrid
● bazat pe o mai buna cunoastere a utilizatorilor + produselor

39. cum?

40. 40
recomandari
● content based:
● more like this
● collaborative filtering:
● select (streaming)

41. 41
more like this
<requestHandler name="/mlt" class="solr.MoreLikeThisHandler">
<lst name="defaults">
<str name="mlt.fl">name, category,
description</str>
<str name="mlt.mintf">1</str>
<str name="mlt.mindf">1</str>
<str name="mlt.boost">true</str>
<str name="mlt.qf">name^90
category^70 description</str>
</lst>
</requestHandler>

42. 42
recomandari
● content based:
● more like this
● collaborative filtering:
● select (streaming)

43. 43
collaborative filtering
top(n="5",
sort="count(*) desc",
nodes(ratings,
top(n="30",
sort="count(*) desc",
nodes(ratings,
search(ratings, q="userId:305", fl="movieId", sort="movieId asc",
fq="rating:[3 TO 5]", qt="/export"),
walk="movieId->movieId",
gather="userId",
fq="rating:[3 TO 5]",
maxDocFreq="10000",
count(*))),
walk="node->userId",
gather="movieId",
fq="rating:[3 TO 5]",
count(*)))

44. 44
ce?
● motor de cautare “full text”
● cautare pe categorii (facets)
● sugestii / corectii
● recomandari
● join multicore-uri
● arhitectura cloud
● moduri de lucru in Java

45. discutie - multicore

46. 46
join multicore-uri
● core1: produse
● core2: cosuri
● core3: utilizatori
● core4: evenimente
● cosuri
● utilizatori
● produse

47. discutie – solr read

48. 48
citire vs scriere

49. cum?

50. 50
join multicore-uri
● clasic
● cloud (streams)

51. 51
join multicore-uri
● clasic
● join
● nested query
● restrictii
● cloud (streams)

52. 52
join – clasic
select on core_1
{!join fromIndex=core_2
from=core_2_field_1
to=core_1_field_1}
core_2_field_2:value

53. 53
join – clasic nested query
http://localhost:8983/solr/parents/
select?
q=alive:yes AND
_query_:"{!
join
fromIndex=children
from=fatherid to=parentid
v='childname:Tom'}"

54. 54
join multicore-uri
● clasic
● subquery
● nested query
● cloud (streams)

55. 55
join – stream-uri
gatherNodes(movies,
gatherNodes(ratings,
walk="389→userId",
gather="movieId"),
walk="node->id",
gather="title")

56. 56
ce?
● motor de cautare “full text”
● cautare pe categorii (facets)
● sugestii / corectii
● recomandari
● join-uri multicore-uri
● arhitectura cloud
● moduri de lucru in Java

57. 57
arhitectura cloud
● scalabilitate
● disponibilitate
● durabilitate

58. cum?

59. 59
arhitectura cloud
Replica 1_1 Replica 1_2
Replica 2_1 Replica 2_2
Zookeeper
Shard 1
Shard 2

60. 60
ce?
● motor de cautare “full text”
● cautare pe categorii (facets)
● sugestii / corectii
● recomandari
● join-uri multicore-uri
● arhitectura cloud
● moduri de lucru in Java

61. cum?

62. 62
moduri de lucru in java
● similar cu lucrul cu BD clasice
● Spring Data
● SOLRJ
● atentie la versiunile suportate

63. multumesc