The document discusses various approaches for performing joins in Lucene/Solr, including query-time joins using JoinUtil and index-time joins using block joins. It proposes a new join index approach that stores join mappings in docvalues to enable faster querying compared to JoinUtil while also allowing incremental updates unlike block joins. The approach aims to address issues like slow querying in JoinUtil due to term enumeration and inability to reorder docs in block joins.

1. Approaching Join Index
yet another one join algorithm
Mikhail Khludnev
principal engineer

2. PRIVILEGED AND CONFIDENTIAL
• Grid Dynamics is a Silicon Valley-based leading provider of scalable, next-generation
commerce technology solutions
• Record of outperformance with Tier 1 retail clients
• Fortune 1000 client relationships

3. About Me
● principal engineer at Grid Dynamics
● spoke at few last LuceneRevolutions
● contributed BlockJoin query parser for Solr - {!parent}
● blogged about it at http://blog.griddynamics.com/
● tried to fix threads at DataImportHandler
http://google.com/+MikhailKhludnev

4. You are expected to know
● how Lucene searches/filters
● how it counts facets
● that there are segments
● what is DocValues
● why to join
● RDBMS joins: nested loop join, sort-merge join and hash join.

5. I’m expected to know
● query-time join
● index-time join
● yet another one join

6. Lucene/Solr/Elastic Is Strong
● searching
○ filtering
● analytics
○ facets
○ pivots
○ stats

7. SKU_ID: 13
PROD_ID: 1
TYPE: SKU
SIZE: 8
COLOR:Blue
SKU_ID: 12
PROD_ID: 1
TYPE: SKU
SIZE: 8
COLOR:Green
There is a weakness
● robust joins
○ multiple entities
○ relations
PROD_ID: 1
TYPE: PROD
BRAND: Nike
NAME: Shoes
PRICE: $50
SKU_ID: 11
PROD_ID: 1
TYPE: SKU
SIZE: 7
COLOR: Black

8. Joins in General

9. Joins in General

10. Joins in General
PK=FK

11. Joins in General
PK=FK

12. Joins in General
PK=FK

13. children
Joins in General
1:M
parents

14. Executing Join Query
q

15. Executing Join Query
q

16. Executing Join Query
q

17. Executing Join Query
q
fq

18. Executing Join Query
q
fq

19. Join in General
parents ∩ join-relation ∩ children

20. JoinUtil
q
“25”
“17”
“17”
“25”
“25”
“56”
“56”
“56”
“25”
“4”
“61”
FK[doc#]

21. JoinUtil
q
FK[doc#]
“17”
“17”
“25”
“25”
“56”
“56”
“56”
“25”
“4”
“61”
“25”
“25”
“17”
...

22. JoinUtil
q
PK
“1” →△
…
“17”→△
“25”→△
...
“25”
“17”
...
“25”
“17”
“17”
“25”
“25”
“56”
“56”
“56”
“25”
“4”
“61”
FK[doc#]

23. JoinUtil
q
“1” →△
…
“17”→△
“25”→△
...
“25”
“17”
...
fq
“25”
“17”
“17”
“25”
“25”
“56”
“56”
“56”
“25”
“4”
“61”
FK[doc#]PK

24. Block Join
doc#

25. Block Join
doc#
4
3
2

26. Block Join
doc#1
0
0
1
0
0
1
0
0
1
0

27. Block Join
doc#1
0
0
1
0
0
1
0
0
1
0
q

28. Block Join
doc#1
0
0
1
0
0
1
0
0
1
0
q
fq

29. Block Join
doc#1
0
0
1
0
0
1
0
0
1
0
q
fq

30. Comparison
JoinUtil BlockJoin
searching slow fast
reindexing

31. Comparison
JoinUtil BlockJoin
searching slow fast
reindexing fast slow

32. Comparison
doc#
JoinUtil BlockJoin
searching slow fast
reindexing fast slow

33. Comparison
doc#
JoinUtil BlockJoin
searching slow fast
reindexing fast slow

34. Comparison
doc#
JoinUtil BlockJoin
searching slow fast
reindexing fast slow

35. Comparison
JoinUtil BlockJoin
searching slow < ? < fast
reindexing fast > ? > slow

36. Join Index
q
doc#[doc#]
3
6
0
3
10
0
6
3

37. Join Index
q
doc#[doc#]
3
6
0
3
10
0
6
3

38. Join Index
q
doc#[doc#]
fq 3
6
0
3
10
0
6
3

39. Join Index
q
fq
doc#[doc#]
2
4
1
6
5 10
9
8

40. Join Index
q
doc#[doc#]
fq
2
4
1
6
5 10
9
8

41. Join Index
q
doc#[doc#]
fq
2
4
1
6
5 10
9
8

42. meanwhile… in LUCENE-6352

43. GlobalOrdinalsQuery
“25”
“17”
“17”
“25”
“25”
“56”
“56”
“56”
“25”
“4”
“61”
SortedDocValues

44. GlobalOrdinalsQuery
“25”
“17”
“17”
“25”
“25”
“56”
“56”
“56”
“25”
“4”
“61”
SortedDocValues
0 “17”
1 “25”
2 “4”
3 “56”
4 “61”
Ordinals
0
0
1
1
3
3
3
1
2
4
1

45. GlobalOrdinalsQuery
q
0
0
1
1
3
3
3
1
2
4
1
1
1
0
0
0

46. fq
GlobalOrdinalsQuery
q
0
0
1
1
3
3
3
1
2
4
1
1
1
0
0
0

47. Benchmarking 2.9 M docs
https://github.com/m-khl/lucene-solr/tree/dvjoin-benchmark-5-1
Latency, ms
the bigger the worse
7
28
14
BlockJoin
(i-time)
Join Index
JoinUtil
(q-time)
GloblOrdinals

48. ●
JoinUtil JoinIndex
Global
Ordinals
Block
Join
searching slow fast fast
faster
anyway
reindexing fast uber slow fast slow
<

49. doc#[doc#]
Indexing is still a problem
2
4
3
6
1
0
3
10
0
6
3
5 10
9
8

50. Further Plans 2.0
● incremental join-index update
● perhaps just calculate and cache it
● or put to dedicated index
● join in both directions
● calculate optimal execution plan of segments enumeration
● edge case for benchmark

51. Summary
● Joins in General
● JoinUtil vs Block-join vs GlobalOrdinals
● updatable DocValues
● opportunities for improving query-time joins:
○ eliminate term enum
○ choose lower cardinality side for enumeration
○ GlobalOrdinalsJoin

52. References
● Searching relational content with Lucene's BlockJoinQuery
http://blog.mikemccandless.com/
● Solr Experience: search parent-child relations. Part I
Solr block-join support
http://blog.griddynamics.com/
● https://wiki.apache.org/solr/Join
● http://www.slideshare.net/martijnvg/document-relations
● SOLR-6234 - {!scorejoin }
● LUCENE-6352
● Updatable DocValues Under the Hood
http://shaierera.blogspot.com/
● Subject: How to openIfChanged the most recent merge?
at: java-dev@lucene.apache.org

53. Thanks for Joining us!
https://goo.gl/hjsYZW

54. Off scope

55. Joins’ Zoo in Lucene
True Joins
● query-time join
○ JoinUtil
○ {!join }
○ {!scorejoin } - SOLR-6234
● index-time join aka block-join {!
parent}

56. Joins’ Zoo in Lucene
Workarounds
● term positions/SpanQueries
● FieldCollapsing/Grouping
● term decoration
○ spatial
● multivalue fields
True Joins
● query-time join
○ JoinUtil
○ {!join },
○ {!scorejoin } - SOLR-6234
● index-time join aka block-join {!
parent}

57. Joins’ Zoo in Lucene
Workarounds
● term positions/SpanQueries
● FieldCollapsing/Grouping
● term decoration
○ spatial
● multivalue fields
True Joins
● query-time join
○ JoinUtil
○ {!join },
○ {!scorejoin } - SOLR-6234
● index-time join aka block-join {!
parent}

58. Two phase update problem
Subject: How to openIfChanged the most recent merge?
at: java-dev@lucene.apache.org

59. JoinUtil
● query-time
● indexing is fast
● searching is slow, why?
○ expensive term enum
○ single enumeration order
BlockJoin
● index-time
● reindexing whole block is as expensive as mandatory
● searching is darn fast, however
○ can’t reorder child docs

60. store ref = segment#, doc#
put ref to previous and current segment in DV
when add new segment, join IDs with previous segments
- for parent, just ref to all children docnums
- for children, add plain field refsToSeg:seg#
-
when score parents on some segment
- buffer them with the link refs, then
- intersect buffered link refs with children query on previous segments
- search all segments for refsToSeg:seg#, intersect with children query, obtain
perent ref from DV intersect with buffered