This presentation was provided at the Lucene/Solr Revolution conference in Washington, DC in 2014 by Mikhail Khludnev and discusses how to work with Joins and opportunities for improving query-time joins.

1. Approaching Join Index
yet another one join algorithm
Mikhail Khludnev
principal engineer

2. • Grid Dynamics is a Silicon Valley-based leading provider of scalable, next-generation
commerce technology solutions
• Record of outperformance with Tier 1 retail clients
• Fortune 1000 client relationships
PRIVILEGED AND CONFIDENTIAL

3. About Me
● principal engineer at Grid Dynamics
● spoke at few last LuceneRevolutions
● contributed BlockJoin query parser for Solr - {!parent}
● blogged about it at http://blog.griddynamics.com/
● tried to fix threads at DataImportHandler
http://google.com/+MikhailKhludnev

4. You are expected to know
● how Lucene searches/filters
● how it counts facets
● that there are segments
● what is DocValues
● why to join
● RDBMS joins: nested loop join, sort-merge join and hash join.

5. I’m expected to know
● query-time join
● index-time join
● yet another one join

6. Lucene/Solr Is Strong
● searching
○ filtering
● analytics
○ facets
○ pivots
○ stats

7. Lucene/Solr Is Weak in
● robust joins
○ multiple entities
○ relations

8. Joins in General

9. Joins in General

10. Joins in General
PK=FK

11. Joins in General
PK=FK

12. Joins in General
PK=FK

13. children
Joins in General
1:M
parents

14. Executing Join Query
q

15. Executing Join Query
q

16. Executing Join Query
q

17. Executing Join Query
q
fq

18. Executing Join Query
q
fq

19. Join in General
parents ∩ join-relation ∩ children

20. Join in General
parents ∩ join-relation ∩ children
● input
● output
● enumeration order

21. JoinUtil
q
FK[doc#]
“17”
“17”
“25”
“25”
“56”
“56”
“56”
“25”
“4”
“61”
“25”

22. JoinUtil
q
FK[doc#]
“17”
“17”
“25”
“25”
“56”
“56”
“56”
“25”
“4”
“61”
“25”
“25”
“17”
...

23. JoinUtil
q
PK
“1” →△
…
“17”→△
“25”→△
...
“25”
“17”
...
FK[doc#]
“17”
“17”
“25”
“25”
“56”
“56”
“56”
“25”
“4”
“61”
“25”

24. JoinUtil
q
PK FK[doc#]
“1” →△
…
“17”→△
“25”→△
...
“25”
“17”
...
fq
“17”
“17”
“25”
“25”
“56”
“56”
“56”
“25”
“4”
“61”
“25”

25. Block Join
doc#

26. Block Join
doc#

27. Block Join
1 doc#
0
0
1
0
0
1
0
0
1
0

28. Block Join
1 doc#
0
0
1
0
0
1
0
0
1
0
q

29. Block Join
1 doc#
0
0
1
0
0
1
0
0
1
0
q
fq

30. Block Join
1 doc#
0
0
1
0
0
1
0
0
1
0
q
fq

31. Comparison
JoinUtil BlockJoin
searching slow fast
reindexing

32. Comparison
JoinUtil BlockJoin
searching slow fast
reindexing fast slow

33. Comparison
doc#
JoinUtil BlockJoin
searching slow fast
reindexing fast slow

34. Comparison
doc#
JoinUtil BlockJoin
searching slow fast
reindexing fast slow

35. Comparison
doc#
JoinUtil BlockJoin
searching slow fast
reindexing fast slow

36. Comparison
JoinUtil BlockJoin
searching slow < ? < fast
reindexing fast > ? > slow

37. Join Index
q
doc#[doc#]
3
6
0
3
10
0
6
3

38. Join Index
q
doc#[doc#]
3
6
0
3
10
0
6
3

39. Join Index
q
doc#[doc#]
fq 3
6
0
3
10
0
6
3

40. Join Index
q
fq
doc#[doc#]
4
1
2
6
8
5 10
9

41. Join Index
q
doc#[doc#]
fq
4
1
2
6
8
5 10
9

42. Join Index
q
doc#[doc#]
fq
4
1
2
6
8
5 10
9

43. Benchmarking 2.9 M docs
Latency, ms
the bigger the worse
27
14
10
BlockJoin
(i-time)
Join Index
JoinUtil
(q-time)
https://github.com/m-khl/lucene-solr/tree/dvjoin-benchmark

44. Indexing is still a problem
doc#[doc#]
4
3
6
1
0
3
2
10
0
6
3
8
5 10
9

45. Further Plans
● incremental join-index update
● perhaps just calculate and cache it
● or put to dedicated index
● join in both directions
● calculate optimal execution plan of segments enumeration

46. Summary
● Joins in General
● JoinUtil vs Block-join
● {!scorejoin } - SOLR-6234
● updatable DocValues
● opportunities for improving query-time joins:
○ eliminate term enum
○ choose lower cardinality side for enumeration

47. References
● Searching relational content with Lucene's BlockJoinQuery
http://blog.mikemccandless.com/
● Solr Experience: search parent-child relations. Part I
Solr block-join support
http://blog.griddynamics.com/
● https://wiki.apache.org/solr/Join
● http://www.slideshare.net/martijnvg/document-relations
● https://issues.apache.org/jira/browse/SOLR-6234 {!scorejoin }
● Updatable DocValues Under the Hood
http://shaierera.blogspot.com/
● Subject: How to openIfChanged the most recent merge?
at: java-dev@lucene.apache.org

48. Thanks for Joining us!
http://goo.gl/aYsxiD

49. Off scope

50. Joins’ Zoo in Lucene
True Joins
● query-time join
○ JoinUtil
○ {!join }
○ {!scorejoin } - SOLR-6234
● index-time join aka block-join
{!parent}

51. Joins’ Zoo in Lucene
Workarounds
● term positions/SpanQueries
● FieldCollapsing/Grouping
● term decoration
○ spatial
● multivalue fields
True Joins
● query-time join
○ JoinUtil
○ {!join },
○ {!scorejoin } - SOLR-6234
● index-time join aka block-join
{!parent}

52. Joins’ Zoo in Lucene
Workarounds
● term positions/SpanQueries
● FieldCollapsing/Grouping
● term decoration
○ spatial
● multivalue fields
True Joins
● query-time join
○ JoinUtil
○ {!join },
○ {!scorejoin } - SOLR-6234
● index-time join aka block-join
{!parent}

53. Two phase update problem
Subject: How to openIfChanged the most recent merge?
at: java-dev@lucene.apache.org

54. JoinUtil
● query-time
● indexing is fast
● searching is slow, why?
○ expensive term enum
○ single enumeration order
BlockJoin
● index-time
● reindexing whole block is as expensive as mandatory
● searching is darn fast, however
○ can’t reorder child docs