This document discusses how graph capabilities have evolved in Lucene, Solr, and Elasticsearch. It provides an overview of how full-text search engines started supporting relationship exploration through indexing and scoring related terms. It then explains how Elasticsearch, Solr, and streaming expressions in Solr added more explicit graph traversal capabilities through features like the graph explorer API, GraphQueryParser, and functions like shortestPath and gatherNodes. Examples of using these graph features are also provided.

1. Graphs, graphs everywhere
Zbyszko Papierski, Senior Dev@JIRA Cloud,
T:@ZPapierski
E: zbyszko.papierski@gmail.com
Lucene powered relation exploration

2. Agenda
1. Introduction to Lucene and friends
2. Evolution of data analysis by Solr and Elasticsearch
3. Graph capabilities of Elasticsearch(briefly)
4. Solr - QueryParserPlugin
5. Solr - Streaming Expressions
6. Examples

3. http://bit.do/graphs-src

4. 1. Create a collection
2. Put schema
3. Run feeder

5. Lucene and friends

6. Lucene
Provides mechanism for fast searching of text data - both full-text
search(analyzed data) and exact match(non-analyzed, or docValues)

7. Step one - indexing
{kitty|kitten|cat|cats|kittens|pussy} —> cat
{is} —>
{GORGEOUS!!!} —> gorgeous, pretty, nice, etc.

8. Step one - searching
{very} —> very
{nice} —> nice
{kitty} —> cat
{nice, cat, …} {very, ugly, cat, …}
{very,nice, dog, …}
{very, nice, bear, …}

9. Step one - scoring
{very} —> very
{nice} —> nice
{kitty} —> cat
{nice, cat, …} {very, ugly, cat, …}
{very,nice, dog, …}
{very, nice, bear, …}

10. Winner!
nice and cat score
higher than very and nice
or very and cat
because cat is rarer than very
this is only an example, all cats are nice…

11. Solr
Older, works closer with Lucene

12. Elasticsearch
Newer, but with more toys

13. Waiter, there is a graph in my full-text search engine!
are relations

14. • full text searching
• faceting/aggregation
• statistical
• relationship exploration
How did we get here?

15. 1. Your standard, full-text search
2. TF-IDF-ish relationship sorting
3. It’s already there

16. It’s still your standard Lucene index

17. • From Elasticsearch 2.3
• REST API - /_graph/explore
• visualization for Kibana
• Part of elastic commercial offering (named
from 5.0 X-Pack)
Elasticsearch+Kibana
Plugin for Elasticsearch and Kibana - Graph
picture from: https://www.elastic.co/guide/en/graph/current/graph-introduction.html

18. • Available from Solr 6.0
• experimental feature
• currently, works for single node, single core
applications (due to change)
• no 1st party visualization
• does not track edges of the traversal
Solr
built-in GraphQueryParser
picture from: http://solr.pl/2016/04/25/wizualizacja-grafow-przy-pomocy-solr-6/

19. • Available from Solr 5.5
• experimental feature
• no 1st party visualization
• does track edges of the traversal and level
Solr
built-in Streaming Expressions
picture from: http://solr.pl/2016/04/25/wizualizacja-grafow-przy-pomocy-solr-6/

20. fq={!graph from=email to=friends maxDepth=2}email:"susan.gardner@example.com"

25. Params
traversalFilter
Filter query used to filter out incoming nodes on each iteration

26. Params
returnRoot
Should the root set of documents (found by initial query) be returned. Default: true

27. Params
returnOnlyLeaf
Should only leaf documents be returned. Default: false

28. Streaming Expressions
• New alternative way of creating and processing queries
• allow chaining functions
• also experimental
• graph functions - shortestPath, gatherNodes, scoreNodes

29. Streaming Expressions
example

30. shortestPath
• one of the source functions - function producing tuple stream
• returns shortest path between to given nodes using iterative breadth-first search of the graph

31. shortestPath - params
• collection - collection to perform the search
• from - starting node
• to - ending node
• edge - definition of edge, in format <from-field>=<to_field>
• fq - filter query, which filters out nodes taken into account
• maxDepth - maximal depth of the traversal

32. gatherNodes
• transforms input document stream to stream of accessible, through graph
traversal, documents
• can return edges
• allows nesting functions
• works for multi-collection streams, irregardless of number of cluster nodes
• is also a source function
• currently does not support multivalued fields

33. gatherNodes - params
• collection - collection on which function will be performed
• walk - defines starting nodes and the field, e.g. „zpapierski@atlassian.com->from”
• gather - defines which fields are gathered
• scatter - parameter that can have values(one or both):
• leaves - emits only leaf nodes (outer-most ones)
• branches - emits nodes leading up to leaves (root node is a branch)
• fq - filter query that filters out nodes
• maxDocFreq - every node in the result over this number is filtered out
Aggregations, cross-collection gathering and combining with other streaming expressions
is possible

34. scoreNodes
• Function user only with output of gatherNodes
• Score document relevancy, using TF-IDF formula
• As TF - how often document appeared on graph traversal
• IDF is fetched from documents original collection
• Adds additional field, nodeScore, to the output stream

35. Thank you!