Proposal for nested document support in Lucene

Nested Documents in Lucene,[object Object],High-performance support for parent/child document relations,[object Object],mark@searcharea.co.uk,[object Object]

Problem:,[object Object],The Lucene data model is based on Documents, Fields and Terms. However many real-world data structures cannot be properly represented when collapsed into a single Lucene document.,[object Object],Single,[object Object],Lucene,[object Object],document,[object Object]

Problem: “Cross-matching”,[object Object],When two or more data structures of the same type are jumbled up into a single Lucene field, matching logic becomes confused e.g. >1 qualification in a resume,[object Object],John,[object Object],Name,[object Object],John,[object Object],A1 in Maths,[object Object],A1, E1,[object Object],Grade,[object Object],E1 in Science,[object Object],Subject,[object Object],Maths, Science,[object Object],!,[object Object],False match for query:,[object Object],Grade:A1 AND Subject:Science,[object Object]

Unacceptable solution #1,[object Object],One modeling approach is to store related items in the same field and use proximity operators in queries,[object Object],Name,[object Object],John,[object Object],A1 Maths….E1 Science,[object Object],GradeAndSubject,[object Object],John,[object Object],Example query:,[object Object], “GradeAndSubject:”A1 Science”~2,[object Object],A1 in Maths,[object Object],E1 in Science,[object Object],!,[object Object],Slow,[object Object],!,[object Object],Not scalable with number of fields ,[object Object],[object Object]

Proximity distances must grow.

Only one choice of Analyzer for given field ,[object Object],[object Object]

Solution: Nested Document Queries,[object Object],Nested documents need to be queried using new NestedDocumentQuery class which understands document relationships,[object Object],John,[object Object],Name,[object Object],A1,[object Object],E1,[object Object],Grade,[object Object],Grade,[object Object],docType,[object Object],resume,[object Object],Subject,[object Object],Maths,[object Object],Subject,[object Object],Science,[object Object],New NestedDocumentQuery,[object Object],[object Object]

Reports any matches as a match on the parent document not the child

Super-fast evaluation of joins between child and parent

Requires an indexed field to identify parent documents?,[object Object]

Solution: Example Query,[object Object],Find resume of person called “John” with A1 grade in Maths,[object Object],John,[object Object],Name,[object Object],E1,[object Object],A1,[object Object],resume,[object Object],Grade,[object Object],docType,[object Object],Grade,[object Object],Subject,[object Object],Science,[object Object],Subject,[object Object],Maths,[object Object],The NestedDocumentQuery wrapper simply translates the stream of reported matches from the child-level query criteria into matches on the parent for evaluation of all the parent-level logic,[object Object]

Solution: Join speed,[object Object],Unlike a database, the cost of a join (child to parent) is blisteringly fast,[object Object],3) Find first prior set bit e.g. position #356,670,[object Object],100000100000000100000001000000010000001000010000000001000000100000100001,[object Object],2) Index directly into cached BitSet at position #356,675,[object Object],1) Match reported on document #356,675,[object Object],ParentQuery,[object Object],4) Attribute match to doc #356,670,[object Object],NestedDocumentQuery,[object Object],ChildQuery,[object Object],The BitSet for defining parents is obtained from a Filter and can be cached aggressively with minimal memory cost (one bit per document in the index),[object Object]

Other advantages,[object Object],Parent-child document relationships can also be used to limit child results from any one parent (e.g. efficiently control the max number of pages returned from any one website),[object Object],Nesting levels can be arbitrarily deep ,[object Object],Very powerful multi-child queries possible e.g. find people likely to know person X using resume’s employment histories (multiple employer names/urls and related date-ranges),[object Object]

Proposal for nested document support in Lucene

Proposal for nested document support in Lucene

Recommended

More Related Content

What's hot

What's hot(20)

Viewers also liked

Viewers also liked(12)

Similar to Proposal for nested document support in Lucene

Similar to Proposal for nested document support in Lucene(20)

Recently uploaded

Recently uploaded(20)

Proposal for nested document support in Lucene