Berlin Buzzwords 2013 - How does lucene store your data?

4,373 views

Published on

This presentation was given at Berlin buzzwords 2013 and gives the main ideas used by Lucene to efficiently store an index on disk.

Published in: Technology

Berlin Buzzwords 2013 - How does lucene store your data?

  1. 1. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedHow does Lucenestore your data?Adrien Grand@jpountzApache Lucene/Solr committerSoftware engineer @ Elasticsearch
  2. 2. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedOutline●Segments●What does a segment store?●Improvements since Lucene 4.0
  3. 3. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedSegments
  4. 4. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedSegments●Every segment is a fullyfunctional index●High numbers ofsegments trigger merges●Merge: Copy all live datafrom several segmentsinto a new one
  5. 5. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedSegments●Immutable (up to deletes)● SSD-friendly (no write amplification)● great for caches (including the FS cache)● easy incremental backups●Merged together when they are too many of them● Expunges deleted documents●An IndexReader is a point-in-time view over a fixednumber of segments● Need to reopen to see changes
  6. 6. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedWhat does asegment store?
  7. 7. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedWhat is in a segment?Stores Useful forSegment &Field infosMetadataGetting doc count / indexoptionsLive docs Non-deleted docsExcluding deleted docsfrom resultsInverted indexThe mapping from terms todocs and positions Finding matching docsNorms Index-time boosts ScoringDoc values Any number or (small) bytesSorting, faceting, customscoringStored fields The original doc Result summariesTerm vectors Single doc inverted index Highlighting, MoreLikeThis
  8. 8. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedWhat is in a segment?APIField infos AtomicReader.getFieldInfos()Live docs AtomicReader.getLiveDocs()Inverted index AtomicReader.fields()Norms AtomicReader.getNormValues(String field)Doc values AtomicReader.get*Values(String field)Stored fields AtomicReader.document(int docID, FieldVisitor visitor)Term vectors AtomicReader.getTermVectors()
  9. 9. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedDoc IDs●Lucene gives sequential doc IDs to all documents in asegment, from 0 (inclusive) to AtomicReader.maxDoc()(exclusive)●Uniquely identifies documents inside a segment● ie. if the inverted index API says that document 42matches the term "bbuzz", I can query the storedfields API with the same ID●Allows for efficient storage● doc IDs can be used as ordinals● Small & dense ints are easy to compress
  10. 10. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedDetour: bit packing●Efficient technique to store blocks of small ints● Supports random access● Special case: bits per value = 1 is a bit set●Say you want to store● 5 30 1 1 10 12● Raw data: 6 * 32 = 192 bits● Packed : 6 * 5 = 30 bits (84% size reduction!)00000000000000000000000000000101 = 500000000000000000000000000011110 = 3000000000000000000000000000000001 = 100000000000000000000000000000001 = 100000000000000000000000000001010 = 1000000000000000000000000000001100 = 12
  11. 11. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedFixed-length data●Dense doc IDs are great for single-valued fixed-lengthdata● Store data sequentially● Data for doc N is at offset N * dataLength● Allows for fast and memory-efficient lookups●Live docs (1 bit per value)●Norms (1 byte per value)●Numeric doc values● Blocks with independent numbers of bits per value4096 values 4096 values 4096 values ● Block idx○ docID / 4096● Idx in block○ docID % 4096
  12. 12. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedVariable-length dataend addressesbytes●Binary doc values●Stored fields●Term vectors●Need one level of indirection: store end addresses● Easy to compress since end addresses areincreasing● Only store endAddress - (docID+1) * avgLength
  13. 13. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedString data●Terms index●Sorted (Set) doc values●MemoryPostingsFormat●Suggesterss/1 t a c kr/1o/2pt/4●FST: automaton with weighted arcs○ compact thanks to shared prefixes/suffixes●Stack = 1●Star = 2●Stop = 3●Top = 4o
  14. 14. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedInverted index●Terms index: map a term prefix to a block in the dict○ FST●Terms dictionary: statistics + pointer in postings lists●Postings lists: encodes matching docs in sorted order○ + positions + offsetsOriginal data 1 2 4 11 42 43 (6 * 4 = 32 bytes)Split into blocks of 3(128 in practice)1 2 4 | 11 42 43Delta-encode 1 1 2 | 11 31 1Pack values 3 [1 1 2] | 5 [11 31 1] (1+1+1+2 = 5 bytes)
  15. 15. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedImprovements sinceLucene 4.0
  16. 16. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedImprovements since Lucene 4.0●LUCENE-4399 (4.1): no seek on write●LUCENE-4498 (4.1): terms "pulsed" when freq=1●Compression:● LUCENE-3892 (4.1): postings encoding moved fromvInt to packed ints: smaller & faster!● LUCENE-4226 (4.1): compressed stored fields● LUCENE-4599 (4.2): compressed term vectors● LUCENE-4547 (4.2): better doc values:● blocks of packed ints for numbers● compression of addresses for binary● FST for Sorted (Set)● LUCENE-4936 (4.4): compression for date DV
  17. 17. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedPerformance●http://people.apache.org/~mikemccand/lucenebench/Term.html
  18. 18. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedDetour: LZ4●Super simple, blazing fast compression codec●http://code.google.com/p/lz4/●https://github.com/jpountz/lz4-java●Example● L: literals● R: reference = (offset decrement, length)● 1 2 3 6 7 6 7 6 7 6 7 8 9 1 2 3 6 7 10● L 1 2 3 6 7 R(2,6) L 8 9 R(13,5) L 10
  19. 19. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedDetour: LZ4●https://github.com/ning/jvm-compressor-benchmark
  20. 20. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedTwitter benchmark●Quick benchmark on a Twitter corpus● 160908 tweets● WhitespaceAnalyzerType Indexed Stored Doc valuesTermvectorsid long yes yes - -created_at long - yes numeric -user.name string yes yes sorted -text text yes yes - yes
  21. 21. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedTwitter benchmarkLucene 4.0Lucene 4.4(not released yet)DifferenceInverted index 23.3M 20.5M -12%Norms 157K 157K +0%Doc values 3.4M 3.1M -9%Stored fields 21.2M 15.7M -26%Term vectors 23.5M 15.5M -34%Overall ~71.5M ~55.0M -23%
  22. 22. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedQuestions?

×