Sematext Group, Inc. , profile picture
Uploaded bySematext Group, Inc.
PPTX, PDF10,625 views

Solr Search Engine: Optimize Is (Not) Bad for You

The document provides an in-depth analysis of segment management and merging in Solr, discussing aspects such as segment creation, modification, merging techniques, and performance implications. It explains the significance of immutable segments for search efficiency while also presenting various configuration options for controlling the merging process. Additionally, it includes technical examples of atomic updates and merge policies to optimize indexing and search performance.

Embed presentation

Downloaded 108 times
Optimize Is (Not) Bad For You Deep Dive Into The Segment Merge Abyss Rafał Kuć Sematext Group, Inc.
Agenda • Segments – where, what & how • Writing segments • Modifying segments • Segment merging – what, where, how, why • Force merging • Force merging & SolrCloud • Performance considerations • Specialized merge policies https://github.com/sematext/lr/tree/master/2017/optimize
3 01 Sematext & I cloud metrics logs &
4 01 Solr Collection Architecture Zookeeper
5 01 Solr Collection Architecture Zookeeper SOLR SOLR SOLR SOLR
6 01 Solr Collection Architecture Zookeeper SOLR shard shard SOLR shard shard SOLR shard shard SOLR shard shard
7 01 Solr Shard Architecture TLOG
8 01 Solr Shard Architecture TLOG Segment Segment Segment Segment
9 01 Lucene Segment Segment Info Field Names Stored Field Values Point Values Term Dictionary Term Frequency Term Proximity Normalization Per Document Vals Live Documents
1 01 Inside the Segment – Term Dictionary TERM DOCID lucene <1>, <2> revolution <1>, <2> washington <1> boston <2> _1.tim Doc1 Title: Lucene Revolution Washington, City: Washington D.C Doc2 Title: Lucene Revolution Boston, City: Boston _1.tip
1 01 Inside the Segment – Doc Values Doc1 Title: Lucene Revolution Washington, City: Washington D.C Doc2 Title: Lucene Revolution Boston, City: Boston DOCID FIELD VALUE 1 Title Lucene Revolution Washington 1 City Washington D.C. 2 Title Lucene Revolution Boston 2 City Boston _1.dvd _1.dvm
1 01 Inside the Segment – Stored Fields Doc1 Title: Lucene Revolution Washington, City: Washington D.C Doc2 Title: Lucene Revolution Boston, City: Boston DOCID VALUE 1 Title: Lucene Revolution Washington City: Washington D.C 2 Title: Lucene Revolution Boston City: Boston _1.fdx _1.fdt
1 01 Inside the Segment – Compound File System _1.fdt _1.fdx _1.fnm _1.nvd _1.nvm _1.si _1.Lucene50_0.doc _1.Lucene50_0.pos _1.Lucene50_0.tim _1.Lucene50_0.tip _1.Lucene50_0.dvd _1.Lucene50_0.dvm
1 01 Inside the Segment – Compound File System _1.fdt _1.fdx _1.fnm _1.nvd _1.nvm _1.si _1.Lucene50_0.doc _1.Lucene50_0.pos _1.Lucene50_0.tim _1.Lucene50_0.tip _1.Lucene50_0.dvd _1.Lucene50_0.dvm
1 01 Inside the Segment – Compound File System _1.fdt _1.fdx _1.fnm _1.nvd _1.nvm _1.si _1.Lucene50_0.doc _1.Lucene50_0.pos _1.Lucene50_0.tim _1.Lucene50_0.tip _1.Lucene50_0.dvd _1.Lucene50_0.dvm _2.cfs _2.cfe
1 01 Indexing
1 01 Indexing
1 01 Indexing
1 01 Indexing level/tier
2 01 Indexing
2 01 Indexing
2 01 Indexing
2 01 Indexing
2 01 Indexing
2 01 Indexing
2 01 Indexing
2 01 Deletes
2 01 Deletes – After Merge
2 01 Atomic Updates $ curl -XPOST -H 'Content-Type: application/json' 'http://localhost:8983/solr/lr/update?commit=true' --data-binary '[ { "id" : "3", "tags" : { "add" : [ "solr" ] } } ]' retrieve document { "id" : 3, "tags" : [ "lucene" ], "awesome" : true }
3 01 Atomic Updates $ curl -XPOST -H 'Content-Type: application/json' 'http://localhost:8983/solr/lr/update?commit=true' --data-binary '[ { "id" : "3", "tags" : { "add" : [ "solr" ] } } ]' { "id" : 3, "tags" : [ "lucene", "solr" ], "awesome" : true } apply changes
3 01 Atomic Updates $ curl -XPOST -H 'Content-Type: application/json' 'http://localhost:8983/solr/lr/update?commit=true' --data-binary '[ { "id" : "3", "tags" : { "add" : [ "solr" ] } } ]' { "id" : 3, "tags" : [ "lucene", "solr" ], "awesome" : true } delete old document
3 01 Atomic Updates $ curl -XPOST -H 'Content-Type: application/json' 'http://localhost:8983/solr/lr/update?commit=true' --data-binary '[ { "id" : "3", "tags" : { "add" : [ "solr" ] } } ]' { "id" : 3, "tags" : [ "lucene", "solr" ], "awesome" : true }
3 01 Atomic Updates – In Place Works on top of numeric, doc values based fields Fields need to be not indexed and not stored Doesn’t require delete/index Support only inc and set modifers $ curl -XPOST -H 'Content-Type: application/json' 'http://localhost:8983/solr/lr/update?commit=true' --data-binary '[ { "id" : "3", "views" : { "inc" : 100 } } ]'
3 01 Atomic Updates – In Place $ curl -XPOST -H 'Content-Type: application/json' 'http://localhost:8983/solr/lr/update?commit=true' --data-binary '[ { "id" : "3", "views" : { "inc" : 100 } } ]' retrieve document { "id" : 3, "tags" : [ "lucene", "solr" ], "awesome" : true }
3 01 Atomic Updates – In Place $ curl -XPOST -H 'Content-Type: application/json' 'http://localhost:8983/solr/lr/update?commit=true' --data-binary '[ { "id" : "3", "views" : { "inc" : 100 } } ]' { "id" : 3, "tags" : [ "lucene", "solr" ], "awesome" : true, "views" : 100 } apply changes
3 01 Atomic Updates – In Place $ curl -XPOST -H 'Content-Type: application/json' 'http://localhost:8983/solr/lr/update?commit=true' --data-binary '[ { "id" : "3", "views" : { "inc" : 100 } } ]' { "id" : 3, "tags" : [ "lucene", "solr" ], "awesome" : true, "views" : 100 } update doc values
3 01 Search – Importance of Segments Immutable – write once read many
3 01 Search – Importance of Segments Immutable – write once read many More segments – slower search speed
3 01 Search – Importance of Segments Immutable – write once read many More segments – slower search speed Fewer segments – faster searches
4 01 Search – Importance of Segments Immutable – write once read many More segments – slower search speed Fewer segments – faster searches Fewer segments – smaller shard size
4 01 Search – Importance of Segments Immutable – write once read many More segments – slower search speed Fewer segments – faster searches Fewer segments – smaller shard size Rapid segment changes – worse I/O cache usage
4 01 Taking Control Merge Policy Factory <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory"> <int name="maxMergeAtOnce">10</int> <int name="maxMergeAtOnceExplicit">30</int> <int name="segmentsPerTier">10</int> <int name="floorSegmentMB">2048</int> <int name="maxMergedSegmentMB">5120</int> <double name="noCFSRatio">0.1</double> <int name="maxCFSSegmentSizeMB">2048</int> <double name="reclaimDeletesWeight">2.0</double> <double name="forceMergeDeletesPctAllowed">10.0</double> </mergePolicyFactory>
4 01 Taking Control Merge Policy Factory <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory"> <int name="maxMergeAtOnce">10</int> <int name="maxMergeAtOnceExplicit">30</int> <int name="segmentsPerTier">10</int> <int name="floorSegmentMB">2048</int> <int name="maxMergedSegmentMB">5120</int> <double name="noCFSRatio">0.1</double> <int name="maxCFSSegmentSizeMB">2048</int> <double name="reclaimDeletesWeight">2.0</double> <double name="forceMergeDeletesPctAllowed">10.0</double> </mergePolicyFactory> Merge Scheduler <mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler" />
4 01 Taking Control Merge Policy Factory <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory"> <int name="maxMergeAtOnce">10</int> <int name="maxMergeAtOnceExplicit">30</int> <int name="segmentsPerTier">10</int> <int name="floorSegmentMB">2048</int> <int name="maxMergedSegmentMB">5120</int> <double name="noCFSRatio">0.1</double> <int name="maxCFSSegmentSizeMB">2048</int> <double name="reclaimDeletesWeight">2.0</double> <double name="forceMergeDeletesPctAllowed">10.0</double> </mergePolicyFactory> Merge Scheduler <mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler" /> Segment Warmer <mergedSegmentWarmer class="org.apache.lucene.index.SimpleMergedSegmentWarmer" />
4 01 Taking Control – Default Indexing Throughput Merge Policy Factory <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory"> <int name="maxMergeAtOnce">10</int> <int name="maxMergeAtOnceExplicit">30</int> <int name="segmentsPerTier">10</int> <int name="floorSegmentMB">2048</int> <int name="maxMergedSegmentMB">5120</int> <double name="noCFSRatio">0.1</double> <int name="maxCFSSegmentSizeMB">2048</int> <double name="reclaimDeletesWeight">2.0</double> <double name="forceMergeDeletesPctAllowed">10.0</double> </mergePolicyFactory>
4 01 Taking Control – Default Indexing Throughput throughput < 5k/sec @ ~14GB
4 01 Taking Control – Max Merged Segment Size Merge Policy Factory <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory"> <int name="maxMergeAtOnce">10</int> <int name="maxMergeAtOnceExplicit">30</int> <int name="segmentsPerTier">10</int> <int name="floorSegmentMB">2048</int> <int name="maxMergedSegmentMB">5120</int> <double name="noCFSRatio">0.1</double> <int name="maxCFSSegmentSizeMB">2048</int> <double name="reclaimDeletesWeight">2.0</double> <double name="forceMergeDeletesPctAllowed">10.0</double> </mergePolicyFactory> Lower higher indexing throughput – smaller segments Higher better search latency (depends) – more merges
4 01 Taking Control – Lowering Max Merged Size Merge Policy Factory <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory"> <int name="maxMergeAtOnce">10</int> <int name="maxMergeAtOnceExplicit">30</int> <int name="segmentsPerTier">10</int> <int name="floorSegmentMB">2048</int> <int name="maxMergedSegmentMB">512</int> <double name="noCFSRatio">0.1</double> <int name="maxCFSSegmentSizeMB">2048</int> <double name="reclaimDeletesWeight">2.0</double> <double name="forceMergeDeletesPctAllowed">10.0</double> </mergePolicyFactory>
4 01 Taking Control – Lowering Max Segment Size throughput < 5k/sec @ ~15.5GB 11% throughput increase
5 01 Taking Control – Merge At Once Merge Policy Factory <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory"> <int name="maxMergeAtOnce">10</int> <int name="maxMergeAtOnceExplicit">30</int> <int name="segmentsPerTier">10</int> <int name="floorSegmentMB">2048</int> <int name="maxMergedSegmentMB">5120</int> <double name="noCFSRatio">0.1</double> <int name="maxCFSSegmentSizeMB">2048</int> <double name="reclaimDeletesWeight">2.0</double> <double name="forceMergeDeletesPctAllowed">10.0</double> </mergePolicyFactory> Lower better search latency (depends) Higher higher indexing throughput
5 01 Taking Control – Lowering Merge At Once Merge Policy Factory <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory"> <int name="maxMergeAtOnce">2</int> <int name="maxMergeAtOnceExplicit">30</int> <int name="segmentsPerTier">10</int> <int name="floorSegmentMB">2048</int> <int name="maxMergedSegmentMB">5120</int> <double name="noCFSRatio">0.1</double> <int name="maxCFSSegmentSizeMB">2048</int> <double name="reclaimDeletesWeight">2.0</double> <double name="forceMergeDeletesPctAllowed">10.0</double> </mergePolicyFactory>
5 01 Taking Control – Lowering Merge At Once throughput < 5k/sec @ ~13GB 8% throughput decrease
5 01 Taking Control – Merge At Once Explicit Merge Policy Factory <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory"> <int name="maxMergeAtOnce">10</int> <int name="maxMergeAtOnceExplicit">30</int> <int name="segmentsPerTier">10</int> <int name="floorSegmentMB">2048</int> <int name="maxMergedSegmentMB">5120</int> <double name="noCFSRatio">0.1</double> <int name="maxCFSSegmentSizeMB">2048</int> <double name="reclaimDeletesWeight">2.0</double> <double name="forceMergeDeletesPctAllowed">10.0</double> </mergePolicyFactory> Controls number of segments merged at once during force merge
5 01 Taking Control – Segments Per Tier Merge Policy Factory <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory"> <int name="maxMergeAtOnce">10</int> <int name="maxMergeAtOnceExplicit">30</int> <int name="segmentsPerTier">10</int> <int name="floorSegmentMB">2048</int> <int name="maxMergedSegmentMB">5120</int> <double name="noCFSRatio">0.1</double> <int name="maxCFSSegmentSizeMB">2048</int> <double name="reclaimDeletesWeight">2.0</double> <double name="forceMergeDeletesPctAllowed">10.0</double> </mergePolicyFactory> Lower value means more merging, but less segments Along with maxMergeAtOnce can smoothen I/O spikes For better indexing throughput set maxMergeAtOnce < segmentsPerTier
5 01 Taking Control – Combined Together Merge Policy Factory <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory"> <int name="maxMergeAtOnce">30</int> <int name="maxMergeAtOnceExplicit">30</int> <int name="segmentsPerTier">30</int> <int name="floorSegmentMB">2048</int> <int name="maxMergedSegmentMB">512</int> <double name="noCFSRatio">0.1</double> <int name="maxCFSSegmentSizeMB">2048</int> <double name="reclaimDeletesWeight">2.0</double> <double name="forceMergeDeletesPctAllowed">10.0</double> </mergePolicyFactory>
5 01 Taking Control – Combined Together throughput < 5k/sec @ ~15GB but look at read difference
5 01 Taking Control – Default vs Combined Read/Write default settings
5 01 Taking Control – Default vs Combined Read/Write default settings combined changes settings
5 01 Taking Control – Reclaim Deletes Weight Merge Policy Factory <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory"> <int name="maxMergeAtOnce">10</int> <int name="maxMergeAtOnceExplicit">30</int> <int name="segmentsPerTier">10</int> <int name="floorSegmentMB">2048</int> <int name="maxMergedSegmentMB">5120</int> <double name="noCFSRatio">0.1</double> <int name="maxCFSSegmentSizeMB">2048</int> <double name="reclaimDeletesWeight">2.0</double> <double name="forceMergeDeletesPctAllowed">10.0</double> </mergePolicyFactory> Controls importance of merging segments with deleted documents Increase to put priority on merging segments with deleted documents
6 01 Taking Control – No CFS Ratio Merge Policy Factory <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory"> <int name="maxMergeAtOnce">10</int> <int name="maxMergeAtOnceExplicit">30</int> <int name="segmentsPerTier">10</int> <int name="floorSegmentMB">2048</int> <int name="maxMergedSegmentMB">5120</int> <double name="noCFSRatio">0.1</double> <int name="maxCFSSegmentSizeMB">2048</int> <double name="reclaimDeletesWeight">2.0</double> <double name="forceMergeDeletesPctAllowed">10.0</double> </mergePolicyFactory> Controls compound file system segments ratio To completely disable CFS set to 0.0
6 01 Taking Control – Merge Scheduler Controls maximum number of concurrent merges Merge Scheduler <mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler"> <int name="maxMergeCount">4</int> <int name="maxThreadCount">4</int> </mergeScheduler>
6 01 Taking Control – Merge Scheduler Controls number of threads dedicated to merging Merge Scheduler <mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler"> <int name="maxMergeCount">4</int> <int name="maxThreadCount">4</int> </mergeScheduler>
6 01 Taking Control – Merge Scheduler Controls number of threads dedicated to merging For spinning drives set maxThreadCount to 1 Merge Scheduler <mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler"> <int name="maxMergeCount">4</int> <int name="maxThreadCount">4</int> </mergeScheduler>
6 01 Taking Control – Merge Scheduler Controls number of threads dedicated to merging For spinning drives set maxThreadCount to 1 For SSD set maxThreadCount to min(4, #CPUs / 2) Merge Scheduler <mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler"> <int name="maxMergeCount">4</int> <int name="maxThreadCount">4</int> </mergeScheduler>
6 01 Optimize aka Force Merge Forces segment merge – usually very expensive
6 01 Optimize aka Force Merge Forces segment merge – usually very expensive Desired number of segments can be specified
6 01 Optimize aka Force Merge Forces segment merge – usually very expensive Desired number of segments can be specified Done on all shards at the same time (by default)
6 01 Optimize aka Force Merge Forces segment merge – usually very expensive Desired number of segments can be specified Done on all shards at the same time (by default) Can be very bad or very good – depending on the use case
6 01 Optimize aka Force Merge Forces segment merge – usually very expensive Desired number of segments can be specified Done on all shards at the same time (by default) Can be very bad or very good – depending on the use case $ curl 'http://solr:8983/solr/lr/update?optimize=true&numSegments=1&waitFlush=false'
7 01 Force Merge – The Good Improves search speed (fewer segments)
7 01 Force Merge – The Good Improves search speed (fewer segments) Removes deleted documents
7 01 Force Merge – The Good Improves search speed (fewer segments) Removes deleted documents Shrinks the index by pruning duplicated data
7 01 Force Merge – The Good Improves search speed (fewer segments) Removes deleted documents Shrinks the index by pruning duplicated data Reduces number of used files
7 01 Force Merge – The Bad Invalidates operating system I/O cache
7 01 Force Merge – The Bad Invalidates operating system I/O cache Very expensive to perform – rewrites all segments
7 01 Force Merge – The Bad Invalidates operating system I/O cache Very expensive to perform – rewrites all segments Not efficient on changing data
7 01 Force Merge – The Bad Invalidates operating system I/O cache Very expensive to perform – rewrites all segments Not efficient on changing data May cause performance issues
7 01 Force Merge – The Bad Invalidates operating system I/O cache Very expensive to perform – rewrites all segments Not efficient on changing data May cause performance issues Will cause temporary increase of disk usage (up to 3x)
7 01 Force Merge – SolrCloud Performance Example
8 01 Force Merge – SolrCloud Performance Example
8 01 Force Merge – Legacy Index on the master server Solr Master Solr Slave Solr Slave Solr Slave index Documents
8 01 Force Merge – Legacy Index on the master server Force merge on the master server Solr Master Solr Slave Solr Slave Solr Slave force merge
8 01 Force Merge – Legacy Index on the master server Force merge on the master server Replicate after optimize is done Solr Master Solr Slave Solr Slave Solr Slave pull after optimize
8 01 Force Merge – SolrCloud (Solr 7 – pull replicas) Create collection Force merge Solr will do the rest Solr Solr Solr Solr Primary 1 Primary 2 Pull Replica 2 Pull Replica 1
8 01 Force Merge – SolrCloud (NRT, pre 7.0) Ask yourself if you really need force merge Solr Solr Solr Solr
8 01 Force Merge – SolrCloud (NRT replicas, pre 7.0) Ask yourself if you really need force merge Create collection on part of the nodes Solr Solr Solr Solr Primary 1 Primary 2
8 01 Force Merge – SolrCloud (NRT replicas, pre 7.0) Ask yourself if you really need force merge Create collection on part of the nodes Index Solr Solr Solr Solr Primary 1 Primary 2 DocumentsDocuments Documents Documents
8 01 Force Merge – SolrCloud (NRT replicas, pre 7.0) Ask yourself if you really need force merge Create collection on part of the nodes Index Force merge Solr Solr Solr Solr Primary 1 Primary 2optimize
8 01 Force Merge – SolrCloud (NRT replicas, pre 7.0) Ask yourself if you really need force merge Create collection on part of the nodes Index Force merge Create replicas Solr Solr Solr Solr Primary 1 Primary 2 Replica 2 Replica 1
9 01 Specialized Merge Policy Example – Sorting Sorting Merge Policy Factory Example <mergePolicyFactory class="org.apache.solr.index.SortingMergePolicyFactory"> <str name="sort">timestamp desc</str> <str name="wrapper.prefix">inner</str> <str name="inner.class">org.apache.solr.index.TieredMergePolicyFactory</str> <int name="inner.maxMergeAtOnce">10</int> <int name="inner.segmentsPerTier">10</int> <double name="inner.noCFSRatio">0.1</double> </mergePolicyFactory>
9 01 Specialized Merge Policy Example – Sorting Sorting Merge Policy Factory Example <mergePolicyFactory class="org.apache.solr.index.SortingMergePolicyFactory"> <str name="sort">timestamp desc</str> <str name="wrapper.prefix">inner</str> <str name="inner.class">org.apache.solr.index.TieredMergePolicyFactory</str> <int name="inner.maxMergeAtOnce">10</int> <int name="inner.segmentsPerTier">10</int> <double name="inner.noCFSRatio">0.1</double> </mergePolicyFactory> Pre-sorts data during merge for: - faster range queries - faster data retrieval - possibility of early query termination - convenient for time based data
9 01 http://sematext.com/jobs You love like we do? You want to work with ? Want to work with open source? You want to do fun stuff?
9 01 Get in touch Rafał rafal.kuc@sematext.com @kucrafal http://sematext.com @sematext http://sematext.com/jobs Come talk to us at the booth
Thank You

More Related Content

PDF
Physical Plans in Spark SQL
byDatabricks
 
PDF
Introduction to elasticsearch
bypmanvi
 
PPTX
Solrcloud Leader Election
byravikgiitk
 
PDF
Introducing DataFrames in Spark for Large Scale Data Science
byDatabricks
 
PDF
OSMC 2021 | Introduction into OpenSearch
byNETWAYS
 
PDF
Scaling paypal workloads with oracle rac ss
byAnil Nair
 
PDF
Introduction to elasticsearch
byhypto
 
PPT
Using galera replication to create geo distributed clusters on the wan
byCodership Oy - Creators of Galera Cluster
 
Physical Plans in Spark SQL
byDatabricks
 
Introduction to elasticsearch
bypmanvi
 
Solrcloud Leader Election
byravikgiitk
 
Introducing DataFrames in Spark for Large Scale Data Science
byDatabricks
 
OSMC 2021 | Introduction into OpenSearch
byNETWAYS
 
Scaling paypal workloads with oracle rac ss
byAnil Nair
 
Introduction to elasticsearch
byhypto
 
Using galera replication to create geo distributed clusters on the wan
byCodership Oy - Creators of Galera Cluster
 

What's hot

PDF
Oracle RAC 19c: Best Practices and Secret Internals
byAnil Nair
 
PPTX
(Re)Indexing Large Repositories in Alfresco
byAngel Borroy López
 
PDF
ACID ORC, Iceberg, and Delta Lake—An Overview of Table Formats for Large Scal...
byDatabricks
 
PPTX
Understanding SQL Trace, TKPROF and Execution Plan for beginners
byCarlos Sierra
 
PPTX
Redis introduction
byFederico Daniel Colombo Gennarelli
 
PDF
Understanding oracle rac internals part 2 - slides
byMohamed Farouk
 
PDF
postgres_빌드_및_디버깅.pdf
byLee Dong Wook
 
PDF
Spark shuffle introduction
bycolorant
 
PPTX
Adapting and adopting spm v04
byCarlos Sierra
 
PPTX
Log analysis using elk
byRushika Shah
 
PDF
SQLd360
byMauro Pagano
 
PDF
Capturing NIC and Kernel TX and RX Timestamps for Packets in Go
byScyllaDB
 
PDF
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
byDatabricks
 
PPTX
Troopers 19 - I am AD FS and So Can You
byDouglas Bienstock
 
PDF
Parquet performance tuning: the missing guide
byRyan Blue
 
PPT
OOUG: Oracle transaction locking
byKyle Hailey
 
PDF
Introduction to Apache Spark
byAnastasios Skarlatidis
 
ODP
Elasticsearch for beginners
byNeil Baker
 
PPTX
Alfresco Process Services REST API - Alfresco DevCon 2018
byDennis Koch
 
PPTX
Understanding my database through SQL*Plus using the free tool eDB360
byCarlos Sierra
 
Oracle RAC 19c: Best Practices and Secret Internals
byAnil Nair
 
(Re)Indexing Large Repositories in Alfresco
byAngel Borroy López
 
ACID ORC, Iceberg, and Delta Lake—An Overview of Table Formats for Large Scal...
byDatabricks
 
Understanding SQL Trace, TKPROF and Execution Plan for beginners
byCarlos Sierra
 
Redis introduction
byFederico Daniel Colombo Gennarelli
 
Understanding oracle rac internals part 2 - slides
byMohamed Farouk
 
postgres_빌드_및_디버깅.pdf
byLee Dong Wook
 
Spark shuffle introduction
bycolorant
 
Adapting and adopting spm v04
byCarlos Sierra
 
Log analysis using elk
byRushika Shah
 
SQLd360
byMauro Pagano
 
Capturing NIC and Kernel TX and RX Timestamps for Packets in Go
byScyllaDB
 
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
byDatabricks
 
Troopers 19 - I am AD FS and So Can You
byDouglas Bienstock
 
Parquet performance tuning: the missing guide
byRyan Blue
 
OOUG: Oracle transaction locking
byKyle Hailey
 
Introduction to Apache Spark
byAnastasios Skarlatidis
 
Elasticsearch for beginners
byNeil Baker
 
Alfresco Process Services REST API - Alfresco DevCon 2018
byDennis Koch
 
Understanding my database through SQL*Plus using the free tool eDB360
byCarlos Sierra
 

Viewers also liked

PDF
Effective Hive Queries
byQubole
 
PDF
Cross Datacenter Replication in Apache Solr 6
byShalin Shekhar Mangar
 
PDF
SolrCloud on Hadoop
byAlex Moundalexis
 
PDF
Solr on Docker - the Good, the Bad and the Ugly
bySematext Group, Inc.
 
PDF
Best practices for highly available and large scale SolrCloud
byAnshum Gupta
 
PDF
How to Run Solr on Docker and Why
bySematext Group, Inc.
 
Effective Hive Queries
byQubole
 
Cross Datacenter Replication in Apache Solr 6
byShalin Shekhar Mangar
 
SolrCloud on Hadoop
byAlex Moundalexis
 
Solr on Docker - the Good, the Bad and the Ugly
bySematext Group, Inc.
 
Best practices for highly available and large scale SolrCloud
byAnshum Gupta
 
How to Run Solr on Docker and Why
bySematext Group, Inc.
 

Similar to Solr Search Engine: Optimize Is (Not) Bad for You

PDF
Optimize Is (Not) Bad For You - Rafał Kuć, Sematext Group, Inc.
byLucidworks
 
PPTX
SolrCloud in Public Cloud: Scaling Compute Independently from Storage - Ilan ...
byLucidworks
 
PDF
Consuming RealTime Signals in Solr
byUmesh Prasad
 
PDF
Recent Additions to Lucene Arsenal
bylucenerevolution
 
PDF
Solr4 nosql search_server_2013
byLucidworks (Archived)
 
PDF
Building a near real time search engine & analytics for logs using solr
bylucenerevolution
 
PDF
Building a Large Scale SEO/SEM Application with Apache Solr: Presented by Rah...
byLucidworks
 
PPTX
Solr
byMohammad Reza Dehghani Tafti
 
PDF
Keynote Yonik Seeley & Steve Rowe lucene solr roadmap
bylucenerevolution
 
PDF
KEYNOTE: Lucene / Solr road map
bylucenerevolution
 
PDF
High Performance Solr
byShalin Shekhar Mangar
 
PPTX
Building a Large Scale SEO/SEM Application with Apache Solr
byRahul Jain
 
PPTX
IT talk SPb "Full text search for lazy guys"
byDataArt
 
PDF
Webinar: Faster Log Indexing with Fusion
byLucidworks
 
PDF
Building a Real-Time News Search Engine: Presented by Ramkumar Aiyengar, Bloo...
byLucidworks
 
PDF
SOLR Power FTW: short version
byAlex Pinkin
 
PDF
Elasticsearch at EyeEm
byLars Fronius
 
PDF
Introduction to SolrCloud
byVarun Thacker
 
PDF
Solr At Scale For Time-Oriented Data: Presented by Brett Hoerner, Rocana
byLucidworks
 
PDF
Solr 4 highlights - Mark Miller
bylucenerevolution
 
Optimize Is (Not) Bad For You - Rafał Kuć, Sematext Group, Inc.
byLucidworks
 
SolrCloud in Public Cloud: Scaling Compute Independently from Storage - Ilan ...
byLucidworks
 
Consuming RealTime Signals in Solr
byUmesh Prasad
 
Recent Additions to Lucene Arsenal
bylucenerevolution
 
Solr4 nosql search_server_2013
byLucidworks (Archived)
 
Building a near real time search engine & analytics for logs using solr
bylucenerevolution
 
Building a Large Scale SEO/SEM Application with Apache Solr: Presented by Rah...
byLucidworks
 
Solr
byMohammad Reza Dehghani Tafti
 
Keynote Yonik Seeley & Steve Rowe lucene solr roadmap
bylucenerevolution
 
KEYNOTE: Lucene / Solr road map
bylucenerevolution
 
High Performance Solr
byShalin Shekhar Mangar
 
Building a Large Scale SEO/SEM Application with Apache Solr
byRahul Jain
 
IT talk SPb "Full text search for lazy guys"
byDataArt
 
Webinar: Faster Log Indexing with Fusion
byLucidworks
 
Building a Real-Time News Search Engine: Presented by Ramkumar Aiyengar, Bloo...
byLucidworks
 
SOLR Power FTW: short version
byAlex Pinkin
 
Elasticsearch at EyeEm
byLars Fronius
 
Introduction to SolrCloud
byVarun Thacker
 
Solr At Scale For Time-Oriented Data: Presented by Brett Hoerner, Rocana
byLucidworks
 
Solr 4 highlights - Mark Miller
bylucenerevolution
 

More from Sematext Group, Inc.

PDF
Tweaking the Base Score: Lucene/Solr Similarities Explained
bySematext Group, Inc.
 
PDF
OOPs, OOMs, oh my! Containerizing JVM apps
bySematext Group, Inc.
 
PPTX
Is observability good for your brain?
bySematext Group, Inc.
 
PDF
Introducing log analysis to your organization
bySematext Group, Inc.
 
PDF
Monitoring and Log Management for
bySematext Group, Inc.
 
PDF
Introduction to solr
bySematext Group, Inc.
 
PDF
Building Resilient Log Aggregation Pipeline with Elasticsearch & Kafka
bySematext Group, Inc.
 
PDF
Elasticsearch for Logs & Metrics - a deep dive
bySematext Group, Inc.
 
PDF
Tuning Solr & Pipeline for Logs
bySematext Group, Inc.
 
PPTX
Running High Performance & Fault-tolerant Elasticsearch Clusters on Docker
bySematext Group, Inc.
 
PDF
Top Node.js Metrics to Watch
bySematext Group, Inc.
 
PPT
Running High Performance and Fault Tolerant Elasticsearch Clusters on Docker
bySematext Group, Inc.
 
PDF
Large Scale Log Analytics with Solr (from Lucene Revolution 2015)
bySematext Group, Inc.
 
PDF
From Zero to Production Hero: Log Analysis with Elasticsearch (from Velocity ...
bySematext Group, Inc.
 
PDF
Docker Logging Webinar
bySematext Group, Inc.
 
PDF
Docker Monitoring Webinar
bySematext Group, Inc.
 
PDF
Metrics, Logs, Transaction Traces, Anomaly Detection at Scale
bySematext Group, Inc.
 
PDF
Side by Side with Elasticsearch & Solr, Part 2
bySematext Group, Inc.
 
PPTX
Tuning Elasticsearch Indexing Pipeline for Logs
bySematext Group, Inc.
 
PDF
Solr Anti Patterns
bySematext Group, Inc.
 
Tweaking the Base Score: Lucene/Solr Similarities Explained
bySematext Group, Inc.
 
OOPs, OOMs, oh my! Containerizing JVM apps
bySematext Group, Inc.
 
Is observability good for your brain?
bySematext Group, Inc.
 
Introducing log analysis to your organization
bySematext Group, Inc.
 
Monitoring and Log Management for
bySematext Group, Inc.
 
Introduction to solr
bySematext Group, Inc.
 
Building Resilient Log Aggregation Pipeline with Elasticsearch & Kafka
bySematext Group, Inc.
 
Elasticsearch for Logs & Metrics - a deep dive
bySematext Group, Inc.
 
Tuning Solr & Pipeline for Logs
bySematext Group, Inc.
 
Running High Performance & Fault-tolerant Elasticsearch Clusters on Docker
bySematext Group, Inc.
 
Top Node.js Metrics to Watch
bySematext Group, Inc.
 
Running High Performance and Fault Tolerant Elasticsearch Clusters on Docker
bySematext Group, Inc.
 
Large Scale Log Analytics with Solr (from Lucene Revolution 2015)
bySematext Group, Inc.
 
From Zero to Production Hero: Log Analysis with Elasticsearch (from Velocity ...
bySematext Group, Inc.
 
Docker Logging Webinar
bySematext Group, Inc.
 
Docker Monitoring Webinar
bySematext Group, Inc.
 
Metrics, Logs, Transaction Traces, Anomaly Detection at Scale
bySematext Group, Inc.
 
Side by Side with Elasticsearch & Solr, Part 2
bySematext Group, Inc.
 
Tuning Elasticsearch Indexing Pipeline for Logs
bySematext Group, Inc.
 
Solr Anti Patterns
bySematext Group, Inc.
 

Recently uploaded

PDF
Spec-Driven Development with Kiro: Elevating Software Quality, Traceability, ...
byHaim Michael
 
PDF
Automated Governance for FME Flow: Smarter Admin at Scale
bySafe Software
 
PPTX
Introducing VisualSim 2610 The Next Leap in System Level Modeling
byDeepak Shankar
 
PPTX
Spacecraft Guidance Quick Research Guide by Arthur Morgan
byArthur Morgan
 
PDF
Effortless Distributed Systems with Aspire.pdf
byParticular Software
 
PDF
final~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~.pdf
byomarbishtawi04
 
PDF
final.pdf
byomarbishtawi04
 
PDF
20260212 Security-JAWS activity results for 2025 and activity goals for 2026
byTyphon 666
 
PDF
Session 1/5: Enhancing Automation with Screenplay & Business Rules
bysuhanisingh58689
 
PDF
UiPath Automation Developer Associate Training Series 2026 - Session 3
byDianaGray10
 
PDF
How does MES(Manufacturing Execution System) work?
byakipii ogaoga
 
PDF
Digital Twin in IBM for Accelerated Discovery of Climate & Sustainability, K...
byMichiaki Tatsubori
 
PDF
apidays Paris 2025 | Zero Trust By Design
byapidays
 
PPTX
Critique Prompting: The Secret to High-Quality AI Outputs
bySemantic SEO BD
 
PPTX
Workflow and decision Automation with Flowable
bychakib ch
 
PDF
GTM-and-Sales-Plan for a cyber security product
byAshish Jangir
 
PPTX
Microsoft Azure News - February 2026 - BAUG
byDaniel Toomey
 
PDF
Chapter 5 Security Mechanisms iin computer security.pdf
byGetnet Tigabie Askale -(GM)
 
PDF
Towards a Vibrant AI Hardware Accelerator Ecosystem, invited talk at the 4th ...
byMichiaki Tatsubori
 
PDF
Escape from the Forbidden Zone: Smuggling green and inclusive tech past the g...
byBookNet Canada
 
Spec-Driven Development with Kiro: Elevating Software Quality, Traceability, ...
byHaim Michael
 
Automated Governance for FME Flow: Smarter Admin at Scale
bySafe Software
 
Introducing VisualSim 2610 The Next Leap in System Level Modeling
byDeepak Shankar
 
Spacecraft Guidance Quick Research Guide by Arthur Morgan
byArthur Morgan
 
Effortless Distributed Systems with Aspire.pdf
byParticular Software
 
final~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~.pdf
byomarbishtawi04
 
final.pdf
byomarbishtawi04
 
20260212 Security-JAWS activity results for 2025 and activity goals for 2026
byTyphon 666
 
Session 1/5: Enhancing Automation with Screenplay & Business Rules
bysuhanisingh58689
 
UiPath Automation Developer Associate Training Series 2026 - Session 3
byDianaGray10
 
How does MES(Manufacturing Execution System) work?
byakipii ogaoga
 
Digital Twin in IBM for Accelerated Discovery of Climate & Sustainability, K...
byMichiaki Tatsubori
 
apidays Paris 2025 | Zero Trust By Design
byapidays
 
Critique Prompting: The Secret to High-Quality AI Outputs
bySemantic SEO BD
 
Workflow and decision Automation with Flowable
bychakib ch
 
GTM-and-Sales-Plan for a cyber security product
byAshish Jangir
 
Microsoft Azure News - February 2026 - BAUG
byDaniel Toomey
 
Chapter 5 Security Mechanisms iin computer security.pdf
byGetnet Tigabie Askale -(GM)
 
Towards a Vibrant AI Hardware Accelerator Ecosystem, invited talk at the 4th ...
byMichiaki Tatsubori
 
Escape from the Forbidden Zone: Smuggling green and inclusive tech past the g...
byBookNet Canada
 

Solr Search Engine: Optimize Is (Not) Bad for You

  • 1.
    Optimize Is (Not)Bad For You Deep Dive Into The Segment Merge Abyss Rafał Kuć Sematext Group, Inc.
  • 2.
    Agenda • Segments –where, what & how • Writing segments • Modifying segments • Segment merging – what, where, how, why • Force merging • Force merging & SolrCloud • Performance considerations • Specialized merge policies https://github.com/sematext/lr/tree/master/2017/optimize
  • 3.
  • 4.
  • 5.
  • 6.
    6 01 Solr Collection Architecture Zookeeper SOLR shardshard SOLR shard shard SOLR shard shard SOLR shard shard
  • 7.
  • 8.
  • 9.
    9 01 Lucene Segment Segment Info FieldNames Stored Field Values Point Values Term Dictionary Term Frequency Term Proximity Normalization Per Document Vals Live Documents
  • 10.
    1 01 Inside the Segment– Term Dictionary TERM DOCID lucene <1>, <2> revolution <1>, <2> washington <1> boston <2> _1.tim Doc1 Title: Lucene Revolution Washington, City: Washington D.C Doc2 Title: Lucene Revolution Boston, City: Boston _1.tip
  • 11.
    1 01 Inside the Segment– Doc Values Doc1 Title: Lucene Revolution Washington, City: Washington D.C Doc2 Title: Lucene Revolution Boston, City: Boston DOCID FIELD VALUE 1 Title Lucene Revolution Washington 1 City Washington D.C. 2 Title Lucene Revolution Boston 2 City Boston _1.dvd _1.dvm
  • 12.
    1 01 Inside the Segment– Stored Fields Doc1 Title: Lucene Revolution Washington, City: Washington D.C Doc2 Title: Lucene Revolution Boston, City: Boston DOCID VALUE 1 Title: Lucene Revolution Washington City: Washington D.C 2 Title: Lucene Revolution Boston City: Boston _1.fdx _1.fdt
  • 13.
    1 01 Inside the Segment– Compound File System _1.fdt _1.fdx _1.fnm _1.nvd _1.nvm _1.si _1.Lucene50_0.doc _1.Lucene50_0.pos _1.Lucene50_0.tim _1.Lucene50_0.tip _1.Lucene50_0.dvd _1.Lucene50_0.dvm
  • 14.
    1 01 Inside the Segment– Compound File System _1.fdt _1.fdx _1.fnm _1.nvd _1.nvm _1.si _1.Lucene50_0.doc _1.Lucene50_0.pos _1.Lucene50_0.tim _1.Lucene50_0.tip _1.Lucene50_0.dvd _1.Lucene50_0.dvm
  • 15.
    1 01 Inside the Segment– Compound File System _1.fdt _1.fdx _1.fnm _1.nvd _1.nvm _1.si _1.Lucene50_0.doc _1.Lucene50_0.pos _1.Lucene50_0.tim _1.Lucene50_0.tip _1.Lucene50_0.dvd _1.Lucene50_0.dvm _2.cfs _2.cfe
  • 16.
  • 17.
  • 18.
  • 19.
  • 20.
  • 21.
  • 22.
  • 23.
  • 24.
  • 25.
  • 26.
  • 27.
  • 28.
  • 29.
    2 01 Atomic Updates $ curl-XPOST -H 'Content-Type: application/json' 'http://localhost:8983/solr/lr/update?commit=true' --data-binary '[ { "id" : "3", "tags" : { "add" : [ "solr" ] } } ]' retrieve document { "id" : 3, "tags" : [ "lucene" ], "awesome" : true }
  • 30.
    3 01 Atomic Updates $ curl-XPOST -H 'Content-Type: application/json' 'http://localhost:8983/solr/lr/update?commit=true' --data-binary '[ { "id" : "3", "tags" : { "add" : [ "solr" ] } } ]' { "id" : 3, "tags" : [ "lucene", "solr" ], "awesome" : true } apply changes
  • 31.
    3 01 Atomic Updates $ curl-XPOST -H 'Content-Type: application/json' 'http://localhost:8983/solr/lr/update?commit=true' --data-binary '[ { "id" : "3", "tags" : { "add" : [ "solr" ] } } ]' { "id" : 3, "tags" : [ "lucene", "solr" ], "awesome" : true } delete old document
  • 32.
    3 01 Atomic Updates $ curl-XPOST -H 'Content-Type: application/json' 'http://localhost:8983/solr/lr/update?commit=true' --data-binary '[ { "id" : "3", "tags" : { "add" : [ "solr" ] } } ]' { "id" : 3, "tags" : [ "lucene", "solr" ], "awesome" : true }
  • 33.
    3 01 Atomic Updates –In Place Works on top of numeric, doc values based fields Fields need to be not indexed and not stored Doesn’t require delete/index Support only inc and set modifers $ curl -XPOST -H 'Content-Type: application/json' 'http://localhost:8983/solr/lr/update?commit=true' --data-binary '[ { "id" : "3", "views" : { "inc" : 100 } } ]'
  • 34.
    3 01 Atomic Updates –In Place $ curl -XPOST -H 'Content-Type: application/json' 'http://localhost:8983/solr/lr/update?commit=true' --data-binary '[ { "id" : "3", "views" : { "inc" : 100 } } ]' retrieve document { "id" : 3, "tags" : [ "lucene", "solr" ], "awesome" : true }
  • 35.
    3 01 Atomic Updates –In Place $ curl -XPOST -H 'Content-Type: application/json' 'http://localhost:8983/solr/lr/update?commit=true' --data-binary '[ { "id" : "3", "views" : { "inc" : 100 } } ]' { "id" : 3, "tags" : [ "lucene", "solr" ], "awesome" : true, "views" : 100 } apply changes
  • 36.
    3 01 Atomic Updates –In Place $ curl -XPOST -H 'Content-Type: application/json' 'http://localhost:8983/solr/lr/update?commit=true' --data-binary '[ { "id" : "3", "views" : { "inc" : 100 } } ]' { "id" : 3, "tags" : [ "lucene", "solr" ], "awesome" : true, "views" : 100 } update doc values
  • 37.
    3 01 Search – Importanceof Segments Immutable – write once read many
  • 38.
    3 01 Search – Importanceof Segments Immutable – write once read many More segments – slower search speed
  • 39.
    3 01 Search – Importanceof Segments Immutable – write once read many More segments – slower search speed Fewer segments – faster searches
  • 40.
    4 01 Search – Importanceof Segments Immutable – write once read many More segments – slower search speed Fewer segments – faster searches Fewer segments – smaller shard size
  • 41.
    4 01 Search – Importanceof Segments Immutable – write once read many More segments – slower search speed Fewer segments – faster searches Fewer segments – smaller shard size Rapid segment changes – worse I/O cache usage
  • 42.
    4 01 Taking Control Merge PolicyFactory <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory"> <int name="maxMergeAtOnce">10</int> <int name="maxMergeAtOnceExplicit">30</int> <int name="segmentsPerTier">10</int> <int name="floorSegmentMB">2048</int> <int name="maxMergedSegmentMB">5120</int> <double name="noCFSRatio">0.1</double> <int name="maxCFSSegmentSizeMB">2048</int> <double name="reclaimDeletesWeight">2.0</double> <double name="forceMergeDeletesPctAllowed">10.0</double> </mergePolicyFactory>
  • 43.
    4 01 Taking Control Merge PolicyFactory <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory"> <int name="maxMergeAtOnce">10</int> <int name="maxMergeAtOnceExplicit">30</int> <int name="segmentsPerTier">10</int> <int name="floorSegmentMB">2048</int> <int name="maxMergedSegmentMB">5120</int> <double name="noCFSRatio">0.1</double> <int name="maxCFSSegmentSizeMB">2048</int> <double name="reclaimDeletesWeight">2.0</double> <double name="forceMergeDeletesPctAllowed">10.0</double> </mergePolicyFactory> Merge Scheduler <mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler" />
  • 44.
    4 01 Taking Control Merge PolicyFactory <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory"> <int name="maxMergeAtOnce">10</int> <int name="maxMergeAtOnceExplicit">30</int> <int name="segmentsPerTier">10</int> <int name="floorSegmentMB">2048</int> <int name="maxMergedSegmentMB">5120</int> <double name="noCFSRatio">0.1</double> <int name="maxCFSSegmentSizeMB">2048</int> <double name="reclaimDeletesWeight">2.0</double> <double name="forceMergeDeletesPctAllowed">10.0</double> </mergePolicyFactory> Merge Scheduler <mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler" /> Segment Warmer <mergedSegmentWarmer class="org.apache.lucene.index.SimpleMergedSegmentWarmer" />
  • 45.
    4 01 Taking Control –Default Indexing Throughput Merge Policy Factory <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory"> <int name="maxMergeAtOnce">10</int> <int name="maxMergeAtOnceExplicit">30</int> <int name="segmentsPerTier">10</int> <int name="floorSegmentMB">2048</int> <int name="maxMergedSegmentMB">5120</int> <double name="noCFSRatio">0.1</double> <int name="maxCFSSegmentSizeMB">2048</int> <double name="reclaimDeletesWeight">2.0</double> <double name="forceMergeDeletesPctAllowed">10.0</double> </mergePolicyFactory>
  • 46.
    4 01 Taking Control –Default Indexing Throughput throughput < 5k/sec @ ~14GB
  • 47.
    4 01 Taking Control –Max Merged Segment Size Merge Policy Factory <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory"> <int name="maxMergeAtOnce">10</int> <int name="maxMergeAtOnceExplicit">30</int> <int name="segmentsPerTier">10</int> <int name="floorSegmentMB">2048</int> <int name="maxMergedSegmentMB">5120</int> <double name="noCFSRatio">0.1</double> <int name="maxCFSSegmentSizeMB">2048</int> <double name="reclaimDeletesWeight">2.0</double> <double name="forceMergeDeletesPctAllowed">10.0</double> </mergePolicyFactory> Lower higher indexing throughput – smaller segments Higher better search latency (depends) – more merges
  • 48.
    4 01 Taking Control –Lowering Max Merged Size Merge Policy Factory <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory"> <int name="maxMergeAtOnce">10</int> <int name="maxMergeAtOnceExplicit">30</int> <int name="segmentsPerTier">10</int> <int name="floorSegmentMB">2048</int> <int name="maxMergedSegmentMB">512</int> <double name="noCFSRatio">0.1</double> <int name="maxCFSSegmentSizeMB">2048</int> <double name="reclaimDeletesWeight">2.0</double> <double name="forceMergeDeletesPctAllowed">10.0</double> </mergePolicyFactory>
  • 49.
    4 01 Taking Control –Lowering Max Segment Size throughput < 5k/sec @ ~15.5GB 11% throughput increase
  • 50.
    5 01 Taking Control –Merge At Once Merge Policy Factory <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory"> <int name="maxMergeAtOnce">10</int> <int name="maxMergeAtOnceExplicit">30</int> <int name="segmentsPerTier">10</int> <int name="floorSegmentMB">2048</int> <int name="maxMergedSegmentMB">5120</int> <double name="noCFSRatio">0.1</double> <int name="maxCFSSegmentSizeMB">2048</int> <double name="reclaimDeletesWeight">2.0</double> <double name="forceMergeDeletesPctAllowed">10.0</double> </mergePolicyFactory> Lower better search latency (depends) Higher higher indexing throughput
  • 51.
    5 01 Taking Control –Lowering Merge At Once Merge Policy Factory <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory"> <int name="maxMergeAtOnce">2</int> <int name="maxMergeAtOnceExplicit">30</int> <int name="segmentsPerTier">10</int> <int name="floorSegmentMB">2048</int> <int name="maxMergedSegmentMB">5120</int> <double name="noCFSRatio">0.1</double> <int name="maxCFSSegmentSizeMB">2048</int> <double name="reclaimDeletesWeight">2.0</double> <double name="forceMergeDeletesPctAllowed">10.0</double> </mergePolicyFactory>
  • 52.
    5 01 Taking Control –Lowering Merge At Once throughput < 5k/sec @ ~13GB 8% throughput decrease
  • 53.
    5 01 Taking Control –Merge At Once Explicit Merge Policy Factory <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory"> <int name="maxMergeAtOnce">10</int> <int name="maxMergeAtOnceExplicit">30</int> <int name="segmentsPerTier">10</int> <int name="floorSegmentMB">2048</int> <int name="maxMergedSegmentMB">5120</int> <double name="noCFSRatio">0.1</double> <int name="maxCFSSegmentSizeMB">2048</int> <double name="reclaimDeletesWeight">2.0</double> <double name="forceMergeDeletesPctAllowed">10.0</double> </mergePolicyFactory> Controls number of segments merged at once during force merge
  • 54.
    5 01 Taking Control –Segments Per Tier Merge Policy Factory <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory"> <int name="maxMergeAtOnce">10</int> <int name="maxMergeAtOnceExplicit">30</int> <int name="segmentsPerTier">10</int> <int name="floorSegmentMB">2048</int> <int name="maxMergedSegmentMB">5120</int> <double name="noCFSRatio">0.1</double> <int name="maxCFSSegmentSizeMB">2048</int> <double name="reclaimDeletesWeight">2.0</double> <double name="forceMergeDeletesPctAllowed">10.0</double> </mergePolicyFactory> Lower value means more merging, but less segments Along with maxMergeAtOnce can smoothen I/O spikes For better indexing throughput set maxMergeAtOnce < segmentsPerTier
  • 55.
    5 01 Taking Control –Combined Together Merge Policy Factory <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory"> <int name="maxMergeAtOnce">30</int> <int name="maxMergeAtOnceExplicit">30</int> <int name="segmentsPerTier">30</int> <int name="floorSegmentMB">2048</int> <int name="maxMergedSegmentMB">512</int> <double name="noCFSRatio">0.1</double> <int name="maxCFSSegmentSizeMB">2048</int> <double name="reclaimDeletesWeight">2.0</double> <double name="forceMergeDeletesPctAllowed">10.0</double> </mergePolicyFactory>
  • 56.
    5 01 Taking Control –Combined Together throughput < 5k/sec @ ~15GB but look at read difference
  • 57.
    5 01 Taking Control –Default vs Combined Read/Write default settings
  • 58.
    5 01 Taking Control –Default vs Combined Read/Write default settings combined changes settings
  • 59.
    5 01 Taking Control –Reclaim Deletes Weight Merge Policy Factory <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory"> <int name="maxMergeAtOnce">10</int> <int name="maxMergeAtOnceExplicit">30</int> <int name="segmentsPerTier">10</int> <int name="floorSegmentMB">2048</int> <int name="maxMergedSegmentMB">5120</int> <double name="noCFSRatio">0.1</double> <int name="maxCFSSegmentSizeMB">2048</int> <double name="reclaimDeletesWeight">2.0</double> <double name="forceMergeDeletesPctAllowed">10.0</double> </mergePolicyFactory> Controls importance of merging segments with deleted documents Increase to put priority on merging segments with deleted documents
  • 60.
    6 01 Taking Control –No CFS Ratio Merge Policy Factory <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory"> <int name="maxMergeAtOnce">10</int> <int name="maxMergeAtOnceExplicit">30</int> <int name="segmentsPerTier">10</int> <int name="floorSegmentMB">2048</int> <int name="maxMergedSegmentMB">5120</int> <double name="noCFSRatio">0.1</double> <int name="maxCFSSegmentSizeMB">2048</int> <double name="reclaimDeletesWeight">2.0</double> <double name="forceMergeDeletesPctAllowed">10.0</double> </mergePolicyFactory> Controls compound file system segments ratio To completely disable CFS set to 0.0
  • 61.
    6 01 Taking Control –Merge Scheduler Controls maximum number of concurrent merges Merge Scheduler <mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler"> <int name="maxMergeCount">4</int> <int name="maxThreadCount">4</int> </mergeScheduler>
  • 62.
    6 01 Taking Control –Merge Scheduler Controls number of threads dedicated to merging Merge Scheduler <mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler"> <int name="maxMergeCount">4</int> <int name="maxThreadCount">4</int> </mergeScheduler>
  • 63.
    6 01 Taking Control –Merge Scheduler Controls number of threads dedicated to merging For spinning drives set maxThreadCount to 1 Merge Scheduler <mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler"> <int name="maxMergeCount">4</int> <int name="maxThreadCount">4</int> </mergeScheduler>
  • 64.
    6 01 Taking Control –Merge Scheduler Controls number of threads dedicated to merging For spinning drives set maxThreadCount to 1 For SSD set maxThreadCount to min(4, #CPUs / 2) Merge Scheduler <mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler"> <int name="maxMergeCount">4</int> <int name="maxThreadCount">4</int> </mergeScheduler>
  • 65.
    6 01 Optimize aka ForceMerge Forces segment merge – usually very expensive
  • 66.
    6 01 Optimize aka ForceMerge Forces segment merge – usually very expensive Desired number of segments can be specified
  • 67.
    6 01 Optimize aka ForceMerge Forces segment merge – usually very expensive Desired number of segments can be specified Done on all shards at the same time (by default)
  • 68.
    6 01 Optimize aka ForceMerge Forces segment merge – usually very expensive Desired number of segments can be specified Done on all shards at the same time (by default) Can be very bad or very good – depending on the use case
  • 69.
    6 01 Optimize aka ForceMerge Forces segment merge – usually very expensive Desired number of segments can be specified Done on all shards at the same time (by default) Can be very bad or very good – depending on the use case $ curl 'http://solr:8983/solr/lr/update?optimize=true&numSegments=1&waitFlush=false'
  • 70.
    7 01 Force Merge –The Good Improves search speed (fewer segments)
  • 71.
    7 01 Force Merge –The Good Improves search speed (fewer segments) Removes deleted documents
  • 72.
    7 01 Force Merge –The Good Improves search speed (fewer segments) Removes deleted documents Shrinks the index by pruning duplicated data
  • 73.
    7 01 Force Merge –The Good Improves search speed (fewer segments) Removes deleted documents Shrinks the index by pruning duplicated data Reduces number of used files
  • 74.
    7 01 Force Merge –The Bad Invalidates operating system I/O cache
  • 75.
    7 01 Force Merge –The Bad Invalidates operating system I/O cache Very expensive to perform – rewrites all segments
  • 76.
    7 01 Force Merge –The Bad Invalidates operating system I/O cache Very expensive to perform – rewrites all segments Not efficient on changing data
  • 77.
    7 01 Force Merge –The Bad Invalidates operating system I/O cache Very expensive to perform – rewrites all segments Not efficient on changing data May cause performance issues
  • 78.
    7 01 Force Merge –The Bad Invalidates operating system I/O cache Very expensive to perform – rewrites all segments Not efficient on changing data May cause performance issues Will cause temporary increase of disk usage (up to 3x)
  • 79.
    7 01 Force Merge –SolrCloud Performance Example
  • 80.
    8 01 Force Merge –SolrCloud Performance Example
  • 81.
    8 01 Force Merge –Legacy Index on the master server Solr Master Solr Slave Solr Slave Solr Slave index Documents
  • 82.
    8 01 Force Merge –Legacy Index on the master server Force merge on the master server Solr Master Solr Slave Solr Slave Solr Slave force merge
  • 83.
    8 01 Force Merge –Legacy Index on the master server Force merge on the master server Replicate after optimize is done Solr Master Solr Slave Solr Slave Solr Slave pull after optimize
  • 84.
    8 01 Force Merge –SolrCloud (Solr 7 – pull replicas) Create collection Force merge Solr will do the rest Solr Solr Solr Solr Primary 1 Primary 2 Pull Replica 2 Pull Replica 1
  • 85.
    8 01 Force Merge –SolrCloud (NRT, pre 7.0) Ask yourself if you really need force merge Solr Solr Solr Solr
  • 86.
    8 01 Force Merge –SolrCloud (NRT replicas, pre 7.0) Ask yourself if you really need force merge Create collection on part of the nodes Solr Solr Solr Solr Primary 1 Primary 2
  • 87.
    8 01 Force Merge –SolrCloud (NRT replicas, pre 7.0) Ask yourself if you really need force merge Create collection on part of the nodes Index Solr Solr Solr Solr Primary 1 Primary 2 DocumentsDocuments Documents Documents
  • 88.
    8 01 Force Merge –SolrCloud (NRT replicas, pre 7.0) Ask yourself if you really need force merge Create collection on part of the nodes Index Force merge Solr Solr Solr Solr Primary 1 Primary 2optimize
  • 89.
    8 01 Force Merge –SolrCloud (NRT replicas, pre 7.0) Ask yourself if you really need force merge Create collection on part of the nodes Index Force merge Create replicas Solr Solr Solr Solr Primary 1 Primary 2 Replica 2 Replica 1
  • 90.
    9 01 Specialized Merge PolicyExample – Sorting Sorting Merge Policy Factory Example <mergePolicyFactory class="org.apache.solr.index.SortingMergePolicyFactory"> <str name="sort">timestamp desc</str> <str name="wrapper.prefix">inner</str> <str name="inner.class">org.apache.solr.index.TieredMergePolicyFactory</str> <int name="inner.maxMergeAtOnce">10</int> <int name="inner.segmentsPerTier">10</int> <double name="inner.noCFSRatio">0.1</double> </mergePolicyFactory>
  • 91.
    9 01 Specialized Merge PolicyExample – Sorting Sorting Merge Policy Factory Example <mergePolicyFactory class="org.apache.solr.index.SortingMergePolicyFactory"> <str name="sort">timestamp desc</str> <str name="wrapper.prefix">inner</str> <str name="inner.class">org.apache.solr.index.TieredMergePolicyFactory</str> <int name="inner.maxMergeAtOnce">10</int> <int name="inner.segmentsPerTier">10</int> <double name="inner.noCFSRatio">0.1</double> </mergePolicyFactory> Pre-sorts data during merge for: - faster range queries - faster data retrieval - possibility of early query termination - convenient for time based data
  • 92.
    9 01 http://sematext.com/jobs You love likewe do? You want to work with ? Want to work with open source? You want to do fun stuff?
  • 93.
  • 94.