Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
High Performance Solr
Shalin Shekhar Mangar
shalin@apache.org
https://twitter.com/shalinmangar
Performance constraints
• CPU
• Memory
• Disk
• Network
Tuning (CPU) Queries
• Phrase query
• Boolean query (AND)
• Boolean query (OR)
• Wildcard
• Fuzzy
• Soundex
• …roughly in ...
Tuning (CPU) Queries
• Reduce frequent-term queries
• Remove stopwords
• Try CommonGramsFilter
• Index pruning (advanced)
...
Tuning (CPU) Queries
• Make efficient use of caches
• Watch those eviction counts
• Beware of NOW in date range queries. Us...
Tuning (CPU) Queries
• Warm those caches
• Auto-warming
• Warming queries
• firstSearcher
• newSearcher
Tuning (CPU) Queries
• Stop using primitive number/date fields if you are performing range queries
• facet.query (sometimes...
Tuning (CPU) Queries
• Faceting methods
• facet.method=enum - great for less unique values
• facet.enum.cache.minDf - use ...
Tuning (CPU) Queries
• ReRankQueryParser
• Like a PostFilter but for queries!
• Run expensive queries at the very last
• S...
Tuning (CPU) Queries
• Divide and conquer
• Shard’em out
• Use multiple CPUs
• Sometime multiple cores are the answer even...
Tuning Memory Usage
• Use DocValues for sorting/faceting/grouping
• There are docValueFormats: {‘default’, ‘memory’,
‘dire...
Tuning Memory usage
• termIndexInterval - Choose how often terms are
loaded into term dictionary. Default is 128.
Tuning Memory Usage
• Garbage Collection pauses kill search performance
• GC pauses expire ZK sessions in SolrCloud
leadin...
Tuning Disk usage
• Atomic updates are costly
• Lookup from transaction log
• Lookup from Index (all stored fields)
• Combi...
Tuning Disk Usage
• Experiment with merge policies
• TieredMergePolicy is great but
LogByteSizeMergePolicy can be better i...
Tuning Disk Usage
• Always hard commit once in a while
• Best to use autoCommit and maxDocs
• Trims transaction logs
• Sol...
Tuning Network
• Batch writes together as much as possible
• Use CloudSolrServer in SolrCloud always
• Routes updates inte...
Tuning network
• Share HttpClient instance for all Solrj clients or just
re-use the same client object
• Disable retries o...
Tuning Network
• Distributed Search is optimised if you ask for
fl=id,score only
• Avoid numShard*rows stored field lookups
...
Tuning Network
• Consider setting up a caching proxy such as squid or varnish in front of
your Solr cluster
• Solr can emi...
Avoid wastage
• Don’t store what you don’t need back
• Use stored=false
• Don’t index what you don’t search
• Use indexed=...
Reduce indexed info
• omitNorms=true - Use if you don’t need index-time boosts
• omitTermFreqAndPositions=true - Use if yo...
DocValue tricks & gotchas
• DocValue field should be stored=false, indexed=false
• It can still be retrieved using fl=field(m...
Deep paging
• Bulk exporting documents from Solr will bring it to
its knees
• Enter deep paging and cursorMark parameter
•...
Classic paging vs Deep paging
LucidWorks Open Source
• Effortless AWS deployment and monitoring http://
www.github.com/lucidworks/solr-scale-tk
• Logsta...
LucidWorks
• We’re hiring!
• Work on open source Apache Lucene/Solr
• Help our customers win
• Work remotely from home! Lo...
Upcoming SlideShare
Loading in …5
×

High Performance Solr

19,990 views

Published on

How to make the most of your Apache Solr deployment by tuning CPU, Memory, Disk and Network usage.

Published in: Software, Technology

High Performance Solr

  1. 1. High Performance Solr Shalin Shekhar Mangar shalin@apache.org https://twitter.com/shalinmangar
  2. 2. Performance constraints • CPU • Memory • Disk • Network
  3. 3. Tuning (CPU) Queries • Phrase query • Boolean query (AND) • Boolean query (OR) • Wildcard • Fuzzy • Soundex • …roughly in order of increasing cost • Query performance inversely proportional to matches (doc frequency)
  4. 4. Tuning (CPU) Queries • Reduce frequent-term queries • Remove stopwords • Try CommonGramsFilter • Index pruning (advanced) • Some function queries match ALL documents - terribly inefficient
  5. 5. Tuning (CPU) Queries • Make efficient use of caches • Watch those eviction counts • Beware of NOW in date range queries. Use NOW/DAY or NOW/HOUR • No need to cache every filter • Use fq={!cache=false}year:[2005 TO *] • Specify cost for non-cached filters for efficiency • fq={!geofilt sfield=location pt=22,-127 d=50 cache=false cost=50} • Use PostFilters for very expensive filters (cache=false, cost > 100)
  6. 6. Tuning (CPU) Queries • Warm those caches • Auto-warming • Warming queries • firstSearcher • newSearcher
  7. 7. Tuning (CPU) Queries • Stop using primitive number/date fields if you are performing range queries • facet.query (sometimes) or facet.range are also range queries • Use Trie* Fields • When performing range queries on a string field (rare use-case), use frange to trade off memory for speed • It will un-invert the field • No additional cost is paid if the field is already being used for sorting or other function queries • fq={!frange l=martin u=rowling}author_last_name instead of fq=author_last_name:[martin TO rowling]
  8. 8. Tuning (CPU) Queries • Faceting methods • facet.method=enum - great for less unique values • facet.enum.cache.minDf - use filter cache or iterate through DocsEnum • facet.method=fc • facet.method=fcs (per-segment) • facet.sort=index faster than facet.sort=count but useless in typical cases
  9. 9. Tuning (CPU) Queries • ReRankQueryParser • Like a PostFilter but for queries! • Run expensive queries at the very last • Solr 4.9+ only (soon to be released)
  10. 10. Tuning (CPU) Queries • Divide and conquer • Shard’em out • Use multiple CPUs • Sometime multiple cores are the answer even for small indexes and specially for high-updates
  11. 11. Tuning Memory Usage • Use DocValues for sorting/faceting/grouping • There are docValueFormats: {‘default’, ‘memory’, ‘direct’} with different trade-offs. • default - Helps avoid OOM but uses disk and OS page cache • memory - compressed in-memory format • direct - no-compression, in-memory format
  12. 12. Tuning Memory usage • termIndexInterval - Choose how often terms are loaded into term dictionary. Default is 128.
  13. 13. Tuning Memory Usage • Garbage Collection pauses kill search performance • GC pauses expire ZK sessions in SolrCloud leading to many problems • Large heap sizes are almost never the answer • Leave a lot of memory for the OS page cache • http://wiki.apache.org/solr/ShawnHeisey
  14. 14. Tuning Disk usage • Atomic updates are costly • Lookup from transaction log • Lookup from Index (all stored fields) • Combine • Index
  15. 15. Tuning Disk Usage • Experiment with merge policies • TieredMergePolicy is great but LogByteSizeMergePolicy can be better if multiple indexes are sharing a single disk • Increase buffer size - ramBufferSizeMB (>1024M doesn’t help, may reduce performance)
  16. 16. Tuning Disk Usage • Always hard commit once in a while • Best to use autoCommit and maxDocs • Trims transaction logs • Solution for slow startup times • Use autoSoftCommit for new searchers • commitWithin is a great way to commit frequently
  17. 17. Tuning Network • Batch writes together as much as possible • Use CloudSolrServer in SolrCloud always • Routes updates intelligently to correct leader • ConcurrentUpdateSolrServer (previously known as StreamingUpdateSolrServer) for indexing in non- Cloud mode • Don’t use it for querying!
  18. 18. Tuning network • Share HttpClient instance for all Solrj clients or just re-use the same client object • Disable retries on HttpClient
  19. 19. Tuning Network • Distributed Search is optimised if you ask for fl=id,score only • Avoid numShard*rows stored field lookups • Saves numShard network calls
  20. 20. Tuning Network • Consider setting up a caching proxy such as squid or varnish in front of your Solr cluster • Solr can emit the right cache headers if configured in solrconfig.xml • Last-Modified and ETag headers are generated based on the properties of the index such as last searcher open time • You can even force new ETag headers by changing the ETag seed value • <httpCaching never304=“true”><cacheControl>max-age=30, public</ cacheControl></httpCaching> • The above config will set responses to be cached for 30s by your caching proxy unless the index is modifed.
  21. 21. Avoid wastage • Don’t store what you don’t need back • Use stored=false • Don’t index what you don’t search • Use indexed=false • Don’t retrieve what you don’t need back • Don’t use fl=* unless necessary • Don’t use rows=10 when all you need is numFound
  22. 22. Reduce indexed info • omitNorms=true - Use if you don’t need index-time boosts • omitTermFreqAndPositions=true - Use if you don’t need term frequencies and positions • No fuzzy query, no phrase queries • Can do simple exists check, can do simple AND/OR searches on terms • No scoring difference whether the term exists once or a thousand times
  23. 23. DocValue tricks & gotchas • DocValue field should be stored=false, indexed=false • It can still be retrieved using fl=field(my_dv_field) • If you store DocValue field, it uses extra space as a stored field also. • In future, update-able doc value fields will be supported by Solr but they’ll work only if stored=false, indexed=false • DocValues save disk space also (all values, next to each other lead to very efficient compression)
  24. 24. Deep paging • Bulk exporting documents from Solr will bring it to its knees • Enter deep paging and cursorMark parameter • Specify cursorMark=* on the first request • Use the returned ‘nextCursorMark’ value as the nextCursorMark parameter
  25. 25. Classic paging vs Deep paging
  26. 26. LucidWorks Open Source • Effortless AWS deployment and monitoring http:// www.github.com/lucidworks/solr-scale-tk • Logstash for Solr: https://github.com/LucidWorks/ solrlogmanager • Banana (Kibana for Solr): https://github.com/LucidWorks/ banana • Data Quality Toolkit: https://github.com/LucidWorks/data-quality • Coming Soon for Big Data: Hadoop, Pig, Hive 2-way support w/ Lucene and Solr, different file formats, pipelines, Logstash
  27. 27. LucidWorks • We’re hiring! • Work on open source Apache Lucene/Solr • Help our customers win • Work remotely from home! Location no bar! • Contact me at shalin@apache.org

×