Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Large Scale Log Analytics with Solr (from Lucene Revolution 2015)

10,639 views

Published on

In this talk from Lucene/Solr Revolution 2015, Solr and centralized logging experts Radu Gheorghe and Rafal Kuć cover topics like: flow in Logstash, flow in rsyslog, parsing JSON, log shipping, Solr tuning, time-based collections and tiered clusters.

Published in: Data & Analytics
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Large Scale Log Analytics with Solr (from Lucene Revolution 2015)

  1. 1. OCTOBER 13-16, 2016 • AUSTIN, TX
  2. 2. Large Scale Log Analytics with Solr Rafał Kuć and Radu Gheorghe Sematext Group
  3. 3. 3 01 About Us RaduRafał Logsene
  4. 4. 4 02 Agenda Logstash + Solr rsyslog + Solr rsyslog + Redis + Logstash + Solr Solr
  5. 5. 5 01 Flow in Logstash /var/log/apache.log redis https://cdn2.iconfinder.com/data/icons/gconstruct/2118/gconstruct1-14.png input
  6. 6. 6 01 Flow in Logstash /var/log/apache.log redis https://cdn2.iconfinder.com/data/icons/gconstruct/2118/gconstruct1-14.png plain {json} input codec
  7. 7. 7 01 Flow in Logstash /var/log/apache.log redis Rafał @kucrafal grok { "user": "Rafał", "twitter": "@kucrafal" } - w $numberOfWorkers https://cdn2.iconfinder.com/data/icons/gconstruct/2118/gconstruct1-14.png plain {json} input codec filter
  8. 8. 8 01 Flow in Logstash /var/log/apache.log redis Rafał @kucrafal grok { "user": "Rafał", "twitter": "@kucrafal" } - w $numberOfWorkers https://cdn2.iconfinder.com/data/icons/gconstruct/2118/gconstruct1-14.png workers => 2 plain {json} input codec filter output
  9. 9. 9 01 Simple Config https://github.com/sematext/lucene-revolution-samples/tree/master/2015 input { file { path => "/opt/logs/example.log" start_position => "beginning" } } output { solr_http { solr_url => "http://localhost:8983/solr/gettingstarted" flush_size => 5000 workers => 4 } } bin/plugin install logstash-output-solr_http apache combined logs
  10. 10. 10 01 Base Result
  11. 11. 11 01 Parse JSON input { file { path => "/opt/logs/example.log.parsed" start_position => "beginning" … filter { json { source => "message" } } output { solr_http { … apache combined logs in JSON bin/logstash -f logstash.conf -w 4 # filterWorkers=4
  12. 12. 12 01 JSON Result
  13. 13. input { file { path => "/opt/logs/example.log" start_position => "beginning" … filter { grok { match => [ "message", "%{COMBINEDAPACHELOG}" ] } } output { solr_http { … 13 01 Grok
  14. 14. 14 01 Grok Result
  15. 15. 15 01 Flow Options https://upload.wikimedia.org/wikipedia/commons/thumb/b/bb/Gorilla-server.svg/2000px-Gorilla-server.svg.png https://www.elastic.co/assets/blt69f6410148efbab8/logstash.png
  16. 16. 16 01 Flow Options (cont.) http://www.hanselman.com/blog/content/binary/Windows-Live-Writer/ef572a4c3e50_13F7B/redis_logo_a83f44f3-708d-4fad-aa6e-6eb0d6f82001.png https://upload.wikimedia.org/wikipedia/commons/thumb/f/f8/Question_mark_alternate.svg/2000px-Question_mark_alternate.svg.png or Kafka or *MQ or... something light here rsyslog rsyslog rsyslog
  17. 17. 17 01 Flow in rsyslog /var/log/apache.log syslog socket input
  18. 18. 18 01 Flow in rsyslog /var/log/apache.log syslog socket main queue (RAM+Disk) input queue.type queue.size ...
  19. 19. 19 01 Flow in rsyslog /var/log/apache.log syslog socket main queue (RAM+Disk) input queue.type queue.size ... queue.workerThreads (filter, parse and send events)
  20. 20. 20 01 Flow in rsyslog /var/log/apache.log syslog socket main queue (RAM+Disk) input queue.type queue.size ... queue.workerThreads (filter, parse and send events) queue.dequeueBatchSize
  21. 21. 21 01 Flow in rsyslog /var/log/apache.log syslog socket main queue (RAM+Disk) input queue.type queue.size ... queue.workerThreads (filter, parse and send events) queue.dequeueBatchSize rsyslog_solr.py rsyslog_solr.py rsyslog_solr.py action template {JSON}
  22. 22. 22 01 Flow in rsyslog /var/log/apache.log syslog socket main queue (RAM+Disk) input queue.type queue.size ... queue.workerThreads (filter, parse and send events) queue.dequeueBatchSize rsyslog_solr.py rsyslog_solr.py rsyslog_solr.py action template {JSON}
  23. 23. 23 01 Simple Config (1/2) https://github.com/sematext/lucene-revolution-samples/tree/master/2015 module(load="imfile") module(load="omprog") input(type="imfile" File="/opt/logs/example.log" Tag="apache:") main_queue( queue.highWatermark="100000" queue.lowWatermark="50000" queue.maxDiskSpace="5g" queue.fileName="solr_action" queue.spoolDirectory="/opt/rsyslog/queues" queue.saveOnShutdown="on" queue.workerThreads="4" queue.dequeueBatchSize="500" ) apache combined logs
  24. 24. 24 01 Simple Config (2/2) template(name="json_lines" type="list" option.json="on") { constant(value="{") constant(value=""timestamp":"") property(name="timereported" dateFormat="rfc3339") constant(value="","message":"") property(name="msg") ... constant(value="","syslog-tag":"") property(name="syslogtag") constant(value=""}n") } action( type="omprog" binary="/opt/rsyslog/rsyslog_solr.py" template="json_lines" ) get from https://github.com/rsyslog/rsyslog/tree/master/plugins/external/solr
  25. 25. 25 01 Base Result
  26. 26. 26 01 Base Result 15% rsyslog, 4x1% rsyslog_solr.py
  27. 27. 27 01 Base Result 15% rsyslog, 4x1% rsyslog_solr.py 125MB rsyslog, 4x15MB rsyslog_solr.py Depends on queue. Here up to 100K events in RAM
  28. 28. 28 01 JSON Config # same main queue settings and modules input(type="imfile" File="/opt/logs/example.log.parsed" Tag="apache:") module(load="mmnormalize") action(type="mmnormalize" rulebase="/opt/rsyslog/json.rb" ) template(name="json_lines" type="list") { property(name="$!root") constant(value="n") } action(type="omprog" ... apache combined logs already parsed in JSON version=2 rule=:%root:json%
  29. 29. 29 01 JSON Result
  30. 30. 30 01 Normalizing Config input(type="imfile" File="/opt/logs/example.log" Tag="apache") action(type="mmnormalize" rulebase="/opt/rsyslog/apache_combined.rb" ) template(name="json_lines" type="list") { property(name="$!all-json") constant(value="n") } version=2 rule=:%[ {"type": "word", "name": "clientip"}, {"type": "literal", "text": " "}, ... {"type": "char-to", "name": "agent", "extradata": """}, {"type": "literal", "text": """}, {"type": "rest", "name": "blob"} ]%
  31. 31. 31 01 Normalizing Result
  32. 32. 32 01 Normalizing “Should Scale”* sys tem log d -ng performance depends mostly on log length and not on the number of rules: http://blog.gerhards.net/2013/01/performance-of-liblognormrsyslog-parse.html
  33. 33. rule=apache_combined:%[ {"type": "word", "name": "clientip"}, ... {"type": "char-to", "name": "agent", "extradata": """}, {"type": "literal", "text": """}, {"type": "rest", "name": "blob"} ]% rule=apache_common:%[ {"type": "word", "name": "clientip"}, ... {"type": "number", "name": "bytes"}, {"type": "rest", "name": "blob", "priority": 65535} ]% ... 33 01 Normalizing with Five Rules input(type="imfile" File="/opt/logs/example*" Tag="apache") action(type="mmnormalize" rulebase="/opt/rsyslog/multiple_rules.rb" ) if $!root <> "" then { set $.final-json = $!root; } else { set $.final-json = $!all-json; } template(name="json_lines" type="list") { property(name="$.final-json") constant(value="n") }
  34. 34. 34 01 5 Rules Result
  35. 35. 35 01 OK, so this works: rsyslog rsyslog rsyslog
  36. 36. 36 01 How about this: rsyslog rsyslog rsyslog
  37. 37. 37 01 rsyslog.conf module(load="imfile") module(load="omhiredis") input(type="imfile" File="/opt/logs/example.log" Tag="apache:") template(name="json_lines" type="list" option.json="on") {...} main_queue(queue.workerthreads="1" queue.dequeueBatchSize="100" queue.size="10000") action(type="omhiredis" mode="publish" key="rsyslog_logstash" template="json_lines") ./configure --enable-omhiredis small&light queue
  38. 38. 38 01 logstash.conf input { redis { data_type => "channel" key => "rsyslog_logstash" batch_count => 100 } } output { solr_http { ... } } JSON codec is implied
  39. 39. 39 01 Combined Result rsyslog 1% Redis 2% Logstash 200% rsyslog 10MB (10K queue) Redis 1000MB (configurable) Logstash 380MB
  40. 40. 40 01 5-Rule Normalizing Result rsyslog 100% Redis 2% Logstash 200% rsyslog 30MB Redis 1000MB Logstash 450MB
  41. 41. 41 01 Shipper conclusions rsyslog rsyslog rsyslog rsyslog rsyslog rsyslog easy setup; flexible heavy light; fast less flexible&easy offloads buffers and Logstash processing; flexible and efficient setup and maintenance overhead
  42. 42. 42 01 Solr Tuning Agenda Schema and config adjustments Time-based collections Tiered cluster (e.g. hot vs cold nodes)
  43. 43. 43 01 Schema: Two Kinds of Fields message:failed "docValues": true "omitNorms": true, "omitTermFreqAndPositions": true
  44. 44. 44 01 Schema: Two Kinds of Fields message:failed "docValues": true "omitNorms": true, "omitTermFreqAndPositions": true +20 to 100% capacity* 10% faster indexing* * http://blog.sematext.com/2014/11/17/solr-presentations-lucene-solr-revolution/
  45. 45. 45 01 Commits "updateHandler.autoSoftCommit.maxTime": 5000 "updateHandler.autoCommit.maxTime": 60000 <ramBufferSizeMB>200</ramBufferSizeMB> 5s feels near- realtime while searching Flush to disk every minute or 200MB
  46. 46. 46 01 Commits "updateHandler.autoSoftCommit.maxTime": 5000 "updateHandler.autoCommit.maxTime": 60000 <ramBufferSizeMB>200</ramBufferSizeMB> 5s feels near- realtime while searching Flush to disk every minute of 200MB +10% capacity; 10% faster indexing*
  47. 47. 47 01 Time-Based Collections indexing, merges, most searches doesn’t change => cache friendly can be optimized delete without triggering merges
  48. 48. 48 01 Time-Based Collections indexing, merges, most searches doesn’t change => cache friendly => can be optimized delete without triggering merges 20-30x capacity; less indexing degradation* * http://www.slideshare.net/sematext/side-by-side-with-elasticsearch-solr-part-2
  49. 49. 49 01 Tiered Cluster hot1 hot2 cold1 cold2 cold3 cold4
  50. 50. 50 01 Tiered Cluster hot1 hot2 cold1 cold2 cold3 cold4
  51. 51. 51 01 Tiered Cluster hot1 hot2 cold1 cold2 cold3 cold4 ADDREPLICA
  52. 52. 52 01 Tiered Cluster hot1 hot2 cold1 cold2 cold3 cold4
  53. 53. 53 01 Tiered Cluster hot1 hot2 cold1 cold2 cold3 cold4
  54. 54. 54 01 Tiered Cluster hot1 hot2 cold1 cold2 cold3 cold4
  55. 55. 55 01 Tiered Cluster hot1 hot2 cold1 cold2 cold3 cold4 quick recent searches and indexing rare lengthy requests
  56. 56. 56 01 Tiered Cluster cold1 cold2 cold3 cold4 quick recent searches and indexing rare lengthy requests hot1 hot2 buffer for indexing spikes
  57. 57. 57 01 Tiered Cluster cold1 cold2 cold3 cold4 quick recent searches and indexing rare lengthy requests hot1 hot2 buffer for indexing spikes less shards per collection and the cluster is still balanced
  58. 58. 58 01 Tiered Cluster cold1 cold2 cold3 cold4 quick recent searches and indexing rare lengthy requests hot1 hot2 buffer for indexing spikes less shards per collection and the cluster is still balanced CPU++ RAM++ IO++
  59. 59. 59 01 Wrap-Up
  60. 60. 60 01 Wrap-Up DocValues commits
  61. 61. 61 01 Wrap-Up DocValues commits https://cdn0.iconfinder.com/data/icons/dance-fitness/72/13-512.png https://www.standardlife.co.uk/resources/custom/uk/images/heroes/illustration/easy-box.png
  62. 62. 62 01 Wrap-Up DocValues commits https://cdn0.iconfinder.com/data/icons/dance-fitness/72/13-512.png https://www.standardlife.co.uk/resources/custom/uk/images/heroes/illustration/easy-box.png
  63. 63. 63 01 Wrap-Up DocValues commits http://www.funnyshirts.net/media/catalog/product/cache/1/image/9df78eab33525d08d6e5fb8d27136e95/z/o/zombies-hate-fast-food-funny-tshirt-preview.png https://cdn0.iconfinder.com/data/icons/dance-fitness/72/13-512.png https://www.standardlife.co.uk/resources/custom/uk/images/heroes/illustration/easy-box.png
  64. 64. 64 01 Wrap-Up DocValues commits http://www.funnyshirts.net/media/catalog/product/cache/1/image/9df78eab33525d08d6e5fb8d27136e95/z/o/zombies-hate-fast-food-funny-tshirt-preview.png https://cdn0.iconfinder.com/data/icons/dance-fitness/72/13-512.png https://www.standardlife.co.uk/resources/custom/uk/images/heroes/illustration/easy-box.png rsyslog
  65. 65. 65 01 Wrap-Up DocValues commits http://www.funnyshirts.net/media/catalog/product/cache/1/image/9df78eab33525d08d6e5fb8d27136e95/z/o/zombies-hate-fast-food-funny-tshirt-preview.png https://cdn0.iconfinder.com/data/icons/dance-fitness/72/13-512.png https://www.standardlife.co.uk/resources/custom/uk/images/heroes/illustration/easy-box.png rsyslog rsyslog rsyslog rsyslog
  66. 66. 66 01 Questions? Rafał Kuć @kucrafal rafal.kuc@sematext.com Radu Gheorghe @radu0gheorghe radu.gheorghe@sematext.com Sematext @sematext http://sematext.com
  67. 67. 67 01 Questions? Rafał Kuć @kucrafal rafal.kuc@sematext.com Radu Gheorghe @radu0gheorghe radu.gheorghe@sematext.com Sematext @sematext http://sematext.com we’re hiring, too!

×