Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Central LogFile Storage. ELK stack Elasticsearch, Logstash and Kibana.

Central LogFile Storage. ELK stack Elasticsearch, Logstash and Kibana. Central Log Storage. Diagram. FileBeat ssl Logstash. LogStash, FileBeat config file example.

  • Login to see the comments

Central LogFile Storage. ELK stack Elasticsearch, Logstash and Kibana.

  1. 1. Central Log Storage. ELK stack(a.k.a., Elasticsearch, Logstash, and Kibana) 1
  2. 2. List of Log Files. Central Log Storage. 2
  3. 3. Central Log Storage. Diagram. 3
  4. 4. FileBeat config file. YML standard FileBeat config file. YML 4 vim /etc/filebeat/filebeat.yml - paths: - ${API_APP_LOG_PATH}/file.log encoding: plain input_type: log fields: level: apico_backend review: 1 fields_under_root: false ignore_older: 0 document_type: api_backend_requests scan_frequency: 2s multiline: pattern: ^[[:digit:]]{4} negate: true match: after max_lines: 500 timeout: 2s tail_files: false ### Logstash as output logstash: # The Logstash hosts hosts: [""] # Number of workers per Logstash host. worker: 2 # Optional TLS. By default is off. tls: certificate_authorities:["/etc/ssl/certs/rootCA.crt"] insecure: false
  5. 5. Using Environment Variblaes in Configuration Logstash config source Link environment-variables @see /etc/sysconfig/filebeat Filebeat_using_env_vars Environment Logstash config result input { tcp { port => "${TCP_PORT}" } } export TCP_PORT=12345 input { tcp { port => 12345 } } input { tcp { port => "${TCP_PORT}" } } No TCP_PORT defined Raise a logstash configuration error input { tcp { port => "${TCP_PORT:54321}" } } No TCP_PORT defined input { tcp { port => 54321 } } filter { mutate { add_field => { "my_path" => "${HOME}/file.log" } } export HOME="/path" filter { mutate { add_field => { "my_path" => "/path/file.log" } } 5
  6. 6. SSL Certificate Authority Certificate Authority — в 5 OpenSSL команд Using TLS between Beats and Logstash Validate crt key: [root@MSG-predprod lostash_pp]# pwd /etc/ssl/certs/lostash_pp [root@MSG-predprod lostash_pp]# curl -v --cacert rootCA.crt * Rebuilt URL to: * Connected to ( port 9999 (#0) * CAfile: rootCA.crt * Server certificate: * subject:,,OU=ITY,O=Default Company Ltd,L=Moscow,ST=Moscow,C=MT * start date: Apr 19 11:59:50 2016 GMT * expire date: Sep 05 11:59:50 2043 GMT * common name: > GET / HTTP/1.1 > Host: > Accept: */* * Empty reply from server [root@MSG-predprod lostash_pp]# 6 Security:SSL Certificate AUthority.
  7. 7. Beats Platform: Collect, Parse, and Ship Don't stop the Beat Packetbeat - the best way to understand the value of a network packet analytics system like Packetbeat is to try it on your own traffic. Topbeat - helps you monitor the availability of your website or service by providing system-wide and per-process statistics along with a disk usage overview. Filebeat - read data from log files 7
  8. 8. LogStash Input config file. LogStash Input config file. 8 vim /etc/logstash/conf.d/logstash.conf input { beats { #plugins port => 9999 host => "" ssl_certificate => "/etc/ssl/certs/rootCA.crt" ssl_key => "/etc/pki/tls/private/rootCA.key" } rabbitmq { exchange => "event-sms-in" queue => "logstash-queue-pp" } } filter { ... } output { file{ path => "/logstash/%{+YYYY-MM-dd}-input.log" } rabbitmq { exchange => "logstash-exchange" key => "logstash-routing-key" } }
  9. 9. Get data from Rabbit. Get data from RabbitMq (Logstash plugin) 9 Get data from RabbitMq.
  10. 10. RabbitMq configuration. Output RabbitMq configuration. 10 Hostname IP Exchange logstash-exchange Exchange_type direct Routing Key logstash-routing-key User user
  11. 11. LogStash Filter config file. LogStash Filters. Filter for “log_status”. WIKI: Добавление параметра message_sid в файла логирования для компонентов SmsController RadiusController. Filter messages from log file by STATUS: 11 log file message status error warning info log_status error warning info
  12. 12. LogStash Filter config file. LogStash Filters. Filter for “account_sid”. Filter API request by User Auth Status/Role: 12 user auth status/role User unauthorized User:: ROLE_SYSTEM User::ROLE_ACCOUNT account_sid unknown system acc01fe181e-741b-3693-88bb-3847abfb6e86 sac01fe181e-741b-3693-88bb-3847abfb6e55
  13. 13. LogStash Filter config file. LogStash Filters. Filter for “message_sid”. Filter SmsController RadiusController SmsSender by message_sid: 13 Source Consoler_app: [MessageSid:msgXXX] SmsSender_app:"message_sid":"msgXXX" message_sid msgXXX msgXXX
  14. 14. LogStash Filter config file. LogStash Filters. Filter for “message_sid”. Filter SmsController RadiusController SmsSender by message_sid: 14 Source Consoler_app: [MessageSid:msgXXX] SmsSender_app:"message_sid":"msgXXX" message_sid msgXXX msgXXX
  15. 15. LogStash List og plugins.. LogStash Plugins. 15 logstash-codec logstash-filter logstash-input logstash-output collectd dots edn edn_lines es_bulk fluent graphite json json_lines line msgpack multiline netflow oldlogstashjson plain rubydebug anonymize checksum clone csv date dns drop Fingerprint geoip grok json Kv metrics multiline mutate ruby sleep split syslog_pri throttle urldecode useragent uuid xm beats couchdb_changes elasticsearch eventlog exec file ganglia gelf generator graphite heartbeat http http_poller Imap irc Jdbc kafka log4j lumberjack pipe rabbitmq redis s3 snmptrap sqs stdin syslog tcp twitter udp unix xmpp zeromq cloudwatch csv elasticsearch email exec file ganglia gelf graphite hipchat http irc juggernaut kafka lumberjack nagios nagios_nsca null opentsdb pagerduty pipe rabbitmq redis s3 sns sqs statsd stdout tcp udp xmpp zeromq logstash-patterns-core
  16. 16. Nginx Kibana: Authentication. Nginx-to-proxy Nginx Restricting Access Nginx Reverse Proxy 16 Nginx Kibana: Authentication. Nginx-to-proxy
  17. 17. Kibana. Discover. Search. ElasticSearch Query. 17 Kibana: The Main Components. Discover.
  18. 18. Kibana. Visualize. Visualization type. 18 Kibana: The Main Components. Visualize.
  19. 19. Kibana. Dashboard. EL Query + Visualization types 19 Kibana: The Main Components.
  20. 20. ElasticSearch Query DSL. ES Query DSL ES Query DSL (Webinar Video) 20 ElasticSearch Query DSL
  21. 21. Debug. Troubleshooting. Filebeat: 1. filebeat -e -d "publish" -c /etc/filebeat/filebeat.yml -configtest 2. .(точка)(пробел) /etc/sysconfig/filebeat 3. filebeat -e -d "*" -c /etc/filebeat/filebeat.yml Logstash: 1. /opt/logstash/bin/logstash --config /etc/logstash/conf.d/logstash.conf --configtest 2. /opt/logstash/bin/logstash --verbose --config /etc/logstash/conf.d/logstash.conf 21 Debug. Troubleshooting.
  22. 22. Inspiration manual. 1.Собираем и анализируем логи с помощью Lumberjack+Logstash+Elasticsearch+RabbitMQ (The Guardian ) 2. Wiki MTT.Innovations. АPICO.CentralLogStorage. 3. Import to CvS Excel . Read Comments 4. Logstash alert. The throttle filter is for throttling the number of events received. Plugins outputs email 22 Inspiration manual.
  23. 23. End. No just the beginning. Elastic BackUp: Snapshot and restore. Backing-up-your-cluster File rotation linux BackUp & Restore(5). Elasticsearch Monitoring Java 23 Just the beginning.