The document outlines the configuration and setup of a central log storage system using the ELK stack (Elasticsearch, Logstash, Kibana) along with Filebeat and RabbitMQ. It includes various configuration files, SSL settings, and troubleshooting steps for monitoring and managing log data. Additionally, it discusses Logstash filters for filtering log messages based on status and user authentication roles.

1. Central Log Storage.
ELK stack(a.k.a., Elasticsearch, Logstash, and Kibana)
1

2. List of Log Files. Central Log Storage.
2

3. Central Log Storage. Diagram.
3

4. FileBeat config file. YML standard
FileBeat config file. YML
4
vim /etc/filebeat/filebeat.yml
-
paths:
- ${API_APP_LOG_PATH}/file.log
encoding: plain
input_type: log
fields:
level: apico_backend
review: 1
fields_under_root: false
ignore_older: 0
document_type: api_backend_requests
scan_frequency: 2s
multiline:
pattern: ^[[:digit:]]{4}
negate: true
match: after
max_lines: 500
timeout: 2s
tail_files: false
### Logstash as output
logstash:
# The Logstash hosts
hosts: ["cls.apico.net:9999"]
# Number of workers per Logstash host.
worker: 2
# Optional TLS. By default is off.
tls: certificate_authorities:["/etc/ssl/certs/rootCA.crt"]
insecure: false

5. Using Environment Variblaes in Configuration
Logstash config source
Link environment-variables @see /etc/sysconfig/filebeat
Filebeat_using_env_vars
Environment Logstash config result
input {
tcp {
port => "${TCP_PORT}"
}
}
export TCP_PORT=12345 input {
tcp {
port => 12345
}
}
input {
tcp {
port => "${TCP_PORT}"
}
}
No TCP_PORT defined Raise a logstash configuration error
input {
tcp {
port => "${TCP_PORT:54321}"
}
}
No TCP_PORT defined input {
tcp {
port => 54321
}
}
filter {
mutate {
add_field => {
"my_path" => "${HOME}/file.log"
}
}
export HOME="/path" filter {
mutate {
add_field => {
"my_path" => "/path/file.log"
}
}
5

6. SSL Certificate Authority
Certificate Authority — в 5 OpenSSL команд
https://habrahabr.ru/post/192446/
Using TLS between Beats and Logstash
https://gist.github.com/andrewkroh/fdc7e5f3f0f0ed63a11c
Validate crt key:
[root@MSG-predprod lostash_pp]# pwd
/etc/ssl/certs/lostash_pp
[root@MSG-predprod lostash_pp]# curl -v --cacert rootCA.crt https://cls.apico.net:9999
* Rebuilt URL to: https://cls.apico.net:9999/
* Connected to cls.apico.net (172.31.13.49) port 9999 (#0)
* CAfile: rootCA.crt
* Server certificate:
* subject: E=kh.airat14@gmail.com,CN=cls.apico.net,OU=ITY,O=Default Company Ltd,L=Moscow,ST=Moscow,C=MT
* start date: Apr 19 11:59:50 2016 GMT
* expire date: Sep 05 11:59:50 2043 GMT
* common name: cls.apico.net
> GET / HTTP/1.1
> Host: cls.apico.net:9999
> Accept: */*
* Empty reply from server
[root@MSG-predprod lostash_pp]#
6
Security:SSL Certificate AUthority.

7. Beats Platform: Collect, Parse, and Ship
Don't stop the Beat
Packetbeat - the best way to understand the value of a network packet analytics system like
Packetbeat is to try it on your own traffic.
Topbeat - helps you monitor the availability of your website or service by providing system-wide
and per-process statistics along with a disk usage overview.
Filebeat - read data from log files
7

8. LogStash Input config file.
LogStash Input config file.
8
vim /etc/logstash/conf.d/logstash.conf
input {
beats { #plugins
port => 9999
host => "cls.apico.net"
ssl_certificate => "/etc/ssl/certs/rootCA.crt"
ssl_key => "/etc/pki/tls/private/rootCA.key"
}
rabbitmq {
exchange => "event-sms-in"
queue => "logstash-queue-pp"
}
}
filter {
...
}
output {
file{
path => "/logstash/%{+YYYY-MM-dd}-input.log"
}
rabbitmq {
exchange => "logstash-exchange"
key => "logstash-routing-key"
}
}

9. Get data from Rabbit.
Get data from RabbitMq (Logstash plugin)
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-rabbitmq.html
9
Get data from RabbitMq.

10. RabbitMq configuration.
Output RabbitMq configuration.
10
Hostname IP cls.apico.net
127.0.0.1
Exchange logstash-exchange
Exchange_type direct
Routing Key logstash-routing-key
User user

11. LogStash Filter config file.
LogStash Filters. Filter for “log_status”.
WIKI: Добавление параметра message_sid в файла логирования для компонентов SmsController RadiusController.
Filter messages from log file by STATUS:
11
log file message
status
error warning info
log_status error warning info

12. LogStash Filter config file.
LogStash Filters. Filter for “account_sid”.
Filter API request by User Auth Status/Role:
12
user auth
status/role
User
unauthorized
User::
ROLE_SYSTEM
User::ROLE_ACCOUNT
account_sid unknown system acc01fe181e-741b-3693-88bb-3847abfb6e86
sac01fe181e-741b-3693-88bb-3847abfb6e55

13. LogStash Filter config file.
LogStash Filters. Filter for “message_sid”.
Filter SmsController RadiusController
SmsSender by message_sid:
13
Source Consoler_app: [MessageSid:msgXXX] SmsSender_app:"message_sid":"msgXXX"
message_sid msgXXX msgXXX

14. LogStash Filter config file.
LogStash Filters. Filter for “message_sid”.
Filter SmsController RadiusController
SmsSender by message_sid:
14
Source Consoler_app: [MessageSid:msgXXX] SmsSender_app:"message_sid":"msgXXX"
message_sid msgXXX msgXXX

15. LogStash List og plugins..
LogStash Plugins.
15
logstash-codec logstash-filter logstash-input logstash-output
collectd
dots
edn
edn_lines
es_bulk
fluent
graphite
json
json_lines
line
msgpack
multiline
netflow
oldlogstashjson
plain
rubydebug
anonymize
checksum
clone
csv date
dns drop
Fingerprint geoip
grok json
Kv metrics
multiline mutate
ruby
sleep
split
syslog_pri
throttle
urldecode
useragent
uuid
xm
beats
couchdb_changes
elasticsearch
eventlog
exec
file
ganglia
gelf generator
graphite
heartbeat
http http_poller
Imap irc
Jdbc kafka
log4j lumberjack
pipe
rabbitmq
redis s3
snmptrap sqs
stdin syslog
tcp twitter
udp unix
xmpp zeromq
cloudwatch csv
elasticsearch
email
exec
file ganglia
gelf graphite
hipchat
http irc
juggernaut
kafka
lumberjack
nagios
nagios_nsca
null opentsdb
pagerduty pipe
rabbitmq redis
s3 sns
sqs statsd
stdout tcp
udp xmpp
zeromq
logstash-patterns-core

16. Nginx Kibana: Authentication. Nginx-to-proxy
Nginx Restricting Access
Nginx Reverse Proxy
16
Nginx Kibana: Authentication. Nginx-to-proxy

17. Kibana. Discover. Search. ElasticSearch Query.
17
Kibana: The Main Components. Discover.

18. Kibana. Visualize. Visualization type.
18
Kibana: The Main Components. Visualize.

19. Kibana. Dashboard. EL Query + Visualization types
19
Kibana: The Main Components.

20. ElasticSearch Query DSL.
ES Query DSL
ES Query DSL (Webinar Video)
20
ElasticSearch Query DSL

21. Debug. Troubleshooting.
Filebeat:
1. filebeat -e -d "publish" -c /etc/filebeat/filebeat.yml -configtest
2. .(точка)(пробел) /etc/sysconfig/filebeat
3. filebeat -e -d "*" -c /etc/filebeat/filebeat.yml
Logstash:
1. /opt/logstash/bin/logstash --config /etc/logstash/conf.d/logstash.conf --configtest
2. /opt/logstash/bin/logstash --verbose --config /etc/logstash/conf.d/logstash.conf
21
Debug. Troubleshooting.

22. Inspiration manual.
1.Собираем и анализируем логи с помощью Lumberjack+Logstash+Elasticsearch+RabbitMQ
(The Guardian )
2. Wiki MTT.Innovations. АPICO.CentralLogStorage.
3. Import to CvS Excel . Read Comments
https://habrahabr.ru/post/236341/
4. Logstash alert.
The throttle filter is for throttling the number of events received.
https://www.elastic.co/guide/en/logstash/current/plugins-filters-throttle.html
Plugins outputs email
https://www.elastic.co/guide/en/logstash/current/plugins-outputs-email.html
22
Inspiration manual.

23. End. No just the beginning.
Elastic BackUp: Snapshot and restore.
Backing-up-your-cluster
File rotation linux BackUp & Restore(5).
Elasticsearch Monitoring Java
23
Just the beginning.