rsyslog v8 provides more capabilities than just basic syslog functionality. It can parse, modify, and convert log messages to JSON before sending them to ElasticSearch for storage and analysis. It supports reliable delivery with RELP, metrics collection, lookup tables, and runtime control through a control socket. Rsyslog provides powerful configuration options through RainerScript for flexible parsing, modification, and routing of log messages.

1. rsyslog v8:
more than just syslog!
Yury Bushmelev
Lazada

2. what people usually think
rsyslog is
/dev/log
udp:514
some
magic
/var/log/some.log
remote udp:514

3. what rsyslog says it is

4. "some magic"
queue
input
pre-
processor
action
queue
parser
&
filter
action
processor
output
input
input
action
queue
action
queue
action
processor
action
processor
output
output
ruleset

5. rulesets
input(...)
input(...)
if ($syslogtag ==
"prog1") then {
action(...)
} else {
action(...)
}
input(...
ruleset="rules1")
input(...
ruleset="rules2")
ruleset(name="rules1") {
action(...)
}
ruleset(name="rules2") {
action(...)
}

6. configuration formats
• basic AKA sysklogd (*.* /var/log/file.log) – still OK in
simple cases
• obsolete legacy ($UglyScaryThings everywhere) – Do not
use it at all! "It will make your life miserable" © Rainer
Gerhards
• advanced AKA RainerScript – new (2010) readable
configuration language

7. queues
• most confusing part of rsyslog configuration

8. queues
• most confusing part of rsyslog configuration
• 2 places
• ruleset queue
• action queue

9. queues
• most confusing part of rsyslog configuration
• 2 places
• ruleset queue
• action queue
• 4 modes
• direct (AKA no queue, sync)
• in-memory
• on-disk
• disk-assisted (DA, in-memory + on-disk)

10. queues and performance
• try to increase ruleset queue workers number first
• action queue is mostly about string building
• for fast action (omfile) thread syncing overhead may be
significant
• for single-action ruleset everything is complicated...
• read more details in Rainer's blog: http://
blog.gerhards.net/2013/06/rsyslog-performance-main-
and-action.html

11. – ELK?
• read log messages from file/network
• parse
• modify
• convert to JSON
• send to ElasticSearch

12. – ERK!
• read log messages from file/network (inputs)
• parse (builtin parser/RainerScript/mm* modules)
• modify (RainerScript/mm* modules/templates)
• convert to JSON (templates)
• send to ElasticSearch (omelasticsearch)

13. parse (mmnormalize)
• "mmnormalize" is unique feature of rsyslog based on
liblognorm library which...
• ... is amazingly fast
• ... because it's using parse tree (not regexp/grok)
• ... may try multiple rules for same message (first win)
• ... have some documentation:
http://www.liblognorm.com/files/manual/index.html

14. liblognorm rules
version=2
rule=v0:MyAuthAPI: Invalid user %user:word%
from %src-ip:ipv4%
rule=v1:MyAuthAPI[%pid:number%]: Invalid
user %user:word% from %src-ip:ipv4%

15. modify/send as JSON v1
template(name="tmpl_to_json_v1" type="string" string="%$!all-json%")
template(name="timereported_rfc3339" type="string"
string="%timereported:::date-rfc3339%")
ruleset(name="test") {
action(type="mmjsonparse")
set $!_log_time = exec_template("timereported_rfc3339");
set $!_log_tag = $!tag;
unset $!tag;
...
action(type="omelasticsearch"
server="es.example.com"
template="tmpl_to_json_v1"
...)
}

16. modify/send as JSON v2
template(name="tmpl_to_json_v2" type="list" option.json="on") {
constant(value="{")
property(outname="_log_time" name="timereported" dateFormat="rfc3339"
format="jsonf")
constant(value=",")
property(outname="_log_tag" name="tag" format="jsonf")
constant(value=",")
property(outname="message" name="msg" dropLastLF="on" format="jsonf")
constant(value="}")
}
ruleset(name="test") {
action(type="omelasticsearch"
server="es.example.com"
template="tmpl_to_json_v2"
...)
}

17. reliable delivery
• message will be lost when using UDP delivery
• message may be lost when using TCP delivery

18. reliable delivery
• message will be lost when using UDP delivery
• message may be lost when using TCP delivery
• ... so there is RELP (imrelp/omrelp)
• ... it may do TLS with compression easy as well

19. reliable delivery
• message will be lost when using UDP delivery
• message may be lost when using TCP delivery
• ... so there is RELP (imrelp/omrelp)
• ... it may do TLS with compression easy as well
• ... but it's single-threaded :(

20. metrics (impstats)
{ "name": "global", "origin": "dynstats", "values": { } }
{ "name": "imuxsock", "origin": "imuxsock", "submitted": 3612711,
"ratelimit.discarded": 0, "ratelimit.numratelimiters": 0 }
{ "name": "action 0", "origin": "core.action", "processed":
33429, "failed": 0, "suspended": 0, "suspended.duration": 0,
"resumed": 0 }
{ "name": "resource-usage", "origin": "impstats", "utime":
290424444, "stime": 314385112, "maxrss": 3176, "minflt": 509,
"majflt": 105, "inblock": 5920, "oublock": 11030360, "nvcsw":
7281385, "nivcsw": 7506 }
{ "name": "main Q", "origin": "core.queue", "size": 0,
"enqueued": 3612765, "full": 0, "discarded.full": 0,
"discarded.nf": 0, "maxqsize": 30 }
• read more here: https://www.rsyslog.com/doc/v8-stable/configuration/
rsyslog_statistic_counter.html

21. metrics (dyn_stats)
dyn_stats(name="msg_per_host" resettable="on"
maxCardinality="3000" unusedMetricLife="600")
set $.inc = dyn_inc("msg_per_host", $hostname);
# impstats output
# { "name": "msg_per_host", "origin":
"dynstats.bucket", "values": { "lab3.sgdc":
1002540, "lab4.sgdc": 1318551, "lab7.sgdc":
1788136} }

22. lookup tables
{ "version" : 1,
"nomatch" : "unknown",
"type" : "string",
"table" : [
{"index" : "10.0.1.1", "value" : "A" },
{"index" : "10.0.1.2", "value" : "A" },
{"index" : "10.0.1.3", "value" : "A" },
{"index" : "10.0.2.1", "value" : "B" },
{"index" : "10.0.2.2", "value" : "B" },
{"index" : "10.0.2.3", "value" : "B" }]}

23. lookup tables
lookup_table(name="host_bu"
file="/var/lib/host_billing_unit_mapping.json"
reloadOnHUP="on"
)
set $.bu = lookup("host_bu", $hostname);
if ($.bu != "unknown") then {
...
}
# Go to "bu_A"/"bu_B"/"bu_unknown" ruleset:
# call_indirect "bu_" & $.bu;

24. lookup tables
if ($.do_reload == "y") then {
reload_lookup_table("host_bu", "unknown")
}

25. runtime control
input(type="imuxsock" socket="/run/rsyslog-control.sock"
ruleset="control")
ruleset(name="control") {
if ($msg == "reload bu") then {
reload_lookup_table("host_bu", "unknown")
} else if ($msg == "run cmd") then {
action(type="omprog" binary="/path/to/script.sh")
} else {
action(type="omfile" file="/var/log/rsyslog-control.log")
}
}

26. any questions?
• Yury Bushmelev
• https://twitter.com/Jay7t
• https://github.com/jay7x