1. rsyslog v8 improvements
and plugins in ANY language
Rainer Gerhards

2. First things first:
Version check :-)
• Current stable is 8.2.x
• Don't use outdated cruft like 5.8.x (!)
• Packages available at
http://www.rsyslog.com

3. What's in this talk?
• v8 improvements
▫ Infrastructure things
▫ The v8 engine
• Writing plugins in languages other than C
• I will probably “run out of time” - but that's fine, the
slides at the end are optional.

4. The rsyslog doc project
• The doc

5. The rsyslog doc project
• The doc just sucks...
• Spawned a new project to create better one:
https://github.com/rsyslog/rsyslog-doc
• Initiated by James Boylan (a sysadmin)
• Please help
▫ Complain ;-)
▫ open issues
▫ Write some doc...
• We are especially interested to learn what is hard for
beginners!

6. Community involvement
• Rsyslog was traditionally open to contributions and
has a lively community
• Yet we still try to improve things and have made a
couple of moves to encourage further contributions
• Rsyslog & subprojects on github
• Project Admins are David Lang (not affiliated),
Tomas Heinrich (Red Hat) and me
• Subprojects with different commit levels

7. What is new in the v8 engine?
• Output part totally revamped
• Simplified script execution
▫ Even faster execution speed
▫ Less complex code
• Even higher scalability
• Global variable support
• Required changes to output module interface
• Support for plugins in any language

8. The v7 rule engine
rsyslog
core
Queue worker
Queue queue worker Action instance
Queue worker
Single-thread
compartment
 Filter processing
 Message formatting
 Actual output action, like sending msg
 Kept simple & single threaded
 Works well with fast actions
 Has problems with slow ones, e.g.
via HTTP (like Elasticsearch)

9. What was the problem with the v7
engine and slow outputs?
• Traditionally, an action instance is non-reentrant
• Multiple threads are used for
▫ Filtering
▫ Template generation
▫ Execution of independent actions
• But the SAME action could not benefit from the
thread pool
• Not a problem for traditional fast outputs

10. Why is this a problem e.g. for
Elasticsearch?
• Rsyslog-processing is NOT time-dominant
• Transfer via HTTP takes its time
▫ Need to wait on result
▫ Both ES and rsyslog mostly idle in this process
• ES request relatively slow, scaling via multiple
connections
• ES can process many requests concurrently
• With the new engine, we can spawn multiple
concurrent connections from the same
action, and thus keep both ES and rsyslog busy and
de-tangle it from the slow parts!

11. The v8 rule engine
rsyslog
core
Queue worker
Queue worker
Queue worker
Action wrkr inst.
queue
 Now multiple instances per action!
 Queue worker pool automatically
scales outbound connection count
by spawning more worker instances
 Works well with Elasticsearch etc.
 Inherently serial outputs (e.g. local files!)
must serialize themselves
Action wrkr inst.
Action wrkr inst.

12. What if the destination cannot
handle multiple workers?
• A typical example is file output
• Framework will still call into multiple workers
• Worker must use pData mutex to protect itself – this is
not done automatically!
• Basically like in v7, but plugin needs to take care
Action
instance
1
Action
instance
2
wrkr 1
wrkr 2
wrkr 1
wrkr 2
wrkr 3
“real” code
Action-instance mutexes
“real” code

13. Writing plugins
• Traditionally, plugins
▫ Are written in C
▫ Macros hide interface plumbing
▫ Fairly easy to write for the C-literate
▫ Still perceived as “complicated”
• V8 goal
▫ Enable everyone to write plugins (sysadmins!)
▫ Support any language (Python, Perl, ...)
▫ Ability to execute security-sensitive plugin out of
rsyslog security context

14. Types of Plugins
• Output (actions)
▫ Deliver message to some destination system, e.g. file,
ElasticSearch, MongoDB, Solr, ...
▫ Any language supported in v8.2.0+
• Message Modification Plugins (Modules)
▫ Permit on-the-fly modification of message content
(e.g. anonymization, credit card removal)
▫ Any language supported in v8.3.0+
• Input
▫ Accept input messages
▫ Currently being worked on (target 8.3.[3-5]+)

15. Rainer Gerhards
Writing external output plugins
for rsyslog

16. Interface Overview
rsyslog
core
engine
Internal plugin
external plugin
connector
process border
perl plugin
python plugin

17. External Interface Design Goals
• Keep it stupid simple
▫ Must support almost any language
▫ Dumb easy to use even for novice programmer
▫ Do not require explicit threading
• Speed is NOT the most important goal
▫ Don't make it unnecessarily slow
▫ Many real-world log destinations are slow in any case
(like when you connect via http...)
▫ Focus on “enable to build solution”
▫ If necessary, conversion to internal module can be
done later

18. Interface Details: communication
• use pipes
• stdin
▫ one message per line
▫ format can be customized via rsyslog templates
▫ multi-line messags via JSON
• stdout/stderr
▫ Must NOT be written in initial version
▫ Will later convey back state information via plain text
(e.g. “ERR”, “ERRMSG:xxx”, ...)
• Template specifies input format (with JSON
recommended for more complex cases)

19. Interface Details: Threading
• Do NOT care about threading
• Write app according to single-thread paradigm
• rsyslog will spawn multiple instances of your plugin
if there is need to do so
▫ Happens based on config in busy cases
▫ Works well in most cases (e.g. http connects)
▫ Can be disabled if necessary
▫ If your program can run in multiple ter-minal
sessions concurrently, it can also be run as
multiple rsyslog action instances.

20. Startup & Termination
• rsyslog will startup the plugin automatically
• Plugin needs to read stdin until EOF
• Do NOT terminate before EOF is reached
• On EOF, cleanup and terminate
• If the plugin dies, rsyslog restarts a new instance
• Some signals (like sigint) are blocked and should
remain so

21. Skeletons
• The rsyslog project provides sample plugin
skeletons
• Available in ./plugins/external/skeletons
• These contain
▫ the necessary plumbing
▫ often a kind of abstraction layer to make writing
plugins even easier
▫ often performance-enhancement features
• Can simply be copied to create your own plugins,
don't care about the (minimal) plumbing!

22. Call to Action
• If you need to send logs to a destination that is not
yet supported, you can quickly write an external
plugin – in any language you know!
• Writing rsyslog plugins is easy
▫ If there is already a skeleton for your language, copy it
and add your app-specific code
▫ If not ... no problem, the interface is dumb easy
If you can write a script that reads stdin and does
something useful with it, you can also write a
rsyslog plugin!

23. Rainer Gerhards
Writing external output plugins
for RSysLog
IN 2 MINUTES

24. Write the plugin itself
• Choose any language you like
• Implement the pseudocode below
▫ Messages arrive via stdin, one message per line
▫ Read from stdin until EOF
▫ Process each message read as you like
▫ Terminate when EOF is reached
• That's it!
While not EOF(stdin) do {
Read msg from stdin
Process msg
}

25. Make RsysLog call plugin
• Regular filtering applies (as with any action)
• You can specify message format via a template
• Use omprog for the call
module(load=”omprog”) # needed only once in config!
if $rawmsg contains “sometrigger” then
action(type=”omprog”
binary=”/path/to/your/plugin”)

26. Optional: debugging your plugin
• If something doesn't work, it's best to debug outside
of rsyslog
• Do this as you usually debug your programs (e.g. use
your favorite debugger!)
• For example, do
$ echo “testmessage” | /path/to/your/plugin
• Questions about the plugin interface or plugin
integration? Visit
http://kb.monitorware.com/external-plugins-f53.html

27. Questions?
rgerhards@adiscon.com
www.rsyslog.com
https://github.com/rsyslog