Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
rsyslog v8 improvements 
and plugins in ANY language 
Rainer Gerhards
First things first: 
Version check :-) 
• Current stable is 8.2.x 
• Don't use outdated cruft like 5.8.x (!) 
• Packages a...
What's in this talk? 
• v8 improvements 
▫ Infrastructure things 
▫ The v8 engine 
• Writing plugins in languages other th...
The rsyslog doc project 
• The doc
The rsyslog doc project 
• The doc just sucks... 
• Spawned a new project to create better one: 
https://github.com/rsyslo...
Community involvement 
• Rsyslog was traditionally open to contributions and 
has a lively community 
• Yet we still try t...
What is new in the v8 engine? 
• Output part totally revamped 
• Simplified script execution 
▫ Even faster execution spee...
The v7 rule engine 
rsyslog 
core 
Queue worker 
Queue queue worker Action instance 
Queue worker 
Single-thread 
compartm...
What was the problem with the v7 
engine and slow outputs? 
• Traditionally, an action instance is non-reentrant 
• Multip...
Why is this a problem e.g. for 
Elasticsearch? 
• Rsyslog-processing is NOT time-dominant 
• Transfer via HTTP takes its t...
The v8 rule engine 
rsyslog 
core 
Queue worker 
Queue worker 
Queue worker 
Action wrkr inst. 
queue 
 Now multiple inst...
What if the destination cannot 
handle multiple workers? 
• A typical example is file output 
• Framework will still call ...
Writing plugins 
• Traditionally, plugins 
▫ Are written in C 
▫ Macros hide interface plumbing 
▫ Fairly easy to write fo...
Types of Plugins 
• Output (actions) 
▫ Deliver message to some destination system, e.g. file, 
ElasticSearch, MongoDB, So...
Rainer Gerhards 
Writing external output plugins 
for rsyslog
Interface Overview 
rsyslog 
core 
engine 
Internal plugin 
external plugin 
connector 
process border 
perl plugin 
pytho...
External Interface Design Goals 
• Keep it stupid simple 
▫ Must support almost any language 
▫ Dumb easy to use even for ...
Interface Details: communication 
• use pipes 
• stdin 
▫ one message per line 
▫ format can be customized via rsyslog tem...
Interface Details: Threading 
• Do NOT care about threading 
• Write app according to single-thread paradigm 
• rsyslog wi...
Startup & Termination 
• rsyslog will startup the plugin automatically 
• Plugin needs to read stdin until EOF 
• Do NOT t...
Skeletons 
• The rsyslog project provides sample plugin 
skeletons 
• Available in ./plugins/external/skeletons 
• These c...
Call to Action 
• If you need to send logs to a destination that is not 
yet supported, you can quickly write an external ...
Rainer Gerhards 
Writing external output plugins 
for RSysLog 
IN 2 MINUTES
Write the plugin itself 
• Choose any language you like 
• Implement the pseudocode below 
▫ Messages arrive via stdin, on...
Make RsysLog call plugin 
• Regular filtering applies (as with any action) 
• You can specify message format via a templat...
Optional: debugging your plugin 
• If something doesn't work, it's best to debug outside 
of rsyslog 
• Do this as you usu...
Questions? 
rgerhards@adiscon.com 
www.rsyslog.com 
https://github.com/rsyslog
Upcoming SlideShare
Loading in …5
×

RSYSLOG v8 improvements and how to write plugins in any language.

10,549 views

Published on

RSYSLOG is a next generation log processing tool. In the frist part, we will explain the new RSYSLOG v8 engine, its motivation and its benefits. Learn, for example, why writing to Elasticsearch is much faster with the new engine. We will describe the tuning parameters vital for making best use of the new features.

In the second part we will explain how to write RSYSLOG plugins in any language. Traditionally, writing rsyslog plugins has been considered quite hard, with at least C knowledge necessary. In v8, we have introduced new interfaces which make it possible to write plugins in any language - be it Python, Perl or Java. Even bash will do. In essence, this is a great tool for any admin to add special needs with just a bit of scripting. We will proivde concrete instructions on how to write a plugin, point to read-to-copy samples and tell how to integrate this into rsyslog.

NOTE: This is my LinuxTag Berlin 2014 talk.

Published in: Technology

RSYSLOG v8 improvements and how to write plugins in any language.

  1. 1. rsyslog v8 improvements and plugins in ANY language Rainer Gerhards
  2. 2. First things first: Version check :-) • Current stable is 8.2.x • Don't use outdated cruft like 5.8.x (!) • Packages available at http://www.rsyslog.com
  3. 3. What's in this talk? • v8 improvements ▫ Infrastructure things ▫ The v8 engine • Writing plugins in languages other than C • I will probably “run out of time” - but that's fine, the slides at the end are optional.
  4. 4. The rsyslog doc project • The doc
  5. 5. The rsyslog doc project • The doc just sucks... • Spawned a new project to create better one: https://github.com/rsyslog/rsyslog-doc • Initiated by James Boylan (a sysadmin) • Please help ▫ Complain ;-) ▫ open issues ▫ Write some doc... • We are especially interested to learn what is hard for beginners!
  6. 6. Community involvement • Rsyslog was traditionally open to contributions and has a lively community • Yet we still try to improve things and have made a couple of moves to encourage further contributions • Rsyslog & subprojects on github • Project Admins are David Lang (not affiliated), Tomas Heinrich (Red Hat) and me • Subprojects with different commit levels
  7. 7. What is new in the v8 engine? • Output part totally revamped • Simplified script execution ▫ Even faster execution speed ▫ Less complex code • Even higher scalability • Global variable support • Required changes to output module interface • Support for plugins in any language
  8. 8. The v7 rule engine rsyslog core Queue worker Queue queue worker Action instance Queue worker Single-thread compartment  Filter processing  Message formatting  Actual output action, like sending msg  Kept simple & single threaded  Works well with fast actions  Has problems with slow ones, e.g. via HTTP (like Elasticsearch)
  9. 9. What was the problem with the v7 engine and slow outputs? • Traditionally, an action instance is non-reentrant • Multiple threads are used for ▫ Filtering ▫ Template generation ▫ Execution of independent actions • But the SAME action could not benefit from the thread pool • Not a problem for traditional fast outputs
  10. 10. Why is this a problem e.g. for Elasticsearch? • Rsyslog-processing is NOT time-dominant • Transfer via HTTP takes its time ▫ Need to wait on result ▫ Both ES and rsyslog mostly idle in this process • ES request relatively slow, scaling via multiple connections • ES can process many requests concurrently • With the new engine, we can spawn multiple concurrent connections from the same action, and thus keep both ES and rsyslog busy and de-tangle it from the slow parts!
  11. 11. The v8 rule engine rsyslog core Queue worker Queue worker Queue worker Action wrkr inst. queue  Now multiple instances per action!  Queue worker pool automatically scales outbound connection count by spawning more worker instances  Works well with Elasticsearch etc.  Inherently serial outputs (e.g. local files!) must serialize themselves Action wrkr inst. Action wrkr inst.
  12. 12. What if the destination cannot handle multiple workers? • A typical example is file output • Framework will still call into multiple workers • Worker must use pData mutex to protect itself – this is not done automatically! • Basically like in v7, but plugin needs to take care Action instance 1 Action instance 2 wrkr 1 wrkr 2 wrkr 1 wrkr 2 wrkr 3 “real” code Action-instance mutexes “real” code
  13. 13. Writing plugins • Traditionally, plugins ▫ Are written in C ▫ Macros hide interface plumbing ▫ Fairly easy to write for the C-literate ▫ Still perceived as “complicated” • V8 goal ▫ Enable everyone to write plugins (sysadmins!) ▫ Support any language (Python, Perl, ...) ▫ Ability to execute security-sensitive plugin out of rsyslog security context
  14. 14. Types of Plugins • Output (actions) ▫ Deliver message to some destination system, e.g. file, ElasticSearch, MongoDB, Solr, ... ▫ Any language supported in v8.2.0+ • Message Modification Plugins (Modules) ▫ Permit on-the-fly modification of message content (e.g. anonymization, credit card removal) ▫ Any language supported in v8.3.0+ • Input ▫ Accept input messages ▫ Currently being worked on (target 8.3.[3-5]+)
  15. 15. Rainer Gerhards Writing external output plugins for rsyslog
  16. 16. Interface Overview rsyslog core engine Internal plugin external plugin connector process border perl plugin python plugin
  17. 17. External Interface Design Goals • Keep it stupid simple ▫ Must support almost any language ▫ Dumb easy to use even for novice programmer ▫ Do not require explicit threading • Speed is NOT the most important goal ▫ Don't make it unnecessarily slow ▫ Many real-world log destinations are slow in any case (like when you connect via http...) ▫ Focus on “enable to build solution” ▫ If necessary, conversion to internal module can be done later
  18. 18. Interface Details: communication • use pipes • stdin ▫ one message per line ▫ format can be customized via rsyslog templates ▫ multi-line messags via JSON • stdout/stderr ▫ Must NOT be written in initial version ▫ Will later convey back state information via plain text (e.g. “ERR”, “ERRMSG:xxx”, ...) • Template specifies input format (with JSON recommended for more complex cases)
  19. 19. Interface Details: Threading • Do NOT care about threading • Write app according to single-thread paradigm • rsyslog will spawn multiple instances of your plugin if there is need to do so ▫ Happens based on config in busy cases ▫ Works well in most cases (e.g. http connects) ▫ Can be disabled if necessary ▫ If your program can run in multiple ter-minal sessions concurrently, it can also be run as multiple rsyslog action instances.
  20. 20. Startup & Termination • rsyslog will startup the plugin automatically • Plugin needs to read stdin until EOF • Do NOT terminate before EOF is reached • On EOF, cleanup and terminate • If the plugin dies, rsyslog restarts a new instance • Some signals (like sigint) are blocked and should remain so
  21. 21. Skeletons • The rsyslog project provides sample plugin skeletons • Available in ./plugins/external/skeletons • These contain ▫ the necessary plumbing ▫ often a kind of abstraction layer to make writing plugins even easier ▫ often performance-enhancement features • Can simply be copied to create your own plugins, don't care about the (minimal) plumbing!
  22. 22. Call to Action • If you need to send logs to a destination that is not yet supported, you can quickly write an external plugin – in any language you know! • Writing rsyslog plugins is easy ▫ If there is already a skeleton for your language, copy it and add your app-specific code ▫ If not ... no problem, the interface is dumb easy If you can write a script that reads stdin and does something useful with it, you can also write a rsyslog plugin!
  23. 23. Rainer Gerhards Writing external output plugins for RSysLog IN 2 MINUTES
  24. 24. Write the plugin itself • Choose any language you like • Implement the pseudocode below ▫ Messages arrive via stdin, one message per line ▫ Read from stdin until EOF ▫ Process each message read as you like ▫ Terminate when EOF is reached • That's it! While not EOF(stdin) do { Read msg from stdin Process msg }
  25. 25. Make RsysLog call plugin • Regular filtering applies (as with any action) • You can specify message format via a template • Use omprog for the call module(load=”omprog”) # needed only once in config! if $rawmsg contains “sometrigger” then action(type=”omprog” binary=”/path/to/your/plugin”)
  26. 26. Optional: debugging your plugin • If something doesn't work, it's best to debug outside of rsyslog • Do this as you usually debug your programs (e.g. use your favorite debugger!) • For example, do $ echo “testmessage” | /path/to/your/plugin • Questions about the plugin interface or plugin integration? Visit http://kb.monitorware.com/external-plugins-f53.html
  27. 27. Questions? rgerhards@adiscon.com www.rsyslog.com https://github.com/rsyslog

×