The rsyslog v8 engine
A developer's view
Rainer Gerhards
What is New?
• Output part totally revamped
• Simplified script execution
▫ Even faster execution speed
▫ Less complex cod...
The v7 rule engine
Queue worker
rsyslog
core

Queue worker

queue

Single-thread
compartment

Action instance

Queue worke...
V7 Action data structures
• Global data, once per module
▫ Solely for setting default parameters

• pData: per action-inst...
Real-World Sample: ommysql
typedef struct _instanceData {
MYSQL *f_hmysql;
char
f_dbsrv[MAXHOSTNAMELEN+1];
unsigned int f_...
The v8 rule engine
Queue worker
queue

Queue worker

Action wrkr inst.

Queue worker

rsyslog
core

Action wrkr inst.

Act...
V8 Action data structures
• Global data, once per module
▫ Solely for setting default parameters

• pData: per action-inst...
Real-World Sample: ommysql
typedef struct _instanceData {
char
dbsrv[MAXHOSTNAMELEN+1];
unsigned int dbsrvPort;
char
dbnam...
V8 data structure sample
• One action module
• With 2 actions
• One being executed on two, one on three threads

Action
in...
What if the destination cannot
handle multiple workers?
• A typical example is file output
• Framework will still call int...
Real-World Sample: omfile
BEGINcommitTransaction
instanceData *__restrict__ const pData = pWrkrData->pData;
unsigned i;
CO...
Anything else?
• A couple of more things have changed,e.g.
▫ New callbacks for wrkr Instance handling
▫ New transaction pr...
Even more?
• Yup...
• We now support external plugins, which need
not be written in C!
• Use any language you like, e.g. P...
Upcoming SlideShare
Loading in...5
×

The rsyslog v8 engine (developer's view)

7,013

Published on

Which enhancements does the rsyslog v8 engine contain? This short presentation focuses on those changes in the output engine that greatly helps to improve performance. We describe what exactly was changed, how this looks in real code and what developers need to know before upgrading a plugin to v8.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
7,013
On Slideshare
0
From Embeds
0
Number of Embeds
19
Actions
Shares
0
Downloads
10
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

The rsyslog v8 engine (developer's view)

  1. 1. The rsyslog v8 engine A developer's view Rainer Gerhards
  2. 2. What is New? • Output part totally revamped • Simplified script execution ▫ Even faster execution speed ▫ Less complex code • Even higher scalability • Global variable support • Required changes to output module interface Rainer Gerhards * http://blog.gerhards.net
  3. 3. The v7 rule engine Queue worker rsyslog core Queue worker queue Single-thread compartment Action instance Queue worker Filter processing  Message formatting  Actual output action, like sending msg  Kept simple & single threaded  Works well with fast actions  Has problems with slow ones, e.g. via HTTP (like Elasticsearch)  Rainer Gerhards * http://blog.gerhards.net
  4. 4. V7 Action data structures • Global data, once per module ▫ Solely for setting default parameters • pData: per action-instance ▫ Automatically created “by framework” ▫ Holds all data necessary for doing action  Config parameters  Connection handles  Other state Rainer Gerhards * http://blog.gerhards.net
  5. 5. Real-World Sample: ommysql typedef struct _instanceData { MYSQL *f_hmysql; char f_dbsrv[MAXHOSTNAMELEN+1]; unsigned int f_dbsrvPort; char f_dbname[_DB_MAXDBLEN+1]; char f_dbuid[_DB_MAXUNAMELEN+1]; char f_dbpwd[_DB_MAXPWDLEN+1]; unsigned uLastMySQLErrno; uchar *f_configfile; uchar *f_configsection; uchar *tplName; } instanceData; Green: per-connection parameters Rest is config part. Rainer Gerhards * http://blog.gerhards.net
  6. 6. The v8 rule engine Queue worker queue Queue worker Action wrkr inst. Queue worker rsyslog core Action wrkr inst. Action wrkr inst. Now multiple instances per action!  Queue worker pool automatically scales outbound connection count by spawning more worker instances  Works well with ElastiSearch etc.  Inherantly serial outputs (e.g. local files!) must serialize themselves  Rainer Gerhards * http://blog.gerhards.net
  7. 7. V8 Action data structures • Global data, once per module ▫ Solely for setting default parameters • pData: per action-instance ▫ Automatically created “by framework” ▫ Primarily config parameters ▫ Maybe some global state like overall counters, mutexes, ... • pWrkrData: per action-worker-instance ▫ Connection handles ▫ Other “live” state Rainer Gerhards * http://blog.gerhards.net
  8. 8. Real-World Sample: ommysql typedef struct _instanceData { char dbsrv[MAXHOSTNAMELEN+1]; unsigned int dbsrvPort; char dbname[_DB_MAXDBLEN+1]; char dbuid[_DB_MAXUNAMELEN+1]; char dbpwd[_DB_MAXPWDLEN+1]; uchar *configfile; uchar *configsection; uchar *tplName; } instanceData; typedef struct wrkrInstanceData { instanceData pData; /* points back to action instance */ MYSQL *hmysql; unsigned uLastMySQLErrno; } wrkrInstanceData_t; Config and per-connection parameters now split, multiple instances of per-connection data. Rainer Gerhards * http://blog.gerhards.net
  9. 9. V8 data structure sample • One action module • With 2 actions • One being executed on two, one on three threads Action instance 2 Rainer Gerhards * http://blog.gerhards.net wrkr 1 wrkr 2 wrkr 1 wrkr 2 wrkr 3 External Connections module global data Action instance 1
  10. 10. What if the destination cannot handle multiple workers? • A typical example is file output • Framework will still call into multiple workers • Worker must use pData mutex to protect itself – this is not done automatically! • Basically like in v7, but plugin needs to take care Action instance 1 Action instance 2 wrkr 1 wrkr 2 “real” code Action-instance mutexes wrkr 1 wrkr 2 “real” code wrkr 3 Rainer Gerhards * http://blog.gerhards.net
  11. 11. Real-World Sample: omfile BEGINcommitTransaction instanceData *__restrict__ const pData = pWrkrData->pData; unsigned i; CODESTARTcommitTransaction pthread_mutex_lock(&pData->mutWrite); for(i = 0 ; i < nParams ; ++i) writeFile(pData, pParams, i); if(pData->bFlushOnTXEnd && pData->pStrm != NULL) { /* if we have an async writer, it controls the flush via * a timeout. */ if(!pData->bUseAsyncWriter) CHKiRet(strm.Flush(pData->pStrm)); } finalize_it: pthread_mutex_unlock(&pData->mutWrite); ENDcommitTransaction Rainer Gerhards * http://blog.gerhards.net
  12. 12. Anything else? • A couple of more things have changed,e.g. ▫ New callbacks for wrkr Instance handling ▫ New transaction processing callback ▫ New data structures for message data • Changes are hidden mostly in macro layer • But still there is work to do ▫ Usually quite easy ▫ Presentation on how to do this will follow, in the meantime use http://blog.gerhards.net/2013/11/samples-for-v8-module-conversion.html Rainer Gerhards * http://blog.gerhards.net
  13. 13. Even more? • Yup... • We now support external plugins, which need not be written in C! • Use any language you like, e.g. Python, Perl or Java... • More info at http://www.slideshare.net/rainergerhards1/writing-rsyslog-p • Note: action worker instances facilitate writing great performance external plugins Rainer Gerhards * http://blog.gerhards.net
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×