The rsyslog v8 engine (developer's view)


Published on

Which enhancements does the rsyslog v8 engine contain? This short presentation focuses on those changes in the output engine that greatly helps to improve performance. We describe what exactly was changed, how this looks in real code and what developers need to know before upgrading a plugin to v8.

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

The rsyslog v8 engine (developer's view)

  1. 1. The rsyslog v8 engine A developer's view Rainer Gerhards
  2. 2. What is New? • Output part totally revamped • Simplified script execution ▫ Even faster execution speed ▫ Less complex code • Even higher scalability • Global variable support • Required changes to output module interface Rainer Gerhards *
  3. 3. The v7 rule engine Queue worker rsyslog core Queue worker queue Single-thread compartment Action instance Queue worker Filter processing  Message formatting  Actual output action, like sending msg  Kept simple & single threaded  Works well with fast actions  Has problems with slow ones, e.g. via HTTP (like Elasticsearch)  Rainer Gerhards *
  4. 4. V7 Action data structures • Global data, once per module ▫ Solely for setting default parameters • pData: per action-instance ▫ Automatically created “by framework” ▫ Holds all data necessary for doing action  Config parameters  Connection handles  Other state Rainer Gerhards *
  5. 5. Real-World Sample: ommysql typedef struct _instanceData { MYSQL *f_hmysql; char f_dbsrv[MAXHOSTNAMELEN+1]; unsigned int f_dbsrvPort; char f_dbname[_DB_MAXDBLEN+1]; char f_dbuid[_DB_MAXUNAMELEN+1]; char f_dbpwd[_DB_MAXPWDLEN+1]; unsigned uLastMySQLErrno; uchar *f_configfile; uchar *f_configsection; uchar *tplName; } instanceData; Green: per-connection parameters Rest is config part. Rainer Gerhards *
  6. 6. The v8 rule engine Queue worker queue Queue worker Action wrkr inst. Queue worker rsyslog core Action wrkr inst. Action wrkr inst. Now multiple instances per action!  Queue worker pool automatically scales outbound connection count by spawning more worker instances  Works well with ElastiSearch etc.  Inherantly serial outputs (e.g. local files!) must serialize themselves  Rainer Gerhards *
  7. 7. V8 Action data structures • Global data, once per module ▫ Solely for setting default parameters • pData: per action-instance ▫ Automatically created “by framework” ▫ Primarily config parameters ▫ Maybe some global state like overall counters, mutexes, ... • pWrkrData: per action-worker-instance ▫ Connection handles ▫ Other “live” state Rainer Gerhards *
  8. 8. Real-World Sample: ommysql typedef struct _instanceData { char dbsrv[MAXHOSTNAMELEN+1]; unsigned int dbsrvPort; char dbname[_DB_MAXDBLEN+1]; char dbuid[_DB_MAXUNAMELEN+1]; char dbpwd[_DB_MAXPWDLEN+1]; uchar *configfile; uchar *configsection; uchar *tplName; } instanceData; typedef struct wrkrInstanceData { instanceData pData; /* points back to action instance */ MYSQL *hmysql; unsigned uLastMySQLErrno; } wrkrInstanceData_t; Config and per-connection parameters now split, multiple instances of per-connection data. Rainer Gerhards *
  9. 9. V8 data structure sample • One action module • With 2 actions • One being executed on two, one on three threads Action instance 2 Rainer Gerhards * wrkr 1 wrkr 2 wrkr 1 wrkr 2 wrkr 3 External Connections module global data Action instance 1
  10. 10. What if the destination cannot handle multiple workers? • A typical example is file output • Framework will still call into multiple workers • Worker must use pData mutex to protect itself – this is not done automatically! • Basically like in v7, but plugin needs to take care Action instance 1 Action instance 2 wrkr 1 wrkr 2 “real” code Action-instance mutexes wrkr 1 wrkr 2 “real” code wrkr 3 Rainer Gerhards *
  11. 11. Real-World Sample: omfile BEGINcommitTransaction instanceData *__restrict__ const pData = pWrkrData->pData; unsigned i; CODESTARTcommitTransaction pthread_mutex_lock(&pData->mutWrite); for(i = 0 ; i < nParams ; ++i) writeFile(pData, pParams, i); if(pData->bFlushOnTXEnd && pData->pStrm != NULL) { /* if we have an async writer, it controls the flush via * a timeout. */ if(!pData->bUseAsyncWriter) CHKiRet(strm.Flush(pData->pStrm)); } finalize_it: pthread_mutex_unlock(&pData->mutWrite); ENDcommitTransaction Rainer Gerhards *
  12. 12. Anything else? • A couple of more things have changed,e.g. ▫ New callbacks for wrkr Instance handling ▫ New transaction processing callback ▫ New data structures for message data • Changes are hidden mostly in macro layer • But still there is work to do ▫ Usually quite easy ▫ Presentation on how to do this will follow, in the meantime use Rainer Gerhards *
  13. 13. Even more? • Yup... • We now support external plugins, which need not be written in C! • Use any language you like, e.g. Python, Perl or Java... • More info at • Note: action worker instances facilitate writing great performance external plugins Rainer Gerhards *