0
Using NoSQL databases tostore RADIUS and Syslog     data, part 1: Idea       Karri Huhtanen         18.9.2012
Some background•   currently RADIUS accounting data is stored usually    in SQL databases with fixed database schema•   for...
RADIUS accounting message             Wed	  Aug	  	  8	  13:49:33	  2012             	  	  	  	  	  	  	  	  User-­‐Name	 ...
Syslog message  Until researching    The	  syslog	  message	  has	  the	  following	  ABNF	  [RFC5234]	  definition: into ...
So what happens next?•   Selection of NoSQL database:    •   Likely Column Family Store if no one can suggest a        bet...
Results (hopefully)•   Is storing RADIUS accounting and Syslog messages into    NoSQL database: a brilliant idea, brillian...
Using NoSQL databases tostore RADIUS and Syslog data,  part 1I: The Saga Continues        Karri Huhtanen          27.11.2012
Happened earlier•   currently RADIUS accounting data is stored usually    in SQL databases with fixed database schema•   fo...
Results (luckily)•   Is storing RADIUS accounting and Syslog messages into    NoSQL database: a brilliant idea, brilliantl...
So what happened?•   Selection of NoSQL database:    •   Likely Column Family Store if no one can suggest a               ...
That was the executive summary. Thank you.
Now some moredetailed informationand even some code.
storing RADIUS accounting and Syslog           messages into NoSQL database•   It is a good idea because:    •   When we h...
storing RADIUS accounting and Syslog           messages into NoSQL database•   It is not a brilliant idea because:    •   ...
How hard can it be?•   With Ubuntu Linux Server 12.04 LTS:    •   sudo apt-get install python-pymongo mongodb syslog-ng   ...
Demo
Syslog-ng              # /etc/syslog-ng/syslog-ng.conf              # mongodb log destination              destination kar...
Radiator RADIUS server# /etc/radiator/radiator.cfg## send all RADIUS accounting requests to external script#<Handler Reque...
#!/usr/bin/env pythonfrom pymongo import Connectionimport datetime                                                        ...
Does it actually work? What       can you do with data?•   Yes it does actually work, but once again it does not solve or ...
Performance?•   Would need to be measured and verified and with    real production environment or solution.•   Would also n...
Conclusions•   NoSQL should be at least considered as an option    when designing and implementing large scale Syslog or  ...
Upcoming SlideShare
Loading in...5
×

Using NoSQL databases to store RADIUS and Syslog data

5,029

Published on

A seminar presentation done for TUT's NoSQL course. A brief look into the possibility and the feasibility of using NoSQL databases to store RADIUS accounting and Syslog data. In this particular case, Syslog-NG, Radiator RADIUS server and MongoDB were used as trial platforms. The presentation includes configuration examples and also some code.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
5,029
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
44
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "Using NoSQL databases to store RADIUS and Syslog data"

  1. 1. Using NoSQL databases tostore RADIUS and Syslog data, part 1: Idea Karri Huhtanen 18.9.2012
  2. 2. Some background• currently RADIUS accounting data is stored usually in SQL databases with fixed database schema• for Syslog messages an SQL database can be used, but commercial log analyzers (like Splunk) usually use their own solutions which may or may not be SQL databases• Started thinking if NoSQL database could be applied to both or one of these?
  3. 3. RADIUS accounting message Wed  Aug    8  13:49:33  2012                User-­‐Name  =  "jotain@realm"                NAS-­‐Port  =  8 One message                NAS-­‐IP-­‐Address  =  192.168.229.131 contains                Framed-­‐IP-­‐Address  =  192.168.163.226 undetermined                NAS-­‐Identifier  =  "Cisco_66:77:88"                Airespace-­‐WLAN-­‐Id  =  4 number of                Acct-­‐Session-­‐Id  =  "50223ea9/00:11:22:33:44:55/2292" attributes.                Acct-­‐Authentic  =  Remote                Tunnel-­‐Type  =  0:VLANinterpreted                Tunnel-­‐Medium-­‐Type  =  0:802 Some can beattributes,                Tunnel-­‐Private-­‐Group-­‐ID  =  0:222 interpreted, somethe unknown                Event-­‐Timestamp  =  1344422780 stay unknown.                Acct-­‐Status-­‐Type  =  Aliveattributes are                Acct-­‐Input-­‐Octets  =  1262012usually left in                Acct-­‐Input-­‐Gigawords  =  0 Because there canOID:FieldDataTyp                Acct-­‐Output-­‐Octets  =  13518133 be a changing                Acct-­‐Output-­‐Gigawords  =  0e binary format                Acct-­‐Input-­‐Packets  =  11692 number of                Acct-­‐Output-­‐Packets  =  11154 changing type of                Acct-­‐Session-­‐Time  =  1235 attributes I began                Acct-­‐Delay-­‐Time  =  19                Calling-­‐Station-­‐Id  =  "00:11:22:33:44:55" to wonder if                Called-­‐Station-­‐Id  =  "f4:7f:35:5e:bf:b0" NoSQL could be                cisco-­‐avpair  =  "nas-­‐update=true" used for storing                Digest-­‐Response  =  "P"C<188>"                Digest-­‐Response  =  "P"C<194>" these?                Timestamp  =  1344422954
  4. 4. Syslog message Until researching The  syslog  message  has  the  following  ABNF  [RFC5234]  definition: into this I thought            SYSLOG-­‐MSG            =  HEADER  SP  STRUCTURED-­‐DATA  [SP  MSG]Syslog messages had            HEADER                    =  PRI  VERSION  SP  TIMESTAMP  SP  HOSTNAME                                                SP  APP-­‐NAME  SP  PROCID  SP  MSGID fixed structure and            PRI                          =  "<"  PRIVAL  ">"            PRIVAL                    =  1*3DIGIT  ;  range  0  ..  191 could be then            VERSION                  =  NONZERO-­‐DIGIT  0*2DIGIT            HOSTNAME                =  NILVALUE  /  1*255PRINTUSASCII handled with fixed            APP-­‐NAME                =  NILVALUE  /  1*48PRINTUSASCII database schema.            PROCID                    =  NILVALUE  /  1*128PRINTUSASCII            MSGID                      =  NILVALUE  /  1*32PRINTUSASCII            TIMESTAMP              =  NILVALUE  /  FULL-­‐DATE  "T"  FULL-­‐TIME Then I read the            FULL-­‐DATE              =  DATE-­‐FULLYEAR  "-­‐"  DATE-­‐MONTH  "-­‐"  DATE-­‐MDAY            DATE-­‐FULLYEAR      =  4DIGIT RFC5424: http://            DATE-­‐MONTH            =  2DIGIT    ;  01-­‐12            DATE-­‐MDAY              =  2DIGIT    ;  01-­‐28,  01-­‐29,  01-­‐30,  01-­‐31  based  ontools.ietf.org/html/                                                                ;  month/year            FULL-­‐TIME              =  PARTIAL-­‐TIME  TIME-­‐OFFSET rfc5424            PARTIAL-­‐TIME        =  TIME-­‐HOUR  ":"  TIME-­‐MINUTE  ":"  TIME-­‐SECOND                                                [TIME-­‐SECFRAC]            TIME-­‐HOUR              =  2DIGIT    ;  00-­‐23            TIME-­‐MINUTE          =  2DIGIT    ;  00-­‐59            TIME-­‐SECOND          =  2DIGIT    ;  00-­‐59            TIME-­‐SECFRAC        =  "."  1*6DIGIT            TIME-­‐OFFSET          =  "Z"  /  TIME-­‐NUMOFFSET            TIME-­‐NUMOFFSET    =  ("+"  /  "-­‐")  TIME-­‐HOUR  ":"  TIME-­‐MINUTE Here we have once            STRUCTURED-­‐DATA  =  NILVALUE  /  1*SD-­‐ELEMENT again parameters,            SD-­‐ELEMENT            =  "["  SD-­‐ID  *(SP  SD-­‐PARAM)  "]"            SD-­‐PARAM                =  PARAM-­‐NAME  "="  %d34  PARAM-­‐VALUE  %d34 although they are            SD-­‐ID                      =  SD-­‐NAME            PARAM-­‐NAME            =  SD-­‐NAME within one defined            PARAM-­‐VALUE          =  UTF-­‐8-­‐STRING  ;  characters  ",    and                                                                          ;  ]  MUST  be  escaped. STRUCTURED-            SD-­‐NAME                  =  1*32PRINTUSASCII                                                ;  except  =,  SP,  ],  %d34  (") DATA field.            MSG                          =  MSG-­‐ANY  /  MSG-­‐UTF8            MSG-­‐ANY                  =  *OCTET  ;  not  starting  with  BOM            MSG-­‐UTF8                =  BOM  UTF-­‐8-­‐STRING So could NoSQL be            BOM                          =  %xEF.BB.BF used also for Syslog?
  5. 5. So what happens next?• Selection of NoSQL database: • Likely Column Family Store if no one can suggest a better one? • Something easy to setup and use, will concentrate into getting RADIUS server and/or Syslogd transferring data to database.• Setting up a WiFi access point and/or controller to provide real RADIUS and Syslog data• Storing data, retrieving data, searching data, deleting data to see what works• Writing and presenting Part II: “Implementation and Results” of these slides
  6. 6. Results (hopefully)• Is storing RADIUS accounting and Syslog messages into NoSQL database: a brilliant idea, brilliantly stupid idea or something else?• How hard can it be? What does it require to do this, is it possible and how?• Does it actually work? What can you do with data? Is there some indication of performance improvements or problems?• Will not do complete performance measurements though, designing and setting up reliable measurement environment will probably take too much time.
  7. 7. Using NoSQL databases tostore RADIUS and Syslog data, part 1I: The Saga Continues Karri Huhtanen 27.11.2012
  8. 8. Happened earlier• currently RADIUS accounting data is stored usually in SQL databases with fixed database schema• for Syslog messages an SQL database can be used, but commercial log analyzers (like Splunk) usually use their own solutions which may or may not be SQL databases• Started thinking if NoSQL database could be applied to both or one of these?
  9. 9. Results (luckily)• Is storing RADIUS accounting and Syslog messages into NoSQL database: a brilliant idea, brilliantly stupid idea or something else? a good idea• How hard can it be? What does it require to do this, is it possible and how? easy, 1 night before presentation required• Does it actually work? What can you do with data? Is there some indication of performance improvements or problems? Yes. Store and Process. Unknown. Some issues to be considered.• Will not do complete performance measurements though, designing and setting up reliable measurement environment will probably take too much time. Coded one Python script.
  10. 10. So what happened?• Selection of NoSQL database: • Likely Column Family Store if no one can suggest a MongoDB better one? • Something easy to setup and use, will concentrate into getting RADIUS server and/or Syslogd transferring data to database.• Setting up a WiFi access point and/or controller to provide real RADIUS and Syslog data• Storing data, retrieving data, searching data, deleting data to see what works Done, but not thoroughly• Writing and presenting Part II: “Implementation and Results” of these slides Done
  11. 11. That was the executive summary. Thank you.
  12. 12. Now some moredetailed informationand even some code.
  13. 13. storing RADIUS accounting and Syslog messages into NoSQL database• It is a good idea because: • When we have massive amount of log or accounting data, we need massive database clusters. • Data is mainly stored, read, analyzed and occasionally deleted. Data will not be updated or changed and is relatively simple (few tables with a lot of columns). • NoSQL may provide better way to scale this horizontally by distribution and sharding. • It is already being done. Several log analyzers, stores already use NoSQL databases as backends. There exists projects such as Greylog2 etc. which provide complete solutions from log storage, visualization, analysis etc. • Logs and accounting data are actually use cases for some NoSQL databases, for example: http://docs.mongodb.org/manual/use-cases/storing-log-data/
  14. 14. storing RADIUS accounting and Syslog messages into NoSQL database• It is not a brilliant idea because: • If we look what we need to do to optimize the performance it starts to look like a lot like designing and optimizing a SQL database: http://docs.mongodb.org/ manual/use-cases/storing-log-data/ • You cannot forget datatypes or database design even with NoSQL databases especially when going into production. • Prototypes may be faster and easier for developers, but creating a design and configuration which survices production use may be as hard as it has ever been. The difference is that instead of SQL database expert, you know need a NoSQL expert.• ... but it is not a brilliantly stupid idea either, it is an idea worth considering depending of the project.
  15. 15. How hard can it be?• With Ubuntu Linux Server 12.04 LTS: • sudo apt-get install python-pymongo mongodb syslog-ng syslog-ng-mod-mongodb • for Syslog-NG, just some configuration • for Radiator, some configuration and coding an external Python script to handle accounting messages• But this is far from production use, it is more like proto or proof of concept implementation done in 1 work day.
  16. 16. Demo
  17. 17. Syslog-ng # /etc/syslog-ng/syslog-ng.conf # mongodb log destination destination karrin_net_mongodb { mongodb(); }; # ... log { source(s_src); source(s_net); destination(karrin_net_mongodb); }; # that’s ithttps://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guides/syslog- ng-ose-v3.3-guide-admin-en.html/reference_destination_mongodb.html
  18. 18. Radiator RADIUS server# /etc/radiator/radiator.cfg## send all RADIUS accounting requests to external script#<Handler Request-Type = Accounting-Request> <AuthBy EXTERNAL> Command %D/acct2mongo.py </AuthBy> AcctLogFileName %L/acct-acct2mongodb-%Y-%M.log</Handler>
  19. 19. #!/usr/bin/env pythonfrom pymongo import Connectionimport datetime acct2mongo.pyimport sysdef main(): line = str() post = dict() # opening connection connection = Connection( localhost, 27017) # database radius db = connection[radius] # collection accounting collection = db[accounting] post[acct2mongotimestamp] = datetime.datetime.utcnow() for line in sys.stdin.readlines(): pieces = line.split( = , 1) if len(pieces) == 2: post[pieces[0].strip().strip(")]=pieces[1].strip().strip(") collection.insert(post) connection.end_request() connection.disconnect() # 0 Means reply with an acceptance. For Access-Requests, # an Access-Accept will be sent. For Accounting-Requests, # an Accounting-Response will be sent. return 0if __name__ == __main__: main()
  20. 20. Does it actually work? What can you do with data?• Yes it does actually work, but once again it does not solve or be applicable to everything.• One can store, read, search and delete data supposedly very efficiently, but anything more complicated is harder and must be implemented by developer.• For example: MongoDB does not have a reliable decimal datatype. It is better to keep numbers as a string and convert them when processing data.• Repeating earlier statement: “You cannot forget datatypes or database design even with NoSQL databases especially when going into production.”
  21. 21. Performance?• Would need to be measured and verified and with real production environment or solution.• Would also need to be compared with well designed and optimised SQL database, maybe even one functioning as NoSQL one.• In the implementation this was not tested as the datasets were very small compared to real datasets.
  22. 22. Conclusions• NoSQL should be at least considered as an option when designing and implementing large scale Syslog or Radius Accounting storages.• For development it is flexible.• For production use NoSQL solution still needs design, careful planning and testing to verify if the performance, reliability and security is enough. Probably as much as SQL database design.• Key issue will probably be can the SQL database handle the data or is horizontal scaling required.
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×