Logging ApplicationBehavior to MongoDB	       Robert Stewart	       @wombatnation	         1/1/2012
TOC	•  Why Log to MongoDB?	•  Logging Library Basics	•  Logging by Language	•  Other Ways to Distribute Your Logging
Why Log to MongoDB?	•  Centralized application logging	•  Easy remote access, compared to files	•  Log events good fit for d...
Event Storage	•  Create a database for your app	•  Create a capped collection to store log events	  •  Time-based eviction...
Log Analysis	•  Compared to files, easier to analyze app behavior  across servers and days	•  Log values to custom keys whe...
Miscellaneous Free Advice	     •  Take advantage of separating storage       from presentation	     •  Be aware of which c...
Logging Library Basics	•  Some provide API with pluggable implementation	•  Configuration	  •  Programmatic	  •  Declarativ...
Logging Concepts	•  Named, Hierarchical Logger Objects	  •  Identify a class or an area of an app	•  Log Level, e.g., erro...
Logging Concepts	•  Handler - determines event destination	  •  File, console, socket, email, syslog, MongoDB, etc.	  •  H...
Common Flow for Logging	        App calls	             For each handler:	logger.info( freak out )	                        ...
Polyglotism	•  Good news - lots of client language bindings  and logging libraries for MongoDB	•  Mostly good news - Log4J...
Logging by Language	•  Java	•  Python	•  Ruby	•  PHP	•  C#
Java	•  APIs	  •  Apache Commons Logging	  •  SLF4J 	•  java.util.logging, a.k.a., JUL	  •  Limited feature set 	  •  Dest...
Log4J	•  Apache Log4J is dominant logging library	•  Logback is potential successor	•  Logger - usually, represents class ...
Log4mongo-java	•  Set of Log4J Appenders for MongoDB	•  log4mongo.org	•  Open source project on GitHub	•  I m a committer ...
MongoDbAppender	•  Stores a Log4J LoggingEvent object as a  document in a collection	•  Format is very similar to structur...
MongoDbPatternLayoutAppender	 •  Stores log event based on user-specified pattern	 •  Standard Log4J pattern layout, parser...
BsonAppender	•  Custom log format by extending BsonAppender	•  Can use it to log BSON objects to other data  stores	•  Inc...
Enabling log4mongo-java	• Associate a name with an appender implementation	log4j.appender.MongoDB = org.log4mongo.MongoDbA...
Using Pattern Layout                  Appender	• Specify pattern layout appender	log4j.appender.MongoDB = org.log4mongo.Mo...
Shell and Tail
In the Shell	> use app_nameswitched to db app_name> db.log.findOne(){    "_id" : ObjectId("4c200c4d28cf037460e82363"),    ...
Recent Log Events	If using mongo 1.9.1+, put useful functions like this in .mongorc.js> function last(count) {       if (c...
Tailing Logs	•  You’ll really miss ability to tail logfiles	•  Or, ... will you?	•  MongoDB offers tailable cursors	•  Spec...
Python Tailing Example 	from pymongo import Connectionimport timedb = Connection().my_dbcoll = db.my_collectioncursor = co...
Python	•  Feature-rich logging module in Python 2.3	•  Influenced by Log4J
Components	•  Logger - represents area of library or project	•  LogRecord	•  Handler	•  Filter	•  Formatter	•  LoggerAdapt...
mongodb-log	•  Python logging handler to log to MongoDB	•  Stores dictionary directly as BSON doc	•  https://github.com/an...
log4mongo-python	• Similar design to log4mongo PHP and .NET appenders 	• Example programmatic handler config and usage:	imp...
Ruby	•  Built-in Logger library	•  Limited feature set	•  Console and file outputs	•  Message format is fixed	•  Datetime fo...
Log4r	•  Influenced by Log4J	  •  Logger	  •  Outputter	  •  Formatter	  •  Configurator	•  http://log4r.rubyforge.org	•  Co...
Logging	•  Similar to Log4r with a few enhancements	•  Supports JSON as output format	•  http://logging.rubyforge.org/
mongo_db_logger	•  Rails plugin which uses Rails Logger, which can use  Ruby Logger, Log4r, etc.	•  Logs a single document...
Central Logger	•  Forked from mongo_db_logger	•  Added Rails 3 support and gemified	•  Web UI for searching, filtering and a...
MongodbLogger	•  Based on central_logger	•  Different web UI for searching, filtering and  analyzing logs	•  Configure via d...
PHP	•  error_log() logs to file, email address or  system logger (e.g., syslog or NT EventLog)	•  Log4PHP - heavily influenc...
Log4mongo-php	•  Log4PHP appender	•  Programmatic & declarative config	•  Timestamp stored as Mongo Date	•  Exception store...
PEAR Log	•  PEAR Log class - lots of handlers (Firebug!), but  none yet for MongoDB	•  Programmatic config	•  Specify handl...
Log Writer for Zend                	•  Project - Recordshelf_Log_Writer_MongoDb	•  Described as a prototypesque implementa...
C#/.NET	•  Log4net	•  Heavily influenced by Log4J	•  http://logging.apache.org/log4net/
Log4mongo-net	•  Uses community developed C# driver, but may  move to official driver	•  Tested with .NET 3.5+ and Mono 2.8...
Other Options	•  Destinations include MongoDB	  •  Flume, logstash, Fluentd	•  Primary Destination is HDFS	  •  Chukwa and...
Flume	•  If you can t change the source of logs ...	•  Can aggregate files, syslog, etc.	•  Agent -- Collector -- Storage	•...
logstash	•  Log event storage, search and graphing	•  Read from file, syslog, AMQP, tcp, etc.	•  Store to ElasticSearch, AM...
Fluentd	•  Ruby-based open source log collector	•  Input, output and buffer plug-ins	•  For example, plug-ins for tailing ...
Questions?	•  Photo/Image Credits	  •    Log analysis - ww.flickr.com/photos/linuxtuxguy/2579295702 	  •    Don’t Stop Beli...
Upcoming SlideShare
Loading in...5
×

Logging Application Behavior to MongoDB

43,170

Published on

Presentation given at Mongo SV conference in Mountain View on December 3, 2010. Covers reasons for logging to MongoDB, logging library basics and library options for Java, Python, Ruby, PHP and C#. Updated 1/1/2012 with more info on logging in Ruby and tailable cursors.

Published in: Technology, Education
1 Comment
31 Likes
Statistics
Notes
  • logging to MongoDB, includes examples for a number of languages and libraries
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
43,170
On Slideshare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
312
Comments
1
Likes
31
Embeds 0
No embeds

No notes for slide

Logging Application Behavior to MongoDB

  1. 1. Logging ApplicationBehavior to MongoDB Robert Stewart @wombatnation 1/1/2012
  2. 2. TOC •  Why Log to MongoDB? •  Logging Library Basics •  Logging by Language •  Other Ways to Distribute Your Logging
  3. 3. Why Log to MongoDB? •  Centralized application logging •  Easy remote access, compared to files •  Log events good fit for document model •  Flexible schema •  Indexes for frequent queries •  Capped collections are very efficient •  Analyze data in-place with JavaScript MapReduce
  4. 4. Event Storage •  Create a database for your app •  Create a capped collection to store log events •  Time-based eviction more natural, but less efficient •  Capped collection reduces chance of filling up disk
  5. 5. Log Analysis •  Compared to files, easier to analyze app behavior across servers and days •  Log values to custom keys where possible •  JavaScript MapReduce
  6. 6. Miscellaneous Free Advice •  Take advantage of separating storage from presentation •  Be aware of which classes your Mongo driver knows how to BSONify •  If logging from multiple languages, sticking to Log4Lang family can help
  7. 7. Logging Library Basics •  Some provide API with pluggable implementation •  Configuration •  Programmatic •  Declarative •  Dynamic
  8. 8. Logging Concepts •  Named, Hierarchical Logger Objects •  Identify a class or an area of an app •  Log Level, e.g., error, warn, info, debug •  Thresholds determine if event is logged •  Logging Event •  The message •  Metadata, e.g., time, level, class, method, line
  9. 9. Logging Concepts •  Handler - determines event destination •  File, console, socket, email, syslog, MongoDB, etc. •  Handler level threshold •  Async vs. sync •  Filter - for logger and/or handler •  Formatter - event and message •  Pattern with placeholder tokens
  10. 10. Common Flow for Logging App calls For each handler: logger.info( freak out ) Is handler info No Is logger enabled? infoNo enabled? Yes Passes Yes No filter, if Passes any? filter, if Yes Yes any? Format and store event No
  11. 11. Polyglotism •  Good news - lots of client language bindings and logging libraries for MongoDB •  Mostly good news - Log4J very influential •  Sort of bad news - default log event formats can be very different
  12. 12. Logging by Language •  Java •  Python •  Ruby •  PHP •  C#
  13. 13. Java •  APIs •  Apache Commons Logging •  SLF4J •  java.util.logging, a.k.a., JUL •  Limited feature set •  Destinations - console, file, socket, memory •  JVM-level config
  14. 14. Log4J •  Apache Log4J is dominant logging library •  Logback is potential successor •  Logger - usually, represents class logging the event •  LoggingEvent - message and metadata •  Appender - sends LoggingEvent to destination •  Layout - formats the event •  Converter - convert placeholder to value
  15. 15. Log4mongo-java •  Set of Log4J Appenders for MongoDB •  log4mongo.org •  Open source project on GitHub •  I m a committer with Peter Monks and Jozef Šev ík •  I used it at Voxify with speech reco applications and infrastructure services •  Peter has used it at Alfresco
  16. 16. MongoDbAppender •  Stores a Log4J LoggingEvent object as a document in a collection •  Format is very similar to structure of LoggingEvent Java class •  Where possible, data is stored as appropriate BSON objects, rather than just strings
  17. 17. MongoDbPatternLayoutAppender •  Stores log event based on user-specified pattern •  Standard Log4J pattern layout, parser and converter functionality •  Very flexible formatting at cost of additional config •  Values stored as strings or arrays •  Custom values determined at time event is logged
  18. 18. BsonAppender •  Custom log format by extending BsonAppender •  Can use it to log BSON objects to other data stores •  Includes code for BSONifying Java exceptions
  19. 19. Enabling log4mongo-java • Associate a name with an appender implementation log4j.appender.MongoDB = org.log4mongo.MongoDbAppender!• Specify host (optional port, username and password). For replica set, specify all active hosts. log4j.appender.MongoDB.hostname = localhost • Specify database and collection log4j.appender.MongoDB.databaseName = app_name log4j.appender.MongoDB.collectionName = log • Add named appender to rootLogger list log4j.rootLogger = INFO, file, MongoDB
  20. 20. Using Pattern Layout Appender • Specify pattern layout appender log4j.appender.MongoDB = org.log4mongo.MongoDbPatternLayoutAppender • Specify  your  custom  Pa2ernLayout  class   log4j.appender.MongoDB.layout = com.voxify.log.log4j.MongoDBPatternLayout• Specify a conversion pattern •  Must be a valid JSON document •  Can have sub-documents •  Values can be strings or arrays log4j.appender.MongoDB.layout.ConversionPattern = {"ts":"%d{yyyy-MM-dd HH:mm:ss,SSS}","level":"%p","class":"%c{1}","message":"%m"}
  21. 21. Shell and Tail
  22. 22. In the Shell > use app_nameswitched to db app_name> db.log.findOne(){ "_id" : ObjectId("4c200c4d28cf037460e82363"), "ts" : "2010-10-21 18:05:17,237", "level" : "INFO", "class" : "NodeFlagGenerator", "message" : "Generating me some node flags”}
  23. 23. Recent Log Events If using mongo 1.9.1+, put useful functions like this in .mongorc.js> function last(count) { if (count==null) count = 1; return db.log.find({}, {_id:0}).sort({ts:-1}).limit(count); }> last(2){ "ts" : "2010-10-28 23:49:14,567", "level" : "INFO", "session" :"122", "class" : "Site", "message" : "uninformative message" }{ "ts" : "2010-10-28 23:49:14,566", "level" : "INFO", "session" :"122", "class" : "Site", "message" : "some info kind of message" }
  24. 24. Tailing Logs •  You’ll really miss ability to tail logfiles •  Or, ... will you? •  MongoDB offers tailable cursors •  Specify cursor, e.g., from find, as tailable •  While cursor alive, print new docs then sleep
  25. 25. Python Tailing Example from pymongo import Connectionimport timedb = Connection().my_dbcoll = db.my_collectioncursor = coll.find(tailable=True)while cursor.alive: try: doc = cursor.next() print doc except StopIteration: time.sleep(1)
  26. 26. Python •  Feature-rich logging module in Python 2.3 •  Influenced by Log4J
  27. 27. Components •  Logger - represents area of library or project •  LogRecord •  Handler •  Filter •  Formatter •  LoggerAdapter (2.6) - add contextual info
  28. 28. mongodb-log •  Python logging handler to log to MongoDB •  Stores dictionary directly as BSON doc •  https://github.com/andreisavu/mongodb-log
  29. 29. log4mongo-python • Similar design to log4mongo PHP and .NET appenders • Example programmatic handler config and usage: import loggingfrom log4mongo.handlers import MongoHandlerlogger = logging.getLogger(test)logger.addHandler(MongoHandler(host=localhost))logger.warning(test) • http://log4mongo.org/display/PUB/Log4mongo+for+Python • https://github.com/log4mongo/log4mongo-python
  30. 30. Ruby •  Built-in Logger library •  Limited feature set •  Console and file outputs •  Message format is fixed •  Datetime format is configurable
  31. 31. Log4r •  Influenced by Log4J •  Logger •  Outputter •  Formatter •  Configurator •  http://log4r.rubyforge.org •  Code for using Log4r with MongoMapper •  http://gist.github.com/207347
  32. 32. Logging •  Similar to Log4r with a few enhancements •  Supports JSON as output format •  http://logging.rubyforge.org/
  33. 33. mongo_db_logger •  Rails plugin which uses Rails Logger, which can use Ruby Logger, Log4r, etc. •  Logs a single document per request •  IP address, time, etc. •  All messages logged during request •  Custom request data •  Configure via database.yml •  http://github.com/peburrows/mongo_db_logger
  34. 34. Central Logger •  Forked from mongo_db_logger •  Added Rails 3 support and gemified •  Web UI for searching, filtering and analyzing logs •  Configure via database.yml (or mongoid.yml or central_logger.yml) •  http://www.slideshare.net/leopard_me/logging-rails-application-behavior-to-mongodb •  http://files.meetup.com/1742411/Logging%20With%20MongoDB.pdf •  https://github.com/customink/central_logger
  35. 35. MongodbLogger •  Based on central_logger •  Different web UI for searching, filtering and analyzing logs •  Configure via database.yml mongodb_logger.yml) (or mongoid.yml or•  http://mongodb-logger.catware.org/
  36. 36. PHP •  error_log() logs to file, email address or system logger (e.g., syslog or NT EventLog) •  Log4PHP - heavily influenced by Log4J •  PEAR Log - PHP-specific logging library •  Zend_Log - Logging for Zend Framework
  37. 37. Log4mongo-php •  Log4PHP appender •  Programmatic & declarative config •  Timestamp stored as Mongo Date •  Exception stored as array - msg, code, stacktrace •  Inner exception stored if available •  https://github.com/log4mongo/log4mongo-php
  38. 38. PEAR Log •  PEAR Log class - lots of handlers (Firebug!), but none yet for MongoDB •  Programmatic config •  Specify handler when creating logger object •  Composite handler for multiple destinations •  http://pear.php.net/package/Log/
  39. 39. Log Writer for Zend •  Project - Recordshelf_Log_Writer_MongoDb •  Described as a prototypesque implementation •  Standard Zend tools to config and init •  Zend log event is associative array •  Project has code for accessing filtering events •  http://raphaelstolt.blogspot.com/2009/09/logging-to-mongodb-and-accessing-log.html
  40. 40. C#/.NET •  Log4net •  Heavily influenced by Log4J •  http://logging.apache.org/log4net/
  41. 41. Log4mongo-net •  Uses community developed C# driver, but may move to official driver •  Tested with .NET 3.5+ and Mono 2.8 •  Standard App.config/Web.config or programmatic •  http://log4mongo.org/display/PUB/Log4mongo+for+.NET
  42. 42. Other Options •  Destinations include MongoDB •  Flume, logstash, Fluentd •  Primary Destination is HDFS •  Chukwa and Scribe •  Commercial Hosted •  Loggly •  Commercial On-Premise •  Splunk
  43. 43. Flume •  If you can t change the source of logs ... •  Can aggregate files, syslog, etc. •  Agent -- Collector -- Storage •  Storage plugins for HDFS, MongoDB, etc. •  Decorators transform data extract metadata •  https://github.com/cloudera/flume •  https://github.com/mongodb/mongo-hadoop •  Chukwa, Honu and Scribe more HDFS-centric
  44. 44. logstash •  Log event storage, search and graphing •  Read from file, syslog, AMQP, tcp, etc. •  Store to ElasticSearch, AMQP queue, websockets, MongoDB, etc. •  http://logstash.net/ •  http://code.google.com/p/logstash
  45. 45. Fluentd •  Ruby-based open source log collector •  Input, output and buffer plug-ins •  For example, plug-ins for tailing Apache log files and storing log events to MongoDB •  http://blog.treasure-data.com/post/13766262632/real-time-log-collection-with- fluentd-and-mongodb •  http://fluentd.org/doc/ •  https://github.com/fluent/fluentd
  46. 46. Questions? •  Photo/Image Credits •  Log analysis - ww.flickr.com/photos/linuxtuxguy/2579295702 •  Don’t Stop Believing - www.flickr.com/photos/loozrboy/3908830690/ •  Homeland Security Advisory System - www.theonion.com/articles/iraq-adopts-terror-alert-system,1258/ •  Advice - www.flickr.com/photos/wurzle/659315/ •  Shell - http://www.flickr.com/photos/geishaboy500/305555652/ •  Tail - http://www.flickr.com/photos/autumnsonata/2632194442/ •  Log flume - www.flickr.com/photos/halloweenjack/1330806722/ •  Cufflinks - www.flickr.com/photos/oberazzi/318947873/
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×