Logging Application Behavior to MongoDB

Logging ApplicationBehavior to MongoDB Robert Stewart @wombatnation 1/1/2012

TOC •  Why Log to MongoDB? •  Logging Library Basics •  Logging by Language •  Other Ways to Distribute Your Logging

Why Log to MongoDB? •  Centralized application logging •  Easy remote access, compared to files •  Log events good fit for document model •  Flexible schema •  Indexes for frequent queries •  Capped collections are very efficient •  Analyze data in-place with JavaScript MapReduce

Event Storage •  Create a database for your app •  Create a capped collection to store log events •  Time-based eviction more natural, but less efﬁcient •  Capped collection reduces chance of ﬁlling up disk

Log Analysis •  Compared to ﬁles, easier to analyze app behavior across servers and days •  Log values to custom keys where possible •  JavaScript MapReduce

Miscellaneous Free Advice •  Take advantage of separating storage from presentation •  Be aware of which classes your Mongo driver knows how to BSONify •  If logging from multiple languages, sticking to Log4Lang family can help

Logging Library Basics •  Some provide API with pluggable implementation •  Conﬁguration •  Programmatic •  Declarative •  Dynamic

Logging Concepts •  Named, Hierarchical Logger Objects •  Identify a class or an area of an app •  Log Level, e.g., error, warn, info, debug •  Thresholds determine if event is logged •  Logging Event •  The message •  Metadata, e.g., time, level, class, method, line

Logging Concepts •  Handler - determines event destination •  File, console, socket, email, syslog, MongoDB, etc. •  Handler level threshold •  Async vs. sync •  Filter - for logger and/or handler •  Formatter - event and message •  Pattern with placeholder tokens

Common Flow for Logging App calls For each handler: logger.info( freak out ) Is handler info No Is logger enabled? infoNo enabled? Yes Passes Yes No ﬁlter, if Passes any? ﬁlter, if Yes Yes any? Format and store event No

Polyglotism •  Good news - lots of client language bindings and logging libraries for MongoDB •  Mostly good news - Log4J very inﬂuential •  Sort of bad news - default log event formats can be very different

Logging by Language •  Java •  Python •  Ruby •  PHP •  C#

Java •  APIs •  Apache Commons Logging •  SLF4J •  java.util.logging, a.k.a., JUL •  Limited feature set •  Destinations - console, ﬁle, socket, memory •  JVM-level conﬁg

Log4J •  Apache Log4J is dominant logging library •  Logback is potential successor •  Logger - usually, represents class logging the event •  LoggingEvent - message and metadata •  Appender - sends LoggingEvent to destination •  Layout - formats the event •  Converter - convert placeholder to value

Log4mongo-java •  Set of Log4J Appenders for MongoDB •  log4mongo.org •  Open source project on GitHub •  I m a committer with Peter Monks and Jozef Šev ík •  I used it at Voxify with speech reco applications and infrastructure services •  Peter has used it at Alfresco

MongoDbAppender •  Stores a Log4J LoggingEvent object as a document in a collection •  Format is very similar to structure of LoggingEvent Java class •  Where possible, data is stored as appropriate BSON objects, rather than just strings

MongoDbPatternLayoutAppender •  Stores log event based on user-specified pattern •  Standard Log4J pattern layout, parser and converter functionality •  Very flexible formatting at cost of additional config •  Values stored as strings or arrays •  Custom values determined at time event is logged

BsonAppender •  Custom log format by extending BsonAppender •  Can use it to log BSON objects to other data stores •  Includes code for BSONifying Java exceptions

Enabling log4mongo-java • Associate a name with an appender implementation log4j.appender.MongoDB = org.log4mongo.MongoDbAppender!• Specify host (optional port, username and password). For replica set, specify all active hosts. log4j.appender.MongoDB.hostname = localhost • Specify database and collection log4j.appender.MongoDB.databaseName = app_name log4j.appender.MongoDB.collectionName = log • Add named appender to rootLogger list log4j.rootLogger = INFO, ﬁle, MongoDB

Using Pattern Layout Appender • Specify pattern layout appender log4j.appender.MongoDB = org.log4mongo.MongoDbPatternLayoutAppender • Specify your custom Pa2ernLayout class log4j.appender.MongoDB.layout = com.voxify.log.log4j.MongoDBPatternLayout• Specify a conversion pattern •  Must be a valid JSON document •  Can have sub-documents •  Values can be strings or arrays log4j.appender.MongoDB.layout.ConversionPattern = {"ts":"%d{yyyy-MM-dd HH:mm:ss,SSS}","level":"%p","class":"%c{1}","message":"%m"}

Shell and Tail

In the Shell > use app_nameswitched to db app_name> db.log.findOne(){ "_id" : ObjectId("4c200c4d28cf037460e82363"), "ts" : "2010-10-21 18:05:17,237", "level" : "INFO", "class" : "NodeFlagGenerator", "message" : "Generating me some node flags”}

Recent Log Events If using mongo 1.9.1+, put useful functions like this in .mongorc.js> function last(count) { if (count==null) count = 1; return db.log.find({}, {_id:0}).sort({ts:-1}).limit(count); }> last(2){ "ts" : "2010-10-28 23:49:14,567", "level" : "INFO", "session" :"122", "class" : "Site", "message" : "uninformative message" }{ "ts" : "2010-10-28 23:49:14,566", "level" : "INFO", "session" :"122", "class" : "Site", "message" : "some info kind of message" }

Tailing Logs •  You’ll really miss ability to tail logﬁles •  Or, ... will you? •  MongoDB offers tailable cursors •  Specify cursor, e.g., from ﬁnd, as tailable •  While cursor alive, print new docs then sleep

Python Tailing Example from pymongo import Connectionimport timedb = Connection().my_dbcoll = db.my_collectioncursor = coll.find(tailable=True)while cursor.alive: try: doc = cursor.next() print doc except StopIteration: time.sleep(1)

Python •  Feature-rich logging module in Python 2.3 •  Inﬂuenced by Log4J

Components •  Logger - represents area of library or project •  LogRecord •  Handler •  Filter •  Formatter •  LoggerAdapter (2.6) - add contextual info

mongodb-log •  Python logging handler to log to MongoDB •  Stores dictionary directly as BSON doc •  https://github.com/andreisavu/mongodb-log

log4mongo-python • Similar design to log4mongo PHP and .NET appenders • Example programmatic handler conﬁg and usage: import loggingfrom log4mongo.handlers import MongoHandlerlogger = logging.getLogger(test)logger.addHandler(MongoHandler(host=localhost))logger.warning(test) • http://log4mongo.org/display/PUB/Log4mongo+for+Python • https://github.com/log4mongo/log4mongo-python

Ruby •  Built-in Logger library •  Limited feature set •  Console and file outputs •  Message format is fixed •  Datetime format is configurable

Log4r •  Inﬂuenced by Log4J •  Logger •  Outputter •  Formatter •  Conﬁgurator •  http://log4r.rubyforge.org •  Code for using Log4r with MongoMapper •  http://gist.github.com/207347

Logging •  Similar to Log4r with a few enhancements •  Supports JSON as output format •  http://logging.rubyforge.org/

mongo_db_logger •  Rails plugin which uses Rails Logger, which can use Ruby Logger, Log4r, etc. •  Logs a single document per request •  IP address, time, etc. •  All messages logged during request •  Custom request data •  Conﬁgure via database.yml •  http://github.com/peburrows/mongo_db_logger

Central Logger •  Forked from mongo_db_logger •  Added Rails 3 support and gemified •  Web UI for searching, filtering and analyzing logs •  Configure via database.yml (or mongoid.yml or central_logger.yml) •  http://www.slideshare.net/leopard_me/logging-rails-application-behavior-to-mongodb •  http://files.meetup.com/1742411/Logging%20With%20MongoDB.pdf •  https://github.com/customink/central_logger

MongodbLogger •  Based on central_logger •  Different web UI for searching, ﬁltering and analyzing logs •  Conﬁgure via database.yml mongodb_logger.yml) (or mongoid.yml or•  http://mongodb-logger.catware.org/

PHP •  error_log() logs to file, email address or system logger (e.g., syslog or NT EventLog) •  Log4PHP - heavily influenced by Log4J •  PEAR Log - PHP-specific logging library •  Zend_Log - Logging for Zend Framework

Log4mongo-php •  Log4PHP appender •  Programmatic & declarative conﬁg •  Timestamp stored as Mongo Date •  Exception stored as array - msg, code, stacktrace •  Inner exception stored if available •  https://github.com/log4mongo/log4mongo-php

PEAR Log •  PEAR Log class - lots of handlers (Firebug!), but none yet for MongoDB •  Programmatic conﬁg •  Specify handler when creating logger object •  Composite handler for multiple destinations •  http://pear.php.net/package/Log/

Log Writer for Zend •  Project -> Recordshelf_Log_Writer_MongoDb •  Described as a prototypesque implementation •  Standard Zend tools to conﬁg and init •  Zend log event is associative array •  Project has code for accessing & ﬁltering events •  http://raphaelstolt.blogspot.com/2009/09/logging-to-mongodb-and-accessing-log.html

C#/.NET •  Log4net •  Heavily inﬂuenced by Log4J •  http://logging.apache.org/log4net/

Log4mongo-net •  Uses community developed C# driver, but may move to official driver •  Tested with .NET 3.5+ and Mono 2.8 •  Standard App.config/Web.config or programmatic •  http://log4mongo.org/display/PUB/Log4mongo+for+.NET

Other Options •  Destinations include MongoDB •  Flume, logstash, Fluentd •  Primary Destination is HDFS •  Chukwa and Scribe •  Commercial Hosted •  Loggly •  Commercial On-Premise •  Splunk

Flume •  If you can t change the source of logs ... •  Can aggregate ﬁles, syslog, etc. •  Agent --> Collector --> Storage •  Storage plugins for HDFS, MongoDB, etc. •  Decorators transform data & extract metadata •  https://github.com/cloudera/ﬂume •  https://github.com/mongodb/mongo-hadoop •  Chukwa, Honu and Scribe more HDFS-centric

logstash •  Log event storage, search and graphing •  Read from ﬁle, syslog, AMQP, tcp, etc. •  Store to ElasticSearch, AMQP queue, websockets, MongoDB, etc. •  http://logstash.net/ •  http://code.google.com/p/logstash

Fluentd •  Ruby-based open source log collector •  Input, output and buffer plug-ins •  For example, plug-ins for tailing Apache log files and storing log events to MongoDB •  http://blog.treasure-data.com/post/13766262632/real-time-log-collection-with- fluentd-and-mongodb •  http://fluentd.org/doc/ •  https://github.com/fluent/fluentd

Questions? •  Photo/Image Credits •  Log analysis - ww.flickr.com/photos/linuxtuxguy/2579295702 •  Don’t Stop Believing - www.flickr.com/photos/loozrboy/3908830690/ •  Homeland Security Advisory System - www.theonion.com/articles/iraq-adopts-terror-alert-system,1258/ •  Advice - www.flickr.com/photos/wurzle/659315/ •  Shell - http://www.flickr.com/photos/geishaboy500/305555652/ •  Tail - http://www.flickr.com/photos/autumnsonata/2632194442/ •  Log flume - www.flickr.com/photos/halloweenjack/1330806722/ •  Cufflinks - www.flickr.com/photos/oberazzi/318947873/

http://log4mongo.org	4713
http://www.wombatnation.com	1486
http://localhost	9
http://translate.googleusercontent.com	4
http://reddo.us	3
http://us-w1.rockmelt.com	2
http://www.plurk.com	2
http://webcache.googleusercontent.com	1
https://si0.twimg.com	1
http://www.webpagetest.org	1
http://codescale.net	1

Logging Application Behavior to MongoDB

by Robert Stewart

Accessibility

Categories

Upload Details

Usage Rights

11 Embeds 6,223

Statistics

Logging Application Behavior to MongoDB Presentation Transcript