SlideShare a Scribd company logo
1 of 46
Download to read offline
Logging Application
Behavior to MongoDB	

       Robert Stewart	

       @wombatnation	

         1/1/2012
TOC	


•  Why Log to MongoDB?	

•  Logging Library Basics	

•  Logging by Language	

•  Other Ways to Distribute Your Logging
Why Log to MongoDB?	

•  Centralized application logging	

•  Easy remote access, compared to files	

•  Log events good fit for document model	

•  Flexible schema	

•  Indexes for frequent queries	

•  Capped collections are very efficient	

•  Analyze data in-place with JavaScript MapReduce
Event Storage	


•  Create a database for your app	

•  Create a capped collection to store log events	

  •  Time-based eviction more natural, but less efficient	

  •  Capped collection reduces chance of filling up disk
Log Analysis	



•  Compared to files, easier to analyze app behavior
  across servers and days	

•  Log values to custom keys where possible	

•  JavaScript MapReduce
Miscellaneous Free Advice	

     •  Take advantage of separating storage
       from presentation	

     •  Be aware of which classes your
       Mongo driver knows how to BSONify	

     •  If logging from multiple languages,
       sticking to Log4Lang family can help
Logging Library Basics	


•  Some provide API with pluggable implementation	

•  Configuration	

  •  Programmatic	

  •  Declarative	

  •  Dynamic
Logging Concepts	

•  Named, Hierarchical Logger Objects	

  •  Identify a class or an area of an app	

•  Log Level, e.g., error, warn, info, debug	

  •  Thresholds determine if event is logged	

•  Logging Event	

  •  The message	

  •  Metadata, e.g., time, level, class, method, line
Logging Concepts	

•  Handler - determines event destination	

  •  File, console, socket, email, syslog, MongoDB, etc.	

  •  Handler level threshold	

  •  Async vs. sync	

•  Filter - for logger and/or handler	

•  Formatter - event and message	

  •  Pattern with placeholder tokens
Common Flow for Logging	

        App calls	

             For each handler:	

logger.info( freak out )	

                                       Is handler
                                           info
                                                        No	

         Is logger                      enabled?	

            info
No	

    enabled?	

                         Yes	

                                        Passes
               Yes	

                                   No	

                                        filter, if
         Passes                          any?	

         filter, if      Yes	

                                           Yes	

          any?	

                                   Format and store event	

               No
Polyglotism	


•  Good news - lots of client language bindings
  and logging libraries for MongoDB	

•  Mostly good news - Log4J very influential	

•  Sort of bad news - default log event formats
  can be very different
Logging by Language	

•  Java	

•  Python	

•  Ruby	

•  PHP	

•  C#
Java	

•  APIs	

  •  Apache Commons Logging	

  •  SLF4J 	

•  java.util.logging, a.k.a., JUL	

  •  Limited feature set 	

  •  Destinations - console, file, socket, memory	

  •  JVM-level config
Log4J	

•  Apache Log4J is dominant logging library	

•  Logback is potential successor	

•  Logger - usually, represents class logging the event	

•  LoggingEvent - message and metadata	

•  Appender - sends LoggingEvent to destination	

•  Layout - formats the event	

  •  Converter - convert placeholder to value
Log4mongo-java	

•  Set of Log4J Appenders for MongoDB	

•  log4mongo.org	

•  Open source project on GitHub	

•  I m a committer with Peter Monks and Jozef Šev ík 	

•  I used it at Voxify with speech reco applications and
  infrastructure services	

•  Peter has used it at Alfresco
MongoDbAppender	


•  Stores a Log4J LoggingEvent object as a
  document in a collection	

•  Format is very similar to structure of
  LoggingEvent Java class	

•  Where possible, data is stored as appropriate
  BSON objects, rather than just strings
MongoDbPatternLayoutAppender	

 •  Stores log event based on user-specified pattern	

 •  Standard Log4J pattern layout, parser and
   converter functionality	

 •  Very flexible formatting at cost of additional config	

 •  Values stored as strings or arrays 	

 •  Custom values determined at time event is logged
BsonAppender	


•  Custom log format by extending BsonAppender	

•  Can use it to log BSON objects to other data
  stores	

•  Includes code for BSONifying Java exceptions
Enabling log4mongo-java	

• Associate a name with an appender implementation	

log4j.appender.MongoDB = org.log4mongo.MongoDbAppender!

• Specify host (optional port, username and
  password). For replica set, specify all active hosts.	

log4j.appender.MongoDB.hostname = localhost	


• Specify database and collection	

log4j.appender.MongoDB.databaseName = app_name	


log4j.appender.MongoDB.collectionName = log  	

• Add named appender to rootLogger list	

log4j.rootLogger = INFO, file, MongoDB
Using Pattern Layout
                  Appender	

• Specify pattern layout appender	

log4j.appender.MongoDB = org.log4mongo.MongoDbPatternLayoutAppender	


• Specify	
  your	
  custom	
  Pa2ernLayout	
  class	
  
                                                                        	

log4j.appender.MongoDB.layout = com.voxify.log.log4j.MongoDBPatternLayout

• Specify a conversion pattern	

   •  Must be a valid JSON document	

   •  Can have sub-documents	

   •  Values can be strings or arrays	

log4j.appender.MongoDB.layout.ConversionPattern = {"ts":"%d{yyyy-MM-dd HH:mm:ss,SSS}",
"level":"%p","class":"%c{1}","message":"%m"}
Shell and Tail
In the Shell	

> use app_name
switched to db app_name

> db.log.findOne()
{
    "_id" : ObjectId("4c200c4d28cf037460e82363"),
    "ts" : "2010-10-21 18:05:17,237",
    "level" : "INFO",
    "class" : "NodeFlagGenerator",
    "message" : "Generating me some node flags”
}
Recent Log Events	

If using mongo 1.9.1+, put useful functions like this in .mongorc.js

> function last(count) {

       if (count==null) count = 1;

       return db.log.find({}, {_id:0}).sort({ts:-1}).limit(count);

   }



> last(2)

{ "ts" : "2010-10-28 23:49:14,567", "level" : "INFO", "session" :
"122", "class" : "Site", "message" : "uninformative message" }

{ "ts" : "2010-10-28 23:49:14,566", "level" : "INFO", "session" :
"122", "class" : "Site", "message" : "some info kind of message" }
Tailing Logs	

•  You’ll really miss ability to tail logfiles	

•  Or, ... will you?	

•  MongoDB offers tailable cursors	

•  Specify cursor, e.g., from find, as tailable	

•  While cursor alive, print new docs then sleep
Python Tailing Example 	

from pymongo import Connection
import time

db = Connection().my_db
coll = db.my_collection
cursor = coll.find(tailable=True)
while cursor.alive:
    try:
         doc = cursor.next()
         print doc
    except StopIteration:
         time.sleep(1)
Python	


•  Feature-rich logging module in Python 2.3	

•  Influenced by Log4J
Components	

•  Logger - represents area of library or project	

•  LogRecord	

•  Handler	

•  Filter	

•  Formatter	

•  LoggerAdapter (2.6) - add contextual info
mongodb-log	


•  Python logging handler to log to MongoDB	

•  Stores dictionary directly as BSON doc	

•  https://github.com/andreisavu/mongodb-log
log4mongo-python	

• Similar design to log4mongo PHP and .NET
 appenders 	

• Example programmatic handler config and usage:	

import logging
from log4mongo.handlers import MongoHandler

logger = logging.getLogger('test')
logger.addHandler(MongoHandler(host='localhost'))
logger.warning('test')	

• http://log4mongo.org/display/PUB/Log4mongo+for+Python	

• https://github.com/log4mongo/log4mongo-python
Ruby	

•  Built-in Logger library	

•  Limited feature set	

•  Console and file outputs	

•  Message format is fixed	

•  Datetime format is configurable
Log4r	

•  Influenced by Log4J	

  •  Logger	

  •  Outputter	

  •  Formatter	

  •  Configurator	

•  http://log4r.rubyforge.org	

•  Code for using Log4r with MongoMapper	

  •  http://gist.github.com/207347
Logging	


•  Similar to Log4r with a few enhancements	

•  Supports JSON as output format	

•  http://logging.rubyforge.org/
mongo_db_logger	

•  Rails plugin which uses Rails Logger, which can use
  Ruby Logger, Log4r, etc.	

•  Logs a single document per request	

  •  IP address, time, etc.	

  •  All messages logged during request	

  •  Custom request data	

•  Configure via database.yml	

•  http://github.com/peburrows/mongo_db_logger
Central Logger	

•  Forked from mongo_db_logger	

•  Added Rails 3 support and gemified	

•  Web UI for searching, filtering and analyzing logs	

•  Configure via database.yml                        (or mongoid.yml or central_logger.yml)	


•    http://www.slideshare.net/leopard_me/logging-rails-application-behavior-to-mongodb	


•    http://files.meetup.com/1742411/Logging%20With%20MongoDB.pdf	


•  https://github.com/customink/central_logger
MongodbLogger	

•  Based on central_logger	

•  Different web UI for searching, filtering and
  analyzing logs	

•  Configure via database.yml
  mongodb_logger.yml)	

                              (or mongoid.yml or



•  http://mongodb-logger.catware.org/
PHP	


•  error_log() logs to file, email address or
  system logger (e.g., syslog or NT EventLog)	

•  Log4PHP - heavily influenced by Log4J 	

•  PEAR Log - PHP-specific logging library	

•  Zend_Log - Logging for Zend Framework
Log4mongo-php	


•  Log4PHP appender	

•  Programmatic & declarative config	

•  Timestamp stored as Mongo Date	

•  Exception stored as array - msg, code, stacktrace	

•  Inner exception stored if available	

•  https://github.com/log4mongo/log4mongo-php
PEAR Log	

•  PEAR Log class - lots of handlers (Firebug!), but
  none yet for MongoDB	

•  Programmatic config	

•  Specify handler when creating logger object	

•  Composite handler for multiple destinations	

•  http://pear.php.net/package/Log/
Log Writer for Zend
                	

•  Project - Recordshelf_Log_Writer_MongoDb	

•  Described as a prototypesque implementation 	

•  Standard Zend tools to config and init	

•  Zend log event is associative array	

•  Project has code for accessing  filtering events	

•    http://raphaelstolt.blogspot.com/2009/09/logging-to-mongodb-and-accessing-log.html
C#/.NET	


•  Log4net	

•  Heavily influenced by Log4J	

•  http://logging.apache.org/log4net/
Log4mongo-net	


•  Uses community developed C# driver, but may
  move to official driver	

•  Tested with .NET 3.5+ and Mono 2.8	

•  Standard App.config/Web.config or programmatic	

•  http://log4mongo.org/display/PUB/Log4mongo+for+.NET
Other Options	

•  Destinations include MongoDB	

  •  Flume, logstash, Fluentd	

•  Primary Destination is HDFS	

  •  Chukwa and Scribe	

•  Commercial Hosted	

  •  Loggly 	

•  Commercial On-Premise	

  •  Splunk
Flume	

•  If you can t change the source of logs ...	

•  Can aggregate files, syslog, etc.	

•  Agent -- Collector -- Storage	

•  Storage plugins for HDFS, MongoDB, etc.	

•  Decorators transform data  extract metadata	

•  https://github.com/cloudera/flume	

•  https://github.com/mongodb/mongo-hadoop	

•  Chukwa, Honu and Scribe more HDFS-centric
logstash	

•  Log event storage, search and graphing	

•  Read from file, syslog, AMQP, tcp, etc.	

•  Store to ElasticSearch, AMQP queue,
  websockets, MongoDB, etc.	

•  http://logstash.net/	

•  http://code.google.com/p/logstash
Fluentd	

•  Ruby-based open source log collector	

•  Input, output and buffer plug-ins	

•  For example, plug-ins for tailing Apache log files
     and storing log events to MongoDB	

•    http://blog.treasure-data.com/post/13766262632/real-time-log-collection-with-
     fluentd-and-mongodb	


•  http://fluentd.org/doc/	

•  https://github.com/fluent/fluentd
Questions?	



•  Photo/Image Credits	

  •    Log analysis - ww.flickr.com/photos/linuxtuxguy/2579295702 	


  •    Don’t Stop Believing - www.flickr.com/photos/loozrboy/3908830690/	


  •    Homeland Security Advisory System - www.theonion.com/articles/iraq-adopts-terror-alert-system,1258/	


  •    Advice - www.flickr.com/photos/wurzle/659315/	


  •    Shell - http://www.flickr.com/photos/geishaboy500/305555652/	


  •    Tail - http://www.flickr.com/photos/autumnsonata/2632194442/	


  •    Log flume - www.flickr.com/photos/halloweenjack/1330806722/	


  •    Cufflinks - www.flickr.com/photos/oberazzi/318947873/

More Related Content

What's hot

Whats new in was liberty security and cloud readiness
Whats new in was liberty   security and cloud readinessWhats new in was liberty   security and cloud readiness
Whats new in was liberty security and cloud readiness
sflynn073
 

What's hot (20)

Master the Monorepo
Master the MonorepoMaster the Monorepo
Master the Monorepo
 
Distributed Tracing in Practice
Distributed Tracing in PracticeDistributed Tracing in Practice
Distributed Tracing in Practice
 
Ad108 - XPages in the IBM Lotus Notes Client - A Deep Dive!
Ad108 - XPages in the IBM Lotus Notes Client - A Deep Dive!Ad108 - XPages in the IBM Lotus Notes Client - A Deep Dive!
Ad108 - XPages in the IBM Lotus Notes Client - A Deep Dive!
 
Salesforce DX 201 - Advanced Implementation for ISVs
Salesforce DX 201 - Advanced Implementation for ISVsSalesforce DX 201 - Advanced Implementation for ISVs
Salesforce DX 201 - Advanced Implementation for ISVs
 
Accelerating DevOps with ChatOps
Accelerating DevOps with ChatOpsAccelerating DevOps with ChatOps
Accelerating DevOps with ChatOps
 
Building Repeatable Infrastructure using Terraform
Building Repeatable Infrastructure using TerraformBuilding Repeatable Infrastructure using Terraform
Building Repeatable Infrastructure using Terraform
 
The basics of fluentd
The basics of fluentdThe basics of fluentd
The basics of fluentd
 
Jenkins Pipelines
Jenkins PipelinesJenkins Pipelines
Jenkins Pipelines
 
Detecting secrets in code committed to gitlab (in real time)
Detecting secrets in code committed to gitlab (in real time)Detecting secrets in code committed to gitlab (in real time)
Detecting secrets in code committed to gitlab (in real time)
 
GitHub Actions with Node.js
GitHub Actions with Node.jsGitHub Actions with Node.js
GitHub Actions with Node.js
 
Janus + Audio @ Open Source World
Janus + Audio @ Open Source WorldJanus + Audio @ Open Source World
Janus + Audio @ Open Source World
 
Intro to GitOps & Flux.pdf
Intro to GitOps & Flux.pdfIntro to GitOps & Flux.pdf
Intro to GitOps & Flux.pdf
 
KITE Network Instrumentation: Advanced WebRTC Testing
KITE Network Instrumentation: Advanced WebRTC TestingKITE Network Instrumentation: Advanced WebRTC Testing
KITE Network Instrumentation: Advanced WebRTC Testing
 
Whats new in was liberty security and cloud readiness
Whats new in was liberty   security and cloud readinessWhats new in was liberty   security and cloud readiness
Whats new in was liberty security and cloud readiness
 
Engage 2019: Introduction to Node-Red
Engage 2019: Introduction to Node-RedEngage 2019: Introduction to Node-Red
Engage 2019: Introduction to Node-Red
 
Why NGINX Plus/NGINX Controller for NGINX OSS users
Why NGINX Plus/NGINX Controller for NGINX OSS usersWhy NGINX Plus/NGINX Controller for NGINX OSS users
Why NGINX Plus/NGINX Controller for NGINX OSS users
 
Git
GitGit
Git
 
Intro to Github Actions @likecoin
Intro to Github Actions @likecoinIntro to Github Actions @likecoin
Intro to Github Actions @likecoin
 
Sandbox vs manual malware analysis v1.1
Sandbox vs manual malware analysis v1.1Sandbox vs manual malware analysis v1.1
Sandbox vs manual malware analysis v1.1
 
LISA17 Container Performance Analysis
LISA17 Container Performance AnalysisLISA17 Container Performance Analysis
LISA17 Container Performance Analysis
 

Viewers also liked

Day 1 1505 - 1550 - pearl 1 - vimal kumar khanna
Day 1   1505 - 1550 - pearl 1 - vimal kumar khannaDay 1   1505 - 1550 - pearl 1 - vimal kumar khanna
Day 1 1505 - 1550 - pearl 1 - vimal kumar khanna
PMI2011
 
Holmes / A2 / Lab Design
Holmes / A2 / Lab DesignHolmes / A2 / Lab Design
Holmes / A2 / Lab Design
Rama Chandra
 

Viewers also liked (20)

Creating a MongoDB Based Logging System in a Webservice Heavy Environment
Creating a MongoDB Based Logging System in a Webservice Heavy EnvironmentCreating a MongoDB Based Logging System in a Webservice Heavy Environment
Creating a MongoDB Based Logging System in a Webservice Heavy Environment
 
Large Scale Log collection using LogStash & mongoDB
Large Scale Log collection using LogStash & mongoDB Large Scale Log collection using LogStash & mongoDB
Large Scale Log collection using LogStash & mongoDB
 
Practice Fusion & MongoDB: Transitioning a 4 TB Audit Log from SQL Server to ...
Practice Fusion & MongoDB: Transitioning a 4 TB Audit Log from SQL Server to ...Practice Fusion & MongoDB: Transitioning a 4 TB Audit Log from SQL Server to ...
Practice Fusion & MongoDB: Transitioning a 4 TB Audit Log from SQL Server to ...
 
An Open Source NoSQL solution for Internet Access Logs Analysis
An Open Source NoSQL solution for Internet Access Logs AnalysisAn Open Source NoSQL solution for Internet Access Logs Analysis
An Open Source NoSQL solution for Internet Access Logs Analysis
 
Hadoop
HadoopHadoop
Hadoop
 
Day 1 1505 - 1550 - pearl 1 - vimal kumar khanna
Day 1   1505 - 1550 - pearl 1 - vimal kumar khannaDay 1   1505 - 1550 - pearl 1 - vimal kumar khanna
Day 1 1505 - 1550 - pearl 1 - vimal kumar khanna
 
Hadoop trainting in hyderabad@kelly technologies
Hadoop trainting in hyderabad@kelly technologiesHadoop trainting in hyderabad@kelly technologies
Hadoop trainting in hyderabad@kelly technologies
 
Attacking MongoDB
Attacking MongoDBAttacking MongoDB
Attacking MongoDB
 
Rainyday
RainydayRainyday
Rainyday
 
No sql matters_2012_keynote
No sql matters_2012_keynoteNo sql matters_2012_keynote
No sql matters_2012_keynote
 
Sync is hard: building offline-first Android apps from the ground up
Sync is hard: building offline-first Android apps from the ground up	Sync is hard: building offline-first Android apps from the ground up
Sync is hard: building offline-first Android apps from the ground up
 
Holmes / A2 / Lab Design
Holmes / A2 / Lab DesignHolmes / A2 / Lab Design
Holmes / A2 / Lab Design
 
Web 10,20,30
Web 10,20,30 Web 10,20,30
Web 10,20,30
 
Cv orlan
Cv orlanCv orlan
Cv orlan
 
MongoDB basics in Russian
MongoDB basics in RussianMongoDB basics in Russian
MongoDB basics in Russian
 
Pracital application logging and monitoring
Pracital application logging and monitoringPracital application logging and monitoring
Pracital application logging and monitoring
 
FLTK Summer Course - Part VII - Seventh Impact
FLTK Summer Course - Part VII  - Seventh ImpactFLTK Summer Course - Part VII  - Seventh Impact
FLTK Summer Course - Part VII - Seventh Impact
 
FLTK Summer Course - Part II - Second Impact - Exercises
FLTK Summer Course - Part II - Second Impact - Exercises FLTK Summer Course - Part II - Second Impact - Exercises
FLTK Summer Course - Part II - Second Impact - Exercises
 
TMS - Schedule of Presentations and Reports
TMS - Schedule of Presentations and ReportsTMS - Schedule of Presentations and Reports
TMS - Schedule of Presentations and Reports
 
FLTK Summer Course - Part VI - Sixth Impact - Exercises
FLTK Summer Course - Part VI - Sixth Impact - ExercisesFLTK Summer Course - Part VI - Sixth Impact - Exercises
FLTK Summer Course - Part VI - Sixth Impact - Exercises
 

Similar to Logging Application Behavior to MongoDB

Logging and Exception
Logging and ExceptionLogging and Exception
Logging and Exception
Azeem Mumtaz
 
Distributed Logging System Using Elasticsearch Logstash,Beat,Kibana Stack and...
Distributed Logging System Using Elasticsearch Logstash,Beat,Kibana Stack and...Distributed Logging System Using Elasticsearch Logstash,Beat,Kibana Stack and...
Distributed Logging System Using Elasticsearch Logstash,Beat,Kibana Stack and...
Sanjog Kumar Dash
 
Messaging, interoperability and log aggregation - a new framework
Messaging, interoperability and log aggregation - a new frameworkMessaging, interoperability and log aggregation - a new framework
Messaging, interoperability and log aggregation - a new framework
Tomas Doran
 

Similar to Logging Application Behavior to MongoDB (20)

The new OSGi LogService 1.4 and integrating with SLF4J
The new OSGi LogService 1.4 and integrating with SLF4JThe new OSGi LogService 1.4 and integrating with SLF4J
The new OSGi LogService 1.4 and integrating with SLF4J
 
Integrating SLF4J and the new OSGi LogService 1.4 - BJ Hargrave (IBM)
Integrating SLF4J and the new OSGi LogService 1.4 - BJ Hargrave (IBM)Integrating SLF4J and the new OSGi LogService 1.4 - BJ Hargrave (IBM)
Integrating SLF4J and the new OSGi LogService 1.4 - BJ Hargrave (IBM)
 
FireBug And FirePHP
FireBug And FirePHPFireBug And FirePHP
FireBug And FirePHP
 
State of the art logging
State of the art loggingState of the art logging
State of the art logging
 
Log4j is a reliable, fast and flexible
Log4j is a reliable, fast and flexibleLog4j is a reliable, fast and flexible
Log4j is a reliable, fast and flexible
 
Log4e
Log4eLog4e
Log4e
 
Atmosphere 2014: Centralized log management based on Logstash and Kibana - ca...
Atmosphere 2014: Centralized log management based on Logstash and Kibana - ca...Atmosphere 2014: Centralized log management based on Logstash and Kibana - ca...
Atmosphere 2014: Centralized log management based on Logstash and Kibana - ca...
 
Logging in Scala
Logging in ScalaLogging in Scala
Logging in Scala
 
Conditional Logging Considered Harmful - Sean Reilly
Conditional Logging Considered Harmful - Sean ReillyConditional Logging Considered Harmful - Sean Reilly
Conditional Logging Considered Harmful - Sean Reilly
 
Scalable Log Analysis with WSO2 BAM
Scalable Log Analysis with WSO2 BAMScalable Log Analysis with WSO2 BAM
Scalable Log Analysis with WSO2 BAM
 
NSLogger - Cocoaheads Paris Presentation - English
NSLogger - Cocoaheads Paris Presentation - EnglishNSLogger - Cocoaheads Paris Presentation - English
NSLogger - Cocoaheads Paris Presentation - English
 
Logging and Exception
Logging and ExceptionLogging and Exception
Logging and Exception
 
Django
DjangoDjango
Django
 
Logs aggregation and analysis
Logs aggregation and analysisLogs aggregation and analysis
Logs aggregation and analysis
 
Distributed Logging System Using Elasticsearch Logstash,Beat,Kibana Stack and...
Distributed Logging System Using Elasticsearch Logstash,Beat,Kibana Stack and...Distributed Logging System Using Elasticsearch Logstash,Beat,Kibana Stack and...
Distributed Logging System Using Elasticsearch Logstash,Beat,Kibana Stack and...
 
Fluent Bit: Log Forwarding at Scale
Fluent Bit: Log Forwarding at ScaleFluent Bit: Log Forwarding at Scale
Fluent Bit: Log Forwarding at Scale
 
PRMA - Introduction
PRMA - IntroductionPRMA - Introduction
PRMA - Introduction
 
Messaging, interoperability and log aggregation - a new framework
Messaging, interoperability and log aggregation - a new frameworkMessaging, interoperability and log aggregation - a new framework
Messaging, interoperability and log aggregation - a new framework
 
Distributed Logging Architecture in the Container Era
Distributed Logging Architecture in the Container EraDistributed Logging Architecture in the Container Era
Distributed Logging Architecture in the Container Era
 
Distributed Logging Architecture in Container Era
Distributed Logging Architecture in Container EraDistributed Logging Architecture in Container Era
Distributed Logging Architecture in Container Era
 

Recently uploaded

Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
FIDO Alliance
 

Recently uploaded (20)

WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024
 
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfHow Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
 
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties ReimaginedEasier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
 
Working together SRE & Platform Engineering
Working together SRE & Platform EngineeringWorking together SRE & Platform Engineering
Working together SRE & Platform Engineering
 
Overview of Hyperledger Foundation
Overview of Hyperledger FoundationOverview of Hyperledger Foundation
Overview of Hyperledger Foundation
 
Google I/O Extended 2024 Warsaw
Google I/O Extended 2024 WarsawGoogle I/O Extended 2024 Warsaw
Google I/O Extended 2024 Warsaw
 
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
 
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdfSimplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
 
Microsoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - QuestionnaireMicrosoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - Questionnaire
 
2024 May Patch Tuesday
2024 May Patch Tuesday2024 May Patch Tuesday
2024 May Patch Tuesday
 
The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?
 
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
 
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
 
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
 
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
 
AI mind or machine power point presentation
AI mind or machine power point presentationAI mind or machine power point presentation
AI mind or machine power point presentation
 
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe
 
ADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptxADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptx
 
Event-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream ProcessingEvent-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream Processing
 
TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024
 

Logging Application Behavior to MongoDB

  • 1. Logging Application Behavior to MongoDB Robert Stewart @wombatnation 1/1/2012
  • 2. TOC •  Why Log to MongoDB? •  Logging Library Basics •  Logging by Language •  Other Ways to Distribute Your Logging
  • 3. Why Log to MongoDB? •  Centralized application logging •  Easy remote access, compared to files •  Log events good fit for document model •  Flexible schema •  Indexes for frequent queries •  Capped collections are very efficient •  Analyze data in-place with JavaScript MapReduce
  • 4. Event Storage •  Create a database for your app •  Create a capped collection to store log events •  Time-based eviction more natural, but less efficient •  Capped collection reduces chance of filling up disk
  • 5. Log Analysis •  Compared to files, easier to analyze app behavior across servers and days •  Log values to custom keys where possible •  JavaScript MapReduce
  • 6. Miscellaneous Free Advice •  Take advantage of separating storage from presentation •  Be aware of which classes your Mongo driver knows how to BSONify •  If logging from multiple languages, sticking to Log4Lang family can help
  • 7. Logging Library Basics •  Some provide API with pluggable implementation •  Configuration •  Programmatic •  Declarative •  Dynamic
  • 8. Logging Concepts •  Named, Hierarchical Logger Objects •  Identify a class or an area of an app •  Log Level, e.g., error, warn, info, debug •  Thresholds determine if event is logged •  Logging Event •  The message •  Metadata, e.g., time, level, class, method, line
  • 9. Logging Concepts •  Handler - determines event destination •  File, console, socket, email, syslog, MongoDB, etc. •  Handler level threshold •  Async vs. sync •  Filter - for logger and/or handler •  Formatter - event and message •  Pattern with placeholder tokens
  • 10. Common Flow for Logging App calls For each handler: logger.info( freak out ) Is handler info No Is logger enabled? info No enabled? Yes Passes Yes No filter, if Passes any? filter, if Yes Yes any? Format and store event No
  • 11. Polyglotism •  Good news - lots of client language bindings and logging libraries for MongoDB •  Mostly good news - Log4J very influential •  Sort of bad news - default log event formats can be very different
  • 12. Logging by Language •  Java •  Python •  Ruby •  PHP •  C#
  • 13. Java •  APIs •  Apache Commons Logging •  SLF4J •  java.util.logging, a.k.a., JUL •  Limited feature set •  Destinations - console, file, socket, memory •  JVM-level config
  • 14. Log4J •  Apache Log4J is dominant logging library •  Logback is potential successor •  Logger - usually, represents class logging the event •  LoggingEvent - message and metadata •  Appender - sends LoggingEvent to destination •  Layout - formats the event •  Converter - convert placeholder to value
  • 15. Log4mongo-java •  Set of Log4J Appenders for MongoDB •  log4mongo.org •  Open source project on GitHub •  I m a committer with Peter Monks and Jozef Šev ík •  I used it at Voxify with speech reco applications and infrastructure services •  Peter has used it at Alfresco
  • 16. MongoDbAppender •  Stores a Log4J LoggingEvent object as a document in a collection •  Format is very similar to structure of LoggingEvent Java class •  Where possible, data is stored as appropriate BSON objects, rather than just strings
  • 17. MongoDbPatternLayoutAppender •  Stores log event based on user-specified pattern •  Standard Log4J pattern layout, parser and converter functionality •  Very flexible formatting at cost of additional config •  Values stored as strings or arrays •  Custom values determined at time event is logged
  • 18. BsonAppender •  Custom log format by extending BsonAppender •  Can use it to log BSON objects to other data stores •  Includes code for BSONifying Java exceptions
  • 19. Enabling log4mongo-java • Associate a name with an appender implementation log4j.appender.MongoDB = org.log4mongo.MongoDbAppender! • Specify host (optional port, username and password). For replica set, specify all active hosts. log4j.appender.MongoDB.hostname = localhost • Specify database and collection log4j.appender.MongoDB.databaseName = app_name log4j.appender.MongoDB.collectionName = log • Add named appender to rootLogger list log4j.rootLogger = INFO, file, MongoDB
  • 20. Using Pattern Layout Appender • Specify pattern layout appender log4j.appender.MongoDB = org.log4mongo.MongoDbPatternLayoutAppender • Specify  your  custom  Pa2ernLayout  class   log4j.appender.MongoDB.layout = com.voxify.log.log4j.MongoDBPatternLayout • Specify a conversion pattern •  Must be a valid JSON document •  Can have sub-documents •  Values can be strings or arrays log4j.appender.MongoDB.layout.ConversionPattern = {"ts":"%d{yyyy-MM-dd HH:mm:ss,SSS}", "level":"%p","class":"%c{1}","message":"%m"}
  • 22. In the Shell > use app_name switched to db app_name > db.log.findOne() { "_id" : ObjectId("4c200c4d28cf037460e82363"), "ts" : "2010-10-21 18:05:17,237", "level" : "INFO", "class" : "NodeFlagGenerator", "message" : "Generating me some node flags” }
  • 23. Recent Log Events If using mongo 1.9.1+, put useful functions like this in .mongorc.js > function last(count) { if (count==null) count = 1; return db.log.find({}, {_id:0}).sort({ts:-1}).limit(count); } > last(2) { "ts" : "2010-10-28 23:49:14,567", "level" : "INFO", "session" : "122", "class" : "Site", "message" : "uninformative message" } { "ts" : "2010-10-28 23:49:14,566", "level" : "INFO", "session" : "122", "class" : "Site", "message" : "some info kind of message" }
  • 24. Tailing Logs •  You’ll really miss ability to tail logfiles •  Or, ... will you? •  MongoDB offers tailable cursors •  Specify cursor, e.g., from find, as tailable •  While cursor alive, print new docs then sleep
  • 25. Python Tailing Example from pymongo import Connection import time db = Connection().my_db coll = db.my_collection cursor = coll.find(tailable=True) while cursor.alive: try: doc = cursor.next() print doc except StopIteration: time.sleep(1)
  • 26. Python •  Feature-rich logging module in Python 2.3 •  Influenced by Log4J
  • 27. Components •  Logger - represents area of library or project •  LogRecord •  Handler •  Filter •  Formatter •  LoggerAdapter (2.6) - add contextual info
  • 28. mongodb-log •  Python logging handler to log to MongoDB •  Stores dictionary directly as BSON doc •  https://github.com/andreisavu/mongodb-log
  • 29. log4mongo-python • Similar design to log4mongo PHP and .NET appenders • Example programmatic handler config and usage: import logging from log4mongo.handlers import MongoHandler logger = logging.getLogger('test') logger.addHandler(MongoHandler(host='localhost')) logger.warning('test') • http://log4mongo.org/display/PUB/Log4mongo+for+Python • https://github.com/log4mongo/log4mongo-python
  • 30. Ruby •  Built-in Logger library •  Limited feature set •  Console and file outputs •  Message format is fixed •  Datetime format is configurable
  • 31. Log4r •  Influenced by Log4J •  Logger •  Outputter •  Formatter •  Configurator •  http://log4r.rubyforge.org •  Code for using Log4r with MongoMapper •  http://gist.github.com/207347
  • 32. Logging •  Similar to Log4r with a few enhancements •  Supports JSON as output format •  http://logging.rubyforge.org/
  • 33. mongo_db_logger •  Rails plugin which uses Rails Logger, which can use Ruby Logger, Log4r, etc. •  Logs a single document per request •  IP address, time, etc. •  All messages logged during request •  Custom request data •  Configure via database.yml •  http://github.com/peburrows/mongo_db_logger
  • 34. Central Logger •  Forked from mongo_db_logger •  Added Rails 3 support and gemified •  Web UI for searching, filtering and analyzing logs •  Configure via database.yml (or mongoid.yml or central_logger.yml) •  http://www.slideshare.net/leopard_me/logging-rails-application-behavior-to-mongodb •  http://files.meetup.com/1742411/Logging%20With%20MongoDB.pdf •  https://github.com/customink/central_logger
  • 35. MongodbLogger •  Based on central_logger •  Different web UI for searching, filtering and analyzing logs •  Configure via database.yml mongodb_logger.yml) (or mongoid.yml or •  http://mongodb-logger.catware.org/
  • 36. PHP •  error_log() logs to file, email address or system logger (e.g., syslog or NT EventLog) •  Log4PHP - heavily influenced by Log4J •  PEAR Log - PHP-specific logging library •  Zend_Log - Logging for Zend Framework
  • 37. Log4mongo-php •  Log4PHP appender •  Programmatic & declarative config •  Timestamp stored as Mongo Date •  Exception stored as array - msg, code, stacktrace •  Inner exception stored if available •  https://github.com/log4mongo/log4mongo-php
  • 38. PEAR Log •  PEAR Log class - lots of handlers (Firebug!), but none yet for MongoDB •  Programmatic config •  Specify handler when creating logger object •  Composite handler for multiple destinations •  http://pear.php.net/package/Log/
  • 39. Log Writer for Zend •  Project - Recordshelf_Log_Writer_MongoDb •  Described as a prototypesque implementation •  Standard Zend tools to config and init •  Zend log event is associative array •  Project has code for accessing filtering events •  http://raphaelstolt.blogspot.com/2009/09/logging-to-mongodb-and-accessing-log.html
  • 40. C#/.NET •  Log4net •  Heavily influenced by Log4J •  http://logging.apache.org/log4net/
  • 41. Log4mongo-net •  Uses community developed C# driver, but may move to official driver •  Tested with .NET 3.5+ and Mono 2.8 •  Standard App.config/Web.config or programmatic •  http://log4mongo.org/display/PUB/Log4mongo+for+.NET
  • 42. Other Options •  Destinations include MongoDB •  Flume, logstash, Fluentd •  Primary Destination is HDFS •  Chukwa and Scribe •  Commercial Hosted •  Loggly •  Commercial On-Premise •  Splunk
  • 43. Flume •  If you can t change the source of logs ... •  Can aggregate files, syslog, etc. •  Agent -- Collector -- Storage •  Storage plugins for HDFS, MongoDB, etc. •  Decorators transform data extract metadata •  https://github.com/cloudera/flume •  https://github.com/mongodb/mongo-hadoop •  Chukwa, Honu and Scribe more HDFS-centric
  • 44. logstash •  Log event storage, search and graphing •  Read from file, syslog, AMQP, tcp, etc. •  Store to ElasticSearch, AMQP queue, websockets, MongoDB, etc. •  http://logstash.net/ •  http://code.google.com/p/logstash
  • 45. Fluentd •  Ruby-based open source log collector •  Input, output and buffer plug-ins •  For example, plug-ins for tailing Apache log files and storing log events to MongoDB •  http://blog.treasure-data.com/post/13766262632/real-time-log-collection-with- fluentd-and-mongodb •  http://fluentd.org/doc/ •  https://github.com/fluent/fluentd
  • 46. Questions? •  Photo/Image Credits •  Log analysis - ww.flickr.com/photos/linuxtuxguy/2579295702 •  Don’t Stop Believing - www.flickr.com/photos/loozrboy/3908830690/ •  Homeland Security Advisory System - www.theonion.com/articles/iraq-adopts-terror-alert-system,1258/ •  Advice - www.flickr.com/photos/wurzle/659315/ •  Shell - http://www.flickr.com/photos/geishaboy500/305555652/ •  Tail - http://www.flickr.com/photos/autumnsonata/2632194442/ •  Log flume - www.flickr.com/photos/halloweenjack/1330806722/ •  Cufflinks - www.flickr.com/photos/oberazzi/318947873/