Logging rails application behavior to MongoDB

Logging rails application behavior to MongoDB Vasiliev Alexey Yuriyovich leopard.in.ua 2011 Vasiliev A.Y. Logging rails application behavior to MongoDB

MongoDB Introduction MongoDB is an open source document-oriented database. Features Consistent UTF-8 encoding; Cross-platform support; Eﬃcient storage for large binary data, for example photos and video; Replication and failover; MapReduce. Vasiliev A.Y. Logging rails application behavior to MongoDB

Why application should log to DB? Centralized application logging; Easy remote access, compared to ﬁles; Can be fast and scalable if you go with database partitioning and/or multi-log databases. Vasiliev A.Y. Logging rails application behavior to MongoDB

What are the advantages appear logging in MongoDB? Flexible schema; Asynchronous inserts; Capped collection; Fast enough compared to ﬁles; Indexed and searched; Analyze data in-place. Vasiliev A.Y. Logging rails application behavior to MongoDB

Gem for RailsMongoDB as Rails logger Central Logger (https://github.com/ customink/central_logger). You should do only several steps: gem ’central_logger’; include CentralLogger::Filter; Conﬁg the connection to mongodb; That’s all! Vasiliev A.Y. Logging rails application behavior to MongoDB

Rails log structureMongoDB as Rails logger { ’ action ’ : action_name , ’ application_name ’ : application_name ( r a i l s root ) , ’ controller ’ : controller_name , ’ ip ’ : ip_address , ’ messages ’ : { ’ info ’ : [ ], ’ debug ’ : [ ] , ’ error ’ : [ ] , ’ warn ’ : [ ], ’ fatal ’ : [ ] }, ’ params ’ : { }, ’ path ’ : path , ’ request_time ’ : date_of_request , ’ runtime ’ : execution_time_in_milliseconds , ’ url ’ : full_url } Vasiliev A.Y. Logging rails application behavior to MongoDB

Add data to logMongoDB as Rails logger Add additional data to log # make s u r e we ’ r e u s i n g t h e C e n t r a l L o g g e r i n t h i s e n v i r o n m e n t i f R a i l s . l o g g e r . r e s p o n d _ t o ? ( : add_metadata ) R a i l s . l o g g e r . add_metadata ( : u s e r _ g u i d => @ u s e r . i d ) end Vasiliev A.Y. Logging rails application behavior to MongoDB

Querying via the Rails consoleMongoDB as Rails logger Handle on the MongoDB >> db = R a i l s . l o g g e r . mongo_connection => #&l t ; Mongo : : DB: 0 x 1 0 2 f 1 9 a c 0 @ s l a v e _ o k= n i l ... >> c o l l e c t i o n = db [ R a i l s . l o g g e r . m o n g o _ c o l l e c t i o n _ n a m e ] => #&l t ; Mongo : : C o l l e c t i o n : 0 x 1 0 3 1 b 3 e e 8 . . . Find all requests for a speciﬁc user (with guid) >> c u r s o r = c o l l e c t i o n . f i n d ( : u s e r _ g u i d => 1 2 3 5 5 ) => #&l t ; Mongo : : C u r s o r : 0 x 1 0 3 1 a 3 e 3 0 . . . &g t ; >> c u r s o r . count => 255 Vasiliev A.Y. Logging rails application behavior to MongoDB

Querying via the Rails consoleMongoDB as Rails logger Find all requests that took more that two second to complete >> c o l l e c t i o n . f i n d ( { : r u n t i m e => { ’ $gt ’ => 2 0 0 0 } } ) . c o u n t => 3 Find all requests with an exception >> c o l l e c t i o n . f i n d ( { " m e s s a g e s . e r r o r " => / R o u t i n g E r r o r / } ) Vasiliev A.Y. Logging rails application behavior to MongoDB

Querying via the Rails consoleMongoDB as Rails logger Find by params values >> c o l l e c t i o n . f i n d ( " params . a c t i v i t y " => { ’ $ e x i s t s ’ => t r u e } , " params . a c t i v i t y . s t o r i e s " => { ’ $ e x i s t s ’ => t r u e } , " r e q u e s t _ t i m e " => { ’ $ g t e ’ => s t a r t _ d a t e . u t c , ’ $ l t e ’ => end_date . u t c } ) . s o r t ( " params . a c t i v i t y . o c c u r r e d _ a t " , 1 ) . c o u n t => 542 Vasiliev A.Y. Logging rails application behavior to MongoDB

Querying via the Rails consoleMongoDB as Rails logger Find current status of story using logs >> c o l l e c t i o n . f i n d ( " params . a c t i v i t y . s t o r i e s . i d " => s t o r y _ i d . to_i , " params . a c t i v i t y . s t o r i e s . c u r r e n t _ s t a t e " => { ’ $ e x i s t s ’ => t r u e } ) . s o r t ( " params . a c t i v i t y . o c c u r r e d _ a t " , −1). f i r s t => {" _id"=>BSON : : O b j e c t I d ( ’ 4 e 7 c 6 2 e d 7 6 1 2 6 8 0 4 d 4 0 0 0 0 0 f ’ ) , " a c t i o n "=>"hook " , " c o n t r o l l e r "=>" s e r v i c e s " , " p a t h "=>"/ a p i / v1 / hook " , " params"=>{" a c t i v i t y "=>{ . . . , " s t o r i e s " = > [ { . . . , " c u r r e n t _ s t a t e "=>" f i n i s h e d " } ] } } Vasiliev A.Y. Logging rails application behavior to MongoDB

Querying via the Rails consoleMongoDB as Rails logger Tailable Cursors v a r c o l l = db . some . c a p p e d . c o l l e c t i o n ; var l a s t V a l = c o l l . f i n d ( ) . s o r t ({ ’ $ n a t u r a l ’ : 1 }) . l i m i t ( 1 ) . next ( ) [ ’ increasing ’ ]; while (1){ c u r s o r = c o l l . f i n d ({ ’ i n c r e a s i n g ’ : { ’ $gte ’ : l a s t V a l } }); // t a i l a b l e c u r s o r . addOption ( 2 ) ; // a w a i t d a t a c u r s o r . a d d O p t i o n ( 32 ) ; // W a i t s s e v e r a l s e c f o r more d a t a w h i l e ( c u r s o r . hasNext ( ) ){ v a r doc = c u r s o r . n e x t ( ) ; l a s t V a l = doc [ ’ i n c r e a s i n g ’ ] ; p r i n t j s o n ( doc ) ; } } Vasiliev A.Y. Logging rails application behavior to MongoDB

Using MapReduceMongoDB as Rails logger Find average time of page loading >> c o l l e c t i o n . g r o u p ( " f u n c t i o n ( x ) { r e t u r n { month : x . r e q u e s t _ t i m e . getMonth ( ) , year : x . request_time . g e t F u l l Y e a r ( ) } ; }" , { : r e q u e s t _ t i m e => { ’ $ g t e ’ => Time . u t c ( 2 0 1 1 , 0 1 , 0 1 , 0 0 , 0 0 , 0 0 ) , " $ l t " => Time . u t c ( 2 0 1 1 , 8 , 0 1 , 0 0 , 0 0 , 0 0 ) } } , { : c o u n t => 0 , : t o t a l _ t i m e => 0 } , " f u n c t i o n ( doc , o u t ) { o u t . c o u n t ++; o u t . t o t a l _ t i m e+=doc . r u n t i m e } " , " f u n c t i o n ( out ){ o u t . avg_time = o u t . t o t a l _ t i m e / o u t . c o u n t } " ) => [ {" month "= >4.0 , " y e a r "= >2011.0 , " c o u n t "= >5385.0 , " t o t a l _ t i m e "= >1027015.0 , " avg_time "= >190.717734447539} , ... ] Vasiliev A.Y. Logging rails application behavior to MongoDB

RestrictionsMongoDB as Rails logger The database doesn’t allow update the existing objects in the collection (but there is possibility). The database doesn’t allow deleting objects from a capped collection. Capped collection are not shardable. More indexes - slower inserts. Vasiliev A.Y. Logging rails application behavior to MongoDB

Q&A Thank you for attention! Vasiliev A.Y. Logging rails application behavior to MongoDB

http://coderwall.com	33
http://www.hanrss.com	2
http://b.hatena.ne.jp	1
http://paper.li	1

Logging rails application behavior to MongoDB

by Alexey Vasiliev on Oct 10, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

4 Embeds 37

Statistics