Centralized Logging System Using MongoDB
@vparihar
AVP Engineering,Webonise Lab
Vivek Parihar
Who Am I?
● A Weboniser and Rubyist
● Blogger(vparihar01.github.com)
● MongoDb user
● Geek
● DevOps
● Mainly write Ruby, b...
● What is Logging?
● Why we need Logging?
● Logging DO’s and Don’t
● Logs are Streams, Not FIles
● Problems managing Logs ...
What is Logging?
Mmmm Logging: It is the most important part of any
application.
In General, Logging refers to keeping tra...
Why we need Logging?
Logging: Helps me finding and fixing bugs
Logging: Extensively used for Debugging
Logging: Helps us diagnose & understand the
behaviour of application.
Logging: Tells us exactly what happened
when, where and why?
Who did it ?
At what time ?
What did he steal ?
Logging: Do’s and Don’t
#1 It should be FAST
Logging: Do’s and Don’t
#2 Should not affect user
Prevent DISK BLOAT
It should not be like-:
{
● "#########its working#########"
● "!!!!!coming here in to get secondary users!!!!!"
● "#######...
Logging: Do’s and Don’t
4. Differentiate Log Levels
Logs Are Streams, Not Files
Logs are a stream, and it behooves everyone to treat them as such. Your
programs should log to...
Problems managing Logs for huge Infra
What about infra like these ?
Problems managing Logs for huge Infra
Expression like:
How can we solve huge Infra problem ?
Solution: Centralized Logging System
What Centralized Logging System can do
for us?
What Centralized Logging System can do for
us?
All of the logs are in one place, this makes things like searching
through ...
#2 Aggregation
Scaled-out servers behind load balancers each produce their
own log files, making it impossible to debug a ...
#3 High Availability
Suppose your system is down or overloaded and unable to tell
you what happened.
What Centralized Logg...
Local logs from the server may be lost in the event of an
intrusion or system failure. But by having the logs elsewhere
yo...
It reduces disk space usage and disk I/O on core servers
that should be busy doing something else.
#5 Prevent Disk BLOAT
W...
#6 Visual Indicators
Abnormal behaviors can be detected faster when we see
them in a visual instrument such as a graph, wh...
Centralized Logging System Architecture
What and Why ?
What’s Fluentd?
It’s like syslogd, but uses JSON for log messages
What’s Fluentd?
What’s Fluentd?
time
tag
record
What’s Fluentd?
What’s Fluentd?
Plug-in Plug-in Plug-in
So Fluentd is a:
Buffer
Router
Collector
Converter
Aggregator
…….
What’s Fluentd?
It’s written in RUBY :)
Why Fluentd?
Extensibility - Plugin Architecture
Why Fluentd?
Unified log format - JSON format
Why Fluentd?
Reliable - HA configuration
Why Fluentd?
Easy to install - RPM/deb packages
> sudo fluentd --setup && fluentd
Very small footprint
> small engine (3,000 line) + pl...
Why is good fit ?
1. It’s Schemaless
Document-oriented / JSON is
a great format for log
information. Very flexible and
“schemaless” in the s...
2. Fire and Forget
MongoDB inserts can be done asynchronously.
Why ?
3. Scalable and easy to replicate.
Built in ReplicaSet and Sharding provides high availability.
Why ?
4. Centralized and easy remote access
Why ?
5. Capped Collection
● They "remember" the insertion order of their documents
● They store inserted documents in the inser...
6. Tailing Logs
● You’ll really miss ability to tail logfiles
● Or, .. will you?
● MongoDB offers tailable cursors
Why ?
Tailable Cursors
What with Tailable Cursors ?
We can implement the pub/sub using
Node.js and MongoDB
https://github.com/sc...
Thanks
Would Love to answer your queries...
Vivek Parihar
@vparihar
Upcoming SlideShare
Loading in …5
×

Centralized logging system using mongoDB

2,274 views

Published on

This talk will cover the need of a centralized logging system, showcasing the architecture of the system. Also, I talk about how we ended up building this centralized logging system, What was the need for such a system, what problems we faced, how MongoDB fits into this and what others can learn from this.
I also covered some how can we use mongoDB to make our logging system for realtime analytics and alerting system
. The major use case of this system it to keeping track of meaningful events. This could be -:
1. How many users registered ?
2. How many registrations fails ?
3. Most occurred errors while doing something.
4. Realtime Analytics and Alerts
5. Identifying the possible threats.

Published in: Technology, Business

Centralized logging system using mongoDB

  1. 1. Centralized Logging System Using MongoDB @vparihar AVP Engineering,Webonise Lab Vivek Parihar
  2. 2. Who Am I? ● A Weboniser and Rubyist ● Blogger(vparihar01.github.com) ● MongoDb user ● Geek ● DevOps ● Mainly write Ruby, but have great passion for Javascript and Cloud Platforms...
  3. 3. ● What is Logging? ● Why we need Logging? ● Logging DO’s and Don’t ● Logs are Streams, Not FIles ● Problems managing Logs for huge INFRA ● What Central Logging System can do for us? ● Central Logging System Architecture ● What and why Fluentd? ● Why MongoDB is good fit. Agenda
  4. 4. What is Logging? Mmmm Logging: It is the most important part of any application. In General, Logging refers to keeping track of something.
  5. 5. Why we need Logging?
  6. 6. Logging: Helps me finding and fixing bugs
  7. 7. Logging: Extensively used for Debugging
  8. 8. Logging: Helps us diagnose & understand the behaviour of application.
  9. 9. Logging: Tells us exactly what happened when, where and why? Who did it ? At what time ? What did he steal ?
  10. 10. Logging: Do’s and Don’t #1 It should be FAST
  11. 11. Logging: Do’s and Don’t #2 Should not affect user Prevent DISK BLOAT
  12. 12. It should not be like-: { ● "#########its working#########" ● "!!!!!coming here in to get secondary users!!!!!" ● "#########I am Here#########" ● "#########Task completed#######" } Logging: Do’s and Don’t #3 Do Log only useful INFO
  13. 13. Logging: Do’s and Don’t 4. Differentiate Log Levels
  14. 14. Logs Are Streams, Not Files Logs are a stream, and it behooves everyone to treat them as such. Your programs should log to stdout and/or stderr and omit any attempt to handle log paths, log rotation, or sending logs over the syslog protocol. Directing where the program’s log stream goes can be left up to the runtime container: a local terminal or IDE (in development environments), an Upstart / Systemd launch script (in traditional hosting environments), or a system like Logplex/Heroku (in a platform environment). By: Adam Wiggins, Heroku co-founder.
  15. 15. Problems managing Logs for huge Infra
  16. 16. What about infra like these ? Problems managing Logs for huge Infra
  17. 17. Expression like:
  18. 18. How can we solve huge Infra problem ?
  19. 19. Solution: Centralized Logging System
  20. 20. What Centralized Logging System can do for us?
  21. 21. What Centralized Logging System can do for us? All of the logs are in one place, this makes things like searching through logs and analysis across multiple servers easier than bouncing around between boxes. Greatly simplifying log analysis and correlation tasks. #1 Log Collections
  22. 22. #2 Aggregation Scaled-out servers behind load balancers each produce their own log files, making it impossible to debug a single action flow that distributed between servers, unless the logs converge into a single article. What Centralized Logging System can do for us?
  23. 23. #3 High Availability Suppose your system is down or overloaded and unable to tell you what happened. What Centralized Logging System can do for us?
  24. 24. Local logs from the server may be lost in the event of an intrusion or system failure. But by having the logs elsewhere you at least have a chance of finding something useful about what happened. #4 Security What Centralized Logging System can do for us?
  25. 25. It reduces disk space usage and disk I/O on core servers that should be busy doing something else. #5 Prevent Disk BLOAT What Centralized Logging System can do for us?
  26. 26. #6 Visual Indicators Abnormal behaviors can be detected faster when we see them in a visual instrument such as a graph, where peak points are easily noticed. What Centralized Logging System can do for us?
  27. 27. Centralized Logging System Architecture
  28. 28. What and Why ?
  29. 29. What’s Fluentd? It’s like syslogd, but uses JSON for log messages
  30. 30. What’s Fluentd?
  31. 31. What’s Fluentd? time tag record
  32. 32. What’s Fluentd?
  33. 33. What’s Fluentd? Plug-in Plug-in Plug-in
  34. 34. So Fluentd is a: Buffer Router Collector Converter Aggregator ……. What’s Fluentd?
  35. 35. It’s written in RUBY :) Why Fluentd?
  36. 36. Extensibility - Plugin Architecture Why Fluentd?
  37. 37. Unified log format - JSON format Why Fluentd?
  38. 38. Reliable - HA configuration Why Fluentd?
  39. 39. Easy to install - RPM/deb packages > sudo fluentd --setup && fluentd Very small footprint > small engine (3,000 line) + plugins Why Fluentd?
  40. 40. Why is good fit ?
  41. 41. 1. It’s Schemaless Document-oriented / JSON is a great format for log information. Very flexible and “schemaless” in the sense we can throw in an extra field any time we want. Why ?
  42. 42. 2. Fire and Forget MongoDB inserts can be done asynchronously. Why ?
  43. 43. 3. Scalable and easy to replicate. Built in ReplicaSet and Sharding provides high availability. Why ?
  44. 44. 4. Centralized and easy remote access Why ?
  45. 45. 5. Capped Collection ● They "remember" the insertion order of their documents ● They store inserted documents in the insertion order on disk ● They remove the oldest documents in the collection automatically as new documents are inserted However, you give up some things with capped collections: ● They have a fixed maximum size ● You cannot shared a capped collection ● Any updates to documents in a capped collection must not cause a document to grow. (i.e. not all$set operations will work, and no $push or $pushAll will) ● You may not explicitly .remove() documents from a capped collection Why ?
  46. 46. 6. Tailing Logs ● You’ll really miss ability to tail logfiles ● Or, .. will you? ● MongoDB offers tailable cursors Why ?
  47. 47. Tailable Cursors What with Tailable Cursors ? We can implement the pub/sub using Node.js and MongoDB https://github.com/scttnlsn/mubsub Why ?
  48. 48. Thanks Would Love to answer your queries... Vivek Parihar @vparihar

×