Log managementforthecloudwithlogstash installationguide

401 views
294 views

Published on

Opensource, Highly available and Scalable solution that can accommodate your Log Management needs with a centralized Dashboard with Filtering capabilities using elastic search

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
401
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Log managementforthecloudwithlogstash installationguide

  1. 1. How To Deploy Logstash 1.1.13 on Centos 6.x Author : Kanwar Batra Audience : System Administrators, NOC Monitoring Team, DBA's, Developers Relevance : This document outlines the deployment of Logstash server components What is Logstash Logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). Speaking of searching, logstash comes with a web interface for searching and drilling into all of your logs. How to Download the Software The software can be downloaded here Software Details This document is based on a 2 node deployment as a POC without redundancy. Logstash is recommended to be deployed as a HA Cluster for redundancy and avoid loss of log data due to individual node outages.  First Node (LogStash Master Node) o Centos 6.4 64 bit o Logstash 1.1.13 o Elasticsearch v0.90 or higher o Java v1.6 or Higher o redis 2.6 o httpd 2.4 o apr 1.4.6 o grok 1.2 o geoip-geolite 2013.04.1  Second Node (Elasticsearch Node) o Centos 6.4 64 bit o Logstash 1.1.13 (For Agent) o Elasticsearch v0.90 or higher o Java v1.6 or Higher O/S Configuration Changes On Centos 6.4 Server modify the following files  /etc/sysctl.conf add to bottom of file o sudo vi /etc/sysctl.conf  vm.overcommit_memory = 1  /etc/security/limits.conf  o * soft core unlimited o * soft nofile 65535 o * hard nofile 65535 o elsearch soft memlock unlimited o elsearch hard memlock unlimited o elsearch soft nofile 256000 o elsearch hard nofile 256000 o elsearch soft rss unlimited o elsearch hard rss unlimited o elsearch soft stack unlimited o elsearch hard stack unlimited
  2. 2. o elsearch soft cpu unlimited o elsearch hard cpu unlimited o elsearch soft nproc unlimited o elsearch hard nproc unlimited o elsearch soft as unlimited o elsearch hard as unlimited  /etc/sysctl/selinux o SELINUX=disabled  /etc/sysconfig/iptables & ip6tables o Modify the files and add relevant ports. This document is created based on iptables being disabled. o service iptables stop o service ip6tables stop o chkconfig iptables off o chkconfig ip6tables off  Reboot the Host after above Changes Pre-Install Checks  o service iptables status ( output - iptables: Firewall is not running) o service ip6tables status ( output - ip6tables: Firewall is not running) o sestatus ( output - SELinux status: disabled) Software Install Logstash Node (Install rpm's in the following order )  sudo yum install java-1.6.0-sun-1.6.0.32-1jpp.x86_64.rpm  sudo yum install elasticsearch-0.90.2-1.el6.x86_64.rpm logstash-1.1.13-1.el6.noarch.rpm redis-2.6.13-1.el6.x86_64.rpm grok-1.20110708.1-1.el6.x86_64.rpm  sudo yum install geoip-geolite-2013.04-1.el6.noarch.rpm  Backup the default Logstash file logstash.conf in /etc/logstash directory to logstash.conf.default  Create logstash.conf  Modify the elastic search yml file also and update it with relevant node details  if you are using GeoIP license change the logstash GOIP to ls /usr/share/GeoIP/GeoIPCity.dat if using lite us the value in the attached logstash.conf  Install sudo yum install httpd-* apr-*  Create a link to /usr/lib64 in /etc/httpd  Modify the httpd.conf Please pay special attention to the LoadModules .  Unzip the kibana software downloaded earlier and move the directory to /var/www/html  Change directory to location of your kibana (/var/www/html/kibana3), copy kibana3.conf to /etc/httpd/conf.d  kibana conf should be configured  config.js is updated as  To have all services startup at boot run chkconfig o chkconfig httpd on o chkconfig elasticsearch on o chkconfig logstash on  This completes the setup of Logstash software on the First Host. The second host is configured as an elastic search server. Elasticsearch Node (Install rpm's in the following order )  sudo yum install java-1.6.0-sun-1.6.0.32-1jpp.x86_64.rpm  sudo yum install elasticsearch-0.90.2-1.el6.x86_64.rpm logstash-1.1.13- 1.el6.noarch.rpm grok-1.20110708.1-1.el6.x86_64.rpm  sudo yum install geoip-geolite-2013.04-1.el6.noarch.rpm
  3. 3.  Backup the default Logstash file logstash.conf in /etc/logstash directory to logstash.conf.default  Create logstash.conf as for the agent  Update the elastic search yml as  To have all services startup at boot run chkconfig o chkconfig httpd on o chkconfig elasticsearch on o chkconfig logstash on  Now we have a running Logstash environment. At this time you can access the Kibana frontend  Run the curl command for template mapping from logstash server. Configuration Files for references https://drive.google.com/folderview?id=0B2jSbXbYuSe_MVotR3ZDdzlwaFE&usp=sharing Disclaimer: The install of this product and opinions are listed above are solely based on my experience in the implementation of Logstash for a Customer and is a working solution copy from that experience.. You can reference and use this document and send questions which I can answer based on my experience. This however is not an official document from Logstash team and they have not evaluated this document for it’s accuracy.

×