• Save
Log managementforthecloudwithlogstash installationguide
Upcoming SlideShare
Loading in...5
×
 

Log managementforthecloudwithlogstash installationguide

on

  • 341 views

Opensource, Highly available and Scalable solution that can accommodate your Log Management needs with a centralized Dashboard with Filtering capabilities using elastic search

Opensource, Highly available and Scalable solution that can accommodate your Log Management needs with a centralized Dashboard with Filtering capabilities using elastic search

Statistics

Views

Total Views
341
Views on SlideShare
341
Embed Views
0

Actions

Likes
0
Downloads
0
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Log managementforthecloudwithlogstash installationguide Log managementforthecloudwithlogstash installationguide Document Transcript

  • How To Deploy Logstash 1.1.13 on Centos 6.x Author : Kanwar Batra Audience : System Administrators, NOC Monitoring Team, DBA's, Developers Relevance : This document outlines the deployment of Logstash server components What is Logstash Logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). Speaking of searching, logstash comes with a web interface for searching and drilling into all of your logs. How to Download the Software The software can be downloaded here Software Details This document is based on a 2 node deployment as a POC without redundancy. Logstash is recommended to be deployed as a HA Cluster for redundancy and avoid loss of log data due to individual node outages.  First Node (LogStash Master Node) o Centos 6.4 64 bit o Logstash 1.1.13 o Elasticsearch v0.90 or higher o Java v1.6 or Higher o redis 2.6 o httpd 2.4 o apr 1.4.6 o grok 1.2 o geoip-geolite 2013.04.1  Second Node (Elasticsearch Node) o Centos 6.4 64 bit o Logstash 1.1.13 (For Agent) o Elasticsearch v0.90 or higher o Java v1.6 or Higher O/S Configuration Changes On Centos 6.4 Server modify the following files  /etc/sysctl.conf add to bottom of file o sudo vi /etc/sysctl.conf  vm.overcommit_memory = 1  /etc/security/limits.conf  o * soft core unlimited o * soft nofile 65535 o * hard nofile 65535 o elsearch soft memlock unlimited o elsearch hard memlock unlimited o elsearch soft nofile 256000 o elsearch hard nofile 256000 o elsearch soft rss unlimited o elsearch hard rss unlimited o elsearch soft stack unlimited o elsearch hard stack unlimited
  • o elsearch soft cpu unlimited o elsearch hard cpu unlimited o elsearch soft nproc unlimited o elsearch hard nproc unlimited o elsearch soft as unlimited o elsearch hard as unlimited  /etc/sysctl/selinux o SELINUX=disabled  /etc/sysconfig/iptables & ip6tables o Modify the files and add relevant ports. This document is created based on iptables being disabled. o service iptables stop o service ip6tables stop o chkconfig iptables off o chkconfig ip6tables off  Reboot the Host after above Changes Pre-Install Checks  o service iptables status ( output - iptables: Firewall is not running) o service ip6tables status ( output - ip6tables: Firewall is not running) o sestatus ( output - SELinux status: disabled) Software Install Logstash Node (Install rpm's in the following order )  sudo yum install java-1.6.0-sun-1.6.0.32-1jpp.x86_64.rpm  sudo yum install elasticsearch-0.90.2-1.el6.x86_64.rpm logstash-1.1.13-1.el6.noarch.rpm redis-2.6.13-1.el6.x86_64.rpm grok-1.20110708.1-1.el6.x86_64.rpm  sudo yum install geoip-geolite-2013.04-1.el6.noarch.rpm  Backup the default Logstash file logstash.conf in /etc/logstash directory to logstash.conf.default  Create logstash.conf  Modify the elastic search yml file also and update it with relevant node details  if you are using GeoIP license change the logstash GOIP to ls /usr/share/GeoIP/GeoIPCity.dat if using lite us the value in the attached logstash.conf  Install sudo yum install httpd-* apr-*  Create a link to /usr/lib64 in /etc/httpd  Modify the httpd.conf Please pay special attention to the LoadModules .  Unzip the kibana software downloaded earlier and move the directory to /var/www/html  Change directory to location of your kibana (/var/www/html/kibana3), copy kibana3.conf to /etc/httpd/conf.d  kibana conf should be configured  config.js is updated as  To have all services startup at boot run chkconfig o chkconfig httpd on o chkconfig elasticsearch on o chkconfig logstash on  This completes the setup of Logstash software on the First Host. The second host is configured as an elastic search server. Elasticsearch Node (Install rpm's in the following order )  sudo yum install java-1.6.0-sun-1.6.0.32-1jpp.x86_64.rpm  sudo yum install elasticsearch-0.90.2-1.el6.x86_64.rpm logstash-1.1.13- 1.el6.noarch.rpm grok-1.20110708.1-1.el6.x86_64.rpm  sudo yum install geoip-geolite-2013.04-1.el6.noarch.rpm
  •  Backup the default Logstash file logstash.conf in /etc/logstash directory to logstash.conf.default  Create logstash.conf as for the agent  Update the elastic search yml as  To have all services startup at boot run chkconfig o chkconfig httpd on o chkconfig elasticsearch on o chkconfig logstash on  Now we have a running Logstash environment. At this time you can access the Kibana frontend  Run the curl command for template mapping from logstash server. Configuration Files for references https://drive.google.com/folderview?id=0B2jSbXbYuSe_MVotR3ZDdzlwaFE&usp=sharing Disclaimer: The install of this product and opinions are listed above are solely based on my experience in the implementation of Logstash for a Customer and is a working solution copy from that experience.. You can reference and use this document and send questions which I can answer based on my experience. This however is not an official document from Logstash team and they have not evaluated this document for it’s accuracy.