LogStash in action

Data ingestion workhorse
Events enrichment and transformation
Extensible plugin ecosystem
Pluggable pipeline architecture
Horizontally scalable data processing pipeline
Strong Elasticsearch and Kibana synergy
Handles data of all shapes and sizes
Key Features of LogStash
What’s In It for You?

Installation and Configuration

The Pre-requisites
Prerequisites
requires Java 7 or higher
Installation steps
Download from elastic.co web site
Use Linux package manager to install
LogStash
Install LogStash as a service

The Service Commands

Video Demonstration
Installing LogStash

The Service Commands
Start or stop service commands :
sudo /etc/init.d/logstash start
sudo /etc/init.d/logstash stop
sudo /etc/init.d/logstash restart
sudo /etc/init.d/logstash status

Simple Pipeline
Verify LogStash installation
with a simple pipeline
Will take input from command
line and output it back to the
command line
Pipeline configuration
information is passed as text on
command line
Takes input from standard
input “stdin”
Outputs to standard output
“stdout” in a structured format

bin/logstash -e 'input { stdin { } } output { stdout {} }'
Simple Pipeline

4
5 # Simple LogStash configuration
6
7 Inputs {
8 Stdin { }
9 }
10
11 Output {
12 Stdout { }
13 }
14
Simple Pipeline

Video Demonstration
Configuring a Simple Pipeline

Advanced Pipeline
Real world pipelines contain one or
more input, filter and outputs
Is generally provided in a
configuration file rather than command
line
Supplied to LogStash with –f
command line argument
Test the configuration using --
configtest argument

Skeleton LogStash Configuration
4 # The # character at the beginning of
5 # a line indicates comment
6 # Use the comments to describe your configuration
7 input {
8 input1 { }
9 input2 { }
10 }
11
12 # The filter part of this file is
13 # commented out to indicate that
14 # it is optional.
15
16 # filter {
17 # }
18
19 output {
20 output1 { }
21 output2 { }
22 }

LogStash Plugins
LogStash Instance
Data Source ElasticSearch
Filter
Plugin
Output
Plugin
Input
Plugin

Input
Plugin
elasticsearch
file
imap
jdbc
stdin
s3
syslog
tcp
twitter
udp

Filter
Plugin
csv
date
drop
grok
mutate
range
sleep
translate

Output
Plugin
csv
elasticsearch
email
file
mongodb
stdout
s3
syslog
tcp
udp

4 # ElasticSearch input plugin
5
6 Input {
7
8 # Read all documents from ElasticSearch
9 # matching the given query
10
11 elasticsearch {
12 hosts => “localhost”
13 index => “blogs”
14 query => ‘{ “localhost” : { “match_all” : { } } }’
15 type => “my-data-elasticsearch”
16 }
17 }
18
Input-ElasticSearch

3
4 # File input
5
6 Inputs {
7
8 # Read events from file or folder
9
10 file {
11 path => “/var/log/ * ”
12 exclude => “ * .gz ”
13 sincedb_path => “ /dev/null ”
14 start_position => “ beginning “
15 type => “ my-data-csv”
16 }
17 }
18
Input-File

4 # JDBC input
5
6 Input {
7
8 # Read all records from mySQL
9 # database
10
11 jdbc {
12
13 jdbc_driver_library => “/opt/logstash/lib/mysql-connector-java-5.1.6-bin.jar “
14 jdbc_driver_class => “com.mysql.jdbc.Driver ”
15 jdbc_connection_string => “jdbc : mysql : // localhost : 3306 / mydb ”
16 jdbc_user => “root ”
17 jdbc_password => “password ”
18 statement => “SELECT * from users “
19
20 }
21 }
22
Input-jbdc

3 # AWS S3 input
4
5 Input {
6
7 # Read all documents from AWS S3
8
9 s3 {
10
11 bucket => “my-bucket “
12 credentials => [ “my-aws-key “ , “my-aws-token “ ]
13 region_endpoint => “us-east-1 “
14 codec => “json “
15
16 }
17 }
18
Input-s3

4 # TCP input
5
6 Input {
7
8 # Read all events over TCP socket
9
10 tcp {
11 port => “ 5000 “
12 type => “ syslog “
13 }
14
15 }
Input-tcp

4 # UDP input
5
6 input {
7
8 # Read all events over UDP port
9
10 udp {
11 port => “5001”
12 type => “netflow”
13 }
14
15 }
16
Input-udp

Filter-csv
2
3 # CSV filter
4 filter {
5
6 csv {
7
8 # List of columns as they appear in csv
9 column => [ “column_1” , “column_2” ]
10 column => { “column_3” => “integer” , “column_4” => “boolean” }
11 type => “syslog”
12
13 }
14
15 }
16

2
3 # date filter
4
5 filter {
6
7
8 date {
9
10 match => [ “logdate” , “MMM dd HH:mm:ss” ]
11 # Default for target is @timestamp
12 target => “logdate_modified”
13
14 }
15 }
Filter-date
• Used for parsing dates and use as LogStash event timestamp in ISO8601 format
• For example “Jan 01 10:40:01” can be parsed using the pattern “MMM dd HH:mm:ss”

Filter-drop
2
3 # drop filter
4
5 filter {
6
7 # drop the events of their loglevel is debug
8 drop {
9 if [ loglevel ] = = “debug” {
10 drop { }
11 }
12 }
13 }
14
15

Filter-range
2
3 # range filter
4
5 filter {
6 range {
7 ranges => [“request_time” , 0, 10, “tag: short” ,
8 “request_time” , 11, 100, “tag: medium”,
9 “request_time” , 101, 1000, “tag: long”,
10 “request_time” , 1001, 100000, “drop”,
11 “request_length” , 0, 100, “field: size: small”,
12 “request_length” , 101, 200, “field: size: normal”,
13 “request_length” , 201, 1000, “field: size: big”,
14 “request_length” , 1001, 100000, “field: size: hugel”,
15 “number_of_requests” , 0, 10, “tag: request_from_%{host}” ]
16 }
17 }

Filter-grok
Grok is one of the most
widely used plugin
It is instrumental in parsing
arbitrary and unstructured text
into structed and queryable
data field
It is widely used to parse
syslog, apache logs, mySQL
logs, custom application logs,
postfix logs etc.
Grok works based on
patterns
Syntax for grok pattern is
%{SYNTAX:SEMANTIC}
Custom patterns can be
added

Filter-grok
2
3 # grok filter
4
5 input {
6 file {
7 path => “/ var/log/http.log”
8
9 # sample log entry
10 # 55.11.55.11 GET/ index.html 453 12
11 }
12 }
13
14 filter {
15 # parse http log
16
17 grok {
18
19 match => { “message” => “% { IP: client} %{WORD: method} %{URIPATHPARAM: request} %{NUMBER: duration}” }
20
21 }
22 }
23

Filter-grok
2
3 # grok filter
4
5 input {
6 file {
7 path => “/ var/log/http.log”
8
9 # sample log entry
10 # 55.11.55.11 GET/ index.html 453 12
11 }
12 }
13
14 filter {
15 # parse http log
16
17 grok {
18
19 match => { “message” => “% { IP: client} %{WORD: method} %{URIPATHPARAM: request} %{NUMBER: duration}” }
20
21 }
22 }
23
Grok supports custom patterns
• inline custom pattern using Oniguruma syntax
• file based custom patterns

Filter-grok
43 # ( ? < field_name> the pattern here
44
45
46 ( ? < message_id> [0-9A-F] {10, 11}

24
25 # grok filter
26
27 filter {
28
29 grok {
30
31 patterns_dir => [ “~/patterns” ]
32 match => { “message” => “% {SYSLOGBASE} %{POSTFIXQUEUEID: queue_id}: %{GREEDYDATA:syslog_message}” }
33
34 }
35 }
Filter-grok

Filter-mutate

Filter-sleep

Filter-translate

Output-csv

Output-file

Output-stdout

Output-elasticsearch

Output-email

Output-s3

Output-tcp

LogStash in action

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to LogStash in action

Similar to LogStash in action (20)

More from Manuj Aggarwal

More from Manuj Aggarwal (7)

Recently uploaded

Recently uploaded (20)

LogStash in action