SlideShare a Scribd company logo
Logstash!
                 Get to know your logs




Dan Ivovich
BMore on Rails
4/9/13
Dan Ivovich
          SmartLogic Solutions

http://smartlogicsolutions.com
Twitter - @danivovich
What is the goal?
● Collect, Parse, and Store your log events

● Make log events searchable

● Analyze log events
Why bother?
● Got logs?
  ○ syslog
  ○ nginx access log
  ○ application logs
  ○ database logs



 Are they all formatted the
           same?
Logstash: Get to know your logs
3 Parts
● Inputs



● Filters



● Outputs
Inputs
●   Files
●   TCP/UDP
●   Redis
●   AMQP
●   rsyslog
●   xmpp

http://logstash.net/docs/1.1.9/ - Full list
Filters
●   grep
●   mutate
●   anonymize
●   date
●   grok

http://logstash.net/docs/1.1.9/ - Full list
Outputs
●   Files
●   TCP/UDP
●   Redis
●   AMQP
●   elasticsearch

http://logstash.net/docs/1.1.9/ - Full list
Getting Started
input { stdin { type => "stdin-type"} }
output { stdout { debug => true debug_format =>
"json"} }

java -jar logstash-1.1.9-monolithic.jar agent -f
logstash-simple.conf

          Type something!
See our message!
Parse something!
input { stdin { type => "stdin-type"} }
filter { grok { type => "stdin-type" pattern =>
"Hello %{DATA:message}!" } }
output { stdout { debug => true debug_format =>
"json"} }

java -jar logstash-1.1.9-monolithic.jar agent -f
logstash-simple.conf

                Say Hello!
See our message in a field!
Life is better with search
input { stdin { type => "stdin-type" } }
output {
  stdout { debug => true debug_format => "json" }
  elasticsearch { embedded => true }
}

java -jar logstash-1.1.9-monolithic.jar agent -f
logstash-search.conf

              cURL for it!
Search for the data
Well that isn't pretty




    Enter Kibana
Kibana is a friendly
interface for your logs
Kibana Connects to Elasticsearch
● Logstash parses and structures data into
    Elasticsearch
●   Kibana makes that data available
●   Apache Lucene Query Syntax (from elasticsearch)
●   Field statistics
●   Range searches




How do we put it together?
It Was Simple to Start
input { stdin { type => "stdin-type" } }
output {
  stdout { debug => true debug_format => "json" }
  elasticsearch { embedded => true }
}

java -jar logstash-1.1.9-monolithic.jar agent -f
logstash-search.conf

         But Let's Get Real
On a server with logs
Logstash/Elasticsearch
Demo
Thoughts....
● Easy to try out, but for anything real, you'll want a much
   more complicated configuration
● The variety of inputs is great
● Easy to build up a nice stack of filters
More Thoughts....
● Slow to boot monolithic jar file can be frustrating
   ○ Flatjar?
● Hard to track down why logs aren't flowing
● Elasticsearch node discovery can be difficult
   ○ If your cluster doesn't have a node added to it when
      your client starts, your client isn't connected
More Information
● logstash.net
● grokdebug.herokuapp.com
● www.elasticsearch.org
Questions?
http://smartlogicsolutions.com
http://twitter.com/smartlogic
http://github.com/smartlogic
 
http://fb.me/smartlogic

More Related Content

What's hot

Logstash + Elasticsearch + Kibana Presentation on Startit Tech Meetup
Logstash + Elasticsearch + Kibana Presentation on Startit Tech MeetupLogstash + Elasticsearch + Kibana Presentation on Startit Tech Meetup
Logstash + Elasticsearch + Kibana Presentation on Startit Tech Meetup
Startit
 
Attack monitoring using ElasticSearch Logstash and Kibana
Attack monitoring using ElasticSearch Logstash and KibanaAttack monitoring using ElasticSearch Logstash and Kibana
Attack monitoring using ElasticSearch Logstash and Kibana
Prajal Kulkarni
 
Journée DevOps : Des dashboards pour tous avec ElasticSearch, Logstash et Kibana
Journée DevOps : Des dashboards pour tous avec ElasticSearch, Logstash et KibanaJournée DevOps : Des dashboards pour tous avec ElasticSearch, Logstash et Kibana
Journée DevOps : Des dashboards pour tous avec ElasticSearch, Logstash et Kibana
Publicis Sapient Engineering
 
Monitoramento com ELK - Elasticsearch - Logstash - Kibana
Monitoramento com ELK - Elasticsearch - Logstash - KibanaMonitoramento com ELK - Elasticsearch - Logstash - Kibana
Monitoramento com ELK - Elasticsearch - Logstash - Kibana
Waldemar Neto
 
Logging logs with Logstash - Devops MK 10-02-2016
Logging logs with Logstash - Devops MK 10-02-2016Logging logs with Logstash - Devops MK 10-02-2016
Logging logs with Logstash - Devops MK 10-02-2016
Steve Howe
 
ELK stack at weibo.com
ELK stack at weibo.comELK stack at weibo.com
ELK stack at weibo.com
琛琳 饶
 
How ElasticSearch lives in my DevOps life
How ElasticSearch lives in my DevOps lifeHow ElasticSearch lives in my DevOps life
How ElasticSearch lives in my DevOps life
琛琳 饶
 
'Scalable Logging and Analytics with LogStash'
'Scalable Logging and Analytics with LogStash''Scalable Logging and Analytics with LogStash'
'Scalable Logging and Analytics with LogStash'
Cloud Elements
 
Elk scilifelab
Elk scilifelabElk scilifelab
Large Scale Log collection using LogStash & mongoDB
Large Scale Log collection using LogStash & mongoDB Large Scale Log collection using LogStash & mongoDB
Large Scale Log collection using LogStash & mongoDB
Gaurav Bhardwaj
 
Application Logging With The ELK Stack
Application Logging With The ELK StackApplication Logging With The ELK Stack
Application Logging With The ELK Stack
benwaine
 
elk_stack_alexander_szalonnas
elk_stack_alexander_szalonnaselk_stack_alexander_szalonnas
elk_stack_alexander_szalonnas
Alexander Szalonnas
 
Logstash
LogstashLogstash
Logstash
Rajgourav Jain
 
LogStash - Yes, logging can be awesome
LogStash - Yes, logging can be awesomeLogStash - Yes, logging can be awesome
LogStash - Yes, logging can be awesome
James Turnbull
 
From zero to hero - Easy log centralization with Logstash and Elasticsearch
From zero to hero - Easy log centralization with Logstash and ElasticsearchFrom zero to hero - Easy log centralization with Logstash and Elasticsearch
From zero to hero - Easy log centralization with Logstash and Elasticsearch
Rafał Kuć
 
Null Bachaav - May 07 Attack Monitoring workshop.
Null Bachaav - May 07 Attack Monitoring workshop.Null Bachaav - May 07 Attack Monitoring workshop.
Null Bachaav - May 07 Attack Monitoring workshop.
Prajal Kulkarni
 
Central LogFile Storage. ELK stack Elasticsearch, Logstash and Kibana.
Central LogFile Storage. ELK stack Elasticsearch, Logstash and Kibana.Central LogFile Storage. ELK stack Elasticsearch, Logstash and Kibana.
Central LogFile Storage. ELK stack Elasticsearch, Logstash and Kibana.
Airat Khisamov
 
Tuning Elasticsearch Indexing Pipeline for Logs
Tuning Elasticsearch Indexing Pipeline for LogsTuning Elasticsearch Indexing Pipeline for Logs
Tuning Elasticsearch Indexing Pipeline for Logs
Sematext Group, Inc.
 
Mobile Analytics mit Elasticsearch und Kibana
Mobile Analytics mit Elasticsearch und KibanaMobile Analytics mit Elasticsearch und Kibana
Mobile Analytics mit Elasticsearch und Kibana
inovex GmbH
 
ELK Stack
ELK StackELK Stack
ELK Stack
Phuc Nguyen
 

What's hot (20)

Logstash + Elasticsearch + Kibana Presentation on Startit Tech Meetup
Logstash + Elasticsearch + Kibana Presentation on Startit Tech MeetupLogstash + Elasticsearch + Kibana Presentation on Startit Tech Meetup
Logstash + Elasticsearch + Kibana Presentation on Startit Tech Meetup
 
Attack monitoring using ElasticSearch Logstash and Kibana
Attack monitoring using ElasticSearch Logstash and KibanaAttack monitoring using ElasticSearch Logstash and Kibana
Attack monitoring using ElasticSearch Logstash and Kibana
 
Journée DevOps : Des dashboards pour tous avec ElasticSearch, Logstash et Kibana
Journée DevOps : Des dashboards pour tous avec ElasticSearch, Logstash et KibanaJournée DevOps : Des dashboards pour tous avec ElasticSearch, Logstash et Kibana
Journée DevOps : Des dashboards pour tous avec ElasticSearch, Logstash et Kibana
 
Monitoramento com ELK - Elasticsearch - Logstash - Kibana
Monitoramento com ELK - Elasticsearch - Logstash - KibanaMonitoramento com ELK - Elasticsearch - Logstash - Kibana
Monitoramento com ELK - Elasticsearch - Logstash - Kibana
 
Logging logs with Logstash - Devops MK 10-02-2016
Logging logs with Logstash - Devops MK 10-02-2016Logging logs with Logstash - Devops MK 10-02-2016
Logging logs with Logstash - Devops MK 10-02-2016
 
ELK stack at weibo.com
ELK stack at weibo.comELK stack at weibo.com
ELK stack at weibo.com
 
How ElasticSearch lives in my DevOps life
How ElasticSearch lives in my DevOps lifeHow ElasticSearch lives in my DevOps life
How ElasticSearch lives in my DevOps life
 
'Scalable Logging and Analytics with LogStash'
'Scalable Logging and Analytics with LogStash''Scalable Logging and Analytics with LogStash'
'Scalable Logging and Analytics with LogStash'
 
Elk scilifelab
Elk scilifelabElk scilifelab
Elk scilifelab
 
Large Scale Log collection using LogStash & mongoDB
Large Scale Log collection using LogStash & mongoDB Large Scale Log collection using LogStash & mongoDB
Large Scale Log collection using LogStash & mongoDB
 
Application Logging With The ELK Stack
Application Logging With The ELK StackApplication Logging With The ELK Stack
Application Logging With The ELK Stack
 
elk_stack_alexander_szalonnas
elk_stack_alexander_szalonnaselk_stack_alexander_szalonnas
elk_stack_alexander_szalonnas
 
Logstash
LogstashLogstash
Logstash
 
LogStash - Yes, logging can be awesome
LogStash - Yes, logging can be awesomeLogStash - Yes, logging can be awesome
LogStash - Yes, logging can be awesome
 
From zero to hero - Easy log centralization with Logstash and Elasticsearch
From zero to hero - Easy log centralization with Logstash and ElasticsearchFrom zero to hero - Easy log centralization with Logstash and Elasticsearch
From zero to hero - Easy log centralization with Logstash and Elasticsearch
 
Null Bachaav - May 07 Attack Monitoring workshop.
Null Bachaav - May 07 Attack Monitoring workshop.Null Bachaav - May 07 Attack Monitoring workshop.
Null Bachaav - May 07 Attack Monitoring workshop.
 
Central LogFile Storage. ELK stack Elasticsearch, Logstash and Kibana.
Central LogFile Storage. ELK stack Elasticsearch, Logstash and Kibana.Central LogFile Storage. ELK stack Elasticsearch, Logstash and Kibana.
Central LogFile Storage. ELK stack Elasticsearch, Logstash and Kibana.
 
Tuning Elasticsearch Indexing Pipeline for Logs
Tuning Elasticsearch Indexing Pipeline for LogsTuning Elasticsearch Indexing Pipeline for Logs
Tuning Elasticsearch Indexing Pipeline for Logs
 
Mobile Analytics mit Elasticsearch und Kibana
Mobile Analytics mit Elasticsearch und KibanaMobile Analytics mit Elasticsearch und Kibana
Mobile Analytics mit Elasticsearch und Kibana
 
ELK Stack
ELK StackELK Stack
ELK Stack
 

Similar to Logstash: Get to know your logs

Handout: 'Open Source Tools & Resources'
Handout: 'Open Source Tools & Resources'Handout: 'Open Source Tools & Resources'
Handout: 'Open Source Tools & Resources'
BDPA Education and Technology Foundation
 
PGConf APAC 2018 - High performance json postgre-sql vs. mongodb
PGConf APAC 2018 - High performance json  postgre-sql vs. mongodbPGConf APAC 2018 - High performance json  postgre-sql vs. mongodb
PGConf APAC 2018 - High performance json postgre-sql vs. mongodb
PGConf APAC
 
Turbo charge your logs
Turbo charge your logsTurbo charge your logs
Turbo charge your logs
Jeremy Cook
 
(Fios#02) 2. elk 포렌식 분석
(Fios#02) 2. elk 포렌식 분석(Fios#02) 2. elk 포렌식 분석
(Fios#02) 2. elk 포렌식 분석
INSIGHT FORENSIC
 
Journey through high performance django application
Journey through high performance django applicationJourney through high performance django application
Journey through high performance django application
bangaloredjangousergroup
 
High performance json- postgre sql vs. mongodb
High performance json- postgre sql vs. mongodbHigh performance json- postgre sql vs. mongodb
High performance json- postgre sql vs. mongodb
Wei Shan Ang
 
Dart the better Javascript 2015
Dart the better Javascript 2015Dart the better Javascript 2015
Dart the better Javascript 2015
Jorg Janke
 
Experiences building a distributed shared log on RADOS - Noah Watkins
Experiences building a distributed shared log on RADOS - Noah WatkinsExperiences building a distributed shared log on RADOS - Noah Watkins
Experiences building a distributed shared log on RADOS - Noah Watkins
Ceph Community
 
.NET @ apache.org
 .NET @ apache.org .NET @ apache.org
.NET @ apache.org
Ted Husted
 
Eko10 - Security Monitoring for Big Infrastructures without a Million Dollar ...
Eko10 - Security Monitoring for Big Infrastructures without a Million Dollar ...Eko10 - Security Monitoring for Big Infrastructures without a Million Dollar ...
Eko10 - Security Monitoring for Big Infrastructures without a Million Dollar ...
Hernan Costante
 
Application Logging in the 21st century - 2014.key
Application Logging in the 21st century - 2014.keyApplication Logging in the 21st century - 2014.key
Application Logging in the 21st century - 2014.key
Tim Bunce
 
Building a Unified Logging Layer with Fluentd, Elasticsearch and Kibana
Building a Unified Logging Layer with Fluentd, Elasticsearch and KibanaBuilding a Unified Logging Layer with Fluentd, Elasticsearch and Kibana
Building a Unified Logging Layer with Fluentd, Elasticsearch and Kibana
Mushfekur Rahman
 
Security Monitoring for big Infrastructures without a Million Dollar budget
Security Monitoring for big Infrastructures without a Million Dollar budgetSecurity Monitoring for big Infrastructures without a Million Dollar budget
Security Monitoring for big Infrastructures without a Million Dollar budget
Juan Berner
 
Andriy Shalaenko - GO security tips
Andriy Shalaenko - GO security tipsAndriy Shalaenko - GO security tips
Andriy Shalaenko - GO security tips
OWASP Kyiv
 
Finding OOMS in Legacy Systems with the Syslog Telegraf Plugin
Finding OOMS in Legacy Systems with the Syslog Telegraf PluginFinding OOMS in Legacy Systems with the Syslog Telegraf Plugin
Finding OOMS in Legacy Systems with the Syslog Telegraf Plugin
InfluxData
 
Dart the Better JavaScript
Dart the Better JavaScriptDart the Better JavaScript
Dart the Better JavaScript
Jorg Janke
 
Query and audit logging in cassandra
Query and audit logging in cassandraQuery and audit logging in cassandra
Query and audit logging in cassandra
Vinay Kumar Chella
 
Test strategies for data processing pipelines, v2.0
Test strategies for data processing pipelines, v2.0Test strategies for data processing pipelines, v2.0
Test strategies for data processing pipelines, v2.0
Lars Albertsson
 
Fluentd vs. Logstash for OpenStack Log Management
Fluentd vs. Logstash for OpenStack Log ManagementFluentd vs. Logstash for OpenStack Log Management
Fluentd vs. Logstash for OpenStack Log Management
NTT Communications Technology Development
 
Testing Django APIs
Testing Django APIsTesting Django APIs
Testing Django APIs
tyomo4ka
 

Similar to Logstash: Get to know your logs (20)

Handout: 'Open Source Tools & Resources'
Handout: 'Open Source Tools & Resources'Handout: 'Open Source Tools & Resources'
Handout: 'Open Source Tools & Resources'
 
PGConf APAC 2018 - High performance json postgre-sql vs. mongodb
PGConf APAC 2018 - High performance json  postgre-sql vs. mongodbPGConf APAC 2018 - High performance json  postgre-sql vs. mongodb
PGConf APAC 2018 - High performance json postgre-sql vs. mongodb
 
Turbo charge your logs
Turbo charge your logsTurbo charge your logs
Turbo charge your logs
 
(Fios#02) 2. elk 포렌식 분석
(Fios#02) 2. elk 포렌식 분석(Fios#02) 2. elk 포렌식 분석
(Fios#02) 2. elk 포렌식 분석
 
Journey through high performance django application
Journey through high performance django applicationJourney through high performance django application
Journey through high performance django application
 
High performance json- postgre sql vs. mongodb
High performance json- postgre sql vs. mongodbHigh performance json- postgre sql vs. mongodb
High performance json- postgre sql vs. mongodb
 
Dart the better Javascript 2015
Dart the better Javascript 2015Dart the better Javascript 2015
Dart the better Javascript 2015
 
Experiences building a distributed shared log on RADOS - Noah Watkins
Experiences building a distributed shared log on RADOS - Noah WatkinsExperiences building a distributed shared log on RADOS - Noah Watkins
Experiences building a distributed shared log on RADOS - Noah Watkins
 
.NET @ apache.org
 .NET @ apache.org .NET @ apache.org
.NET @ apache.org
 
Eko10 - Security Monitoring for Big Infrastructures without a Million Dollar ...
Eko10 - Security Monitoring for Big Infrastructures without a Million Dollar ...Eko10 - Security Monitoring for Big Infrastructures without a Million Dollar ...
Eko10 - Security Monitoring for Big Infrastructures without a Million Dollar ...
 
Application Logging in the 21st century - 2014.key
Application Logging in the 21st century - 2014.keyApplication Logging in the 21st century - 2014.key
Application Logging in the 21st century - 2014.key
 
Building a Unified Logging Layer with Fluentd, Elasticsearch and Kibana
Building a Unified Logging Layer with Fluentd, Elasticsearch and KibanaBuilding a Unified Logging Layer with Fluentd, Elasticsearch and Kibana
Building a Unified Logging Layer with Fluentd, Elasticsearch and Kibana
 
Security Monitoring for big Infrastructures without a Million Dollar budget
Security Monitoring for big Infrastructures without a Million Dollar budgetSecurity Monitoring for big Infrastructures without a Million Dollar budget
Security Monitoring for big Infrastructures without a Million Dollar budget
 
Andriy Shalaenko - GO security tips
Andriy Shalaenko - GO security tipsAndriy Shalaenko - GO security tips
Andriy Shalaenko - GO security tips
 
Finding OOMS in Legacy Systems with the Syslog Telegraf Plugin
Finding OOMS in Legacy Systems with the Syslog Telegraf PluginFinding OOMS in Legacy Systems with the Syslog Telegraf Plugin
Finding OOMS in Legacy Systems with the Syslog Telegraf Plugin
 
Dart the Better JavaScript
Dart the Better JavaScriptDart the Better JavaScript
Dart the Better JavaScript
 
Query and audit logging in cassandra
Query and audit logging in cassandraQuery and audit logging in cassandra
Query and audit logging in cassandra
 
Test strategies for data processing pipelines, v2.0
Test strategies for data processing pipelines, v2.0Test strategies for data processing pipelines, v2.0
Test strategies for data processing pipelines, v2.0
 
Fluentd vs. Logstash for OpenStack Log Management
Fluentd vs. Logstash for OpenStack Log ManagementFluentd vs. Logstash for OpenStack Log Management
Fluentd vs. Logstash for OpenStack Log Management
 
Testing Django APIs
Testing Django APIsTesting Django APIs
Testing Django APIs
 

More from SmartLogic

Writing Game Servers with Elixir
Writing Game Servers with ElixirWriting Game Servers with Elixir
Writing Game Servers with Elixir
SmartLogic
 
All Aboard The Stateful Train
All Aboard The Stateful TrainAll Aboard The Stateful Train
All Aboard The Stateful Train
SmartLogic
 
DC |> Elixir Meetup - Going off the Rails into Elixir - Dan Ivovich
DC |> Elixir Meetup - Going off the Rails into Elixir - Dan IvovichDC |> Elixir Meetup - Going off the Rails into Elixir - Dan Ivovich
DC |> Elixir Meetup - Going off the Rails into Elixir - Dan Ivovich
SmartLogic
 
Monitoring Your Elixir Application with Prometheus
Monitoring Your Elixir Application with PrometheusMonitoring Your Elixir Application with Prometheus
Monitoring Your Elixir Application with Prometheus
SmartLogic
 
Going Multi-Node
Going Multi-NodeGoing Multi-Node
Going Multi-Node
SmartLogic
 
Kubernetes and docker
Kubernetes and dockerKubernetes and docker
Kubernetes and docker
SmartLogic
 
Serializing Value Objects-Ara Hacopian
Serializing Value Objects-Ara HacopianSerializing Value Objects-Ara Hacopian
Serializing Value Objects-Ara Hacopian
SmartLogic
 
Guide to food foraging by SmartLogic's Kei Ellerbrock
Guide to food foraging by SmartLogic's Kei EllerbrockGuide to food foraging by SmartLogic's Kei Ellerbrock
Guide to food foraging by SmartLogic's Kei Ellerbrock
SmartLogic
 
Introduction to Type Script by Sam Goldman, SmartLogic
Introduction to Type Script by Sam Goldman, SmartLogicIntroduction to Type Script by Sam Goldman, SmartLogic
Introduction to Type Script by Sam Goldman, SmartLogic
SmartLogic
 
How SmartLogic Uses Chef-Dan Ivovich
How SmartLogic Uses Chef-Dan IvovichHow SmartLogic Uses Chef-Dan Ivovich
How SmartLogic Uses Chef-Dan Ivovich
SmartLogic
 
A Few Interesting Things in Apple's Swift Programming Language
A Few Interesting Things in Apple's Swift Programming LanguageA Few Interesting Things in Apple's Swift Programming Language
A Few Interesting Things in Apple's Swift Programming Language
SmartLogic
 
Effective ActiveRecord
Effective ActiveRecordEffective ActiveRecord
Effective ActiveRecord
SmartLogic
 
An Introduction to Reactive Cocoa
An Introduction to Reactive CocoaAn Introduction to Reactive Cocoa
An Introduction to Reactive Cocoa
SmartLogic
 
iOS Development Methodology
iOS Development MethodologyiOS Development Methodology
iOS Development Methodology
SmartLogic
 
CSS Preprocessors to the Rescue!
CSS Preprocessors to the Rescue!CSS Preprocessors to the Rescue!
CSS Preprocessors to the Rescue!
SmartLogic
 
Deploying Rails Apps with Chef and Capistrano
 Deploying Rails Apps with Chef and Capistrano Deploying Rails Apps with Chef and Capistrano
Deploying Rails Apps with Chef and Capistrano
SmartLogic
 
From Slacker to Hacker, Practical Tips for Learning to Code
From Slacker to Hacker, Practical Tips for Learning to CodeFrom Slacker to Hacker, Practical Tips for Learning to Code
From Slacker to Hacker, Practical Tips for Learning to Code
SmartLogic
 
The Language of Abstraction in Software Development
The Language of Abstraction in Software DevelopmentThe Language of Abstraction in Software Development
The Language of Abstraction in Software Development
SmartLogic
 
Android Testing: An Overview
Android Testing: An OverviewAndroid Testing: An Overview
Android Testing: An Overview
SmartLogic
 
Intro to DTCoreText: Moving Past UIWebView | iOS Development
Intro to DTCoreText: Moving Past UIWebView | iOS DevelopmentIntro to DTCoreText: Moving Past UIWebView | iOS Development
Intro to DTCoreText: Moving Past UIWebView | iOS Development
SmartLogic
 

More from SmartLogic (20)

Writing Game Servers with Elixir
Writing Game Servers with ElixirWriting Game Servers with Elixir
Writing Game Servers with Elixir
 
All Aboard The Stateful Train
All Aboard The Stateful TrainAll Aboard The Stateful Train
All Aboard The Stateful Train
 
DC |> Elixir Meetup - Going off the Rails into Elixir - Dan Ivovich
DC |> Elixir Meetup - Going off the Rails into Elixir - Dan IvovichDC |> Elixir Meetup - Going off the Rails into Elixir - Dan Ivovich
DC |> Elixir Meetup - Going off the Rails into Elixir - Dan Ivovich
 
Monitoring Your Elixir Application with Prometheus
Monitoring Your Elixir Application with PrometheusMonitoring Your Elixir Application with Prometheus
Monitoring Your Elixir Application with Prometheus
 
Going Multi-Node
Going Multi-NodeGoing Multi-Node
Going Multi-Node
 
Kubernetes and docker
Kubernetes and dockerKubernetes and docker
Kubernetes and docker
 
Serializing Value Objects-Ara Hacopian
Serializing Value Objects-Ara HacopianSerializing Value Objects-Ara Hacopian
Serializing Value Objects-Ara Hacopian
 
Guide to food foraging by SmartLogic's Kei Ellerbrock
Guide to food foraging by SmartLogic's Kei EllerbrockGuide to food foraging by SmartLogic's Kei Ellerbrock
Guide to food foraging by SmartLogic's Kei Ellerbrock
 
Introduction to Type Script by Sam Goldman, SmartLogic
Introduction to Type Script by Sam Goldman, SmartLogicIntroduction to Type Script by Sam Goldman, SmartLogic
Introduction to Type Script by Sam Goldman, SmartLogic
 
How SmartLogic Uses Chef-Dan Ivovich
How SmartLogic Uses Chef-Dan IvovichHow SmartLogic Uses Chef-Dan Ivovich
How SmartLogic Uses Chef-Dan Ivovich
 
A Few Interesting Things in Apple's Swift Programming Language
A Few Interesting Things in Apple's Swift Programming LanguageA Few Interesting Things in Apple's Swift Programming Language
A Few Interesting Things in Apple's Swift Programming Language
 
Effective ActiveRecord
Effective ActiveRecordEffective ActiveRecord
Effective ActiveRecord
 
An Introduction to Reactive Cocoa
An Introduction to Reactive CocoaAn Introduction to Reactive Cocoa
An Introduction to Reactive Cocoa
 
iOS Development Methodology
iOS Development MethodologyiOS Development Methodology
iOS Development Methodology
 
CSS Preprocessors to the Rescue!
CSS Preprocessors to the Rescue!CSS Preprocessors to the Rescue!
CSS Preprocessors to the Rescue!
 
Deploying Rails Apps with Chef and Capistrano
 Deploying Rails Apps with Chef and Capistrano Deploying Rails Apps with Chef and Capistrano
Deploying Rails Apps with Chef and Capistrano
 
From Slacker to Hacker, Practical Tips for Learning to Code
From Slacker to Hacker, Practical Tips for Learning to CodeFrom Slacker to Hacker, Practical Tips for Learning to Code
From Slacker to Hacker, Practical Tips for Learning to Code
 
The Language of Abstraction in Software Development
The Language of Abstraction in Software DevelopmentThe Language of Abstraction in Software Development
The Language of Abstraction in Software Development
 
Android Testing: An Overview
Android Testing: An OverviewAndroid Testing: An Overview
Android Testing: An Overview
 
Intro to DTCoreText: Moving Past UIWebView | iOS Development
Intro to DTCoreText: Moving Past UIWebView | iOS DevelopmentIntro to DTCoreText: Moving Past UIWebView | iOS Development
Intro to DTCoreText: Moving Past UIWebView | iOS Development
 

Recently uploaded

Three New Criminal Laws in India 1 July 2024
Three New Criminal Laws in India 1 July 2024Three New Criminal Laws in India 1 July 2024
Three New Criminal Laws in India 1 July 2024
aakash malhotra
 
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
maigasapphire
 
Vulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive OverviewVulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive Overview
Steven Carlson
 
How Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdfHow Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdf
HackersList
 
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
aslasdfmkhan4750
 
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptxRPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
SynapseIndia
 
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
Kief Morris
 
Opencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of MünsterOpencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of Münster
Matthias Neugebauer
 
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyyActive Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
RaminGhanbari2
 
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python CodebaseEuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
Jimmy Lai
 
July Patch Tuesday
July Patch TuesdayJuly Patch Tuesday
July Patch Tuesday
Ivanti
 
Pigging Unit Lubricant Oil Blending Plant
Pigging Unit Lubricant Oil Blending PlantPigging Unit Lubricant Oil Blending Plant
Pigging Unit Lubricant Oil Blending Plant
LINUS PROJECTS (INDIA)
 
Google I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged SlidesGoogle I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged Slides
Google Developer Group - Harare
 
Amul milk launches in US: Key details of its new products ...
Amul milk launches in US: Key details of its new products ...Amul milk launches in US: Key details of its new products ...
Amul milk launches in US: Key details of its new products ...
chetankumar9855
 
Best Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdfBest Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdf
Tatiana Al-Chueyr
 
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
sunilverma7884
 
Feature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptxFeature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptx
ssuser1915fe1
 
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-InTrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc
 
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSECHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
kumarjarun2010
 
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Muhammad Ali
 

Recently uploaded (20)

Three New Criminal Laws in India 1 July 2024
Three New Criminal Laws in India 1 July 2024Three New Criminal Laws in India 1 July 2024
Three New Criminal Laws in India 1 July 2024
 
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
 
Vulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive OverviewVulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive Overview
 
How Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdfHow Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdf
 
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
 
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptxRPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
 
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
 
Opencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of MünsterOpencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of Münster
 
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyyActive Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
 
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python CodebaseEuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
 
July Patch Tuesday
July Patch TuesdayJuly Patch Tuesday
July Patch Tuesday
 
Pigging Unit Lubricant Oil Blending Plant
Pigging Unit Lubricant Oil Blending PlantPigging Unit Lubricant Oil Blending Plant
Pigging Unit Lubricant Oil Blending Plant
 
Google I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged SlidesGoogle I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged Slides
 
Amul milk launches in US: Key details of its new products ...
Amul milk launches in US: Key details of its new products ...Amul milk launches in US: Key details of its new products ...
Amul milk launches in US: Key details of its new products ...
 
Best Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdfBest Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdf
 
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
 
Feature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptxFeature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptx
 
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-InTrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
 
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSECHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
 
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
 

Logstash: Get to know your logs

  • 1. Logstash! Get to know your logs Dan Ivovich BMore on Rails 4/9/13
  • 2. Dan Ivovich SmartLogic Solutions http://smartlogicsolutions.com Twitter - @danivovich
  • 3. What is the goal? ● Collect, Parse, and Store your log events ● Make log events searchable ● Analyze log events
  • 4. Why bother? ● Got logs? ○ syslog ○ nginx access log ○ application logs ○ database logs Are they all formatted the same?
  • 6. 3 Parts ● Inputs ● Filters ● Outputs
  • 7. Inputs ● Files ● TCP/UDP ● Redis ● AMQP ● rsyslog ● xmpp http://logstash.net/docs/1.1.9/ - Full list
  • 8. Filters ● grep ● mutate ● anonymize ● date ● grok http://logstash.net/docs/1.1.9/ - Full list
  • 9. Outputs ● Files ● TCP/UDP ● Redis ● AMQP ● elasticsearch http://logstash.net/docs/1.1.9/ - Full list
  • 10. Getting Started input { stdin { type => "stdin-type"} } output { stdout { debug => true debug_format => "json"} } java -jar logstash-1.1.9-monolithic.jar agent -f logstash-simple.conf Type something!
  • 12. Parse something! input { stdin { type => "stdin-type"} } filter { grok { type => "stdin-type" pattern => "Hello %{DATA:message}!" } } output { stdout { debug => true debug_format => "json"} } java -jar logstash-1.1.9-monolithic.jar agent -f logstash-simple.conf Say Hello!
  • 13. See our message in a field!
  • 14. Life is better with search input { stdin { type => "stdin-type" } } output { stdout { debug => true debug_format => "json" } elasticsearch { embedded => true } } java -jar logstash-1.1.9-monolithic.jar agent -f logstash-search.conf cURL for it!
  • 16. Well that isn't pretty Enter Kibana
  • 17. Kibana is a friendly interface for your logs
  • 18. Kibana Connects to Elasticsearch ● Logstash parses and structures data into Elasticsearch ● Kibana makes that data available ● Apache Lucene Query Syntax (from elasticsearch) ● Field statistics ● Range searches How do we put it together?
  • 19. It Was Simple to Start input { stdin { type => "stdin-type" } } output { stdout { debug => true debug_format => "json" } elasticsearch { embedded => true } } java -jar logstash-1.1.9-monolithic.jar agent -f logstash-search.conf But Let's Get Real
  • 20. On a server with logs
  • 22. Demo
  • 23. Thoughts.... ● Easy to try out, but for anything real, you'll want a much more complicated configuration ● The variety of inputs is great ● Easy to build up a nice stack of filters
  • 24. More Thoughts.... ● Slow to boot monolithic jar file can be frustrating ○ Flatjar? ● Hard to track down why logs aren't flowing ● Elasticsearch node discovery can be difficult ○ If your cluster doesn't have a node added to it when your client starts, your client isn't connected
  • 25. More Information ● logstash.net ● grokdebug.herokuapp.com ● www.elasticsearch.org