0
The ELK Stack @ Linko
Jilles van Gurp - Linko Inc.
Who is Jilles?
@jillesvangurp, www.jillesvangurp.com, and jillesvangurp on Github & just
about everything else.
Java (J)Ru...
Logging
Stuff runs
Produces errors, warnings, debug, telemetry,
analytics events, and other information
How to make sense ...
Old school: Cat, grep, awk, cut, ….
Good luck with that on 200GB of unstructured
logs. Think lots of coffee breaks.
The fi...
Or do the same stuff in Hadoop
Works great for structured data if you know
what you are looking for.
Requires a lot of inf...
ELK Stack?
Elasticsearch
Logstash
Kibana
ELK - Elasticsearch
Sharded, replicated, searchable, json document store.
Used by many big name services out there - Githu...
Scaling Elasticsearch
1 node, 16GB, all of open streetmap in
geojson format (+ some other stuff) ->
reverse geocode in <10...
Bottom line
Elasticsearch scales, probably way beyond
your needs
Log data is actually easy for elasticsearch
Elk - Logstash
Plumbing for your logs
Many different inputs for your logs
Filtering/parsing for your logs
Many outputs for...
ELK - Kibana
Highly configurable dashboard to slice and
dice your logstash logs in elasticsearch.
Real-time dashboards, ea...
ELK at Linko
Java Logback
NGINX
collectd
APP Servers
Linko Logstash - App Server (1)
input {
file {
type => "nginx_access"
path => ["/var/log/nginx/*.log"]
exclude => ["*.gz”,...
Grok pattern for NGINX
NGINXACCESSWITHUPSTR %{IPORHOST:remote_addr} - %{USERNAME:remote_user}
[%{HTTPDATE:time_local}] "%{...
Linko Logstash - App Server (2)
input {
file {
type => "backbone"
path => "/var/log/linko-
backbone/logstash/*.log"
codec ...
Linko Logstash - Elasticsearch
input {
redis {
host => "192.168.1.13"
# these settings should match the output
of the agen...
Experience - mostly good
Many moving parts - each with their odd
problems and issues
All parts are evolving. Prepare to up...
Finding out the hard way ...
Rolling restarts with elasticsearch
Configuring caching because of OOM’s
Clicking together da...
Gotchas
Kibana needs to talk to ES, but you don’t want
that exposed to the world.
ES Fielddata cache is unrestricted, by d...
Getting started
Download es & logstash to your laptop.
Simply run ES as is; worry about config later
Follow logstash cookb...
After getting started
RTFM, play, explore, mess up, google, …
Configure ES properly
Setup nginx/apache to proxy
Think abou...
Links
http://www.elasticsearch.org/
http://linko.io
https://groups.google.com/forum/?fromgroups
=#!forum/elasticsearch
htt...
Thanks!
@jillesvangurp, @linkoapp
Elk stack
Elk stack
Elk stack
Elk stack
Elk stack
Upcoming SlideShare
Loading in...5
×

Elk stack

8,199

Published on

Elasticsearch, Logstash, and Kibana usage at Linko. Presented at geek2geek meetup in May 2014.

Published in: Software, Technology
3 Comments
15 Likes
Statistics
Notes
  • @hua chu, I presented this a year ago when there still was. These days the functionality has been folded into the regular elasticsearch plugin.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • p. 25 why use elasticsearch_http instead of elasticsearch output? There is no elasticsearch_http output in logstash 1.5. See: https://www.elastic.co/guide/en/logstash/current/output-plugins.html#plugins-outputs-letters-e
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Sorry for poor layout. It seems the Google drive import in slideshare just sucks big time. Original: https://docs.google.com/presentation/d/1yAEhT_T6mJesC5sRoiEVWFNcJ7rZvk1W9AmK6bC3h_k/pub?start=false&loop=false&delayms=3000
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
8,199
On Slideshare
0
From Embeds
0
Number of Embeds
15
Actions
Shares
0
Downloads
119
Comments
3
Likes
15
Embeds 0
No embeds

No notes for slide

Transcript of "Elk stack"

  1. 1. The ELK Stack @ Linko Jilles van Gurp - Linko Inc.
  2. 2. Who is Jilles? @jillesvangurp, www.jillesvangurp.com, and jillesvangurp on Github & just about everything else. Java (J)Ruby Python Javascript GEO Server stuffreluctant Devops guy Software Architecture Universities of Utrecht (NL), Blekinge (SE), and Groningen (NL) GX Creative Online Development (NL) Nokia Research (FI), Nokia/Here (DE) Localstream (DE), Linko (DE).
  3. 3. Logging Stuff runs Produces errors, warnings, debug, telemetry, analytics events, and other information How to make sense of it?
  4. 4. Old school: Cat, grep, awk, cut, …. Good luck with that on 200GB of unstructured logs. Think lots of coffee breaks. The fix: ELK
  5. 5. Or do the same stuff in Hadoop Works great for structured data if you know what you are looking for. Requires a lot of infrastructure and hassle. Not real-time, hard to explore data I’m not a data scientist, are you? The fix: ELK
  6. 6. ELK Stack? Elasticsearch Logstash Kibana
  7. 7. ELK - Elasticsearch Sharded, replicated, searchable, json document store. Used by many big name services out there - Github, Soundcloud, Foursquare, Xing, many others. Full text search, geo spatial search, advanced search ranking, suggestions, … much more. It’s awesome. Nice HTTP API
  8. 8. Scaling Elasticsearch 1 node, 16GB, all of open streetmap in geojson format (+ some other stuff) -> reverse geocode in <100ms There are people running ES with thousands of nodes, trillions of documents, and petabytes ...
  9. 9. Bottom line Elasticsearch scales, probably way beyond your needs Log data is actually easy for elasticsearch
  10. 10. Elk - Logstash Plumbing for your logs Many different inputs for your logs Filtering/parsing for your logs Many outputs for your logs: for example redis, elasticsearch, file,
  11. 11. ELK - Kibana Highly configurable dashboard to slice and dice your logstash logs in elasticsearch. Real-time dashboards, easily configurable
  12. 12. ELK at Linko Java Logback NGINX collectd APP Servers
  13. 13. Linko Logstash - App Server (1) input { file { type => "nginx_access" path => ["/var/log/nginx/*.log"] exclude => ["*.gz”, “error.*"] discover_interval => 10 sincedb_path => "/opt/logstash/sincedb- access-nginx" } } filter { grok { type => "nginx_access" patterns_dir => "/opt/logstash/patterns" pattern => ["%{NGINXACCESSWITHUPSTR}","%{NGINXACCESS}"] } date { type => "nginx_access" locale => "en" match => [ "time_local" , "dd/MMM/YYYY:HH:mm:ss Z" ] } }
  14. 14. Grok pattern for NGINX NGINXACCESSWITHUPSTR %{IPORHOST:remote_addr} - %{USERNAME:remote_user} [%{HTTPDATE:time_local}] "%{WORD:method} %{URIPATHPARAM:request} %{GREEDYDATA:protocol}" %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent} %{QS:backend} %{BASE16FLOAT:duration} NGINXACCESS %{IPORHOST:remote_addr} - %{USERNAME:remote_user} [%{HTTPDATE:time_local}] %{QS:request} %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent}
  15. 15. Linko Logstash - App Server (2) input { file { type => "backbone" path => "/var/log/linko- backbone/logstash/*.log" codec => "json" discover_interval => 10 sincedb_path => "/opt/logstash/sincedb- access-backbone" } } input { collectd { type => 'collectd' } } output { redis { host => "192.168.1.13" data_type => "list" key => "logstash" } }
  16. 16. Linko Logstash - Elasticsearch input { redis { host => "192.168.1.13" # these settings should match the output of the agent data_type => "list" key => "logstash" # We use the 'json' codec here because we expect to read # json events from redis. codec => json } } output { elasticsearch_http { host => "192.168.1.13" manage_template => true template_overwrite => true template => "/opt/logstash/index_template.json" } }
  17. 17. Experience - mostly good Many moving parts - each with their odd problems and issues All parts are evolving. Prepare to upgrade. Documentation is not great.
  18. 18. Finding out the hard way ... Rolling restarts with elasticsearch Configuring caching because of OOM’s Clicking together dashboards in Kibana Don’t restart cluster nodes blindly Beware: Split brain Default ES config is not appropriate for production
  19. 19. Gotchas Kibana needs to talk to ES, but you don’t want that exposed to the world. ES Fielddata cache is unrestricted, by default Elasticsearch_http can fail silently, if misconfigured. If you use file input, be sure to set the sincedb
  20. 20. Getting started Download es & logstash to your laptop. Simply run ES as is; worry about config later Follow logstash cookbook to get started Setup some simple inputs Use elasticsearch_http, not elasticsearch output Install kibana plugin in es Open your browser
  21. 21. After getting started RTFM, play, explore, mess up, google, … Configure ES properly Setup nginx/apache to proxy Think about retention policies ...
  22. 22. Links http://www.elasticsearch.org/ http://linko.io https://groups.google.com/forum/?fromgroups =#!forum/elasticsearch http://www.jillesvangurp.com
  23. 23. Thanks! @jillesvangurp, @linkoapp
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×