ELK Stack

2,776 views
2,315 views

Published on

Short presentation about the ELK stack (Elasticsearch, Logstash, Kibana) running on top of Docker / Vagrant.

Published in: Technology
0 Comments
6 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,776
On SlideShare
0
From Embeds
0
Number of Embeds
30
Actions
Shares
0
Downloads
29
Comments
0
Likes
6
Embeds 0
No embeds

No notes for slide

ELK Stack

  1. 1. ELK
  2. 2. Eberhard Wolff - @ewolff Docker •  No true virtualization •  Linux Containers (lxc) •  i.e. shared kernel •  i.e. separate file systems
  3. 3. Eberhard Wolff - @ewolff Docker File Systems •  Read only base images •  +read/write image •  Can be stacked
  4. 4. Eberhard Wolff - @ewolff Docker •  Linux only •  So: Run it in VM •  Controlled by Vagrant
  5. 5. Docker Demo
  6. 6. Eberhard Wolff - @ewolff Communication Between Docker Container •  Via ports •  Via data volumes
  7. 7. Docker is a Component System
  8. 8. Eberhard Wolff - @ewolff Example •  Log file analysis •  ELK Stack •  Elasticsearch: Store •  Logstash: Parse •  Kibana: Web based GUI •  Install three servers •  Automate installation ... L
  9. 9. Eberhard Wolff - @ewolff Dockerized ELK Stack •  1 Vagrantfile (25 lines) •  6 Dockerfiles (total 28 lines) •  1 HTML file (11 lines) •  1 Config (Logstash) (29 lines) •  Total: 93 lines, 9 files •  Including one RegExp L
  10. 10. Eberhard Wolff - @ewolff Vagrant VM Container user-registration 8080 8081 Port 8081 Container elasticsearch Container logstash 9200 9200 Port 9200 9200 link Container kibana Port 8080 8080 8080 Browser Volume /log
  11. 11. Demo ELK Stack
  12. 12. Eberhard Wolff - @ewolff date time level 1988-10-12 19:42:07.350 INFO 683 [http-nio-8080-exec-6] --- c.e.u.service.RegistrationService : email=eberhard.wolff@gmail.com geloesc pid thread class logmessage
  13. 13. Eberhard Wolff - @ewolff Scaling Elasticsearch •  More servers •  Shards •  Easy to set up •  Index per day or week or month •  Smaller dataset
  14. 14. Eberhard Wolff - @ewolff Ideas about Logstash •  No parsing •  Send JSON
  15. 15. Eberhard Wolff - @ewolff Break Logstash apart •  Logstash does parsing and communication •  Not too sensible in a distributed sytem •  Separate process per responsibility
  16. 16. Eberhard Wolff - @ewolff Shipper •  Ship data from logging system •  Logstash •  Beaver https://github.com/josegonzalez/ beaver •  Woodchuck https://github.com/danryan/ woodchuck
  17. 17. Eberhard Wolff - @ewolff Broker •  Buffer data for later processing •  Redis •  In-Memory NoSQL database •  Small •  Fast •  ...and then Parse (Logstash)
  18. 18. Eberhard Wolff - @ewolff Links •  https://github.com/ewolff/user- registration
  19. 19. Eberhard Wolff - @ewolff Thank You!! eberhard.wolff@gmail.com @ewolff

×