0
ELK
Eberhard Wolff - @ewolff
Docker
•  No true virtualization
•  Linux Containers (lxc)
•  i.e. shared kernel
•  i.e. separate...
Eberhard Wolff - @ewolff
Docker File Systems
•  Read only base images
•  +read/write image
•  Can be stacked
Eberhard Wolff - @ewolff
Docker
•  Linux only
•  So: Run it in VM
•  Controlled by Vagrant
Docker Demo
Eberhard Wolff - @ewolff
Communication Between
Docker Container
•  Via ports
•  Via data volumes
Docker is a
Component
System
Eberhard Wolff - @ewolff
Example
•  Log file analysis
•  ELK Stack
•  Elasticsearch: Store
•  Logstash: Parse
•  Kibana: W...
Eberhard Wolff - @ewolff
Dockerized ELK Stack
•  1 Vagrantfile (25 lines)
•  6 Dockerfiles (total 28 lines)
•  1 HTML file...
Eberhard Wolff - @ewolff
Vagrant VM
Container
user-registration
8080
8081
Port
8081
Container
elasticsearch
Container
logs...
Demo ELK
Stack
Eberhard Wolff - @ewolff
date time level
1988-10-12 19:42:07.350 INFO 683 [http-nio-8080-exec-6] --- c.e.u.service.Registr...
Eberhard Wolff - @ewolff
Scaling Elasticsearch
•  More servers
•  Shards
•  Easy to set up
•  Index per day or week or mon...
Eberhard Wolff - @ewolff
Ideas about Logstash
•  No parsing
•  Send JSON
Eberhard Wolff - @ewolff
Break Logstash apart
•  Logstash does parsing and
communication
•  Not too sensible in a distribu...
Eberhard Wolff - @ewolff
Shipper
•  Ship data from logging system
•  Logstash
•  Beaver
https://github.com/josegonzalez/
b...
Eberhard Wolff - @ewolff
Broker
•  Buffer data for later processing
•  Redis
•  In-Memory NoSQL database
•  Small
•  Fast
...
Eberhard Wolff - @ewolff
Links
•  https://github.com/ewolff/user-
registration
Eberhard Wolff - @ewolff
Thank You!!
eberhard.wolff@gmail.com
@ewolff
Upcoming SlideShare
Loading in...5
×

ELK Stack

1,708

Published on

Short presentation about the ELK stack (Elasticsearch, Logstash, Kibana) running on top of Docker / Vagrant.

Published in: Technology

Transcript of "ELK Stack"

  1. 1. ELK
  2. 2. Eberhard Wolff - @ewolff Docker •  No true virtualization •  Linux Containers (lxc) •  i.e. shared kernel •  i.e. separate file systems
  3. 3. Eberhard Wolff - @ewolff Docker File Systems •  Read only base images •  +read/write image •  Can be stacked
  4. 4. Eberhard Wolff - @ewolff Docker •  Linux only •  So: Run it in VM •  Controlled by Vagrant
  5. 5. Docker Demo
  6. 6. Eberhard Wolff - @ewolff Communication Between Docker Container •  Via ports •  Via data volumes
  7. 7. Docker is a Component System
  8. 8. Eberhard Wolff - @ewolff Example •  Log file analysis •  ELK Stack •  Elasticsearch: Store •  Logstash: Parse •  Kibana: Web based GUI •  Install three servers •  Automate installation ... L
  9. 9. Eberhard Wolff - @ewolff Dockerized ELK Stack •  1 Vagrantfile (25 lines) •  6 Dockerfiles (total 28 lines) •  1 HTML file (11 lines) •  1 Config (Logstash) (29 lines) •  Total: 93 lines, 9 files •  Including one RegExp L
  10. 10. Eberhard Wolff - @ewolff Vagrant VM Container user-registration 8080 8081 Port 8081 Container elasticsearch Container logstash 9200 9200 Port 9200 9200 link Container kibana Port 8080 8080 8080 Browser Volume /log
  11. 11. Demo ELK Stack
  12. 12. Eberhard Wolff - @ewolff date time level 1988-10-12 19:42:07.350 INFO 683 [http-nio-8080-exec-6] --- c.e.u.service.RegistrationService : email=eberhard.wolff@gmail.com geloesc pid thread class logmessage
  13. 13. Eberhard Wolff - @ewolff Scaling Elasticsearch •  More servers •  Shards •  Easy to set up •  Index per day or week or month •  Smaller dataset
  14. 14. Eberhard Wolff - @ewolff Ideas about Logstash •  No parsing •  Send JSON
  15. 15. Eberhard Wolff - @ewolff Break Logstash apart •  Logstash does parsing and communication •  Not too sensible in a distributed sytem •  Separate process per responsibility
  16. 16. Eberhard Wolff - @ewolff Shipper •  Ship data from logging system •  Logstash •  Beaver https://github.com/josegonzalez/ beaver •  Woodchuck https://github.com/danryan/ woodchuck
  17. 17. Eberhard Wolff - @ewolff Broker •  Buffer data for later processing •  Redis •  In-Memory NoSQL database •  Small •  Fast •  ...and then Parse (Logstash)
  18. 18. Eberhard Wolff - @ewolff Links •  https://github.com/ewolff/user- registration
  19. 19. Eberhard Wolff - @ewolff Thank You!! eberhard.wolff@gmail.com @ewolff
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×