Log Collection and Analysis with elk Stack

1. ELK(Elastic Search, Logstash, Kibana) Stack
(주) 투라인코드
설상민
2020.06
© TWOLINECODE Inc. All rights reserved.

2. TWOLINECODE
목차
© TWOLINECODE Inc. All rights reserved.
➢ ELK Stack 이란?
➢ Elasticsearch와 관계형 DB
➢ Elasticsearch RESTful API
➢ Elasticsearch Mapping
➢ LogStash
➢ LogStash filter
➢ Kibana

3. TWOLINECODE
ELK Stack 이란?
© TWOLINECODE Inc. All rights reserved.
- Elasticsearch
JSON 기반의 분산형 오픈 소스 RESTful 검색 및 분석 엔진
- LogStash
여러 소스에서 동시에 데이터를 수집하여 변환 stash로 전송하는 데이터 처리 파이프 라인
- Kibana
차트와 그래프를 이용한 데이터 시각화 도구
* Beats
단일 목적 데이터 수집기 제품군
( File, Packet, Window Event Log, etc… )

4. TWOLINECODE
Elasticsearch와 관계형 DB
© TWOLINECODE Inc. All rights reserved.

5. TWOLINECODE
Elasticsearch와 관계형 DB의 데이터 저장
© TWOLINECODE Inc. All rights reserved.
- Elasticsearch의 데이터 저장
- 관계형 DB의 데이터 저장

6. TWOLINECODE
Elasticsearch RESTful API
© TWOLINECODE Inc. All rights reserved.
- Elasticsearc의 Data CRUD - RESTful API
Data CRUD Elasticsearch Restful
INSERT POST
SELECT GET
UPDATE PUT
DELETE DELETE

7. TWOLINECODE
Elasticsearch RESTful API
© TWOLINECODE Inc. All rights reserved.
- Elasticsearc POST API : Data Insert / Create
- Elasticsearc GET API : Data Read / Select

8. TWOLINECODE
Elasticsearch RESTful API
© TWOLINECODE Inc. All rights reserved.
- Elasticsearc PUT API / Data Update

9. TWOLINECODE
Elasticsearch RESTful API
© TWOLINECODE Inc. All rights reserved.
- Elasticsearc DELETE API / Data Delete

10. TWOLINECODE
Elasticsearch RESTful API
© TWOLINECODE Inc. All rights reserved.
- Elasticsearch API 검색 – Index 전체

11. TWOLINECODE
Elasticsearch RESTful API
© TWOLINECODE Inc. All rights reserved.
- Elasticsearch API 검색 – URI 검색

12. TWOLINECODE
Elasticsearch Mapping
© TWOLINECODE Inc. All rights reserved.
- 관계형 Database의 스키마에 해당.
- Document의 type 정보에 대한 기술.
- Mapping 정보 없이 Document Handling 가능.
- Mapping 정보에 따라 Kibana에서 Visualize 기능 등에 제약이 있을 수 있음

13. TWOLINECODE
Logstash
© TWOLINECODE Inc. All rights reserved.
- Logstash는 Conf 파일을 기반으로 동작
- Conf 파일은 기본적으로 Input, filter, output으로 구성.
Input : beats, jdbc, syslog, tcp, udp, file, stdin 등 데이터 소스 선택
output : elasticsearch, jdbc, file, stdout 등
filter : 입력받은 데이터를 가공

14. TWOLINECODE
Logstash
© TWOLINECODE Inc. All rights reserved.
- Logstash 입출력 Conf 파일 예시

15. TWOLINECODE
Logstash filter
© TWOLINECODE Inc. All rights reserved.
- 입력 데이터의 분해, 추가, 삭제, 변형 등의 과정
grok, mutate, date 등
- 입력 순서 대로 적용됨.

16. TWOLINECODE
Logstash filter
© TWOLINECODE Inc. All rights reserved.
- Grok

17. TWOLINECODE
Logstash logfile->elasticsearch conf
© TWOLINECODE Inc. All rights reserved.
- Logstash file input – log File, output – elasticsearch 설정

18. TWOLINECODE
Kibana
© TWOLINECODE Inc. All rights reserved.
- Create Index Patterns
Define index pattern & select Time filter field

19. TWOLINECODE
Kibana
© TWOLINECODE Inc. All rights reserved.
- Discover
Time filter 설정을 통한 데이터 검색 가능

20. TWOLINECODE
Kibana
© TWOLINECODE Inc. All rights reserved.
- Discover
Document JSON

21. TWOLINECODE
Kibana
© TWOLINECODE Inc. All rights reserved.
- Visualization

22. TWOLINECODE
Kibana
© TWOLINECODE Inc. All rights reserved.
- Visualization : Vertical Bar

23. TWOLINECODE
Kibana
© TWOLINECODE Inc. All rights reserved.
- DashBoard

24. TWOLINECODE
참조 사이트 모음
• https://www.elastic.co/kr/what-is/elk-stack
• https://victorydntmd.tistory.com/308
• https://www.slideshare.net/kjmorc/ss-49009522
• https://www.inflearn.com/course/elk-%EC%8A%A4%ED%83%9D-%EB%8D%B0%EC%9D%B4%ED%84%B0-
%EB%B6%84%EC%84%9D#
• https://victorydntmd.tistory.com/313?category=742451 : Elasticsearch Search API
• https://victorydntmd.tistory.com/314?category=742451 : Elasticsearch Query DSL
• https://esbook.kimjmin.net/ : Elasticsearch 가이드 북
© TWOLINECODE Inc. All rights reserved.

25. TWOLINECODE
마무리 하기
© TWOLINECODE Inc. All rights reserved.

26. Thank you
© TWOLINECODE Inc. All rights reserved.