제품 구성 단일 제품 다양한 오픈소스로 구성
데이터 수집 Splunk Universal Forwarder LogStash, Fluentd, FileBeat
검색 Splunk Search Head Kibana
검색 언어
SPL
(Search Processing Language)
KQL (Kibana Query Language),
Query DSL, Json
인덱싱 기술 C++ Apache Lucene
(k8s-venv) root@k8s-master01:~# helm repo add elastic https://helm.elastic.co
"elastic" has been added to your repositories
(k8s-venv) root@k8s-master01:~# tee elastic-values.yaml << EOF
# Permit co-located instances for solitary minikube virtual machines.
antiAffinity: "soft"
# Shrink default JVM heap.
esJavaOpts: "-Xmx128m -Xms128m"
# Allocate smaller chunks of memory per pod.
resources:
requests:
cpu: "100m"
memory: "512M"
limits:
cpu: "1000m"
memory: "512M"
persistence:
enabled: false
EOF
(k8s-venv) root@k8s-master01:~# helm install elasticsearch elastic/elasticsearch --version=7.9.0 --namespace=logs -f elastic-values.yaml
NAME: elasticsearch
LAST DEPLOYED: Thu May 6 01:22:16 2021
NAMESPACE: logs
STATUS: deployed
REVISION: 1
NOTES:
1. Watch all cluster members come up.
$ kubectl get pods --namespace=logs -l app=elasticsearch-master -w
2. Test cluster health using Helm test.
$ helm test elasticsearch --cleanup
(k8s-venv) root@k8s-master01:~# helm repo add fluent https://fluent.github.io/helm-charts
"fluent" has been added to your repositories
(k8s-venv) root@k8s-master01:~# helm install fluent-bit fluent/fluent-bit --version 0.6.3 --namespace=logs
NAME: fluent-bit
LAST DEPLOYED: Thu May 6 01:24:24 2021
NAMESPACE: logs
STATUS: deployed
REVISION: 1
NOTES:
Get Fluent Bit build information by running these commands:
export POD_NAME=$(kubectl get pods --namespace logs -l "app.kubernetes.io/name=fluent-bit,app.kubernetes.io/instance=fluent-bit" -o
jsonpath="{.items[0].metadata.name}")
echo "curl http://127.0.0.1:2020 for Fluent Bit build information"
kubectl --namespace logs port-forward $POD_NAME 2020:2020
(k8s-venv) root@k8s-master01:~# helm install kibana elastic/kibana --version=7.9.0 --namespace=logs --set service.type=NodePort --set
service.nodePort=31000
NAME: kibana
LAST DEPLOYED: Thu May 6 01:25:01 2021
NAMESPACE: logs
STATUS: deployed
REVISION: 1
TEST SUITE: None
(k8s-venv) root@k8s-master01:~# kubectl get deployments,pods,services --namespace=logs
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/kibana-kibana 1/1 1 1 7m11s
NAME READY STATUS RESTARTS AGE
pod/elasticsearch-master-0 1/1 Running 0 9m56s
pod/elasticsearch-master-1 1/1 Running 0 9m56s
pod/elasticsearch-master-2 1/1 Running 0 9m56s
pod/fluent-bit-wj5gt 1/1 Running 0 7m48s
pod/fluent-bit-xp7vn 1/1 Running 0 7m48s
pod/fluent-bit-xxs4z 1/1 Running 0 7m48s
pod/kibana-kibana-6d874c5f46-6d5kf 1/1 Running 0 7m11s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/elasticsearch-master ClusterIP 10.233.63.66 <none> 9200/TCP,9300/TCP 9m56s
service/elasticsearch-master-headless ClusterIP None <none> 9200/TCP,9300/TCP 9m56s
service/fluent-bit ClusterIP 10.233.19.236 <none> 2020/TCP 7m48s
service/kibana-kibana NodePort 10.233.42.99 <none> 5601:31000/TCP 7m11s
(k8s-venv) root@k8s-master01:~# kubectl run random-logger --image=chentex/random-logger
pod/random-logger created
(k8s-venv) root@k8s-master01:~# kubectl logs pod/random-logger
2021-07-08T01:32:45+0000 ERROR An error is usually an exception that has been caught and not handled.
2021-07-08T01:32:47+0000 DEBUG This is a debug log that shows a log that can be ignored.
T. 02-516-0711 E. sales@osci.kr
서울시강남구테헤란로83길32,5층(삼성동,나라키움삼성동A빌딩)
THANK YOU

[오픈소스컨설팅] EFK Stack 소개와 설치 방법

  • 14.
    제품 구성 단일제품 다양한 오픈소스로 구성 데이터 수집 Splunk Universal Forwarder LogStash, Fluentd, FileBeat 검색 Splunk Search Head Kibana 검색 언어 SPL (Search Processing Language) KQL (Kibana Query Language), Query DSL, Json 인덱싱 기술 C++ Apache Lucene
  • 17.
    (k8s-venv) root@k8s-master01:~# helmrepo add elastic https://helm.elastic.co "elastic" has been added to your repositories (k8s-venv) root@k8s-master01:~# tee elastic-values.yaml << EOF # Permit co-located instances for solitary minikube virtual machines. antiAffinity: "soft" # Shrink default JVM heap. esJavaOpts: "-Xmx128m -Xms128m" # Allocate smaller chunks of memory per pod. resources: requests: cpu: "100m" memory: "512M" limits: cpu: "1000m" memory: "512M" persistence: enabled: false EOF
  • 18.
    (k8s-venv) root@k8s-master01:~# helminstall elasticsearch elastic/elasticsearch --version=7.9.0 --namespace=logs -f elastic-values.yaml NAME: elasticsearch LAST DEPLOYED: Thu May 6 01:22:16 2021 NAMESPACE: logs STATUS: deployed REVISION: 1 NOTES: 1. Watch all cluster members come up. $ kubectl get pods --namespace=logs -l app=elasticsearch-master -w 2. Test cluster health using Helm test. $ helm test elasticsearch --cleanup
  • 19.
    (k8s-venv) root@k8s-master01:~# helmrepo add fluent https://fluent.github.io/helm-charts "fluent" has been added to your repositories (k8s-venv) root@k8s-master01:~# helm install fluent-bit fluent/fluent-bit --version 0.6.3 --namespace=logs NAME: fluent-bit LAST DEPLOYED: Thu May 6 01:24:24 2021 NAMESPACE: logs STATUS: deployed REVISION: 1 NOTES: Get Fluent Bit build information by running these commands: export POD_NAME=$(kubectl get pods --namespace logs -l "app.kubernetes.io/name=fluent-bit,app.kubernetes.io/instance=fluent-bit" -o jsonpath="{.items[0].metadata.name}") echo "curl http://127.0.0.1:2020 for Fluent Bit build information" kubectl --namespace logs port-forward $POD_NAME 2020:2020
  • 20.
    (k8s-venv) root@k8s-master01:~# helminstall kibana elastic/kibana --version=7.9.0 --namespace=logs --set service.type=NodePort --set service.nodePort=31000 NAME: kibana LAST DEPLOYED: Thu May 6 01:25:01 2021 NAMESPACE: logs STATUS: deployed REVISION: 1 TEST SUITE: None
  • 21.
    (k8s-venv) root@k8s-master01:~# kubectlget deployments,pods,services --namespace=logs NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/kibana-kibana 1/1 1 1 7m11s NAME READY STATUS RESTARTS AGE pod/elasticsearch-master-0 1/1 Running 0 9m56s pod/elasticsearch-master-1 1/1 Running 0 9m56s pod/elasticsearch-master-2 1/1 Running 0 9m56s pod/fluent-bit-wj5gt 1/1 Running 0 7m48s pod/fluent-bit-xp7vn 1/1 Running 0 7m48s pod/fluent-bit-xxs4z 1/1 Running 0 7m48s pod/kibana-kibana-6d874c5f46-6d5kf 1/1 Running 0 7m11s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/elasticsearch-master ClusterIP 10.233.63.66 <none> 9200/TCP,9300/TCP 9m56s service/elasticsearch-master-headless ClusterIP None <none> 9200/TCP,9300/TCP 9m56s service/fluent-bit ClusterIP 10.233.19.236 <none> 2020/TCP 7m48s service/kibana-kibana NodePort 10.233.42.99 <none> 5601:31000/TCP 7m11s
  • 22.
    (k8s-venv) root@k8s-master01:~# kubectlrun random-logger --image=chentex/random-logger pod/random-logger created (k8s-venv) root@k8s-master01:~# kubectl logs pod/random-logger 2021-07-08T01:32:45+0000 ERROR An error is usually an exception that has been caught and not handled. 2021-07-08T01:32:47+0000 DEBUG This is a debug log that shows a log that can be ignored.
  • 62.
    T. 02-516-0711 E.sales@osci.kr 서울시강남구테헤란로83길32,5층(삼성동,나라키움삼성동A빌딩) THANK YOU