Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

PHP Log Tracking with ELK & Filebeat part#2

48 views

Published on

PHP Log Tracking with ELK & Filebeat part#2

Published in: Technology
  • Be the first to comment

  • Be the first to like this

PHP Log Tracking with ELK & Filebeat part#2

  1. 1. PHP Log Tracking with ELK & Filebeat part#2 appkr(김주원) 2018년 7월
  2. 2. Ends and Means 2 Log Tracking ELK & Filebeat
  3. 3. 지난 이야기 ● AWS Cloud Watch에서 꼭 필요할 때 로그 일부를 찾을 수 없었다. ● 회사에서는 MSA 전환을 위해 표준 로깅 플랫폼으로 ELK를 선정했다. ○ Beats 데이터 수집기 ○ Logstash 데이터 수집 및 가공을 위한 파이프 라인 ○ Elasticsearch JSON 문서 기반의 검색 및 분석 엔진 ○ Kibana 데이터 시각화용 UI 도구 ● 더 해야 할 일 ○ ① 개발 환경 구축 ○ ② filebeat.yml ○ ③ prime-main.conf (Logstash Filter) ○ ④ 서버 배치 스크립트 작성 ○ ⑤ 애플리케이션 로그 관련 코드 수정 3 지난 번에 ②~③ 에서 삽질하던 이야기까지 했어요~
  4. 4. ① 개발 환경 구축 ● elk.zip 내려 받은 후 $HOME 폴더에서 압축 해제 ● ELK Docker 구동 4 ~ $ unzip elk.zip ~ $ docker run -d --name elk -e TZ="Asia/Seoul" -p 9200:9200 -p 9300:9300 -p 5044:5044 -p 5601:5601 -v $HOME/elk/data:/var/lib/elasticsearch -v $HOME/elk/config/logstash:/etc/logstash/conf.d -v $HOME/elk/config/kibana:/opt/kibana/config -v $HOME/elk/logs/logstash:/var/log/logstash sebp/elk:latest
  5. 5. ① 개발 환경 구축 ● Logstash log tailing ● Filebeat 구동 ● Kibana에서 로그 확인 5 ~ $ brew install filebeat ~ $ filebeat --strict.perms=false -e -c $HOME/elk/config/filebeats/filebeat.yml ~ $ tail -f $HOME/elk/logs/logstash/logstash.stdout ~ $ open http://localhost:5601
  6. 6. ② filebeat.yml 6 filebeat.prospectors: - document_type: log paths: - /path/to/storage/logs/laravel.log fields_under_root: true fields: log_type: prime-main log_source: app multiline.pattern: '^[[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}]' multiline.negate: true multiline.match: after multiline.max_lines: 100000 회사의 표준 로그 플랫폼(a.k.a. VoongELK) spec log_source 값에 따라 다른 Logstash 필터가 작동함
  7. 7. ② filebeat.yml(continue) 7 - document_type: log paths: - /path/to/httpd/prime_access_log fields_under_root: true fields: log_type: prime-main log_source: web instance_id: {INSTANCE_ID_TO_BE_REPLACED_DURING_PROVISIONING} channel: {CHANNEL_TO_BE_REPLACED_DURING_PROVISIONING} filebeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: true output.logstash: hosts: ["{HOST_TO_BE_REPLACED_DURING_PROVISIONING}"] 서버 프로비저닝할 때 교체되는 값 서버 프로비저닝할 때 교체되는 값
  8. 8. ③ prime-main.conf for Application Log 8 [2018-06-01 06:56:52] local.DEBUG: Request-Response log: { "request": { "fingerprint": "1206d351fd9ac5b3743334e4a13ed9a871a372a8", "...": "...", }, "response": { "code": 200, "content": { "result": "success", "...": "...", } }, "execution_time": 0.084259986877441 } { "app_version": "dev-build-180531", "instance_id": "i-07ae611c0b9e33a9d", "transaction_id": "WxBwlP9dgKQ8K7cfPxCgWAAAAA0", "trace_number": 0 } @timestamp channel level_name execution_time MONOLOG_HEADER PRIME_LOG_BODY PRIME_LOG_META
  9. 9. ③ prime-main.conf for Web Access Log 9 10.0.1.130 - - [31/May/2018:22:17:01 +0000] "WxB0XQ0YbL2OdR5CMbFkWQAAAAk" "-" "GET /pos/v1/stores/428/options/pickup HTTP/1.1" 200 109 "-" "RestSharp/105.2.3.0" @timestamp transaction_id request_id request_id는 클라이언트가 제출한 X-Mesh- Request-Id 헤더의 값이며, 값이 없으면 transaction_id의 값으로 폴백.Apache Web Server가 할당한 웹 트랜잭션 고유 식별자
  10. 10. ③ prime-main.conf 10 input { beats { port => 5044 } } filter { } output { elasticsearch { hosts => [ "localhost" ] index => "%{log_type}-%{+YYYY.MM.dd}" } stdout { codec => rubydebug } } 개발해야 할 항목
  11. 11. ③ prime-main.conf(continue) 11 if [log_source] == "app" { grok { pattern_definitions => { "MONOLOG_HEADER" => "[%{TIMESTAMP_ISO8601:datetime}] %{GREEDYDATA:channel}.%{LOGLEVEL:level_name}:" "PRIME_LOG_BODY" => "[wWnt]+" "PRIME_LOG_META" => '(?<prime_log_meta>{ns{4}"app_version":.+ns{4}"instance_id":.+ns{4}"transaction_id":.+ns{4}"trace _number":.+n})' } match => { "message" => "%{MONOLOG_HEADER} %{PRIME_LOG_BODY}s{1,2}%{PRIME_LOG_META}" } } # continued
  12. 12. ③ prime-main.conf(continue) 12 if [level_name] == "DEBUG" and [message] =~ /"execution_time":s?[0-9]+.[0-9]*/ { grok { match => { "message" => '"execution_time":s?(?<execution_time>[0-9]+.[0-9]+)' } } } json { source => "prime_log_meta" } date { match => [ "datetime", "yyyy-MM-dd HH:mm:ss" ] timezone => "Asia/Seoul" } } # continued
  13. 13. ③ prime-main.conf(continue) 13 if [log_source] == "web" { if [message] =~ /.+(internal dummy connection|ELB-HealthChecker).+/ { drop { } } grok { match => { "message" => ".+[%{HTTPDATE:timestamp}] "%{NOTSPACE:transaction_id}" "%{NOTSPACE:request_id}".+" } } if [request_id] =~ /[S]{2,}/ { mutate { replace => { "transaction_id" => "%{request_id}" } } } date { match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] } }
  14. 14. ④ Apache uniqueid Module deployment script 14 # .ebextensions/70mod-uniqueid.config files: /etc/httpd/conf.modules.d/10-uniqueid.conf: # ... content: | LoadModule unique_id_module modules/mod_unique_id.so /etc/httpd/conf.d/mod_unique_id.conf: # ... content: | RequestHeader set X-Unique-Id %{UNIQUE_ID}e /etc/httpd/conf.d/prime_access_log.conf: # ... content: | <IfModule log_config_module> LogFormat "%h %l %u %t "%{X-Unique-Id}i" "%{X-Mesh-Request-Id}i" "%r" %>s %b "%{Referer}i" "%{User-Agent}i"" prime_combined CustomLog "logs/prime_access_log" prime_combined </IfModule>
  15. 15. ④ filebeat binary/config deployment script 15 # .ebextensions/30filebeat.config Resources: AWSEBAutoScalingGroup: Metadata: AWS::CloudFormation::Authentication: S3Access: type: S3 roleName: aws-elasticbeanstalk-ec2-role buckets: vroong-server-config files: /tmp/filebeat.zip: # ... source: https://vroong-server-config.s3.amazonaws.com/filebeat.zip commands: 50copy-filebeat: command: /bin/cp -f /opt/elasticbeanstalk/hooks/appdeploy/post/300-install_filebeat.sh /opt/elasticbeanstalk/hooks/configdeploy/post/300-install_filebeat.sh
  16. 16. 16 # .ebextensions/30filebeat.config /opt/elasticbeanstalk/hooks/appdeploy/post/300-install_filebeat.sh: mode: "000755" owner: root group: root content: | #!/usr/bin/env bash set -xe echo "Unzipping filebeat resources." /usr/bin/unzip -o /tmp/filebeat.zip -d /tmp echo "Preparing filebeat configuration." INSTANCE_ID=$(/usr/bin/curl http://169.254.169.254/latest/meta-data/instance-id 2> /dev/null) /bin/sed -i "s/instance_id: .*/instance_id: ${INSTANCE_ID}/g" /tmp/filebeat/filebeat.yml source /opt/elasticbeanstalk/support/envvars APP_ENV=$(printenv APP_ENV) /bin/sed -i "s/channel: .*/channel: ${APP_ENV}/g" /tmp/filebeat/filebeat.yml # continued ④ filebeat binary/config deployment script (continue)
  17. 17. 17 # .ebextensions/30filebeat.config { LOGSTASH_HOST=$(printenv LOGSTASH_HOST) } || { echo "LOGSTASH_HOST variable not found. Falling back to default value." LOGSTASH_HOST="DEFAULT_LOGSTASH_HOST:5043" } /bin/sed -i "s/hosts: .*/hosts: ["${LOGSTASH_HOST}"]/g" /tmp/filebeat/filebeat.yml if /usr/bin/pgrep filebeat; then echo "Filebeat Agent already running. Skipping installation." else echo "Installing filebeat binary." /bin/rpm -vi --replacepkgs /tmp/filebeat/filebeat.rpm fi echo "Placing filebeat configuration." /bin/cp -bv /tmp/filebeat/filebeat.yml /etc/filebeat/filebeat.yml /bin/chmod 600 /etc/filebeat/filebeat.yml ④ filebeat binary/config deployment script (continue)
  18. 18. 18 # .ebextensions/30filebeat.config echo "Starting filebeat service." { /sbin/service filebeat restart } || { /sbin/service filebeat start } ④ filebeat binary/config deployment script (continue)
  19. 19. ⑤ CustomizedLoggingProvider application code 19 // app/Providers/CustomizedLoggingProvider.php class CustomizedLoggingProvider extends ServiceProvider { public function boot() { $logger = $this->app->make(LoggerInterface::class); $formatter = new SnakeContextKeyFormatter(null, null, true, true); $streamHandler = new StreamHandler( $this->app->storagePath().'/logs/laravel.log', $this->app->make('config')->get('app.log_level', Logger::DEBUG) ); $streamHandler->setFormatter($formatter); $monolog = $logger->getMonolog(); $monolog->setHandlers([$streamHandler]); $extraLogContextProcessor = $this->app->make(ExtraLogContextProcessor::class); $monolog->pushProcessor($extraLogContextProcessor); } }
  20. 20. ⑤ SnakeContextKeyFormatter application code 20 // app/Support/Logging/SnakeContextKeyFormatter.php class SnakeContextKeyFormatter extends LineFormatter { public function format(array $record) { $record['context'] = ToSnakeCaseArray::run($record['context']); return parent::format($record); } protected function toJson($data, $ignoreErrors = false) { $json = json_encode($data, JSON_PRETTY_PRINT|JSON_UNESCAPED_UNICODE|JSON_UNESCAPED_SLASHES); if ($json === false) { $json = parent::toJson($data, $ignoreErrors); } return $json; } } 로그 컨텍스트 키를 snake_case로 일괄 변경 가독성을 위해 인코딩하지 않고 Pretty Print
  21. 21. ⑤ ExtraLogContextProcessor application code 21 // app/Support/Logging/ExtraLogContextProcessor class ExtraLogContextProcessor { private $appContext; public function __construct(ApplicationContext $appContext) { $this->appContext = $appContext; } public function __invoke(array $record) { $record['extra']['app_version'] = $this->appContext->getAppVersion(); $record['extra']['instance_id'] = $this->appContext->getInstanceId(); $record['extra']['transaction_id'] = $this->appContext->getTransactionId(); $record['extra']['trace_number'] = $this->appContext->getTraceNumber(); $this->appContext->increaseTraceNumber(); return $record; } } PRIME_LOG_META에 해당하는 ExtraContext
  22. 22. ⑤ LogRequestResponse application code 22 // app/Http/Middleware/LogRequestResponse.php class LogRequestResponse { private $exractor; public function __construct(ExtractFilteredData $extractor){ $this->exractor = $extractor; } public function handle($request, Closure $next) { return $next($request); } public function terminate($request, $response) { $requestData = $this->exractor->fromIlluminateRequest($request); $responseData = $this->exractor->fromSymfonyResponse($response); $data = [ 'request' => $requestData, 'response' => $responseData, 'execution_time' => microtime(true) - LARAVEL_START, ]; Log::debug('Request-Response log:', $data); } } PRIME_LOG_BODY에 해당하는 로그 본문
  23. 23. 미션 완료 ● 해결해야 할 문제점 ○ 로그 유실 최소화 ■ Filebeat log streaming to VroongELK ■ Log rotate ■ Log publish to S3 ○ 로그 검색 성능 향상으로 개발자 피로도 최소화 ○ awslogs 데몬이 일으키는 서버 부하 감소 ● 상위 조직에서 받은 미션 ○ MSA로 진화하기 위한 선행 과제 ○ Cloud Watch 로그 요금 절약 23
  24. 24. 24 DEMO ● 부릉 프라임 서비스는 하루에 100GB+, 25백만+ 로그 인스턴스를 생산하고 있어요. ● 특정 상점의 배송 신청 찾기 log_source:"app" AND message:"POST /pos/v1/stores/16023/deliveries" ● transaction_id로 찾기 "5ab1988a-17df-48f3-b6a6-1bdd505c67ee" ● execution_time > 1 이상인 로그만 찾기 tags:"_exetimeparsed" AND execution_time:>=1 SSH tunneling 해야 접속할 수 있는 비 공개 호스트에요~
  25. 25. 25 필터 조건에 해당하는 로그 카운트 ➁ 조회 기간 선택 ➀ 조회할 인덱스 선택 ➂ 테이블에 노출할 필드 또는 필터 선택 ➂ 쿼리 표현식 입력
  26. 26. Application Log Why? 26 Code Data Log A large part of software developers' lives are monitoring, troubleshooting and debugging.”
  27. 27. Application Log Why? ● 로그는… ● Blackbox에 대한 Visibility 확보 ● "어제까지 작동했는데, 오늘 왜 갑자기 작동하지 않는지 모르겠다"라는 개발자들의 전형적인 질문에 대한 힌트 ● 복잡도가 낮은, 투명한 애플리케이션에서는 로깅을 할 필요 없다. ● 로그가 개발자에게 애플리케이션의 상태를 말하도록 하라. 27 If Dog is a man’s best friend, Log is a developer’s best friend. ” source: https://www.quora.com/Why-is-Logging-an-important-part-of-Software-Development
  28. 28. Logging Best Practice ● 애플리케이션에서 발생한 예외 트레이스, "RFC5424 The Syslog Protocol"에 따라 레벨 적용 권장 (사례). ● 의심스러운 애플리케이션 이벤트 ● 추적하고 싶은 애플리케이션 상태 ● 풀리지 않는 버그를 잡기 위한 디버그 로그 ● SQL statement ● 클라이언트의 HTTP 요청 ● 클라이언트에게 돌려주는 HTTP 응답 ● 프로세스/쓰레드 정보 ● 클라이언트(자바스크립트, 닷넷, ..) 측에서 발생하는 예외 28 source: https://dzone.com/articles/application-logging-what-when , https://geshan.com.np/blog/2015/08/importance-of-logging-in- your-applications/
  29. 29. Logging Best Practice ● Essential Components ○ Who UserName ○ When Timestamp ○ Where Context ServletOrPage,Database ○ What Command ○ Result Exception ● Things to Consider ○ Under clustered application environment → Logging as a Service ○ Trade off between logging and performance → Find optimum 29 source: https://dzone.com/articles/application-logging-what-when

×