Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Let’s reconsider about collecting logs.

Plus, visiting elastic@Mountain View!
Shin Tanimoto 

Acroquest Technology Co., L...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.
Who am I?
2
• 谷本 心 (Shin Tanimoto)
- Acroquest Technology ...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reser...
Copyright © Acroquest Technology Co., Ltd. All rights reserved.
59
Enjoy processing logs using ELK!
Upcoming SlideShare
Loading in …5
×

Let's reconsider about collecting logs. Plus, visiting elastic@Moutain View!

1,398 views

Published on

第13回elasticsearch勉強会
「ログ収集の仕組みを再考しよう! あとマウンテンビューに行ってきました。」

Published in: Technology
  • Be the first to comment

Let's reconsider about collecting logs. Plus, visiting elastic@Moutain View!

  1. 1. Let’s reconsider about collecting logs.
 Plus, visiting elastic@Mountain View! Shin Tanimoto 
 Acroquest Technology Co., LTD.
  2. 2. Copyright © Acroquest Technology Co., Ltd. All rights reserved. Who am I? 2 • 谷本 心 (Shin Tanimoto) - Acroquest Technology Co., LTD.
 (Sales partner of elastic) - Java Troubleshooter - Board member of JJUG
 (Japan Java User Group) - Twitter : @cero_t - Facebook : shin.tainmoto
  3. 3. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. Quiz🙋 3
  4. 4. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. What is the origin
 of the word “log”? 4
  5. 5. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 5
  6. 6. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 1. Ancient Greece people
 record the “date” using
 branches of the tree. 6
  7. 7. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 2. In medieval Europe, people measured “speed” of ship with log (round wood). 7
  8. 8. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 3. In the early 20th century United States, engineers used a logarithm table for “usage history” of computers. 8
  9. 9. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 1. Ancient Greece people’s “date” record. 2. Medieval Europe sailors’ 
 “speed” record. 3. American engineers’
 “usage” record. 9
  10. 10. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 1.
 2. Medieval Europe sailors’ 
 “speed” record. 3.
 10
  11. 11. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. Common sense:
 Log is important 11
  12. 12. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. True common sense:
 Watching log is painful! 12
  13. 13. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. Then log should be
 watched and processed
 by machine (ordinary) 13
  14. 14. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 14 #1
 Ordinal Log Processing
  15. 15. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 1. Ordinal Log Processing ELK stack 15 send
 logs search
  16. 16. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 1. Ordinal Log Processing Access counts (upper) / response time (lower) 16
  17. 17. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 1. Ordinal Log Processing Access counts (upper) / response time (lower) 17 10/sec 100/sec 30sec 20sec 10sec
  18. 18. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 1. Ordinal Log Processing Huge performance issue 18
  19. 19. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 1. Ordinal Log Processing Huge performance issue 19 3000sec 2000sec 1000sec
  20. 20. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 1. Ordinal Log Processing Slow query log of MySQL 20 same shape!
  21. 21. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 1. Ordinal Log Processing Slow query log of MySQL 21 2000sec 1000sec same shape! same scale!
  22. 22. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 1. Ordinal Log Processing But where do these shapes come? 22
  23. 23. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 1. Ordinal Log Processing 23 But where do these shapes come? 1. Lock tables? 2. Up to maximum size of connection pool? 3. CPU bottle neck? 4. Disk I/O bottle neck?
  24. 24. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 1. Ordinal Log Processing 24 Confirm the stored procedure in detail 1. Found 100,000 times of insert into “temporary table” query – (even in the search function … ) – causing high CPU and Disk I/O usage 2. Optimized the stored procedure removing wasting process – Only a drop in the bucket 😩 3. Modify the create temporary table state in the stored procedure to create that temporary table “on memory” – with memory tunings ( tmp_table_size etc. ) – resulted in …
  25. 25. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 1. Ordinal Log Processing Performance issue was resolved! 25
  26. 26. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 1. Ordinal Log Processing Performance issue was resolved! 26 500sec 100sec Never mind, some heavy batch
  27. 27. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 1. Ordinal Log Processing Disk I/O improved!!! 27 Disk I/O on MySQL server before ← →after
  28. 28. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 1. Ordinal Log Processing I/O wait had gone! 28 before ← →after CPU usage on MySQL server
  29. 29. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 29 #2
 Reconsider Log Processing
  30. 30. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 30 Watching logs to detect errors is a responsibility of developers, isn’t it?
  31. 31. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 31 Watching logs is important but painful
  32. 32. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 32 Let’s think about painless log processing system
  33. 33. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. #2 Reconsider Log Processing Logs can be used in various ways Visualizing - as chart Watching - and notifying by e-mail Viewing - by human’s eyes Keeping - backup just in case 33
  34. 34. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. #2 Reconsider Log Processing Logs can be used in various purpose Visualizing - To find “unknown” issues Watching - To find “known” issues Viewing - To find the cause of issues Keeping - To use as necessary 34
  35. 35. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. #2 Reconsider Log Processing Logs retention period are also various Visualizing - last 2 or 4 weeks Watching - last 24 hours Viewing - last 2 or 4 weeks Keeping - entire period 35
  36. 36. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. #2 Reconsider Log Processing Tools for processing logs are different Visualizing - Elasticsearch Watching - Zabbix or some custom batch Viewing - Text editor Keeping - File server 36
  37. 37. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 37 Log is not necessarily files.
  38. 38. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 38 Log can be regarded as events.
  39. 39. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 39 Log streaming hub
  40. 40. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. #2 Reconsider Log Processing Log Streaming Hub 40 Application Agent Streaming
 Hub Viewer Watcher Visualizer Storage Application Agent
  41. 41. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. #2 Reconsider Log Processing Ordinal case 41 Application fluentd Text Editor Zabbix Elasticsearch
 + Kibana NAS Application fluentd
  42. 42. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. #2 Reconsider Log Processing Using fluentd 42 Application fluentd fluentd Zabbix Elasticsearch
 + Kibana Application fluentd Text Editor NAS
  43. 43. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. #2 Processing Logs Using Redis? 43 Application Logstash Redis Zabbix Elasticsearch
 + Kibana Application Logstash Text Editor NAS
  44. 44. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. #2 Reconsider Log Processing An example on AWS 44 Application Kinesis Agent Kinesis Cloudwatch Logs Elasticsearch
 + Kibana Application Kinesis Agent S3 Lambda
  45. 45. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. #2 Processing Logs An example on AWS 45 Application awslogs Cloudwatch
 Logs Elasticsearch
 + Kibana Application awslogs S3 Lambda
  46. 46. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. #2 Processing Logs An example on AWS 46 Application logstash S3 Elasticsearch
 + Kibana Application logstash Cloudwatch Logs Lambda
  47. 47. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. #2 Processing Logs Anyway… 47 Application Agent Streaming
 Hub Viewer Watcher Visualizer Storage Application Agent
  48. 48. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 48 すんなり入る話
 ですよね?
  49. 49. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 49 By the way
  50. 50. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 50 I had visited elastic @Mountain View!!
  51. 51. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 51 写真はブログで
 http://acro-engineer.hatenablog.com/ entry/2015/11/08/150942
  52. 52. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 52 elastic stack .Next
  53. 53. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. #3 elastic stack .Next Elasticsearch Task management API Reindex API https://www.elastic.co/elasticon/2015/sf/whats-next- for-elasticsearch-2x-and-beyond
 Logstash Clustering Persistent https://www.elastic.co/guide/en/logstash-roadmap/ current/index.html 53
  54. 54. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. #3 elastic stack .Next Kibana Custom Apps / plugins https://www.elastic.co/elasticon/2015/sf/whats- cookin-in-kibana-4
 Beats Packetbeat Filebeat Topbeat https://www.elastic.co/products/beats 54
  55. 55. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. #3 elastic stack .Next Commercial plugin Cross-stack monitoring / management Cross-stack security PDF reporting Orchestration / Automation 55
  56. 56. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 56 Using
 elastic stack .Next
  57. 57. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. #3 elastic stack .Next Using next ELK stack + AWS 57 Application Filebeat
 Topbeat Logstash Elasticsearch
 + Kibana Application Filebeat
 Topbeat S3 Cloudwatch Logs
  58. 58. Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 58 Beatsシリーズ
 調べなくちゃって気に
 なりました
  59. 59. Copyright © Acroquest Technology Co., Ltd. All rights reserved. 59 Enjoy processing logs using ELK!

×