Let's reconsider about collecting logs. Plus, visiting elastic@Moutain View!

Let’s reconsider about collecting logs. 
Plus, visiting elastic@Mountain View!
Shin Tanimoto  
Acroquest Technology Co., LTD.

Copyright © Acroquest Technology Co., Ltd. All rights reserved.
Who am I?
2
• 谷本心 (Shin Tanimoto)
- Acroquest Technology Co., LTD. 
(Sales partner of elastic)
- Java Troubleshooter
- Board member of JJUG 
(Japan Java User Group)
- Twitter : @cero_t
- Facebook : shin.tainmoto

Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved.
Quiz🙋
3

What is the origin 
of the word “log”?
4

5

1. Ancient Greece people 
record the “date” using 
branches of the tree.
6

2. In medieval Europe,
people measured “speed” of
ship with log (round wood).
7

3. In the early 20th century United
States, engineers used a
logarithm table for “usage
history” of computers.
8

1. Ancient Greece people’s
“date” record.
2. Medieval Europe sailors’  
“speed” record.
3. American engineers’ 
“usage” record.
9

1. 
2. Medieval Europe sailors’  
“speed” record.
3. 
10

Common sense: 
Log is important
11

True common sense: 
Watching log is painful!
12

Then log should be 
watched and processed 
by machine (ordinary)
13

14
#1 
Ordinal Log
Processing

1. Ordinal Log Processing
ELK stack
15
send 
logs
search

Access counts (upper) / response time (lower)
16

Access counts (upper) / response time (lower)
17
10/sec
100/sec
30sec
20sec
10sec

Huge performance issue
18

Huge performance issue
19
3000sec
2000sec
1000sec

Slow query log of MySQL
20
same shape!

Slow query log of MySQL
21
2000sec
1000sec
same shape! same scale!

But where do these shapes come?
22

23
But where do these shapes come?
1. Lock tables?
2. Up to maximum size of connection pool?
3. CPU bottle neck?
4. Disk I/O bottle neck?

24
Conﬁrm the stored procedure in detail
1. Found 100,000 times of insert into “temporary table” query
– (even in the search function … )
– causing high CPU and Disk I/O usage
2. Optimized the stored procedure removing wasting process
– Only a drop in the bucket 😩
3. Modify the create temporary table state in the stored
procedure to create that temporary table “on memory”
– with memory tunings ( tmp_table_size etc. )
– resulted in …

Performance issue was resolved!
25

Performance issue was resolved!
26
500sec
100sec
Never mind, some heavy batch

Disk I/O improved!!!
27
Disk I/O on MySQL server
before ← →after

I/O wait had gone!
28
before ← →after
CPU usage on MySQL server

29
#2 
Reconsider Log
Processing

30
Watching logs to
detect errors is a
responsibility of
developers, isn’t it?

31
Watching logs is
important but painful

32
Let’s think about
painless log
processing system

#2 Reconsider Log Processing
Logs can be used in various ways
Visualizing - as chart
Watching - and notifying by e-mail
Viewing - by human’s eyes
Keeping - backup just in case
33

Logs can be used in various purpose
Visualizing - To find “unknown” issues
Watching - To find “known” issues
Viewing - To find the cause of issues
Keeping - To use as necessary
34

Logs retention period are also various
Visualizing - last 2 or 4 weeks
Watching - last 24 hours
Viewing - last 2 or 4 weeks
Keeping - entire period
35

Tools for processing logs are different
Visualizing - Elasticsearch
Watching - Zabbix or some custom batch
Viewing - Text editor
Keeping - File server
36

37
Log is not
necessarily ﬁles.

38
Log can be regarded
as events.

39
Log streaming hub

Log Streaming Hub
40
Application
Agent
Streaming 
Hub
Viewer
Watcher
Visualizer
Storage
Application
Agent

Ordinal case
41
Application
fluentd
Text Editor
Zabbix
Elasticsearch 
+ Kibana
NAS
Application
fluentd

Using ﬂuentd
42
Application
fluentd fluentd
Zabbix
Elasticsearch 
+ Kibana
Application
fluentd
Text Editor
NAS

#2 Processing Logs
Using Redis?
43
Application
Logstash Redis
Zabbix
Elasticsearch 
+ Kibana
Application
Logstash
Text Editor
NAS

An example on AWS
44
Application
Kinesis
Agent
Kinesis Cloudwatch
Logs
Elasticsearch 
+ Kibana
Application
Kinesis
Agent
S3
Lambda

#2 Processing Logs
An example on AWS
45
Application
awslogs
Cloudwatch 
Logs
Elasticsearch 
+ Kibana
Application
awslogs
S3
Lambda

#2 Processing Logs
An example on AWS
46
Application
logstash S3
Elasticsearch 
+ Kibana
Application
logstash
Cloudwatch
Logs
Lambda

#2 Processing Logs
Anyway…
47
Application
Agent
Streaming 
Hub
Viewer
Watcher
Visualizer
Storage
Application
Agent

48
すんなり入る話 
ですよね？

49
By the way

50
I had visited elastic
@Mountain View!!

51
写真はブログで 
http://acro-engineer.hatenablog.com/
entry/2015/11/08/150942

52
elastic stack .Next

#3 elastic stack .Next
Elasticsearch
Task management API
Reindex API
https://www.elastic.co/elasticon/2015/sf/whats-next-
for-elasticsearch-2x-and-beyond 
Logstash
Clustering
Persistent
https://www.elastic.co/guide/en/logstash-roadmap/
current/index.html
53

Kibana
Custom Apps / plugins
https://www.elastic.co/elasticon/2015/sf/whats-
cookin-in-kibana-4 
Beats
Packetbeat
Filebeat
Topbeat
https://www.elastic.co/products/beats
54

Commercial plugin
Cross-stack monitoring / management
Cross-stack security
PDF reporting
Orchestration / Automation
55

56
Using 
elastic stack .Next

Using next ELK stack + AWS
57
Application
Filebeat 
Topbeat
Logstash
Elasticsearch 
+ Kibana
Application
Filebeat 
Topbeat
S3
Cloudwatch
Logs

58
Beatsシリーズ 
調べなくちゃって気に 
なりました

Let's reconsider about collecting logs. Plus, visiting elastic@Moutain View!

Recommended

Recommended

More Related Content

Similar to Let's reconsider about collecting logs. Plus, visiting elastic@Moutain View!

Similar to Let's reconsider about collecting logs. Plus, visiting elastic@Moutain View! (20)

More from 心谷本

More from 心谷本 (11)

Recently uploaded

Recently uploaded (20)