1.
使⽤用 Elasticsearch 及 Kibana 進⾏行
巨量資料搜尋及視覺化
Suiting @ DSC 2015

2.
Who
Am
I
曾書庭
(@suitingtseng)
Data
Engineer
Gogolook

3.
Jeff, CEO

4.
As
a
data
engineer
in
Gogolook…
書庭，請問我們的
DAU
是多少？

5.
As
a
data
engineer
in
Gogolook…
書庭，請問我們的
DAU
是多少？
我算⼀一下...

6.
As
a
data
engineer
in
Gogolook…
書庭，請問我們的
DAU
是多少？
資料好了嗎?
我算⼀一下...

7.
As
a
data
engineer
in
Gogolook…
書庭，請問我們的
DAU
是多少？
資料好了嗎?
還在跑...
我算⼀一下...

8.
As
a
data
engineer
in
Gogolook…
書庭，請問我們的
DAU
是多少？

9.
As
a
data
engineer
in
Gogolook…
書庭，請問我們的
DAU
是多少？
可以分國家嗎？

10.
As
a
data
engineer
in
Gogolook…
書庭，請問我們的
DAU
是多少？
可以分版本嗎？
可以分國家嗎？

11.
As
a
data
engineer
in
Gogolook…
書庭，請問我們的
DAU
是多少？
可以分版本嗎？
可以看⼀一年嗎?
可以分國家嗎？

12.
As
a
data
engineer
in
Gogolook…
書庭，請問我們的
DAU
是多少？
可以分版本嗎？
可以看⼀一年嗎?
可以嗎? 可以嗎? 可以嗎?
可以分國家嗎？

13.
⼀一句話激怒⼯工程師⼤大賽
• 可以分XX嗎

14.
⼀一句話激怒⼯工程師⼤大賽
• 可以分XX嗎
• 可以畫成圖嗎
• 可以給我
raw
data
嗎

15.
⼀一句話激怒⼯工程師⼤大賽
• 可以分XX嗎
• 可以畫成圖嗎
• 可以給我
raw
data
嗎
• 有沒有辦法知道
user
住哪裡
• 可以知道哪些
user
⽐比較有錢嗎
• 下⾬雨天
user
會睡⽐比較晚嗎

16.
Table
of
Contents
• Problems
• Solution
Requirements
• Elasticsearch
&
Kibana
• In
Gogolook
• Future

17.
Problems
• Request-­‐response
model
https://medium.com/@samson_hu/building-analytics-at-500px-92e9a7005c83

18.
Problems
• Request-­‐response
model
• Long
cycle
https://medium.com/@samson_hu/building-analytics-at-500px-92e9a7005c83

19.
Problems
• Request-­‐response
model
• Long
cycle
• EAAB
(engineer
as
a
bottleneck)
https://medium.com/@samson_hu/building-analytics-at-500px-92e9a7005c83

20.
Problems
• Request-­‐response
model
• Long
cycle
• EAAB
(engineer
as
a
bottleneck)
• HDC
(Hippo-­‐driven
company)
https://medium.com/@samson_hu/building-analytics-at-500px-92e9a7005c83

21.
Problems
• Request-­‐response
model
• Long
cycle
• EAAB
(engineer
as
a
bottleneck)
• HDC
(Hippo-­‐driven
company)
• Lack
of
speed
https://medium.com/@samson_hu/building-analytics-at-500px-92e9a7005c83

22.
Problems
• Request-­‐response
model
• Long
innovation
cycle
• EAAB
(engineer
as
a
bottleneck)
• HDC
(Hippo-­‐driven
company)
• Lack
of
speed
• =>
We
are
not
alone
(500px)
https://medium.com/@samson_hu/building-analytics-at-500px-92e9a7005c83

23.
Table
of
Contents
• Problems
• Solution
Requirements
• Elasticsearch
&
Kibana
• In
Gogolook
• Future

24.
Possible
solutions
• Approach
1:
SQL
monkey*
zoo
http://www.slideshare.net/GloriaLau1/keynote-at-spark-summit/5

25.
Possible
solutions
• Approach
1:
SQL
monkey
zoo
• Approach
2:
Provide
limited
yet
easy
visualization
http://www.slideshare.net/GloriaLau1/keynote-at-spark-summit/5

26.
Requirement
• Easy:
Even
CEO
can
use
it
• Fast:
Must
be
interactive
• Export:
Provide
the
csv
file
• Big:
Must
be
scalable
• 80-­‐20:
Solves
80%
problems

27.
Table
of
Contents
• Problems
• Solution
Requirements
• Elasticsearch
&
Kibana
• In
Gogolook
• Future

28.
Elasticsearch
• Lucene-­‐based
search
engine
• Document
storage
(JSON)
• Distributed,
scalable
• Serve
search
request
in
ms
• Build
index
for
every
field

29.
Kibana
• ES
visualization
tool
• No
code
required

30.
ES
+
Kibana
• Fast:
index
every
field
• Fast:
columnar
storage*
• Big:
born
distributed/scalable
• Easy:
GUI
without
code
• Export:
csv

31.
Kibana
• Discover
• Visualization
• Dashboard

32.
Discover
• Raw
data
• Check
data
• Find
dirty
data
• Try
query

33.
Discover

34.
Discover

35.
Visualization
• 8
visualization
types
• 9
group
methods
• 9
aggregation
values

36.
Visualization

37.
Visualization
types

38.
Grouping
methods
• Date
histogram
• Histogram
• Range
of
a
value
• Top
N
• Filter

39.
Aggregation
values
• Count
• Avg,
Sum,
Min,
Max,
S.D.
• Unique
count*
(Hyperloglog)
• Percentile*
(T-­‐digest)

40.
Visualization
• Same
concept,
different
graph
• FILTER
• GROUP
• AGGREGATE

41.
DAU
書庭，請問我們的
DAU
是多少？

42.
DAU
by
region
可以分國家嗎？

43.
DAU
by
version
可以分版本嗎？

44.
server
request
log

45.
Request_total
per
minute
GROUP
BY
DATE
HISTOGRAM(minute)
COUNT(*)

46.
Request_total
by
path
GROUP
BY
TOP(path,
5),
DATE
HISTOGRAM(minute)
COUNT(*)

47.
Dashboard
• Collection
of
visualizations

48.
Community
tag
in
MongoDB

49.
Dashboard

50.
Dashboard
-­‐
1st
peak

51.
Dashboard
-­‐
2nd
peak

52.
Table
of
Contents
• Problems
• Solution
Requirements
• Elasticsearch
&
Kibana
• In
Gogolook
• Future

53.
In
Gogolook
(Aug.
2015)
• 200M+
data
point
daily
• 150GB+
data
size
daily
• 24
dashboards,
160
visualizations
• Service
status
e.g.
requests_total
• Application
data
e.g.
tag_total
• Log
data
e.g.
button_ctr

54.
In
Gogolook
(currently)
• Log
user
behavior
on
features
• ⾃自⼰己的
log
⾃自⼰己記
(Planner/PM)
• ⾃自⼰己的
board
⾃自⼰己拉
(every
one)
• Monitor
performance
from
day
1

55.
In
Gogolook

56.
In
Gogolook
• Tracking
Kibana
usage
by
Google
Analytics

57.
Table
of
Contents
• Problems
• Solution
Requirements
• Elasticsearch
&
Kibana
• In
Gogolook
• Future

58.
In
Gogolook
(future)
• Log
all
user-­‐event,
not
feature-­‐based

59.
In
Gogolook
(future)
• Log
all
user-­‐event,
not
feature-­‐based
• {
"userid":
"suiting",
"@timestamp":
"2015-­‐08-­‐23T11:48:00",
"page":
"login",
"button":
"register",
"period":
3500
}

60.
In
Gogolook
(future)
• Answer
questions
A
B
40%
60%

61.
In
Gogolook
(future)
• Answer
questions
A
B
40%, 7000ms
60%, 1500ms

62.
Limit
• No
SQL
JOIN
• Subquery

63.
How
about
20%
• Powerful
engine/tool
required
• Compute
engines:
• Google
BigQuery
• AWS
Redshift
• Visualization
tools:
• Tableau
• Periscope

64.
Thank you

65.
Questions ?