This document proposes implementing a real-time centralized logging system using the Elastic Stack. It introduces Elastic Stack components like Filebeat, Elasticsearch, and Kibana. It then provides a use case of converting log timestamps to a standard sort format using Logstash filters like grok and date. The presenter works at WeMo Scooter, an electric scooter rental startup aiming to reduce emissions. He is interested in technologies like Elastic Stack, PostgreSQL, and Spark.

1. Implement Real-time Centralized logging System
by Elastic Stack
Len Chang, WeMo Scooter
1

2. Who am I
● 軟體工程師, 資料庫架構師, 資料科學家, HadoopCon 2015 Speaker
● 喜歡發呆, 做瑜珈, 跳舞, 和各種球類運動。當然，寫程式已經是生活的一部分
● 目前較關注和熟悉的技術為: C#, Python, Elastic Stack, PostgreSQL, Spark
● 目前任職於 WeMo Scooter, 我們是一間很有趣的小新創公司, 目標是希望推廣城市內的電動機車租貸服務，進而
減少廢氣排放與機車數量。
● Linkedin: https://tw.linkedin.com/in/huailunchang 2

3. Agenda
● Elastic stack
○ Arch.
■ Open Source
■ License
● How to establish a simple Elastic Stack ?
○ filebeat
○ elasticsearch
○ kibana
● Use Case: How to convert “log timestamp” to be “sort standard”
○ logstash
● Q & A
● WeMo Scooter
3

4. Elastic stack
4

5. Overview
● Elastic's open source solutions solve a growing list of search and log analysis.
● Helps you take data from any source, any format and search, analyze, and
visualize it in real time.
5

6. Architecture
DB
Application
sys log
sys log
Web Console
Restful API
6

7. Open Source Hadoop Ecosystem
7

8. License
8

9. How to establish a simple Elastic Stack ?
9

10. Beats
DB
Application
sys log
sys log
10

11. https://www.elastic.co/downloads/beats
Beats - filebeat
11

12. filebeat - core
12

13. filebeat - Index Template Setting
https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-template.html
13

14. Logs
filebeat - Index Template Description
hostname: string (No Analyze)
msg: string (Analyze)
msg_count: number
...
template.json
14

15. filebeat - Index Template Content
15

16. filebeat - Index Template Demo
16

17. filebeat - tab && space
17
Remember to use
“space”

18. Elasticsearch
DB
Application
sys log
sys log
18

19. Elasticsearch - Deploy by RPM
Category Explanation Destination
conf Configuration files elasticsearch.yml and logging.yml. /etc/elasticsearch
conf Environment variables including heap size, file descriptors. /etc/sysconfig/elasticsearch
19

20. Elasticsearch - Heap Tuning
https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html
● Give (less than) Half Your Memory to Lucene
○ Lucene need memory to interact with the OS.
● Don’t Cross 32 GB!
○ Compressed oops(Ordinary object pointers) have a upper boundary (~ 32 GB)
■ 32-bit pointer can reference four billion objects, rather than four billion bytes
20

21. Kibana
DB
Application
sys log
sys log
21

22. <Use Case>
How to convert “log timestamp” to be
“sort standard”
22

23. Kibana - Search
23

24. Kibana - log sample
24

25. Why we can’t use Beats to do it ?
DB
Application
sys log
sys log
25

26. Filebeat config
26

27. Logstash
DB
Application
sys log
sys log
27

28. Logstash - Filter plugins
28
grok
● https://www.elastic.co/guide/en/logstash/current/p
lugins-filters-grok.html
● Parse arbitrary text and structure it.
● Grok is currently the best way in logstash to
parse crappy unstructured log data into
something structured and queryable.
date
● https://www.elastic.co/guide/en/logstash/current/p
lugins-filters-date.html
● The date filter is used for parsing dates from
fields, and then using that date or timestamp as
the logstash timestamp for the event.

29. Logstash - Filter of Config
29
PREFIX_TIMESTAMP ^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}

30. Logstash - Result
30

31. Q & A
31

32. WeMo Scooter
32
Official Website
● http://www.wemoscooter.com/
Video Introduction
● https://www.youtube.com/watch?v=Ne1kg3KeoRs
If you want to be a software engineer with us….
● len.chang@wemoscooter.com
○ Assistant software engineer / software engineer
■ Django / Python
■ ASP.NET MVC 5 / C#
■ Others...

33. Thanks
33