This document summarizes a presentation given by TAGOMORI Satoshi of LINE Corp on January 31, 2014 about inspecting logs. It discusses retrospectively searching stored log data versus prospectively defining queries to process log data. It also covers differences between schema-full and schema-less log data, as well as tools for collecting, storing, processing, and visualizing log data. The presentation emphasizes using stream processing to handle large volumes of schema-less log data through prospective queries that define required fields and types.

1. Retrospection / prospection
and schema
TAGOMORI Satoshi (@tagomoris)
LINE Corp.
2014/01/31 (Fri) at University of Tsukuba
the 1st half
14年1月31日金曜日

2. TAGOMORI Satoshi (@tagomoris)
LINE Corp.
Development Support Team
14年1月31日金曜日

3. 14年1月31日金曜日

4. 14年1月31日金曜日

5. Logs
Service metrics (Users, PageViews, ...)
UX/UI metrics (Access path, Taps/views, ...)
Monitoring metrics (Traffic Gbps, TBytes/day, ...)
System monitoring (Error rates, Response time, ...)
14年1月31日金曜日

6. Software for Logging
Collection: Fluentd, Scribed, Flume, LogStash, ...
Storage: RDBMS, Hadoop HDFS, NoSQLs, Elasticsearch, ....
Processing: SQL, Hadoop MapReduce(Hive), Presto, Impala, ...
Stream-Processing: Storm, Kafka, Norikra, ...
Visualization: Kibana, Tableau Fnordmetric, GrowthForecast, Focuslight, ...
Appliance: DHW + BI Tools
Services: Google BigQuery, Treasure Data, ...
14年1月31日金曜日

7. How inspect logs
Retrospection (reactive search)
Store data, and search
Prospection (proactive search)
Define what should be processed, and store data
14年1月31日金曜日

8. What logs inspected
Schema-full data:
strict schema: pre defined fields w/ types (or reject)
schema on read: try to read known fields (or ignore)
Schema-less data:
any fields (or ignore), any types (implicit/explicit
conversion)
fit for services in-development (all internet services!)
14年1月31日金曜日

9. How/what
HowWhat
Schema-full
Schema-less
Retrospect
RDBMS,
Hive, BigQuery,
Cassandra, HBase, ...
MongoDB,
Hive(SerDe), TD,
Plain text file, ...
Prospect
Esper,
many of stream CEPs,
...
Norikra, ...
14年1月31日金曜日

10. Data size: schema & index
Logs: size is always important (xTB - xPB)
Schema:
size optimization
access optimization on memory/disk
Index:
access optimization on memory/disk
more memory/disk required
hard to distribute
14年1月31日金曜日

11. Query response improvements
of retrospection
Schema-full + indexed (RDBMS)
Query plan optimization
Schema on read
I/O and Task size optimization & scale out
Schema-less + indexed (Mongo)
mmap-ed index & data (!)
14年1月31日金曜日

12. Query response improvements
of prospection
Time window + incremental calculation
Stream processing engines
14年1月31日金曜日

13. Stream processing
and data size
No disks: reduction of failure points
Less memory:
size of just processing and I/O buffers
aggregation results
Easy to distribute:
stream duplication
stream splitting by aggregation key
14年1月31日金曜日

14. Stream processing and schema
Stream processing: query -> data
Prospective schema by queries:
Queries know required fields and its types
Unused fields can be ignored
Implicit type conversion available
Schema-less data + schema-full queries
14年1月31日金曜日

15. My goal:
Schema-less data stream
+ schema-full queries
It’s Norikra!
14年1月31日金曜日