The document discusses real-time log analytics using Cloudera, covering concepts like big data, log analytics, and the architecture of a log analytics system. It details the configuration of tools such as Flume, Morphlines, and Solr, alongside the creation of dashboards using Hue for visualizing log data. Overall, it serves as a guide for implementing log analytics in a big data environment.

1. REAL-TIME LOG ANALYTICS
WITH CLOUDERA
REAL-TIME LOG ANALYTICS
WITH CLOUDERA
H H U TR NG
S o f t w a r e A r c h i t e c t

2. NỘI DUNG TRÌNH BÀY
Big Data là gì?
Bài toán Log Analytics
Mô hình hệ thống Log Analytics
Cài đặt Cloudera và cấu hình ban đầu
Cấu hình Flume, Morphlines và Solr
Hue Dashboard
© 2003 Hyperlogy Corporation. All rights reserved 2

3. BIGDATA LÀ GÌ?
© 2003 Hyperlogy Corporation. All rights reserved 3

4. BIGDATA LÀ GÌ?
© 2003 Hyperlogy Corporation. All rights reserved 4

5. BÀI TOÁN LOG ANALYTICS
© 2003 Hyperlogy Corporation. All rights reserved 5

6. Nginx:
access_log /var/log/nginx/loganalyticscloudera-access.log combined;
Apache:
CustomLog /var/log/httpd/loganalyticscloudera-access.log combined
51.255.71.131 - - [11/Dec/2017:22:27:10 +0700] "GET / HTTP/1.1" 200 16009 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.2;
+http://ahrefs.com/robot/)"
27.73.252.43 - - [11/Dec/2017:22:27:52 +0700] "GET / HTTP/1.1" 200 12756 "https://www.google.com.vn/" "Mozilla/5.0 (Windows
NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36“,
BÀI TOÁN LOG ANALYTICS
© 2003 Hyperlogy Corporation. All rights reserved 6

7. Cloudera
Access
Logs
Flume
Sink
Flume
Morphlines
Sink
Flume
Source
MÔ HÌNH HỆ THỐNG LOG ANALYTICS
© 2003 Hyperlogy Corporation. All rights reserved 7

8. FLUME
© 2003 Hyperlogy Corporation. All rights reserved 8

9. FLUME
© 2003 Hyperlogy Corporation. All rights reserved 9

10. FLUME
© 2003 Hyperlogy Corporation. All rights reserved 10

11. CARD BANKING
MORPHLINES
© 2003 Hyperlogy Corporation. All rights reserved 11

12. Combined Log
Flume Agent
Morphline
Solr Sink
readCSV loadSolrsplituserAgent geoIP
Solr
HDFS
GeoIP split
UserAgent
ReadCSV
MORPHLINES
© 2003 Hyperlogy Corporation. All rights reserved 12

13. SOLR
© 2003 Hyperlogy Corporation. All rights reserved 13

14. HUE DASHBOARD
© 2003 Hyperlogy Corporation. All rights reserved 14

15. HUE DASHBOARD
© 2003 Hyperlogy Corporation. All rights reserved 15

16. Creating Real-Time Search Dashboards using Apache Solr,
Hue, Flume and Cloudera Morphlines
How-to: Do Real-Time Log Analytics with Apache Kafka,
Cloudera Search, and Hue
Flume 1.8.0 User Guide
Morphlines Introduction
Morphlines Reference Guide
Log analytics cloudera
REFERENCE
© 2003 Hyperlogy Corporation. All rights reserved 16