Presentation at 'Fluentd Meetup in Japan'

ﬂuentd@just_do_neet 1

fluentd - The Event Collector Service•‘fluentd’ product made by a Japanese famous hacker ‘Sadayuki FURUHASHI‘ (Treasure Data Inc. ) http://www.scribd.com/doc/70897187/ Fluent-event-collector-update (English) http://blog.treasure-data.com/post/ 13047440992/fluentd-the-missing-log- collector (English) http://d.hatena.ne.jp/viver/20110929/p1 (Japanese) 2

Agenda• ×• ×•• 3

Real cases 4

• - ﬂuentd• - 2012/01/31 5

•NAVER http://matome.naver.jp/ 6

•access_log••ﬂuentd + Hadoop 8

http://tech.naver.jp/blog/?p=1143 9

• - legacy system: log copy = rsync ( every day ) log analysis = batch process ( every day ) - new system : log copy = ﬂuentd ( real-time ) log analysis = Hadoop ( every 1 hour ) 10

• ( )••No More ‘Legacy Log Analysis Systems’!! 11

Apps analysis parse Apps Logs Batch Storage Apps copy rsync serialize•Apps to Logs : copy and analysis access_log(every day)•Logs to Batch : rsync of analysis results(every day)•Batch to Storage : serialize result to Storage. 12

Apps analysis Hadoop Apps Storage Clusters Apps ﬂuentd serialize•Apps to Hadoop : copy access_log using ‘ﬂuentd’ (real-time)•Hadoop to Storage : serialize result to storage. (every 1 hour) 13

fluentd Apps fluentd NameNode put to HDFS Apps fluentd DataNode fluentdDataNode DataNode Apps fluentd•Apps : input = apps log(tail) / output = send to logs(http)•DataNode : input = receive from Apps(http) / output = local log(file) (copy local logs to HDFS every few minutes) 14

hourly daily access_log access_log /a [A,B,C,D,E] /a [A,B,C,D,E] /a 5 access_log /b/a [B,D,E] [A,B,C,D,E] [B,D,E] /b 3( ( (access_log format) format) /b [B,D,E] /c/b [A,D] ( format) format) /c [A,D] /c [A,D] /c 2 .. . ... . ... . . Hadoop Hadoop•hourly :•daily : hourly 15

• am 1:00 - ﬂuentd) - Hadoop. MapReduce HDFS• 16

••• ﬂuentd + Hadoop ( + Azkaban) (Redis + MessagePack ) 19

•access_log - - Twitter/Facebook/ - Referer) - - etc.•hourly daily 20

(ﬂuentd ) Apps analysis Hadoop Apps Storage Clusters Apps ﬂuentd serialize•※ Hadoop 21

fluentd•Hadoop• → Apps Apps Hadoop1 fluentd fluentd fluentd Hadoop1 Hadoop1 Hadoop1 Apps fluentd Hadoop1 Hadoop1 Apps fluentd fluentd fluentd Hadoop1 Hadoop1 Hadoop2 Apps Apps fluentd fluentd 22

Azkaban•Linked In Hadoop http://sna-projects.com/azkaban/ 23

• am 1:00• 24

• ﬂuentd - - - 26

put to HDFS ﬂuentd Apps Apps DataNode DataNode Apps DataNode NameNode access_log copy retry• Apps→Logs DataNode Apps access_log•HDFS DataNode ﬂuentd log HDFS put 27

ﬂuentd•ﬂuentd CPU Apps 3 5 / Logs 1% 1% Apps:8GB / Logs:16GB•※ CPU 28

•tcp port• access_log 29

•Apps ﬂuentd relay - - relay• ﬂuentd 30

• fluentd 0.10.8 ver. <= 0.10.7 : fluetnd ver. >= 0.10.8 : fluentd $ ps -ewf | grep fluentd www 377 28363 0 Jan18 ? 00:34:45 fluentd #<- child process www 28363 1 0 Jan18 ? 00:00:00 fluentd #<- parent process•0.10.8 parent process kill -SIGHUP $ kill -SIGHUP 28363 31

• ﬂuentd kill -2 <pid> restart process naive )•0.10.8 kill →parent process kill -SIGHUP 32

•out_file compress gzip gzip• access.201201201636.b4b6f0be22fc3a5f1 #1. temporary file access.201201201636_0.log #2.1. no compress access.201201201636_0.log.gz #2.2. gzip compress• temporary file(#1) fluentd 33

•out_ﬁle JSON•buf_ﬁle <= 0.10.4 https://twitter.com/#!/frsyuki/status/133422825074728960 34

•input/output•※ Ruby (´ ω ) 35

Demo cases 36

classifier fluentd Hadoop jubatuswebweb web web apache fluentd analysis fluentd Mongo DB storing log data and result of analysis. hoge.js visualization hoge.js hoge.js node.js 37

•Real-time visualizing access_log to world map. <ﬂuentd + mongodb + mongotail + maptail.js(node.js) >•Real-time monitoring stats of service. <ﬂuentd + mongodb + node.js> 38

ﬂuentd•Real-time visualizing access_log to world map.•ﬂuentd + mongodb + mongo-tail + node.js(maptail.js) 39

ﬂuentd 40

MongoDB•http://www.mongodb.org/ 41

mongo-tail•This tool is included in ‘fluent-plugin-mongo’ https://github.com/fluent/fluent-plugin-mongo/tree/ master/bin•“You can tail mongo capped collection.” $ mongo-tail -h <host> -p <port> -d <db name> -c <collection name> -f 42

node.js•http://nodejs.org/ 43

maptail.js•https://github.com/stagas/maptail•“Creates a server, monitors a tail -f output for IP addresses, GeoIPs them and sends them to a map.” 44

Apps Hadoop ClustersApps mongod node.jsApps ﬂuentd mongo-tail socket.io 45

fluent-plugin-mongo$ fluent-gem install fluent-plugin-mongo <match apps.access> type mongo database hoge collection fuga host mongo1 port 27017 </match> Apps Apps fluentd fluentd fluentd Logs1 Apps fluentd Logs1 Apps fluentd fluentd fluentd mongo1 Apps Apps fluentd fluentd 46

http://dayafterneet.blogspot.com/2011/11/ﬂuent-mongodb-nodejs-web.html 47

ﬂuentd•Real-time monitoring stats of service. : user counts / frequent access pages / etc... : data refresh every few seconds.•ﬂuentd + mongodb + node.js 48

ﬂuentd 49

‘ﬂuentd + mongodb + node.js’•• JavaScript - ﬂuentd : Handling JSON-like data. (MessagePack) - mongodb : Storing JSON-like data. (BSON) - node.js : JavaScript environment. 50

Please help me. 51

•access_log / log rotate - in_tail rotate tailing symlink NG Ruby• / 52

• - shell - log rotate shell →ﬂuentd• 53

relay•access_log relay - HTTP Method = GET - Status Code = 200 - API 54

relay•in_evaltail.rb (experimental) <source> type evaltail format ...... time_format %d/%b/%Y:%H:%M:%S %z path /usr/local/apache2/logs/access_log tag access.pcweb expression record["code"] == "200" && record["method"] == "get" </source> 55

relay•in_evaltail.rb (experimental) module Fluent class EvaltailInput < TailInput Plugin.register_input(evaltail, self) config_param :expression, :string, :default => nil def receive_lines(lines) es = MultiEventStream.new lines.each {|line| begin line.rstrip! # remove n time, record = parse_line(line) if time && record if @expression if eval(@expression) es.add(time, record) end else es.add(time, record) end end rescue 56

relay• eval - in_evaltail.rb - 57

ﬂuentd• ﬂuentd - input / output - /• 58

jubatus• ...• 59

Conclusion 60

• ﬂuentd - ﬂuentd -• Ruby 61

Thank you for your attention. 62

http://a0.twimg.com	19
http://us-w1.rockmelt.com	1
https://twitter.com	1
http://www.twylah.com	1

Presentation at 'Fluentd Meetup in Japan'

by NAVER Japan on Feb 01, 2012

Accessibility

Categories

Tags

Upload Details

Usage Rights

4 Embeds 22

Statistics

Presentation at ’Fluentd Meetup in Japan’ — Presentation Transcript