Elasticsearch is a popular tool for log analytics, full text search, application monitoring, and other analytics use cases. Amazon Elasticsearch Service delivers Elasticsearch’s easy-to-use APIs and real-time capabilities along with the availability, scalability, and security required by production workloads. In this tech talk, we will provide an overview of Amazon Elasticsearch Service and review the new features in Elasticsearch 5 and Kibana 5.
Learning Objectives:
• Get an overview of Amazon Elasticsearch Service and latest features including support for ES5
• Understand how to take advantage of the new Elasticsearch 5 features
32. exec_filterの例
• モバイルサイトのアクセスログフィルタ
– キャリアごとのUID→統合UID、静的ファイルの除外
#!/usr/lib64/fluent/ruby/bin/ruby
require 'digest/sha1‘
path_filter = Regexp.new "^/(img|swf|css|js|healthcheck)/"
while line = STDIN.gets
line.chomp!
host,method,path,code,size,referer,agent,dcmuid,auuid,sbuid,spuid = line.split("¥t")
next if path_filter =~ path
uid = ""
[dcmuid, auuid, sbuid,spuid].each do |id|
if id != "-"
uid = Digest::SHA1.hexdigest id
break
end
end
puts [host,method,path,code,size,referer,agent,uid].join("¥t")
end
32
46. TreasureDataの集計例
問合せ調査 「タイムアウトばっかりでアクセスできません」
クエリ
SELECT time, restime, path
FROM game01
WHERE
unix_timestamp()-24*60*60 < time
AND uid = XXXXXXX
ORDER BY time
マイページアクセスが5秒以上かかっている
マイページアクセスが5
※指定ユーザのアクセスパスと処理 →特定ユーザの場合のみ
時間を時系列で出力 タイムアウトが発生するバグがあった
46