More Related Content
Similar to Fluentd Casual Talks LT #fluentd #fluentdcasual
Similar to Fluentd Casual Talks LT #fluentd #fluentdcasual (20)
Fluentd Casual Talks LT #fluentd #fluentdcasual
- 2. WORK
ScalaJP
backend system
data reporting
aws
FAVORITE PLUGINS TWITTER
input: tail Hitoshi ASAI
buffer: memory @hito_asa
プログラマですよ。
output: exec machida, tokyo, jp
#fluentdcasual
- 6. SYSTEM ARCHITECTURE
EXEC
LOG FILE FORWARD
FLUENTD
FLUENTD
SCRIPT
RUBY
FLUENTD
PHP
WORKER
WEB SCALA
REDIS
API
#fluentdcasual
- 8. SYSTEM SCALE
scale of service
133 MILLION REQUESTS / DAY
7.6 MILLION PICTURES
6.5 MILLION USERS
#fluentdcasual
- 10. SYSTEM SCALE
scale of servers WEB
18 WEB SRVS
6 REDIS SRVS
WORKER
4 API SRVS
4 WORKER SRVS
API
REDIS
#fluentdcasual
- 21. IMPLEMENTATION
recommendation script (ruby) 43 LINES
#!/bin/env ruby
require 'logger'
require 'redis'
require 'redis/distributed'
VIEW_PREFIX = 'r_user_gazo_view_'
RELATION_PREFIX = 'r_gazo_relation_'
LOGGER = Logger.new("#{ENV['FLUENTD_HOME']}/logs/gazo_recommend.log", 1, 100 * 1024 * 1024)
REDISM = Redis.new(:host => "xxx", :port => 6380)
REDISS = Redis::Distributed.new(["redis://xxx:6380", "redis://xxx:6380", "redis://xxx:6380", "redis://xxx:6380", "redis://xxx:6380"])
def calc(time, user_id, gazo_id)
time_s = time.strftime('%Y-%m-%d %H:%M:%S')
view_key = VIEW_PREFIX + user_id.to_s
key_org = RELATION_PREFIX + gazo_id.to_s
if REDISS.lrange(view_key, 0, 0) != [gazo_id] then
REDISM.lpush(view_key, gazo_id)
REDISM.expire(view_key, 7200)
REDISM.expire(key_org, 28800)
rel = REDISS.lrange(view_key, 0, 3).find_all{|r| r != gazo_id.to_s}.uniq.reverse.take(2).reverse
rel.each do |r|
key_rel = RELATION_PREFIX + r.to_s
score = REDISM.hincrby(key_rel, gazo_id.to_s, 1)
REDISM.expire(key_rel, 28800)
LOGGER.info("#{time_s} - #{user_id.to_s} : #{r.to_s} => #{gazo_id.to_s} score: #{score}")
score = REDISM.hincrby(key_org, r.to_s, 1)
LOGGER.info("#{time_s} - #{user_id.to_s} : #{gazo_id.to_s} => #{r.to_s} score: #{score}")
end
end
end
while line = gets
begin
l = line.encode!('UTF-8', 'UTF-8', :invalid => :replace).split("¥t")
if l[1] == 'default' && l[2] == 'pic' && l[3] == 'index' then
time = l[0]
user_id = l[5]
gazo_id = l[6]
if user_id && gazo_id then
calc Time.at(time.to_i), user_id.to_i, gazo_id.to_i rescue nil
end
end
rescue => ex
LOGGER.error("error: #{ex}")
end
end
#fluentdcasual
- 23. LOAD OF AGENT PROCESS
load average
リリース時から目立った変化はなし。
#fluentdcasual
- 24. LOAD OF AGENT PROCESS
memory usage (rss)
0.10.15 => 0.10.19 へのバージョンアップで改善。
#fluentdcasual
- 25. LOAD OF AGENT PROCESS
memory usage (rss) - weekly
上昇傾向もありません。
#fluentdcasual
- 28. TROUBLES
あとは特にないです。
とってもカジュアル。とっても安定。
#fluentdcasual
