Monitoring with Syslog and EventMachine

Monitoring with Syslog andEventMachine

About mePatrick HueslerTwitter/Github: phueslerdeveloper@woogaBerlin, Germany

Wooga Social Games3# game developer on facebook ~ 6.5 Million users per day

Monitoring with Syslog andEventMachine

monitor |ˈmänəәtəәr|verb [ with obj. ]observe and check the progress orquality of (something) over a period oftime; keep under systematic review:equipment was installed to monitor air quality. Oxford Dictionary

DashboardDashboardDashboard

Visualization of Ticket Sale

Concurrent users of Magic Land

Monster World Dashboard

Motivation

DevOps

Moving Target

Debugging

“Since the last deploy, the number of signup errorshas gone up by 300 %. We might have broken something.”

Money Hose

Counting Things

Concurrent Users

Logins

Signups

Errors

A rough sketch

Application Application Application Server Server Server Event Aggregation Dashboard

Criteria

Simple

Fire And Forget

Non blocking

CriteriaSimple/Lightweight Polyglot Fire and Forget

A Little Story

Not Supportednode.js NewRelic RPMerlang NewRelic RPM

Load Balancer haproxynode.js node.js node.js

haproxy logshaproxy[674]: 127.0.0.1:33320 [15/Oct/2003:08:32:17.654] px-http px-http/srv1 9/0/7/14/+30 200 +243 - - ---- 3/3/3/1/0 0/0 "GET /image.iso HTTP/1.0"

send logs custom sysloghaproxy server NewRelic Ruby New Relic Agent

haproxy2rpmhttp://github.com/wooga/haproxy2rpm

Syslog

Syslog• Standard logging solution for Unix/Linux• Facility (daemon, cron, user, local0, etc.)• Priority/Level (Alert, Critical, Error, Warning, etc.)• Client and server• Since 1980

... It also provides devices which would otherwise be unable tocommunicate a means to notify administrators of problems or performance. http://en.wikipedia.org/wiki/Syslog

Syslog FormatDate Hostname Program : MessageJan 1 12:12:12 10.245.3.99 foo[421] : this is a message

Fire And Forget

Checklist

Simple/Lightweight ✔ Polyglot ✔ Fire and Forget ✔

Build it

send messageEvent Source UDP : 514 Event Aggregator Dashboard

send messageEvent Source UDP : 514 Syslog forwardDashboard Event Aggregator

Event Source # man loggerlogger -p local0.notice -t HOSTIDM My Message

Event Sourcerequire syslogdef log(message, level = :warning) script_name = $0 syslog_option = Syslog::LOG_PID | Syslog::LOG_CONS Syslog.open($0, syslog_option) do |s| s.send(level, message) endend

Server• Listen on UDP• Or tail a log ﬁle• receive and parse syslog message• update a counter

Serverrequire eventmachineclass Handler < EM::Connection def receive_data(data) log_line = SyslogParser.parse(data) puts log_line.message endendEM.run { host = "127.0.0.1" port = "3000" EM::open_datagram_socket(host, port, Handler)}

Syslog Parserhttps://github.com/jordansissel/experiments/blob/master/ ruby/eventmachine-speed/basic.rb

Websocket

require em-websocketEM.run do channel = EM::Channel.new options = {:host => "0.0.0.0", :port => 8080} EM::WebSocket.start(options) do |ws| ws.onopen do sid = channel.subscribe{|msg| ws.send msg} end ws.onclose do channel.unsubscribe(sid) end endend

Pass on EM:ChannelEM.run { channel = EM::Channel.new # define your websocket server here # start udp server host = "127.0.0.1" port = "3000" # pass in the channel to the data handler EM::open_datagram_socket(host, port, Handler,channel)}

New Handlerclass Handler < EM::Connection def initialize(*args) @channel = args[0] @counter = 0 super *args end def receive_data(data) if data && data.size > 0 @counter += 1 @channel.push(@counter) end endend

JavaScriptvar socket;var host = "ws://localhost:8080";var socket = new WebSocket(host);socket.onopen = function(){ console.log(open);}socket.onmessage = function(msg){ console.log(msg.data);}socket.onclose = function(){ console.log(closed);}

Server-sent events• Push only• http://dev.w3.org/html5/eventsource/• http://en.wikipedia.org/wiki/Server- sent_events• http://www.html5rocks.com/en/tutorials/ casestudies/sunlight_streamcongress.html

What else?

Graphinghttp://graphite.wikidot.com/screen-shots

Steal from statsd https://github.com/etsy/statsd

Inspiration• http://codeascraft.etsy.com/2011/02/15/ measure-anything-measure-everything/• http://code.ﬂickr.com/blog/2008/10/27/ counting-timing/

Thank You

Special thanks to• @wooga for supporting me• @knutin for facebook faces dashboard• @hukl for hummingbird video (http:// vimeo.com/20840998)

Slideshttp://www.slideshare.net/wooga

wooga.com/jobs WANTED dead or alive BACKEND DEVELOPER REWARD www.wooga.com/ jobs/

Monitoring with Syslog and EventMachine

by Wooga

Accessibility

Categories

Upload Details

Usage Rights

2 Embeds 2

Statistics

Monitoring with Syslog and EventMachine Presentation Transcript

http://a0.twimg.com	1
https://si0.twimg.com	1