1.
Monitoring with Syslog andEventMachine

2.
About mePatrick HueslerTwitter/Github: phueslerdeveloper@woogaBerlin, Germany

3.
Wooga Social Games3# game developer on facebook ~ 6.5 Million users per day

4.
Monitoring with Syslog andEventMachine

5.
monitor |ˈmänəәtəәr|verb [ with obj. ]observe and check the progress orquality of (something) over a period oftime; keep under systematic review:equipment was installed to monitor air quality. Oxford Dictionary

6.
DashboardDashboardDashboard

7.
Visualization of Ticket Sale

8.
Concurrent users of Magic Land

9.
Monster World Dashboard

10.
Motivation

11.
DevOps

12.
Moving Target

13.
Debugging

14.
“Since the last deploy, the number of signup errorshas gone up by 300 %. We might have broken something.”

15.
Money Hose

16.
Counting Things

17.
Concurrent Users

18.
Logins

19.
Signups

20.
Errors

21.
A rough sketch

22.
Application Application Application Server Server Server Event Aggregation Dashboard

23.
Criteria

24.
Simple

25.
Fire And Forget

26.
Non blocking

27.
CriteriaSimple/Lightweight Polyglot Fire and Forget

28.
A Little Story

29.
Not Supportednode.js NewRelic RPMerlang NewRelic RPM

30.
Load Balancer haproxynode.js node.js node.js

31.
haproxy logshaproxy[674]: 127.0.0.1:33320 [15/Oct/2003:08:32:17.654] px-http px-http/srv1 9/0/7/14/+30 200 +243 - - ---- 3/3/3/1/0 0/0 "GET /image.iso HTTP/1.0"

32.
send logs custom sysloghaproxy server NewRelic Ruby New Relic Agent

33.
haproxy2rpmhttp://github.com/wooga/haproxy2rpm

34.
Syslog

35.
Syslog• Standard logging solution for Unix/Linux• Facility (daemon, cron, user, local0, etc.)• Priority/Level (Alert, Critical, Error, Warning, etc.)• Client and server• Since 1980

36.
... It also provides devices which would otherwise be unable tocommunicate a means to notify administrators of problems or performance. http://en.wikipedia.org/wiki/Syslog

37.
Syslog FormatDate Hostname Program : MessageJan 1 12:12:12 10.245.3.99 foo[421] : this is a message

38.
UDP

39.
Fire And Forget

40.
Checklist

41.
Simple/Lightweight ✔ Polyglot ✔ Fire and Forget ✔

42.
Build it

43.
send messageEvent Source UDP : 514 Event Aggregator Dashboard

44.
send messageEvent Source UDP : 514 Syslog forwardDashboard Event Aggregator

45.
Event Source # man loggerlogger -p local0.notice -t HOSTIDM My Message

46.
Event Sourcerequire syslogdef log(message, level = :warning) script_name = $0 syslog_option = Syslog::LOG_PID | Syslog::LOG_CONS Syslog.open($0, syslog_option) do |s| s.send(level, message) endend

47.
Server• Listen on UDP• Or tail a log file• receive and parse syslog message• update a counter

48.
Serverrequire eventmachineclass Handler < EM::Connection def receive_data(data) log_line = SyslogParser.parse(data) puts log_line.message endendEM.run { host = "127.0.0.1" port = "3000" EM::open_datagram_socket(host, port, Handler)}

49.
Syslog Parserhttps://github.com/jordansissel/experiments/blob/master/ ruby/eventmachine-speed/basic.rb

50.
Websocket

51.
require em-websocketEM.run do channel = EM::Channel.new options = {:host => "0.0.0.0", :port => 8080} EM::WebSocket.start(options) do |ws| ws.onopen do sid = channel.subscribe{|msg| ws.send msg} end ws.onclose do channel.unsubscribe(sid) end endend

52.
Pass on EM:ChannelEM.run { channel = EM::Channel.new # define your websocket server here # start udp server host = "127.0.0.1" port = "3000" # pass in the channel to the data handler EM::open_datagram_socket(host, port, Handler,channel)}

53.
New Handlerclass Handler < EM::Connection def initialize(*args) @channel = args[0] @counter = 0 super *args end def receive_data(data) if data && data.size > 0 @counter += 1 @channel.push(@counter) end endend

54.
JavaScriptvar socket;var host = "ws://localhost:8080";var socket = new WebSocket(host);socket.onopen = function(){ console.log(open);}socket.onmessage = function(msg){ console.log(msg.data);}socket.onclose = function(){ console.log(closed);}

55.
Server-sent events• Push only• http://dev.w3.org/html5/eventsource/• http://en.wikipedia.org/wiki/Server- sent_events• http://www.html5rocks.com/en/tutorials/ casestudies/sunlight_streamcongress.html

56.
What else?

57.
Graphinghttp://graphite.wikidot.com/screen-shots

58.
Steal from statsd https://github.com/etsy/statsd

59.
Inspiration• http://codeascraft.etsy.com/2011/02/15/ measure-anything-measure-everything/• http://code.flickr.com/blog/2008/10/27/ counting-timing/

60.
Thank You

61.
Special thanks to• @wooga for supporting me• @knutin for facebook faces dashboard• @hukl for hummingbird video (http:// vimeo.com/20840998)

62.
Slideshttp://www.slideshare.net/wooga

63.
wooga.com/jobs WANTED dead or alive BACKEND DEVELOPER REWARD www.wooga.com/ jobs/