This document discusses monitoring systems using syslog and EventMachine. It proposes building a lightweight, polyglot system that aggregates syslog events and displays metrics and visualizations using various protocols like WebSockets, Server-Sent Events, and Graphite. Event sources would send syslog messages which an EventMachine server would parse and pass to an EM:Channel. A JavaScript client could subscribe to the channel for real-time updates.

1. Monitoring with
Syslog and
EventMachine

2. About me
Patrick Huesler
Twitter/Github: phuesler
developer@wooga
Berlin, Germany

6. Wooga
Social Games
3# game developer on facebook
~ 6.5 Million users per day

7. Monitoring with
Syslog and
EventMachine

8. monitor |ˈmänəәtəәr|
verb [ with obj. ]
observe and check the progress or
quality of (something) over a period of
time; keep under systematic review:
equipment was installed to monitor air quality.
Oxford Dictionary

9. Dashboard
Dashboard
Dashboard

11. Visualization of
Ticket Sale

13. Concurrent users of
Magic Land

15. Monster World
Dashboard

17. Motivation

18. DevOps

19. Moving Target

20. Debugging

21. “Since the last deploy, the
number of signup errors
has gone up by 300 %. We
might have broken
something.”

22. Money Hose

23. Counting Things

24. Concurrent Users

25. Logins

26. Signups

27. Errors

28. A rough sketch

29. Application Application Application
Server Server Server
Event Aggregation
Dashboard

30. Criteria

31. Simple

32. Fire And Forget

33. Non blocking

34. Criteria
Simple/Lightweight
Polyglot
Fire and Forget

35. A Little Story

37. Not Supported
node.js NewRelic RPM
erlang NewRelic RPM

38. Load Balancer
haproxy
node.js node.js node.js

39. haproxy logs
haproxy[674]: 127.0.0.1:33320 [15/Oct/2003:08:32:17.654] px-http
px-http/srv1 9/0/7/14/+30 200 +243 - - ---- 3/3/3/1/0 0/0
"GET /image.iso HTTP/1.0"

40. send logs custom syslog
haproxy server
NewRelic Ruby
New Relic
Agent

41. haproxy2rpm
http://github.com/wooga/haproxy2rpm

43. Syslog

44. Syslog
• Standard logging solution for Unix/Linux
• Facility (daemon, cron, user, local0, etc.)
• Priority/Level (Alert, Critical, Error,
Warning, etc.)
• Client and server
• Since 1980

45. ... It also provides devices which
would otherwise be unable to
communicate a means to notify
administrators of problems or
performance.
http://en.wikipedia.org/wiki/Syslog

46. Syslog Format
Date Hostname Program : Message
Jan 1 12:12:12 10.245.3.99 foo[421] : this is a message

47. UDP

48. Fire And Forget

49. Checklist

50. Simple/Lightweight ✔
Polyglot ✔
Fire and Forget ✔

51. Build it

52. send message
Event Source UDP : 514
Event Aggregator
Dashboard

53. send message
Event Source UDP : 514
Syslog
forward
Dashboard Event Aggregator

54. Event Source
# man logger
logger -p local0.notice -t HOSTIDM My Message

55. Event Source
require 'syslog'
def log(message, level = :warning)
script_name = $0
syslog_option = Syslog::LOG_PID | Syslog::LOG_CONS
Syslog.open($0, syslog_option) do |s|
s.send(level, message)
end
end

56. Server
• Listen on UDP
• Or tail a log file
• receive and parse syslog message
• update a counter

57. Server
require 'eventmachine'
class Handler < EM::Connection
def receive_data(data)
log_line = SyslogParser.parse(data)
puts log_line.message
end
end
EM.run {
host = "127.0.0.1"
port = "3000"
EM::open_datagram_socket(host, port, Handler)
}

58. Syslog Parser
https://github.com/jordansissel/experiments/blob/master/
ruby/eventmachine-speed/basic.rb

59. Websocket

60. require 'em-websocket'
EM.run do
channel = EM::Channel.new
options = {:host => "0.0.0.0", :port => 8080}
EM::WebSocket.start(options) do |ws|
ws.onopen do
sid = channel.subscribe{|msg| ws.send msg}
end
ws.onclose do
channel.unsubscribe(sid)
end
end
end

61. Pass on
EM:Channel
EM.run {
channel = EM::Channel.new
# define your websocket server here
# start udp server
host = "127.0.0.1"
port = "3000"
# pass in the channel to the data handler
EM::open_datagram_socket(host, port, Handler,
channel)
}

62. New Handler
class Handler < EM::Connection
def initialize(*args)
@channel = args[0]
@counter = 0
super *args
end
def receive_data(data)
if data && data.size > 0
@counter += 1
@channel.push(@counter)
end
end
end

63. JavaScript
var socket;
var host = "ws://localhost:8080";
var socket = new WebSocket(host);
socket.onopen = function(){
console.log('open');
}
socket.onmessage = function(msg){
console.log(msg.data);
}
socket.onclose = function(){
console.log('closed');
}

64. Server-sent events
• Push only
• http://dev.w3.org/html5/eventsource/
• http://en.wikipedia.org/wiki/Server-
sent_events
• http://www.html5rocks.com/en/tutorials/
casestudies/sunlight_streamcongress.html

65. What else?

66. Graphing
http://graphite.wikidot.com/screen-shots

67. Steal from statsd
https://github.com/etsy/statsd

68. Inspiration
• http://codeascraft.etsy.com/2011/02/15/
measure-anything-measure-everything/
• http://code.flickr.com/blog/2008/10/27/
counting-timing/

69. Thank You

70. Special thanks to
• @wooga for supporting me
• @knutin for facebook faces dashboard
• @hukl for hummingbird video (http://
vimeo.com/20840998)

71. Slides
http://www.slideshare.net/wooga

72. wooga.com/jobs
WANTED
dead or alive
BACKEND DEVELOPER
REWARD
www.wooga.com/ jobs/