Monitoring with Syslog andEventMachine
About mePatrick HueslerTwitter/Github: phueslerdeveloper@woogaBerlin, Germany
Wooga        Social Games3# game developer on facebook  ~ 6.5 Million users per day
Monitoring with Syslog andEventMachine
monitor |ˈmänəәtəәr|verb [ with obj. ]observe and check the progress orquality of (something) over a period oftime; keep u...
DashboardDashboardDashboard
Visualization of  Ticket Sale
Concurrent users of   Magic Land
Monster World Dashboard
Motivation
DevOps
Moving Target
Debugging
“Since the last deploy, the number of signup errorshas gone up by 300 %. We    might have broken       something.”
Money Hose
Counting Things
Concurrent Users
Logins
Signups
Errors
A rough sketch
Application      Application      Application  Server           Server           Server              Event Aggregation    ...
Criteria
Simple
Fire And Forget
Non blocking
CriteriaSimple/Lightweight     Polyglot Fire and Forget
A Little Story
Not Supportednode.js     NewRelic RPMerlang      NewRelic RPM
Load Balancer          haproxynode.js   node.js   node.js
haproxy logshaproxy[674]: 127.0.0.1:33320 [15/Oct/2003:08:32:17.654] px-http   px-http/srv1 9/0/7/14/+30 200 +243 - - ----...
send logs   custom sysloghaproxy                      server                         NewRelic Ruby New Relic              ...
haproxy2rpmhttp://github.com/wooga/haproxy2rpm
Syslog
Syslog• Standard logging solution for Unix/Linux• Facility (daemon, cron, user, local0, etc.)• Priority/Level (Alert, Crit...
... It also provides devices which  would otherwise be unable tocommunicate a means to notify administrators of problems o...
Syslog FormatDate          Hostname Program        : MessageJan 1 12:12:12 10.245.3.99 foo[421]   :   this is a message
UDP
Fire And Forget
Checklist
Simple/Lightweight   ✔     Polyglot        ✔ Fire and Forget     ✔
Build it
send messageEvent Source    UDP : 514                              Event Aggregator                                Dashboard
send messageEvent Source    UDP : 514                                   Syslog                                      forwar...
Event Source                  # man loggerlogger -p local0.notice -t HOSTIDM My Message
Event Sourcerequire syslogdef log(message, level = :warning)  script_name = $0  syslog_option = Syslog::LOG_PID | Syslog::...
Server• Listen on UDP• Or tail a log file• receive and parse syslog message• update a counter
Serverrequire eventmachineclass Handler < EM::Connection  def receive_data(data)    log_line = SyslogParser.parse(data)   ...
Syslog Parserhttps://github.com/jordansissel/experiments/blob/master/             ruby/eventmachine-speed/basic.rb
Websocket
require em-websocketEM.run do  channel = EM::Channel.new  options = {:host => "0.0.0.0", :port => 8080}  EM::WebSocket.sta...
Pass on            EM:ChannelEM.run {  channel = EM::Channel.new  # define your websocket server here  # start udp server ...
New Handlerclass Handler < EM::Connection  def initialize(*args)   @channel = args[0]    @counter = 0    super *args  end ...
JavaScriptvar socket;var host = "ws://localhost:8080";var socket = new WebSocket(host);socket.onopen = function(){    cons...
Server-sent events• Push only• http://dev.w3.org/html5/eventsource/• http://en.wikipedia.org/wiki/Server-  sent_events• ht...
What else?
Graphinghttp://graphite.wikidot.com/screen-shots
Steal from statsd  https://github.com/etsy/statsd
Inspiration• http://codeascraft.etsy.com/2011/02/15/  measure-anything-measure-everything/• http://code.flickr.com/blog/200...
Thank You
Special thanks to• @wooga for supporting me• @knutin for facebook faces dashboard• @hukl for hummingbird video (http://  v...
Slideshttp://www.slideshare.net/wooga
wooga.com/jobs    WANTED        dead or alive   BACKEND DEVELOPER        REWARD   www.wooga.com/ jobs/
Monitoring with Syslog and EventMachine
Monitoring with Syslog and EventMachine
Monitoring with Syslog and EventMachine
Monitoring with Syslog and EventMachine
Monitoring with Syslog and EventMachine
Monitoring with Syslog and EventMachine
Monitoring with Syslog and EventMachine
Monitoring with Syslog and EventMachine
Monitoring with Syslog and EventMachine
Upcoming SlideShare
Loading in...5
×

Monitoring with Syslog and EventMachine

4,936

Published on

Go beyond server and application stack monitoring, collect application specific metrics and push the information to interested parties. With the help of syslog, EventMachine and a few lines of Ruby code, you can build a language agnostic system to gather and process custom tailored reporting data.

Published in: Technology
1 Comment
19 Likes
Statistics
Notes
No Downloads
Views
Total Views
4,936
On Slideshare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
0
Comments
1
Likes
19
Embeds 0
No embeds

No notes for slide

Monitoring with Syslog and EventMachine

  1. 1. Monitoring with Syslog andEventMachine
  2. 2. About mePatrick HueslerTwitter/Github: phueslerdeveloper@woogaBerlin, Germany
  3. 3. Wooga Social Games3# game developer on facebook ~ 6.5 Million users per day
  4. 4. Monitoring with Syslog andEventMachine
  5. 5. monitor |ˈmänəәtəәr|verb [ with obj. ]observe and check the progress orquality of (something) over a period oftime; keep under systematic review:equipment was installed to monitor air quality. Oxford Dictionary
  6. 6. DashboardDashboardDashboard
  7. 7. Visualization of Ticket Sale
  8. 8. Concurrent users of Magic Land
  9. 9. Monster World Dashboard
  10. 10. Motivation
  11. 11. DevOps
  12. 12. Moving Target
  13. 13. Debugging
  14. 14. “Since the last deploy, the number of signup errorshas gone up by 300 %. We might have broken something.”
  15. 15. Money Hose
  16. 16. Counting Things
  17. 17. Concurrent Users
  18. 18. Logins
  19. 19. Signups
  20. 20. Errors
  21. 21. A rough sketch
  22. 22. Application Application Application Server Server Server Event Aggregation Dashboard
  23. 23. Criteria
  24. 24. Simple
  25. 25. Fire And Forget
  26. 26. Non blocking
  27. 27. CriteriaSimple/Lightweight Polyglot Fire and Forget
  28. 28. A Little Story
  29. 29. Not Supportednode.js NewRelic RPMerlang NewRelic RPM
  30. 30. Load Balancer haproxynode.js node.js node.js
  31. 31. haproxy logshaproxy[674]: 127.0.0.1:33320 [15/Oct/2003:08:32:17.654] px-http   px-http/srv1 9/0/7/14/+30 200 +243 - - ---- 3/3/3/1/0 0/0   "GET /image.iso HTTP/1.0"
  32. 32. send logs custom sysloghaproxy server NewRelic Ruby New Relic Agent
  33. 33. haproxy2rpmhttp://github.com/wooga/haproxy2rpm
  34. 34. Syslog
  35. 35. Syslog• Standard logging solution for Unix/Linux• Facility (daemon, cron, user, local0, etc.)• Priority/Level (Alert, Critical, Error, Warning, etc.)• Client and server• Since 1980
  36. 36. ... It also provides devices which would otherwise be unable tocommunicate a means to notify administrators of problems or performance. http://en.wikipedia.org/wiki/Syslog
  37. 37. Syslog FormatDate Hostname Program : MessageJan 1 12:12:12 10.245.3.99 foo[421] : this is a message
  38. 38. UDP
  39. 39. Fire And Forget
  40. 40. Checklist
  41. 41. Simple/Lightweight ✔ Polyglot ✔ Fire and Forget ✔
  42. 42. Build it
  43. 43. send messageEvent Source UDP : 514 Event Aggregator Dashboard
  44. 44. send messageEvent Source UDP : 514 Syslog forwardDashboard Event Aggregator
  45. 45. Event Source # man loggerlogger -p local0.notice -t HOSTIDM My Message
  46. 46. Event Sourcerequire syslogdef log(message, level = :warning)  script_name = $0  syslog_option = Syslog::LOG_PID | Syslog::LOG_CONS  Syslog.open($0, syslog_option) do |s|    s.send(level, message)  endend
  47. 47. Server• Listen on UDP• Or tail a log file• receive and parse syslog message• update a counter
  48. 48. Serverrequire eventmachineclass Handler < EM::Connection  def receive_data(data)  log_line = SyslogParser.parse(data)    puts log_line.message  endendEM.run {  host = "127.0.0.1"  port = "3000"  EM::open_datagram_socket(host, port, Handler)}
  49. 49. Syslog Parserhttps://github.com/jordansissel/experiments/blob/master/ ruby/eventmachine-speed/basic.rb
  50. 50. Websocket
  51. 51. require em-websocketEM.run do  channel = EM::Channel.new  options = {:host => "0.0.0.0", :port => 8080}  EM::WebSocket.start(options) do |ws|    ws.onopen do      sid = channel.subscribe{|msg| ws.send msg}    end    ws.onclose do      channel.unsubscribe(sid)    end  endend
  52. 52. Pass on EM:ChannelEM.run {  channel = EM::Channel.new  # define your websocket server here  # start udp server  host = "127.0.0.1"  port = "3000"  # pass in the channel to the data handler  EM::open_datagram_socket(host, port, Handler,channel)}
  53. 53. New Handlerclass Handler < EM::Connection  def initialize(*args)   @channel = args[0]    @counter = 0    super *args  end  def receive_data(data)   if data && data.size > 0      @counter += 1     @channel.push(@counter)    end  endend
  54. 54. JavaScriptvar socket;var host = "ws://localhost:8080";var socket = new WebSocket(host);socket.onopen = function(){    console.log(open);}socket.onmessage = function(msg){    console.log(msg.data);}socket.onclose = function(){    console.log(closed);}
  55. 55. Server-sent events• Push only• http://dev.w3.org/html5/eventsource/• http://en.wikipedia.org/wiki/Server- sent_events• http://www.html5rocks.com/en/tutorials/ casestudies/sunlight_streamcongress.html
  56. 56. What else?
  57. 57. Graphinghttp://graphite.wikidot.com/screen-shots
  58. 58. Steal from statsd https://github.com/etsy/statsd
  59. 59. Inspiration• http://codeascraft.etsy.com/2011/02/15/ measure-anything-measure-everything/• http://code.flickr.com/blog/2008/10/27/ counting-timing/
  60. 60. Thank You
  61. 61. Special thanks to• @wooga for supporting me• @knutin for facebook faces dashboard• @hukl for hummingbird video (http:// vimeo.com/20840998)
  62. 62. Slideshttp://www.slideshare.net/wooga
  63. 63. wooga.com/jobs WANTED dead or alive BACKEND DEVELOPER REWARD www.wooga.com/ jobs/

×