Monitoring with Syslog andEventMachine
"It’s up doesn’t mean it’s working"
Monitoring
dashboards/monitoring
sketchingdashboards/monitoring
building     sketchingdashboards/monitoring
outlook      building     sketchingdashboards/monitoring
Dashboards
Motivation
Moving Target
Failure is always    an option
“Since the last deploy,the number of signuperrors has gone up by300 %. We might have broken something.”
Counting Things
Concurrent  Users
Signups
Logins
Errors
Sketch it
Application      Application      Application  Server           Server           Server              Event Aggregation    ...
Criteria
Simple
Polyglot
Fire And Forget
CriteriaSimple/Lightweight     Polyglot Fire and Forget
Background story
Not Supportednode.js      NewRelic RPMerlang       NewRelic RPM
Load Balancer          haproxynode.js   node.js   node.js
haproxy logshaproxy[674]: 127.0.0.1:33320 [15/Oct/2003:08:32:17.654] px-http   px-http/srv1 9/0/7/14/+30 200 +243 - - ----...
send logs   custom sysloghaproxy                      server                         NewRelic Ruby New Relic              ...
haproxy2rpmhttp://github.com/wooga/haproxy2rpm
Syslog
Syslog• Standard logging solution for Unix/Linux• Facility (daemon, cron, user, local0, etc.)• Priority/Level (Alert, Crit...
... It also provides devices which would otherwise be  unable to communicate ameans to notify administratorsof problems or...
Syslog FormatDate          Hostname Program        : MessageJan 1 12:12:12 10.245.3.99 foo[421]   :   this is a message
Syslog client                  # man loggerlogger -p local0.notice -t HOSTIDM My Message
UDP
Fire And Forget
Simple/Lightweight     Polyglot Fire and Forget
Simple/Lightweight   ✔     Polyglot Fire and Forget
Simple/Lightweight   ✔     Polyglot        ✔ Fire and Forget
Simple/Lightweight   ✔     Polyglot        ✔ Fire and Forget     ✔
Build it
send messageEvent Emitter    UDP : 514                               Event Aggregator                                 Dash...
send messageEvent Emitter    UDP : 514                                    Syslog                                       for...
Emitting Eventsrequire syslogdef log(message, level = :warning)  script_name = $0  syslog_option = Syslog::LOG_PID | Syslo...
Server• Listen on UDP• receive and parse syslog messages• update a counter• push it out to dashboards
Serverclass UdpServer  HOST = 127.0.0.1  PORT = 3333  def self.run    EM::open_datagram_socket(HOST,                      ...
Handlerrequire eventmachineclass Handler < EM::Connection  def receive_data(data)    puts "Received event: #{data.inspect}...
Syslog Parserhttps://github.com/jordansissel/experiments/blob/master/             ruby/eventmachine-speed/basic.rb
Web App
Event Source
Event Source            Formatdata: Hello dashboardnnevent: userlogonndata: {"username": "John123"}nnevent: updatendata: {...
Sinatrarequire sinatraget /stream do  content_type text/event-stream  stream(:keep_open) do |out|  endend
class WebApp < Sinatra::Base  def self.connections    @connections ||= []  end  get /stream do    content_type text/event-...
class WebApp < Sinatra::Base  # send the message to all clients  def self.send_event(event = {})    connections.each do |c...
# disable sinatras autorundisable :runEM.run do  WebApp.run!  UdpServer.run  Signal.trap("INT") { EventMachine.stop }  Sig...
View
EventSource Basics// Open up an event source socketvar source = new EventSource(/stream); source.addEventListener(message,...
Custom Eventssource.addEventListener(login, function(e){  console.log("login: " + e.data);}, false);source.addEventListene...
Testingrequire socketdesc send test datatask :send_test_data do  socket = UDPSocket.new  host = "127.0.0.1"  port = "3333"...
https://github.com/   phuesler/yada
Missing features• Persistency• Querying• More complex aggregations (avg, timers,  etc.)• Fancy visualization
Existing Solutions• https://github.com/etsy/statsd• https://github.com/noahhl/batsd• https://github.com/eric/metriks• http...
Graphitehttp://graphite.wikidot.com/screen-shots
Librato Metrics https://metrics.librato.com/
Monitor all the  things!!!!
wooga.com/jobs    WANTED        dead or alive   BACKEND DEVELOPER        REWARD   www.wooga.com/ jobs/
Further Reading• http://codeascraft.etsy.com/2011/02/15/  measure-anything-measure-everything/• http://code.flickr.com/blog...
Slideshttp://www.slideshare.net/wooga
CreditsCliff (Flickr)401K (Flickr)Scott Kidder (Flickr)
Monitoring  with  Syslog and EventMachine (RailswayConf 2012)
Monitoring  with  Syslog and EventMachine (RailswayConf 2012)
Monitoring  with  Syslog and EventMachine (RailswayConf 2012)
Monitoring  with  Syslog and EventMachine (RailswayConf 2012)
Monitoring  with  Syslog and EventMachine (RailswayConf 2012)
Monitoring  with  Syslog and EventMachine (RailswayConf 2012)
Monitoring  with  Syslog and EventMachine (RailswayConf 2012)
Monitoring  with  Syslog and EventMachine (RailswayConf 2012)
Monitoring  with  Syslog and EventMachine (RailswayConf 2012)
Upcoming SlideShare
Loading in …5
×

Monitoring with Syslog and EventMachine (RailswayConf 2012)

3,126 views

Published on

Published in: Technology, Education
1 Comment
1 Like
Statistics
Notes
No Downloads
Views
Total views
3,126
On SlideShare
0
From Embeds
0
Number of Embeds
86
Actions
Shares
0
Downloads
33
Comments
1
Likes
1
Embeds 0
No embeds

No notes for slide

Monitoring with Syslog and EventMachine (RailswayConf 2012)

  1. 1. Monitoring with Syslog andEventMachine
  2. 2. "It’s up doesn’t mean it’s working"
  3. 3. Monitoring
  4. 4. dashboards/monitoring
  5. 5. sketchingdashboards/monitoring
  6. 6. building sketchingdashboards/monitoring
  7. 7. outlook building sketchingdashboards/monitoring
  8. 8. Dashboards
  9. 9. Motivation
  10. 10. Moving Target
  11. 11. Failure is always an option
  12. 12. “Since the last deploy,the number of signuperrors has gone up by300 %. We might have broken something.”
  13. 13. Counting Things
  14. 14. Concurrent Users
  15. 15. Signups
  16. 16. Logins
  17. 17. Errors
  18. 18. Sketch it
  19. 19. Application Application Application Server Server Server Event Aggregation Dashboard
  20. 20. Criteria
  21. 21. Simple
  22. 22. Polyglot
  23. 23. Fire And Forget
  24. 24. CriteriaSimple/Lightweight Polyglot Fire and Forget
  25. 25. Background story
  26. 26. Not Supportednode.js NewRelic RPMerlang NewRelic RPM
  27. 27. Load Balancer haproxynode.js node.js node.js
  28. 28. haproxy logshaproxy[674]: 127.0.0.1:33320 [15/Oct/2003:08:32:17.654] px-http   px-http/srv1 9/0/7/14/+30 200 +243 - - ---- 3/3/3/1/0 0/0   "GET /image.iso HTTP/1.0"
  29. 29. send logs custom sysloghaproxy server NewRelic Ruby New Relic Agent
  30. 30. haproxy2rpmhttp://github.com/wooga/haproxy2rpm
  31. 31. Syslog
  32. 32. Syslog• Standard logging solution for Unix/Linux• Facility (daemon, cron, user, local0, etc.)• Priority/Level (Alert, Critical, Error, Warning, etc.)• Client and server• Since 1980
  33. 33. ... It also provides devices which would otherwise be unable to communicate ameans to notify administratorsof problems or performance. http://en.wikipedia.org/wiki/Syslog
  34. 34. Syslog FormatDate Hostname Program : MessageJan 1 12:12:12 10.245.3.99 foo[421] : this is a message
  35. 35. Syslog client # man loggerlogger -p local0.notice -t HOSTIDM My Message
  36. 36. UDP
  37. 37. Fire And Forget
  38. 38. Simple/Lightweight Polyglot Fire and Forget
  39. 39. Simple/Lightweight ✔ Polyglot Fire and Forget
  40. 40. Simple/Lightweight ✔ Polyglot ✔ Fire and Forget
  41. 41. Simple/Lightweight ✔ Polyglot ✔ Fire and Forget ✔
  42. 42. Build it
  43. 43. send messageEvent Emitter UDP : 514 Event Aggregator Dashboard
  44. 44. send messageEvent Emitter UDP : 514 Syslog forward Dashboard Event Aggregator
  45. 45. Emitting Eventsrequire syslogdef log(message, level = :warning)  script_name = $0  syslog_option = Syslog::LOG_PID | Syslog::LOG_CONS  Syslog.open($0, syslog_option) do |s|    s.send(level, message)  endend
  46. 46. Server• Listen on UDP• receive and parse syslog messages• update a counter• push it out to dashboards
  47. 47. Serverclass UdpServer HOST = 127.0.0.1 PORT = 3333 def self.run EM::open_datagram_socket(HOST, PORT, Handler) endend
  48. 48. Handlerrequire eventmachineclass Handler < EM::Connection def receive_data(data) puts "Received event: #{data.inspect}" puts SyslogParser.parse(data) endend
  49. 49. Syslog Parserhttps://github.com/jordansissel/experiments/blob/master/ ruby/eventmachine-speed/basic.rb
  50. 50. Web App
  51. 51. Event Source
  52. 52. Event Source Formatdata: Hello dashboardnnevent: userlogonndata: {"username": "John123"}nnevent: updatendata: {"username": "John123", "foo": "bar"}nn
  53. 53. Sinatrarequire sinatraget /stream do content_type text/event-stream stream(:keep_open) do |out| endend
  54. 54. class WebApp < Sinatra::Base def self.connections @connections ||= [] end get /stream do content_type text/event-stream stream(:keep_open) do |out| WebApp.connections << out end endend
  55. 55. class WebApp < Sinatra::Base # send the message to all clients def self.send_event(event = {}) connections.each do |c| if c.closed connections.delete(c) else c << "event: #{event[:event]}n" c << "data: #{event[:data]}nn" end end end # here go sinatra routes ...end
  56. 56. # disable sinatras autorundisable :runEM.run do WebApp.run! UdpServer.run Signal.trap("INT") { EventMachine.stop } Signal.trap("TERM") { EventMachine.stop }end
  57. 57. View
  58. 58. EventSource Basics// Open up an event source socketvar source = new EventSource(/stream); source.addEventListener(message, function(e) { console.log("message:" + e.data); }, false); source.addEventListener(open, function(e) { console.log(event source opened: + e); }, false); source.addEventListener(error, function(e) { console.log(received an error: + e); }, false);
  59. 59. Custom Eventssource.addEventListener(login, function(e){ console.log("login: " + e.data);}, false);source.addEventListener(loginError,function(e) { console.log("login error: " + e.data);}, false);
  60. 60. Testingrequire socketdesc send test datatask :send_test_data do socket = UDPSocket.new host = "127.0.0.1" port = "3333" events = ["message", "login", "loginError"] while true do random_index = random_index(0..2) random_value = rand(0..1000) message = "#{events[random_index]}:#{random_value}" socket.send(message, 0, host, port sleep(1) endend
  61. 61. https://github.com/ phuesler/yada
  62. 62. Missing features• Persistency• Querying• More complex aggregations (avg, timers, etc.)• Fancy visualization
  63. 63. Existing Solutions• https://github.com/etsy/statsd• https://github.com/noahhl/batsd• https://github.com/eric/metriks• http://square.github.com/cube/
  64. 64. Graphitehttp://graphite.wikidot.com/screen-shots
  65. 65. Librato Metrics https://metrics.librato.com/
  66. 66. Monitor all the things!!!!
  67. 67. wooga.com/jobs WANTED dead or alive BACKEND DEVELOPER REWARD www.wooga.com/ jobs/
  68. 68. Further Reading• http://codeascraft.etsy.com/2011/02/15/ measure-anything-measure-everything/• http://code.flickr.com/blog/2008/10/27/ counting-timing• https://speakerdeck.com/u/roidrage/p/ metrics-monitoring-logging
  69. 69. Slideshttp://www.slideshare.net/wooga
  70. 70. CreditsCliff (Flickr)401K (Flickr)Scott Kidder (Flickr)

×