Monitoring with Syslog and EventMachine (RailswayConf 2012)
•
1 like•2,293 views
The document discusses building monitoring dashboards using syslog and EventMachine. It proposes sending application events to a server over UDP using syslog format. The server would parse, aggregate, and push events to dashboards over server-sent events. A Sinatra app is used to stream events to the browser. Testing involves sending random test events over UDP. Existing solutions like StatsD, Graphite, and Librato Metrics are also mentioned. The document provides motivation, criteria for the solution, implementation details, and areas for further improvement.
Recommended
More Related Content
What's hot
What's hot (19)
Similar to Monitoring with Syslog and EventMachine (RailswayConf 2012)
Similar to Monitoring with Syslog and EventMachine (RailswayConf 2012) (20)
More from Wooga
More from Wooga (20)
Recently uploaded
Recently uploaded (20)
Monitoring with Syslog and EventMachine (RailswayConf 2012)
- 1. Monitoring with Syslog and EventMachine
- 2. "It’s up doesn’t mean it’s working"
- 3. Monitoring
- 6. building sketching dashboards/monitoring
- 7. outlook building sketching dashboards/monitoring
- 8. Dashboards
- 13. Motivation
- 14. Moving Target
- 15. Failure is always an option
- 17. “Since the last deploy, the number of signup errors has gone up by 300 %. We might have broken something.”
- 19. Counting Things
- 20. Concurrent Users
- 21. Signups
- 22. Logins
- 23. Errors
- 24. Sketch it
- 25. Application Application Application Server Server Server Event Aggregation Dashboard
- 26. Criteria
- 27. Simple
- 28. Polyglot
- 29. Fire And Forget
- 30. Criteria Simple/Lightweight Polyglot Fire and Forget
- 31. Background story
- 33. Not Supported node.js NewRelic RPM erlang NewRelic RPM
- 34. Load Balancer haproxy node.js node.js node.js
- 35. haproxy logs haproxy[674]: 127.0.0.1:33320 [15/Oct/2003:08:32:17.654] px-http px-http/srv1 9/0/7/14/+30 200 +243 - - ---- 3/3/3/1/0 0/0 "GET /image.iso HTTP/1.0"
- 36. send logs custom syslog haproxy server NewRelic Ruby New Relic Agent
- 39. Syslog
- 40. Syslog • Standard logging solution for Unix/Linux • Facility (daemon, cron, user, local0, etc.) • Priority/Level (Alert, Critical, Error, Warning, etc.) • Client and server • Since 1980
- 41. ... It also provides devices which would otherwise be unable to communicate a means to notify administrators of problems or performance. http://en.wikipedia.org/wiki/Syslog
- 42. Syslog Format Date Hostname Program : Message Jan 1 12:12:12 10.245.3.99 foo[421] : this is a message
- 43. Syslog client # man logger logger -p local0.notice -t HOSTIDM My Message
- 44. UDP
- 45. Fire And Forget
- 46. Simple/Lightweight Polyglot Fire and Forget
- 47. Simple/Lightweight ✔ Polyglot Fire and Forget
- 48. Simple/Lightweight ✔ Polyglot ✔ Fire and Forget
- 49. Simple/Lightweight ✔ Polyglot ✔ Fire and Forget ✔
- 50. Build it
- 51. send message Event Emitter UDP : 514 Event Aggregator Dashboard
- 52. send message Event Emitter UDP : 514 Syslog forward Dashboard Event Aggregator
- 53. Emitting Events require 'syslog' def log(message, level = :warning) script_name = $0 syslog_option = Syslog::LOG_PID | Syslog::LOG_CONS Syslog.open($0, syslog_option) do |s| s.send(level, message) end end
- 54. Server • Listen on UDP • receive and parse syslog messages • update a counter • push it out to dashboards
- 55. Server class UdpServer HOST = '127.0.0.1' PORT = '3333' def self.run EM::open_datagram_socket(HOST, PORT, Handler) end end
- 56. Handler require 'eventmachine' class Handler < EM::Connection def receive_data(data) puts "Received event: #{data.inspect}" puts SyslogParser.parse(data) end end
- 57. Syslog Parser https://github.com/jordansissel/experiments/blob/master/ ruby/eventmachine-speed/basic.rb
- 58. Web App
- 59. Event Source
- 60. Event Source Format data: Hello dashboardnn event: userlogonn data: {"username": "John123"}nn event: updaten data: {"username": "John123", "foo": "bar"}nn
- 61. Sinatra require 'sinatra' get '/stream' do content_type 'text/event-stream' stream(:keep_open) do |out| end end
- 62. class WebApp < Sinatra::Base def self.connections @connections ||= [] end get '/stream' do content_type 'text/event-stream' stream(:keep_open) do |out| WebApp.connections << out end end end
- 63. class WebApp < Sinatra::Base # send the message to all clients def self.send_event(event = {}) connections.each do |c| if c.closed connections.delete(c) else c << "event: #{event[:event]}n" c << "data: #{event[:data]}nn" end end end # here go sinatra routes ... end
- 64. # disable sinatra's autorun disable :run EM.run do WebApp.run! UdpServer.run Signal.trap("INT") { EventMachine.stop } Signal.trap("TERM") { EventMachine.stop } end
- 65. View
- 66. EventSource Basics // Open up an event source socket var source = new EventSource('/stream'); source.addEventListener('message', function(e) { console.log("message:" + e.data); }, false); source.addEventListener('open', function(e) { console.log('event source opened:' + e); }, false); source.addEventListener('error', function(e) { console.log('received an error:' + e); }, false);
- 67. Custom Events source.addEventListener('login', function(e) { console.log("login: " + e.data); }, false); source.addEventListener('loginError', function(e) { console.log("login error: " + e.data); }, false);
- 68. Testing require 'socket' desc 'send test data' task :send_test_data do socket = UDPSocket.new host = "127.0.0.1" port = "3333" events = ["message", "login", "loginError"] while true do random_index = random_index(0..2) random_value = rand(0..1000) message = "#{events[random_index]}:#{random_value}" socket.send(message, 0, host, port sleep(1) end end
- 69. https://github.com/ phuesler/yada
- 70. Missing features • Persistency • Querying • More complex aggregations (avg, timers, etc.) • Fancy visualization
- 72. Existing Solutions • https://github.com/etsy/statsd • https://github.com/noahhl/batsd • https://github.com/eric/metriks • http://square.github.com/cube/
- 75. Monitor all the things!!!!
- 76. wooga.com/jobs WANTED dead or alive BACKEND DEVELOPER REWARD www.wooga.com/ jobs/
- 77. Further Reading • http://codeascraft.etsy.com/2011/02/15/ measure-anything-measure-everything/ • http://code.flickr.com/blog/2008/10/27/ counting-timing • https://speakerdeck.com/u/roidrage/p/ metrics-monitoring-logging
- 79. Credits Cliff (Flickr) 401K (Flickr) Scott Kidder (Flickr)