Monitoring with Syslog and EventMachine

  • 4,593 views
Uploaded on

Go beyond server and application stack monitoring, collect application specific metrics and push the information to interested parties. With the help of syslog, EventMachine and a few lines of Ruby …

Go beyond server and application stack monitoring, collect application specific metrics and push the information to interested parties. With the help of syslog, EventMachine and a few lines of Ruby code, you can build a language agnostic system to gather and process custom tailored reporting data.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • Thank you. It was very interesting, specially last slide was cool, If I had a chance to join your team i'd obviously join wooga.
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
4,593
On Slideshare
0
From Embeds
0
Number of Embeds
4

Actions

Shares
Downloads
0
Comments
1
Likes
19

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Monitoring with Syslog andEventMachine
  • 2. About mePatrick HueslerTwitter/Github: phueslerdeveloper@woogaBerlin, Germany
  • 3. Wooga Social Games3# game developer on facebook ~ 6.5 Million users per day
  • 4. Monitoring with Syslog andEventMachine
  • 5. monitor |ˈmänəәtəәr|verb [ with obj. ]observe and check the progress orquality of (something) over a period oftime; keep under systematic review:equipment was installed to monitor air quality. Oxford Dictionary
  • 6. DashboardDashboardDashboard
  • 7. Visualization of Ticket Sale
  • 8. Concurrent users of Magic Land
  • 9. Monster World Dashboard
  • 10. Motivation
  • 11. DevOps
  • 12. Moving Target
  • 13. Debugging
  • 14. “Since the last deploy, the number of signup errorshas gone up by 300 %. We might have broken something.”
  • 15. Money Hose
  • 16. Counting Things
  • 17. Concurrent Users
  • 18. Logins
  • 19. Signups
  • 20. Errors
  • 21. A rough sketch
  • 22. Application Application Application Server Server Server Event Aggregation Dashboard
  • 23. Criteria
  • 24. Simple
  • 25. Fire And Forget
  • 26. Non blocking
  • 27. CriteriaSimple/Lightweight Polyglot Fire and Forget
  • 28. A Little Story
  • 29. Not Supportednode.js NewRelic RPMerlang NewRelic RPM
  • 30. Load Balancer haproxynode.js node.js node.js
  • 31. haproxy logshaproxy[674]: 127.0.0.1:33320 [15/Oct/2003:08:32:17.654] px-http   px-http/srv1 9/0/7/14/+30 200 +243 - - ---- 3/3/3/1/0 0/0   "GET /image.iso HTTP/1.0"
  • 32. send logs custom sysloghaproxy server NewRelic Ruby New Relic Agent
  • 33. haproxy2rpmhttp://github.com/wooga/haproxy2rpm
  • 34. Syslog
  • 35. Syslog• Standard logging solution for Unix/Linux• Facility (daemon, cron, user, local0, etc.)• Priority/Level (Alert, Critical, Error, Warning, etc.)• Client and server• Since 1980
  • 36. ... It also provides devices which would otherwise be unable tocommunicate a means to notify administrators of problems or performance. http://en.wikipedia.org/wiki/Syslog
  • 37. Syslog FormatDate Hostname Program : MessageJan 1 12:12:12 10.245.3.99 foo[421] : this is a message
  • 38. UDP
  • 39. Fire And Forget
  • 40. Checklist
  • 41. Simple/Lightweight ✔ Polyglot ✔ Fire and Forget ✔
  • 42. Build it
  • 43. send messageEvent Source UDP : 514 Event Aggregator Dashboard
  • 44. send messageEvent Source UDP : 514 Syslog forwardDashboard Event Aggregator
  • 45. Event Source # man loggerlogger -p local0.notice -t HOSTIDM My Message
  • 46. Event Sourcerequire syslogdef log(message, level = :warning)  script_name = $0  syslog_option = Syslog::LOG_PID | Syslog::LOG_CONS  Syslog.open($0, syslog_option) do |s|    s.send(level, message)  endend
  • 47. Server• Listen on UDP• Or tail a log file• receive and parse syslog message• update a counter
  • 48. Serverrequire eventmachineclass Handler < EM::Connection  def receive_data(data)  log_line = SyslogParser.parse(data)    puts log_line.message  endendEM.run {  host = "127.0.0.1"  port = "3000"  EM::open_datagram_socket(host, port, Handler)}
  • 49. Syslog Parserhttps://github.com/jordansissel/experiments/blob/master/ ruby/eventmachine-speed/basic.rb
  • 50. Websocket
  • 51. require em-websocketEM.run do  channel = EM::Channel.new  options = {:host => "0.0.0.0", :port => 8080}  EM::WebSocket.start(options) do |ws|    ws.onopen do      sid = channel.subscribe{|msg| ws.send msg}    end    ws.onclose do      channel.unsubscribe(sid)    end  endend
  • 52. Pass on EM:ChannelEM.run {  channel = EM::Channel.new  # define your websocket server here  # start udp server  host = "127.0.0.1"  port = "3000"  # pass in the channel to the data handler  EM::open_datagram_socket(host, port, Handler,channel)}
  • 53. New Handlerclass Handler < EM::Connection  def initialize(*args)   @channel = args[0]    @counter = 0    super *args  end  def receive_data(data)   if data && data.size > 0      @counter += 1     @channel.push(@counter)    end  endend
  • 54. JavaScriptvar socket;var host = "ws://localhost:8080";var socket = new WebSocket(host);socket.onopen = function(){    console.log(open);}socket.onmessage = function(msg){    console.log(msg.data);}socket.onclose = function(){    console.log(closed);}
  • 55. Server-sent events• Push only• http://dev.w3.org/html5/eventsource/• http://en.wikipedia.org/wiki/Server- sent_events• http://www.html5rocks.com/en/tutorials/ casestudies/sunlight_streamcongress.html
  • 56. What else?
  • 57. Graphinghttp://graphite.wikidot.com/screen-shots
  • 58. Steal from statsd https://github.com/etsy/statsd
  • 59. Inspiration• http://codeascraft.etsy.com/2011/02/15/ measure-anything-measure-everything/• http://code.flickr.com/blog/2008/10/27/ counting-timing/
  • 60. Thank You
  • 61. Special thanks to• @wooga for supporting me• @knutin for facebook faces dashboard• @hukl for hummingbird video (http:// vimeo.com/20840998)
  • 62. Slideshttp://www.slideshare.net/wooga
  • 63. wooga.com/jobs WANTED dead or alive BACKEND DEVELOPER REWARD www.wooga.com/ jobs/