Is ruby logger thread(process)-safe? at RubyConf 2013
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Is ruby logger thread(process)-safe? at RubyConf 2013

on

  • 3,982 views

 

Statistics

Views

Total Views
3,982
Views on SlideShare
2,183
Embed Views
1,799

Actions

Likes
5
Downloads
0
Comments
0

10 Embeds 1,799

http://blog.livedoor.jp 1718
http://eventifier.com 46
http://eventifier.co 21
http://cloud.feedly.com 4
http://ec2-54-243-189-159.compute-1.amazonaws.com 3
http://www.eventifier.com 2
https://twitter.com 2
http://www.newsblur.com 1
http://news.google.com 1
http://webcache.googleusercontent.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Is ruby logger thread(process)-safe? at RubyConf 2013 Presentation Transcript

  • 1. Is Ruby Logger Thread(Process)safe? RubyConf 2013 @Miami SEO Naotoshi (@sonots) 2013/11/09
  • 2. Naotoshi SEO like Now-toshi Say-oh @sonots (twitter, github) Dev Engineer for Operations Became as a commiter of Fluentd
  • 3. •realtime pipelining of log streams •plugins written in ruby
  • 4. Log Management Tool •syslog-ng / rsyslog Only collection Not Free •splunk No Plugin •scribed Java Plugin •flumeNG Ruby Plugin (Not Gem) •logstash •fluentd Ruby Plugin (Gem Support)
  • 5. Try the fluentd! It s very nice tool, check it out! http://fluentd.org/
  • 6. Today s Talk
  • 7. Logger
  • 8. We needed a Logger which works safely in multiprocess for Fluentd
  • 9. Examined Ruby Logger • 1) Will logs not be mixtured in multi-threads? • 2) Will logs not be mixtured in multi-processes? • 3) Does log rotation work safely in multi-threads? • 4) Does log rotation work safely in multi-processes?
  • 10. 1) Will logs not be mixtured in multi-threads? Code require 'logger' require 'parallel' logger = Logger.new("/tmp/test.log") Parallel.map(['a', 'b'], :in_threads => 2) do |letter| 10000.times do logger.info letter * 5000 end end Result $ egrep -e 'ab' -e 'ba' /tmp/test.log [empty] OK
  • 11. WHY? logger.rb#L559 def write(message) begin @mutex.synchronize do ... @dev.write(message) ... end Ruby’s Logger is taking mutex to be thread-safe.
  • 12. 2) Will logs not be mixtured in multi-processes? Code require 'logger' require 'parallel' logger = Logger.new("/tmp/test.log") Parallel.map(['a', 'b'], :in_processes => 2) do |letter| 10000.times do logger.info letter * 5000 end end Result $ egrep -e 'ab' -e 'ba' /tmp/test.log [empty] OK
  • 13. Really? Why? Mutex does not work to exclusively lock in multiprocesses environment. Atomic/non-atomic: A write is atomic if the whole amount written in one operation is not interleaved with data from any other process. This is useful when there are multiple writers sending data to a single reader. Applications need to know how large a write request can be expected to be performed atomically. This maximum is called {PIPE_BUF}. This volume of IEEE Std 1003.1-2001 does not say whether write requests for more than {PIPE_BUF} bytes are atomic, but requires that writes of {PIPE_BUF} or fewer bytes shall be atomic. But, Linux system call write(2) itself is atomic as long as writing to a local file. OK
  • 14. 3) Does log rotation work safely multi-threads? Code require 'logger' require 'parallel' logger = Logger.new("/tmp/test.log", 3, 1024 * 10) Parallel.map(['a', 'b'], :in_threads => 2) do |letter| 10000.times do logger.info letter * 5000 end end Result $ ls -l /tmp/test* -rw-r--r-- 1 sonots sonots 10806 9月 29 23:03 /tmp/test.log -rw-r--r-- 1 sonots sonots 10806 9月 29 23:03 /tmp/test.log.0 -rw-r--r-- 1 sonots sonots 10806 9月 29 23:03 /tmp/test.log.1 No Error, OK
  • 15. 4) Does log rotation work safely in multi-processes? Code require 'logger' require 'parallel' logger = Logger.new("/tmp/test.log", 3, 1024 * 10) Parallel.map(['a', 'b'], :in_processes => 2) do |letter| 10000.times do logger.info letter * 5000 end end Result log log log log ... writing failed. closed stream shifting failed. closed stream writing failed. closed stream shifting failed. closed stream Oops, Bad!
  • 16. WHY? logger.rb#L559 def write(message) begin @mutex.synchronize do ... check_shift_log ... end Ruby’s Logger is taking mutex to be thread-safe, but was not treating anything for multi-processes.
  • 17. We(@frsyuki and me) Fixed • The approach is to use flock(2) without creating another *.lock file
  • 18. Pull requested to ruby My first contribution to ruby
  • 19. Merged!! Thanks to @naruse for his review and advice!!
  • 20. Will be released with 2.1.0!! on 12/25
  • 21. CONCLUSION • Ruby Logger was not safe in multiprocesses, but it is safe now!!! • Try ruby-trunk or https://github.com/sonots/ process_safe_logger
  • 22. Try the fluentd! It s very nice tool, check it out! http://fluentd.org/