Is Ruby Logger
Thread(Process)safe?
RubyConf 2013 @Miami
SEO Naotoshi (@sonots)
2013/11/09
Naotoshi SEO
like Now-toshi Say-oh

@sonots (twitter, github)
Dev Engineer for Operations
Became as a commiter of Fluentd
•realtime pipelining of log streams
•plugins written in ruby
Log Management Tool
•syslog-ng / rsyslog Only collection
Not Free
•splunk
No Plugin
•scribed
Java Plugin
•flumeNG
Ruby Plug...
Try the
fluentd!
It s very nice tool, check it out!
http://fluentd.org/
Today s
Talk
Logger
We needed a
Logger which
works safely
in multiprocess
for Fluentd
Examined Ruby Logger
• 1) Will logs not be mixtured in multi-threads?
• 2) Will logs not be mixtured in multi-processes?
•...
1) Will logs not be mixtured in multi-threads?
Code

require 'logger'
require 'parallel'
logger = Logger.new("/tmp/test.lo...
WHY?
logger.rb#L559
def write(message)
begin
@mutex.synchronize do
...
@dev.write(message)
...
end

Ruby’s Logger is takin...
2) Will logs not be mixtured in multi-processes?
Code
require 'logger'
require 'parallel'
logger = Logger.new("/tmp/test.l...
Really? Why?
Mutex does not work to exclusively lock in multiprocesses environment.
Atomic/non-atomic: A write is atomic i...
3) Does log rotation work safely multi-threads?
Code
require 'logger'
require 'parallel'
logger = Logger.new("/tmp/test.lo...
4) Does log rotation work safely in multi-processes?
Code
require 'logger'
require 'parallel'
logger = Logger.new("/tmp/te...
WHY?
logger.rb#L559
def write(message)
begin
@mutex.synchronize do
...
check_shift_log
...
end

Ruby’s Logger is taking mu...
We(@frsyuki and me) Fixed

• The approach is to use flock(2) without creating
another *.lock file
Pull requested to ruby

My first contribution to ruby
Merged!!
Thanks to @naruse for his review and advice!!
Will be released
with 2.1.0!!
on 12/25
CONCLUSION
• Ruby Logger was not safe in multiprocesses, but it is safe now!!!

• Try ruby-trunk or
https://github.com/son...
Try the
fluentd!
It s very nice tool, check it out!
http://fluentd.org/
Upcoming SlideShare
Loading in …5
×

Is ruby logger thread(process)-safe? at RubyConf 2013

7,010 views

Published on

Published in: Technology
  • Be the first to comment

Is ruby logger thread(process)-safe? at RubyConf 2013

  1. 1. Is Ruby Logger Thread(Process)safe? RubyConf 2013 @Miami SEO Naotoshi (@sonots) 2013/11/09
  2. 2. Naotoshi SEO like Now-toshi Say-oh @sonots (twitter, github) Dev Engineer for Operations Became as a commiter of Fluentd
  3. 3. •realtime pipelining of log streams •plugins written in ruby
  4. 4. Log Management Tool •syslog-ng / rsyslog Only collection Not Free •splunk No Plugin •scribed Java Plugin •flumeNG Ruby Plugin (Not Gem) •logstash •fluentd Ruby Plugin (Gem Support)
  5. 5. Try the fluentd! It s very nice tool, check it out! http://fluentd.org/
  6. 6. Today s Talk
  7. 7. Logger
  8. 8. We needed a Logger which works safely in multiprocess for Fluentd
  9. 9. Examined Ruby Logger • 1) Will logs not be mixtured in multi-threads? • 2) Will logs not be mixtured in multi-processes? • 3) Does log rotation work safely in multi-threads? • 4) Does log rotation work safely in multi-processes?
  10. 10. 1) Will logs not be mixtured in multi-threads? Code require 'logger' require 'parallel' logger = Logger.new("/tmp/test.log") Parallel.map(['a', 'b'], :in_threads => 2) do |letter| 10000.times do logger.info letter * 5000 end end Result $ egrep -e 'ab' -e 'ba' /tmp/test.log [empty] OK
  11. 11. WHY? logger.rb#L559 def write(message) begin @mutex.synchronize do ... @dev.write(message) ... end Ruby’s Logger is taking mutex to be thread-safe.
  12. 12. 2) Will logs not be mixtured in multi-processes? Code require 'logger' require 'parallel' logger = Logger.new("/tmp/test.log") Parallel.map(['a', 'b'], :in_processes => 2) do |letter| 10000.times do logger.info letter * 5000 end end Result $ egrep -e 'ab' -e 'ba' /tmp/test.log [empty] OK
  13. 13. Really? Why? Mutex does not work to exclusively lock in multiprocesses environment. Atomic/non-atomic: A write is atomic if the whole amount written in one operation is not interleaved with data from any other process. This is useful when there are multiple writers sending data to a single reader. Applications need to know how large a write request can be expected to be performed atomically. This maximum is called {PIPE_BUF}. This volume of IEEE Std 1003.1-2001 does not say whether write requests for more than {PIPE_BUF} bytes are atomic, but requires that writes of {PIPE_BUF} or fewer bytes shall be atomic. But, Linux system call write(2) itself is atomic as long as writing to a local file. OK
  14. 14. 3) Does log rotation work safely multi-threads? Code require 'logger' require 'parallel' logger = Logger.new("/tmp/test.log", 3, 1024 * 10) Parallel.map(['a', 'b'], :in_threads => 2) do |letter| 10000.times do logger.info letter * 5000 end end Result $ ls -l /tmp/test* -rw-r--r-- 1 sonots sonots 10806 9月 29 23:03 /tmp/test.log -rw-r--r-- 1 sonots sonots 10806 9月 29 23:03 /tmp/test.log.0 -rw-r--r-- 1 sonots sonots 10806 9月 29 23:03 /tmp/test.log.1 No Error, OK
  15. 15. 4) Does log rotation work safely in multi-processes? Code require 'logger' require 'parallel' logger = Logger.new("/tmp/test.log", 3, 1024 * 10) Parallel.map(['a', 'b'], :in_processes => 2) do |letter| 10000.times do logger.info letter * 5000 end end Result log log log log ... writing failed. closed stream shifting failed. closed stream writing failed. closed stream shifting failed. closed stream Oops, Bad!
  16. 16. WHY? logger.rb#L559 def write(message) begin @mutex.synchronize do ... check_shift_log ... end Ruby’s Logger is taking mutex to be thread-safe, but was not treating anything for multi-processes.
  17. 17. We(@frsyuki and me) Fixed • The approach is to use flock(2) without creating another *.lock file
  18. 18. Pull requested to ruby My first contribution to ruby
  19. 19. Merged!! Thanks to @naruse for his review and advice!!
  20. 20. Will be released with 2.1.0!! on 12/25
  21. 21. CONCLUSION • Ruby Logger was not safe in multiprocesses, but it is safe now!!! • Try ruby-trunk or https://github.com/sonots/ process_safe_logger
  22. 22. Try the fluentd! It s very nice tool, check it out! http://fluentd.org/

×