0
The Ruby Guide to
*nix Plumbing
on the quest for efficiency with Ruby [M|K]RI


http://slides.games-with-brains.net
who am I?
               Eleanor McHugh
            eleanor@games-with-brains.com
                 twitter.com/feyeleanor
...
hacker 101

every unix hacker was once a clueless noob
who learnt their craft through experimentation
today I hope to whet...
common requirements
access native hardware
“real-time” user interaction
scale to suit demand
that’s systems programming
and it’s our right as coders to get low-level in
whichever language best suits our needs
but unix is all about C...
that’s like saying web development is all about
javascript - it may be a daily experience but i...
no, unix is about the kernel
which was originally implemented in C with a jus
of assembler, hence the confusion
we can script that in ruby
C is just one language for scripting the kernel
but anything you can do in C you can do in ruby...
unix bootcamp
the kernel manages resources
a process is a program execution
a file stores sequences of character data
a blo...
knife goes in...
only build what we need
reuse what we already have
change our tools as [des|requ]ired
stay flexible
...guts come out
cohesive device model
kernel to manage resources
shell for user interaction
userspace partitions risk
everything else is plumbing
hierarchical trees & files
users & permissions
processes, signals & communications
do with it what you will
“if you give people the license to be as
outrageous as they want in absolutely any
fashion they c...
accessing kernel resources
Kernel#syscall & kernel function indices
file descriptors
IO#for_fd
require 'fcntl'
filemode = Fcntl::O_CREAT | Fcntl::O_RDWR | Fcntl::O_APPEND
descriptor = IO.sysopen “test.dat”, filemode
file...
ruby/dl
dynamic libraries
managed memory buffers
ruby callbacks
wrapping c syscall
require ‘dl’                                            file = open “test.dat”, 0x0209
CRT = DL.dlopen ‘...
malloc
DL:PtrData
garbage collection with free and realloc
[String|Array]#to_ptr
PtrData#[struct|union]!
works how you expect
require ‘dl’

memory_buffer = DL::malloc 20
=> #<DL::PtrData:0x2d0870 ptr=0x820600 size=20 free=0x1b0...
most of the time
string = “hello ruby”
memory_buffer = string.to_ptr
=> #<DL::PtrData:0x41bea0 ptr=0x41be60 size=10 free=0...
the callback that never was
require 'dl'
SIGSEGV = DL::dlopen('libsigsegv.dylib')
install_handler = SIGSEGV['sigsegv_insta...
multi-tasking
threads & processes
semaphores & interprocess communications
shared memory
multiple cores
leveraging networks
limitations of ruby threads
green threads
pthreads
the global interpreter lock
pthreads and child processes
the FreeBSD co...
threaded socket I/O
require 'socket'                                                    def serve request
require 'thread'...
fork and be damned
require 'socket'                                                   private
                            ...
arbitrating sequence
  require ‘dl’
  require ‘fcntl’
  LIBC = DL::dlopen ‘libc.dylib’
  open = LIBC[‘sem_open’, ‘ISII’]
 ...
law of diminishing returns
2x the cores never means 2x the performance
this is fundamental communications theory
and appli...
pipes & sockets
point-to-point links between processes
pipes are restricted to the local machine
sockets can be local or r...
the fifo - a persistent pipe
process 1                                     process 2

  File.umask 0                       ...
sharing memory
allows processes to share data directly
no need to bother with sockets or pipes
but concurrency becomes a m...
beyond ruby/dl
ruby-ffi
RubyInline
wilson
the plumber’s reading list

 http://slides.games-with-brains.net

 http://www.jbrowse.com/text/rdl_en.html

 http://www.ke...
Upcoming SlideShare
Loading in...5
×

The Ruby Guide to *nix Plumbing: on the quest for efficiency with Ruby [M|K]RI

2,137

Published on

Yet another look at using native unix features from Ruby.

Published in: Technology, Education
4 Comments
9 Likes
Statistics
Notes
  • You really did present a brilliant idea in terms of plumbing,this is awesome. Please visit this site: http://www.rørlegger-fredrikstad
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Good stuff...hard to find documentation on this kind of thing.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • As I like to tell an audience, even C coding is nothing more than kernel scripting. Ruby coders have just as much right to tackle systems programming as those using any other language.

    Admittedly the language implementations still need a lot of work to unlock its full potential but I'm confident time will take care of that.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • It's wonderful to stumble on to people using Ruby as a systems programming language. I think it has tremendous potential as a glue language. It is my new 'Perl' for systems automation work and I love to find people pushing the envelope with it.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
2,137
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
66
Comments
4
Likes
9
Embeds 0
No embeds

No notes for slide
  • Transcript of "The Ruby Guide to *nix Plumbing: on the quest for efficiency with Ruby [M|K]RI"

    1. 1. The Ruby Guide to *nix Plumbing on the quest for efficiency with Ruby [M|K]RI http://slides.games-with-brains.net
    2. 2. who am I? Eleanor McHugh eleanor@games-with-brains.com twitter.com/feyeleanor usual haunts include ruby-talk LRUG devchix ruby conferences
    3. 3. hacker 101 every unix hacker was once a clueless noob who learnt their craft through experimentation today I hope to whet your appetite by showing you reams of broken code fixing it is an exercise for the reader and one well worth the sweat and tears
    4. 4. common requirements access native hardware “real-time” user interaction scale to suit demand
    5. 5. that’s systems programming and it’s our right as coders to get low-level in whichever language best suits our needs
    6. 6. but unix is all about C... that’s like saying web development is all about javascript - it may be a daily experience but it’s far from immutable truth
    7. 7. no, unix is about the kernel which was originally implemented in C with a jus of assembler, hence the confusion
    8. 8. we can script that in ruby C is just one language for scripting the kernel but anything you can do in C you can do in ruby, python, groovy, scheme, assembler all you need is a syscall function and some magic numbers
    9. 9. unix bootcamp the kernel manages resources a process is a program execution a file stores sequences of character data a block device describes a peripheral a signal is a software interrupt
    10. 10. knife goes in... only build what we need reuse what we already have change our tools as [des|requ]ired stay flexible
    11. 11. ...guts come out cohesive device model kernel to manage resources shell for user interaction userspace partitions risk
    12. 12. everything else is plumbing hierarchical trees & files users & permissions processes, signals & communications
    13. 13. do with it what you will “if you give people the license to be as outrageous as they want in absolutely any fashion they can dream up, they’ll be creative about it, and do something good besides” - Lester Bangs
    14. 14. accessing kernel resources Kernel#syscall & kernel function indices file descriptors IO#for_fd
    15. 15. require 'fcntl' filemode = Fcntl::O_CREAT | Fcntl::O_RDWR | Fcntl::O_APPEND descriptor = IO.sysopen “test.dat”, filemode file = IO.new descriptor file.syswrite “hello” file.sysseek 0 $stdout.puts file.sysread(10) produces: hello
    16. 16. ruby/dl dynamic libraries managed memory buffers ruby callbacks
    17. 17. wrapping c syscall require ‘dl’ file = open “test.dat”, 0x0209 CRT = DL.dlopen ‘libc.dylib’ write file, “textn” F = ‘syscall’ close file def open file, mode file = open “test.dat”, 0x0000 CRT[F, ‘IISI’].call(5, file, mode)[0] text = read file, 10 end close file def write fd, string, bytes = string.length CRT[F, ‘IIISI’].call(4, fd, string, bytes)[0] end def read fd, bytes = 1 buffer = DL.malloc(bytes) CRT[F, ‘IIIsI’].call(3, fd, buffer, bytes)[1][2] end def close fd CRT[F, ‘III’].call(6, fd)[0] end
    18. 18. malloc DL:PtrData garbage collection with free and realloc [String|Array]#to_ptr PtrData#[struct|union]!
    19. 19. works how you expect require ‘dl’ memory_buffer = DL::malloc 20 => #<DL::PtrData:0x2d0870 ptr=0x820600 size=20 free=0x1b0257> memory_buffer[0] = “hello world!” => “hello world!000000000000000000000000" memory_buffer.free => #<DL::Symbol:0x40b760 func=0x1b0257 'void (free)(void *);'> memory_buffer.nil => nil
    20. 20. most of the time string = “hello ruby” memory_buffer = string.to_ptr => #<DL::PtrData:0x41bea0 ptr=0x41be60 size=10 free=0x1b0257> memory_buffer[0] = “goodbye world” memory_buffer += 1 => #<DL::PtrData:0x422000 ptr=0x41be61 size=9 free=0x0> puts memory_buffer, memory_buffer.to_str, string => “oodbye world” => “oodbye wo” => “hello ruby” memory_buffer -= 1 => (irb):51: [BUG] Segmentation fault
    21. 21. the callback that never was require 'dl' SIGSEGV = DL::dlopen('libsigsegv.dylib') install_handler = SIGSEGV['sigsegv_install_handler', 'IP'] deinstall_handler = SIGSEGV['sigsegv_deinstall_handler', '0'] leave_handler = SIGSEGV['sigsegv_leave_handler', 'IPPPP'] continuation = DL.callback('IPPP') do |address, b, c| raise RuntimeError, "segfault at #{address}" end handler = DL.callback('IPI') do |fault_address, serious| leave_handler.call continuation, fault_address, nil, nil end install_handler.call handler
    22. 22. multi-tasking threads & processes semaphores & interprocess communications shared memory multiple cores leveraging networks
    23. 23. limitations of ruby threads green threads pthreads the global interpreter lock pthreads and child processes the FreeBSD conundrum
    24. 24. threaded socket I/O require 'socket' def serve request require 'thread' ["hello", 0] require 'mutex_m' end class UDPServer private include Mutex_m def event_loop attr_reader :address, :port, :log loop do if sockets = select([@socket]) then def initialize address, port sockets[0].each do |s| @address, @port = address, port @workers << Thread.new(s) do |socket| @workers = [] message, peer = *socket.recvfrom 512 end reply, status = *serve message UDPSocket.open.send reply, status, peer[2], peer[1] def start end @socket = UDPSocket.new end @socket.bind @address, @port @workers.compact! @socket.setsockopt Socket::SOL_SOCKET, Socket::SO_REUSEADDR, 1 end event_loop end end end end def stop @workers.each { |thread| thread.kill } lock @socket.close @socket = nil unlock end
    25. 25. fork and be damned require 'socket' private def event_loop class UDPForkingServer loop do attr_reader :address, :port, :log if sockets = select([@socket]) then sockets[0].each do |s| def initialize address, port fork @address, @port = address, port message, peer = *socket.recvfrom(512) end reply, status = *serve message UDPSocket.open.send reply, status, peer[2], peer[1] def start end @socket = UDPSocket.new end @socket.bind @address, @port end @socket.setsockopt Socket::SOL_SOCKET, Socket::SO_REUSEADDR, 1 end event_loop end end end def stop @socket.close @socket = nil end def serve request ["hello", 0] end
    26. 26. arbitrating sequence require ‘dl’ require ‘fcntl’ LIBC = DL::dlopen ‘libc.dylib’ open = LIBC[‘sem_open’, ‘ISII’] try_wait = LIBC[‘sem_trywait’, ‘II’] wait = LIBC[‘sem_wait’, ‘II’] post = LIBC[‘sem_post’, ‘II’] close = LIBC[‘sem_close’, ‘II’] process 1 process 2 s = open.call(“/tmp/s”, Fcntl::O_CREAT, 1911)[0] s = open.call(“/tmp/s”) wait.call s t = Time.now puts “locked at #{Time.now}” if try_wait.call(s)[0] == 0 then sleep 50 puts “locked at #{t}” puts “posted at #{Time.now}” else post.call s puts “busy at #{t}” close.call s wait.call s puts “waited #{Time.now - t} seconds” => locked at Thu May 28 01:03:23 +0100 2009 end => posted at Thu May 28 01:04:13 +0100 2009 => busy at Thu May 28 01:03:36 +0100 2009 => waited 47.056508 seconds
    27. 27. law of diminishing returns 2x the cores never means 2x the performance this is fundamental communications theory and applies to all “real-time” systems including your development processes... leverage Shannon-Nyquist to your advantage
    28. 28. pipes & sockets point-to-point links between processes pipes are restricted to the local machine sockets can be local or remote the fifo is a multiway pipe for special occasions
    29. 29. the fifo - a persistent pipe process 1 process 2 File.umask 0 File.umask 0 MKFIFO = 132 MKFIFO = 132 syscall MKFIFO, fifo_name, 0666 syscall MKFIFO, “client”, 0666 fd = IO.sysopen “server”, File::RDONLY fd = IO.sysopen "server", File::WRONLY server = File.new fd, "r" server = IO.new fd, "w" client_name = server.gets.chomp server.puts fifo_name puts "#{Time.now}: [#{client_name}]" server.puts "hello world!" fd = IO.sysopen client_name, File::WRONLY server.close client = IO.new fd, "w" fd = IO.sysopen “client”, File::RDONLY message = server.gets.chomp client = IO.new fd, "r" client.puts message.reverse puts client.gets client.close client.close server.close File.delete “client” File.delete “server”
    30. 30. sharing memory allows processes to share data directly no need to bother with sockets or pipes but concurrency becomes a major concern another exciting use for semaphores
    31. 31. beyond ruby/dl ruby-ffi RubyInline wilson
    32. 32. the plumber’s reading list http://slides.games-with-brains.net http://www.jbrowse.com/text/rdl_en.html http://www.kegel.com/c10k.html http://www.ecst.csuchico.edu/~beej/guide/ipc/ http://beej.us/guide/bgnet/ http://wiki.netbsd.se/kqueue_tutorial
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×