The Ruby Guide to *nix Plumbing: on the quest for efficiency with Ruby [M|K]RI
Upcoming SlideShare
Loading in...5
×
 

The Ruby Guide to *nix Plumbing: on the quest for efficiency with Ruby [M|K]RI

on

  • 3,920 views

Yet another look at using native unix features from Ruby.

Yet another look at using native unix features from Ruby.

Statistics

Views

Total Views
3,920
Views on SlideShare
3,870
Embed Views
50

Actions

Likes
9
Downloads
65
Comments
4

2 Embeds 50

http://www.slideshare.net 32
http://www.techgig.com 18

Accessibility

Categories

Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • You really did present a brilliant idea in terms of plumbing,this is awesome. Please visit this site: http://www.rørlegger-fredrikstad
    Are you sure you want to
    Your message goes here
    Processing…
  • Good stuff...hard to find documentation on this kind of thing.
    Are you sure you want to
    Your message goes here
    Processing…
  • As I like to tell an audience, even C coding is nothing more than kernel scripting. Ruby coders have just as much right to tackle systems programming as those using any other language.

    Admittedly the language implementations still need a lot of work to unlock its full potential but I'm confident time will take care of that.
    Are you sure you want to
    Your message goes here
    Processing…
  • It's wonderful to stumble on to people using Ruby as a systems programming language. I think it has tremendous potential as a glue language. It is my new 'Perl' for systems automation work and I love to find people pushing the envelope with it.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

The Ruby Guide to *nix Plumbing: on the quest for efficiency with Ruby [M|K]RI The Ruby Guide to *nix Plumbing: on the quest for efficiency with Ruby [M|K]RI Presentation Transcript

  • The Ruby Guide to *nix Plumbing on the quest for efficiency with Ruby [M|K]RI http://slides.games-with-brains.net
  • who am I? Eleanor McHugh eleanor@games-with-brains.com twitter.com/feyeleanor usual haunts include ruby-talk LRUG devchix ruby conferences
  • hacker 101 every unix hacker was once a clueless noob who learnt their craft through experimentation today I hope to whet your appetite by showing you reams of broken code fixing it is an exercise for the reader and one well worth the sweat and tears
  • common requirements access native hardware “real-time” user interaction scale to suit demand
  • that’s systems programming and it’s our right as coders to get low-level in whichever language best suits our needs
  • but unix is all about C... that’s like saying web development is all about javascript - it may be a daily experience but it’s far from immutable truth
  • no, unix is about the kernel which was originally implemented in C with a jus of assembler, hence the confusion
  • we can script that in ruby C is just one language for scripting the kernel but anything you can do in C you can do in ruby, python, groovy, scheme, assembler all you need is a syscall function and some magic numbers
  • unix bootcamp the kernel manages resources a process is a program execution a file stores sequences of character data a block device describes a peripheral a signal is a software interrupt
  • knife goes in... only build what we need reuse what we already have change our tools as [des|requ]ired stay flexible
  • ...guts come out cohesive device model kernel to manage resources shell for user interaction userspace partitions risk
  • everything else is plumbing hierarchical trees & files users & permissions processes, signals & communications
  • do with it what you will “if you give people the license to be as outrageous as they want in absolutely any fashion they can dream up, they’ll be creative about it, and do something good besides” - Lester Bangs
  • accessing kernel resources Kernel#syscall & kernel function indices file descriptors IO#for_fd
  • require 'fcntl' filemode = Fcntl::O_CREAT | Fcntl::O_RDWR | Fcntl::O_APPEND descriptor = IO.sysopen “test.dat”, filemode file = IO.new descriptor file.syswrite “hello” file.sysseek 0 $stdout.puts file.sysread(10) produces: hello
  • ruby/dl dynamic libraries managed memory buffers ruby callbacks
  • wrapping c syscall require ‘dl’ file = open “test.dat”, 0x0209 CRT = DL.dlopen ‘libc.dylib’ write file, “textn” F = ‘syscall’ close file def open file, mode file = open “test.dat”, 0x0000 CRT[F, ‘IISI’].call(5, file, mode)[0] text = read file, 10 end close file def write fd, string, bytes = string.length CRT[F, ‘IIISI’].call(4, fd, string, bytes)[0] end def read fd, bytes = 1 buffer = DL.malloc(bytes) CRT[F, ‘IIIsI’].call(3, fd, buffer, bytes)[1][2] end def close fd CRT[F, ‘III’].call(6, fd)[0] end
  • malloc DL:PtrData garbage collection with free and realloc [String|Array]#to_ptr PtrData#[struct|union]!
  • works how you expect require ‘dl’ memory_buffer = DL::malloc 20 => #<DL::PtrData:0x2d0870 ptr=0x820600 size=20 free=0x1b0257> memory_buffer[0] = “hello world!” => “hello world!000000000000000000000000" memory_buffer.free => #<DL::Symbol:0x40b760 func=0x1b0257 'void (free)(void *);'> memory_buffer.nil => nil
  • most of the time string = “hello ruby” memory_buffer = string.to_ptr => #<DL::PtrData:0x41bea0 ptr=0x41be60 size=10 free=0x1b0257> memory_buffer[0] = “goodbye world” memory_buffer += 1 => #<DL::PtrData:0x422000 ptr=0x41be61 size=9 free=0x0> puts memory_buffer, memory_buffer.to_str, string => “oodbye world” => “oodbye wo” => “hello ruby” memory_buffer -= 1 => (irb):51: [BUG] Segmentation fault
  • the callback that never was require 'dl' SIGSEGV = DL::dlopen('libsigsegv.dylib') install_handler = SIGSEGV['sigsegv_install_handler', 'IP'] deinstall_handler = SIGSEGV['sigsegv_deinstall_handler', '0'] leave_handler = SIGSEGV['sigsegv_leave_handler', 'IPPPP'] continuation = DL.callback('IPPP') do |address, b, c| raise RuntimeError, "segfault at #{address}" end handler = DL.callback('IPI') do |fault_address, serious| leave_handler.call continuation, fault_address, nil, nil end install_handler.call handler
  • multi-tasking threads & processes semaphores & interprocess communications shared memory multiple cores leveraging networks
  • limitations of ruby threads green threads pthreads the global interpreter lock pthreads and child processes the FreeBSD conundrum
  • threaded socket I/O require 'socket' def serve request require 'thread' ["hello", 0] require 'mutex_m' end class UDPServer private include Mutex_m def event_loop attr_reader :address, :port, :log loop do if sockets = select([@socket]) then def initialize address, port sockets[0].each do |s| @address, @port = address, port @workers << Thread.new(s) do |socket| @workers = [] message, peer = *socket.recvfrom 512 end reply, status = *serve message UDPSocket.open.send reply, status, peer[2], peer[1] def start end @socket = UDPSocket.new end @socket.bind @address, @port @workers.compact! @socket.setsockopt Socket::SOL_SOCKET, Socket::SO_REUSEADDR, 1 end event_loop end end end end def stop @workers.each { |thread| thread.kill } lock @socket.close @socket = nil unlock end
  • fork and be damned require 'socket' private def event_loop class UDPForkingServer loop do attr_reader :address, :port, :log if sockets = select([@socket]) then sockets[0].each do |s| def initialize address, port fork @address, @port = address, port message, peer = *socket.recvfrom(512) end reply, status = *serve message UDPSocket.open.send reply, status, peer[2], peer[1] def start end @socket = UDPSocket.new end @socket.bind @address, @port end @socket.setsockopt Socket::SOL_SOCKET, Socket::SO_REUSEADDR, 1 end event_loop end end end def stop @socket.close @socket = nil end def serve request ["hello", 0] end
  • arbitrating sequence require ‘dl’ require ‘fcntl’ LIBC = DL::dlopen ‘libc.dylib’ open = LIBC[‘sem_open’, ‘ISII’] try_wait = LIBC[‘sem_trywait’, ‘II’] wait = LIBC[‘sem_wait’, ‘II’] post = LIBC[‘sem_post’, ‘II’] close = LIBC[‘sem_close’, ‘II’] process 1 process 2 s = open.call(“/tmp/s”, Fcntl::O_CREAT, 1911)[0] s = open.call(“/tmp/s”) wait.call s t = Time.now puts “locked at #{Time.now}” if try_wait.call(s)[0] == 0 then sleep 50 puts “locked at #{t}” puts “posted at #{Time.now}” else post.call s puts “busy at #{t}” close.call s wait.call s puts “waited #{Time.now - t} seconds” => locked at Thu May 28 01:03:23 +0100 2009 end => posted at Thu May 28 01:04:13 +0100 2009 => busy at Thu May 28 01:03:36 +0100 2009 => waited 47.056508 seconds
  • law of diminishing returns 2x the cores never means 2x the performance this is fundamental communications theory and applies to all “real-time” systems including your development processes... leverage Shannon-Nyquist to your advantage
  • pipes & sockets point-to-point links between processes pipes are restricted to the local machine sockets can be local or remote the fifo is a multiway pipe for special occasions
  • the fifo - a persistent pipe process 1 process 2 File.umask 0 File.umask 0 MKFIFO = 132 MKFIFO = 132 syscall MKFIFO, fifo_name, 0666 syscall MKFIFO, “client”, 0666 fd = IO.sysopen “server”, File::RDONLY fd = IO.sysopen "server", File::WRONLY server = File.new fd, "r" server = IO.new fd, "w" client_name = server.gets.chomp server.puts fifo_name puts "#{Time.now}: [#{client_name}]" server.puts "hello world!" fd = IO.sysopen client_name, File::WRONLY server.close client = IO.new fd, "w" fd = IO.sysopen “client”, File::RDONLY message = server.gets.chomp client = IO.new fd, "r" client.puts message.reverse puts client.gets client.close client.close server.close File.delete “client” File.delete “server”
  • sharing memory allows processes to share data directly no need to bother with sockets or pipes but concurrency becomes a major concern another exciting use for semaphores
  • beyond ruby/dl ruby-ffi RubyInline wilson
  • the plumber’s reading list http://slides.games-with-brains.net http://www.jbrowse.com/text/rdl_en.html http://www.kegel.com/c10k.html http://www.ecst.csuchico.edu/~beej/guide/ipc/ http://beej.us/guide/bgnet/ http://wiki.netbsd.se/kqueue_tutorial