0
The Ruby Guide to
*nix Plumbing
Haxor Redux Edition


http://slides.games-with-brains.net
caveat lector
 danger! we’re entering strange territory
 and our guide is no Alan Quartermain
 the map is missing some maj...
cowgirl for hire
                      Eleanor McHugh
                   eleanor@games-with-brains.com


                 ...
systems programming
it’s our right as coders to get low-level
and we shouldn’t have to use C to do it
worse is better
the unix philosophy of life
it’s smart to be lazy
 create small tools
 to solve simple tasks effectively
 perform each task independently
 and pass th...
knife goes in...
only build what we need
reuse what we already have
change our tools as [des|requ]ired
stay flexible
...guts come out
cohesive device model
kernel
shell
userspace
could it get more punk?
“if you give people the license to be as
outrageous as they want in absolutely any
fashion they ca...
the process
a discrete unit of work
anatomy of a process
program instructions
program data
system data (file descriptors, user, group, parent, etc.)
environmen...
process creation
a process may spawn child processes via fork()
this creates a duplicate child process
with duplicate file ...
a costly strategy
 sometimes we only want a blank slate process
 but fork() copies the full process state
 so BSD introduc...
with certain limitations
 only the thread which forks will be in the child process
 a system based on green threads can wo...
Kernel.system [env ,] cmnd [, args] [, options] => true or false or nil
Kernel.spawn [env ,] cmnd [, args] [, options] => ...
IO.popen cmd [, mode = "r"] => io
IO.popen cmd [, mode = "r" [{ | io | block }] => obj
Invokes a command shell in a subpro...
Kernel.fork [{ block }] => fixnum or nil
Process.fork [{ block }] => fixnum or nil
Ruby provides its own fork() implementati...
zombie want brains
a zombie is a subprocess which outlives its parent
zombies are members of the system process group
and ...
Process.wait2 => [ int, Process::status ]
Prcess.waitpid2 pid, [ int = 0 ] => [ int, Process::status ]
Prcess.waitall => [...
the shell
the glue that holds the system together
standard shells
 provide an interactive environment on std[in|out|err]
 using an interpreted language
 for job control and...
the Ruby way
shell.rb ships with the standard library
provides redirection, piping and standard commands
  tee, cat, echo,...
the unix kernel
an overview with a Ruby twist
basic services
 virtual memory
 process management
 hierarchical file system
 user permissions
 interprocess communication
...
file descriptors
 the kernel keeps a table of open files for each process
 an integer index is used to reference entries
 an...
beware the subtleties
 file descriptors are unique to a process
 so closing a descriptor only affects that process
 but the...
IO.sysopen path [, mode [, perm] => fixnum
io.sysread int [, buffer] => string
io.syswrite string => int
io.sysseek offset,...
IO.new fd [, mode] [, opt] => io
IO.for_fd fd [, mode] [, opt] => io
Returns a new IO object (a stream) for the given IO o...
require 'fcntl'
filemode = Fcntl::O_CREAT | Fcntl::O_RDWR | Fcntl::O_APPEND
descriptor = IO.sysopen “test.dat”, filemode
file...
handling many files at once

file access is much slower than memory access
this causes threads and processes to block
so usu...
select rd_ary [, wrt_ary] [, err_ary] [, timeout] => array or nil
Select is the standard low-level method for polling many...
kernel event queues
allow the kernel to raise events for streams
a process can then poll for events without blocking
on BS...
syscall
calling functions in the kernel
from the docs
Kernel#syscall fixnum [, args...] => integer
Calls the operating system function identified by fixnum, passing ...
finding your syscall codes

 MacOS X
   /usr/include/sys/syscall.h
 Linux
   /usr/include/asm/unistd_32.h
   /usr/include/a...
a fragile approach
 the native syscall is a very primitive interface
   error codes
   file descriptors & pointer addresses...
IO.sysx the hard way
On MacOS X

  file = syscall 5, “test.dat”, 1      # open O_WRONLY (fcntl.h)
  syscall 4, file, “some t...
complex parameters
some kernel calls accept structured data types
ruby represents these as String objects
which can be pre...
currently broken
On Ruby 1.8.7 or 1.9.1 (upto patch 129)

  syscall 4, 1, “hellon#{0.chr}goodbyen”, 15

produces:

  Argum...
Interprocess Communication

IPC objects have descriptors like files
but they don’t need to exist in the file system
System V...
signals
 an abstraction for software interrupts
 SIGTERM and SIGKILL are the best known
 but *nixen provide a wide range o...
sockets
two varieties: domain sockets and network sockets
Ruby provides a rich class hierarchy
check out my other presenta...
pipes

pipes provide a pair of end-points for processes to talk
an unnamed pipe connects two related processes
a named pip...
IO.pipe => read_file, write_file
Ruby supports anonymous pipes, which can be used to communicate
between a parent a child pr...
named pipes
process 1                                     process 2

  File.umask 0                                File.um...
semaphores

exist independently of processes
provide blocking access
allowing processes to be synchronised
live in the file...
posix semaphores
process 1                                            process 2

  require ‘fcntl’                        ...
in the works
shared memory
message queues
managed platform access
memory buffers
structs
c runtime access
ruby/dl
the unsung hero of MRI
in the standard library
 introduced in ruby 1.8
 but poorly documented and lightly used
 provides access to dynamically lo...
memory management
memory pointers encapsulated by DL::PtrData
garbage collected
  uses free() or a custom deallocator
  ha...
using native syscall
require ‘dl’
libc = DL.dlopen(‘libc.dylib’)
open = libc[‘syscall’, ‘IISI’]
write = libc[‘syscall’, ‘I...
with a procedural slant
require ‘dl’                                            file = open “test.dat”, 0x0209
CRT = DL.dlo...
worth checking out
ruby-ffi
RubyInline
wilson
the bluffer’s reading list

 http://slides.games-with-brains.net

 http://www.jbrowse.com/text/rdl_en.html

 http://www.ke...
QEMFD?
find out at Lone Star Ruby Conf 2009
QEMFD?
http://slides.games-with-brains.net
Upcoming SlideShare
Loading in...5
×

The Ruby Guide to *nix Plumbing: Hax0R R3dux

3,018

Published on

Expanded version of my guide to using Ruby as a systems programming language.

Published in: Technology, Education
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,018
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
44
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

Transcript of "The Ruby Guide to *nix Plumbing: Hax0R R3dux"

  1. 1. The Ruby Guide to *nix Plumbing Haxor Redux Edition http://slides.games-with-brains.net
  2. 2. caveat lector danger! we’re entering strange territory and our guide is no Alan Quartermain the map is missing some major landmarks and is riddled with inaccuracies so don’t try this on a live production system and read your man pages thoroughly
  3. 3. cowgirl for hire Eleanor McHugh eleanor@games-with-brains.com usual haunts include ruby-talk LRUG devchix conferences
  4. 4. systems programming it’s our right as coders to get low-level and we shouldn’t have to use C to do it
  5. 5. worse is better the unix philosophy of life
  6. 6. it’s smart to be lazy create small tools to solve simple tasks effectively perform each task independently and pass the results from one task to another using a clean interface thus solving complex problems sociably
  7. 7. knife goes in... only build what we need reuse what we already have change our tools as [des|requ]ired stay flexible
  8. 8. ...guts come out cohesive device model kernel shell userspace
  9. 9. could it get more punk? “if you give people the license to be as outrageous as they want in absolutely any fashion they can dream up, they’ll be creative about it, and do something good besides” - Lester Bangs
  10. 10. the process a discrete unit of work
  11. 11. anatomy of a process program instructions program data system data (file descriptors, user, group, parent, etc.) environment data supports one or more threads of execution preemptively scheduled by the kernel
  12. 12. process creation a process may spawn child processes via fork() this creates a duplicate child process with duplicate file descriptors and program state which can be replaced via exec() of a new program all processes ultimately derive from the init process which is created by the kernel at boot
  13. 13. a costly strategy sometimes we only want a blank slate process but fork() copies the full process state so BSD introduced vfork() which shares data memory whilst POSIX introduced spawn() most fork() implementations now use copy-on-write so the advantages of vfork() have diminished
  14. 14. with certain limitations only the thread which forks will be in the child process a system based on green threads can work around this Ruby 1.8 still uses pthreads so doesn’t it’s also possible for exec() to fail so forked processes need to think about this and return meaningful exit() codes
  15. 15. Kernel.system [env ,] cmnd [, args] [, options] => true or false or nil Kernel.spawn [env ,] cmnd [, args] [, options] => pid The simplest way to take advantage of multiple processes in Ruby is to launch a subshell and either block until it terminates or keep a note of its process ID so this can be checked later. This is an opaque method and not useful when the parent process wishes to capture results. Kernel.` command ` => string %x{ command } => string The next step up in utility from system() and spawn(), the backquote method captures standard output from the subshell and as a bonus stores a Process::Status object in the thread-global $? variable.
  16. 16. IO.popen cmd [, mode = "r"] => io IO.popen cmd [, mode = "r" [{ | io | block }] => obj Invokes a command shell in a subprocess and executes the specified commands, providing a [uni|bi]directional pipe which the parent process can use to communicate with the subprocess. Open3.popen3 cmd [{ |stdin, stdout, stderr| block }] Open3.popen3 cmd => array A wrapper for spawn() which provides access to the standard IO streams of the subshell along with a thread waiting on it.
  17. 17. Kernel.fork [{ block }] => fixnum or nil Process.fork [{ block }] => fixnum or nil Ruby provides its own fork() implementation which can either take a block and run it in a subprocess, or can allow a more traditional split in execution paths. No IPC mechanism is provided beyond allowing the parent to wait on the child process and receive an error code. Kernel.exec [env ,] cmnd [, args] [, options] And as you’d expect it also allows access to the exec() family of system calls. This is a rare core library methods which doesn’t return a value for the simple reason that the existing program ceases to exist unless an error occurs, and that’s flagged with a SystemCallError exception. It’s common practice to terminate the program or retry exec() after a delay.
  18. 18. zombie want brains a zombie is a subprocess which outlives its parent zombies are members of the system process group and are eventually terminated but until then they retain system resources and may not flush all of their IO buffers so get in the habit of flushing IO buffers regularly
  19. 19. Process.wait2 => [ int, Process::status ] Prcess.waitpid2 pid, [ int = 0 ] => [ int, Process::status ] Prcess.waitall => [ [ int, Process::status ] *] Wait for a child process to exit. Wait2 watches all child processes and returns when any one process exits whilst waitpid2 waits for a specified child process, optionally masking for WNOHANG and WUNTRACED. Waitall waits on all child processes to exit and returns their statuses.
  20. 20. the shell the glue that holds the system together
  21. 21. standard shells provide an interactive environment on std[in|out|err] using an interpreted language for job control and file system manipulation the bourne shell is the lowest common denominator with a powerful POSIX-compatible syntax which is a pig to parse or maintain
  22. 22. the Ruby way shell.rb ships with the standard library provides redirection, piping and standard commands tee, cat, echo, pwd, [push|pop]d enumerates files and directories with foreach as well as wrapping arbitrary system commands documented in The Ruby Way
  23. 23. the unix kernel an overview with a Ruby twist
  24. 24. basic services virtual memory process management hierarchical file system user permissions interprocess communication signals and interrupts execution environment
  25. 25. file descriptors the kernel keeps a table of open files for each process an integer index is used to reference entries and is known as a file descriptor unix is liberal about what counts as a file and most descriptors are inherited by child processes caution file semantics can vary...
  26. 26. beware the subtleties file descriptors are unique to a process so closing a descriptor only affects that process but the underlying descriptions are shared performing an lseek() changes the description and therefore will affect other processes with much hilarity ensuing...
  27. 27. IO.sysopen path [, mode [, perm] => fixnum io.sysread int [, buffer] => string io.syswrite string => int io.sysseek offset, whence = SEEK_SET => int These methods make direct calls to the kernel to open and manipulate the contents of files, bypassing buffering. They shouldn’t be used with higher-level methods such as IO#read as unpredictable results may occur. Errors are raised as Exceptions in the Errno namespace.
  28. 28. IO.new fd [, mode] [, opt] => io IO.for_fd fd [, mode] [, opt] => io Returns a new IO object (a stream) for the given IO object or integer file descriptor and mode string. See also IO#fileno and IO.for_fd. a = IO.new(2,"w") # '2' is the unix default for standard error $stderr.print "Hello" a.print " World" STDERR.print “!!” produces: Hello World!!
  29. 29. require 'fcntl' filemode = Fcntl::O_CREAT | Fcntl::O_RDWR | Fcntl::O_APPEND descriptor = IO.sysopen “test.dat”, filemode file = IO.new descriptor file.syswrite “hello” file.sysseek 0 $stdout.puts file.sysread(10) produces: hello
  30. 30. handling many files at once file access is much slower than memory access this causes threads and processes to block so usually IO is buffered by the kernel and non-blocking access methods are implemented however this is still ineffient when writing servers
  31. 31. select rd_ary [, wrt_ary] [, err_ary] [, timeout] => array or nil Select is the standard low-level method for polling many IO streams at once and it’s a blocking call, hence the need for a timeout. For writing socket servers etc. this usually works reasonably well and Ruby’s implementation plays nicely with threads.
  32. 32. kernel event queues allow the kernel to raise events for streams a process can then poll for events without blocking on BSD this is handled by kqueue on Linux by epoll and inotify epoll is much more efficient than inotify EventMachine uses these when available
  33. 33. syscall calling functions in the kernel
  34. 34. from the docs Kernel#syscall fixnum [, args...] => integer Calls the operating system function identified by fixnum, passing in the arguments, which must be either String objects, or Integer objects that ultimately fit within a native long. Up to nine parameters may be passed (14 on the Atari-ST). The function identified by fixnum is system dependent. On some Unix systems, the numbers may be obtained from a header file called syscall.h. syscall 4, 1, “hellon”, 6 # ‘4’ is write(2) on our box produces: hello
  35. 35. finding your syscall codes MacOS X /usr/include/sys/syscall.h Linux /usr/include/asm/unistd_32.h /usr/include/asm/unistd_64.h
  36. 36. a fragile approach the native syscall is a very primitive interface error codes file descriptors & pointer addresses modifiable buffers ruby syscall doesn’t support modifiable buffers but it does wrap errors as Errno exceptions
  37. 37. IO.sysx the hard way On MacOS X file = syscall 5, “test.dat”, 1 # open O_WRONLY (fcntl.h) syscall 4, file, “some textn”, 10 # write syscall 3, file, “”, 5 # read syscall 6, file # close produces: Errno::EFAULT: Bad address should be Errno::EBADF
  38. 38. complex parameters some kernel calls accept structured data types ruby represents these as String objects which can be prepared with Array#pack or Joel VanderWerf’s bit-struct library both approaches allow embedded nulls but on 1.9 remember to use 8-bit ASCII encoding
  39. 39. currently broken On Ruby 1.8.7 or 1.9.1 (upto patch 129) syscall 4, 1, “hellon#{0.chr}goodbyen”, 15 produces: ArgumentError: string contains null byte from (irb):1:in `syscall' from (irb):1 from /usr/local/bin/irb19:12:in `<main>'
  40. 40. Interprocess Communication IPC objects have descriptors like files but they don’t need to exist in the file system System V uses keys to identify IPC objects which are generated from paths using ftok() POSIX uses paths to identify IPC objects
  41. 41. signals an abstraction for software interrupts SIGTERM and SIGKILL are the best known but *nixen provide a wide range of signals which can be used for cheap IPC ruby handles all of these via the Signal core class which allows you to write traps on individual signals
  42. 42. sockets two varieties: domain sockets and network sockets Ruby provides a rich class hierarchy check out my other presentations for code examples some gotchas like DNS reverse lookup as default and Ruby doesn’t support kernel event queues but generally robust for ad hoc data streaming
  43. 43. pipes pipes provide a pair of end-points for processes to talk an unnamed pipe connects two related processes a named pipe lives in the file system it has user permissions (restricted by umask) and persists independently of processes
  44. 44. IO.pipe => read_file, write_file Ruby supports anonymous pipes, which can be used to communicate between a parent a child process, taking advantage of the fact that the file descriptors for the end-points will exist across a fork(). Good practice is for the parent process to close the reading end of the pipe whilst the child closes the writing end as soon as they return from fork() and the child should also close the reading end before it terminates to ensure buffered data is properly flushed.
  45. 45. named pipes process 1 process 2 File.umask 0 File.umask 0 MKFIFO = 132 MKFIFO = 132 syscall MKFIFO, fifo_name, 0666 syscall MKFIFO, “client”, 0666 fd = IO.sysopen “server”, File::RDONLY fd = IO.sysopen "server", File::WRONLY server = File.new fd, "r" server = IO.new fd, "w" client_name = server.gets.chomp server.puts fifo_name puts "#{Time.now}: [#{client_name}]" server.puts "hello world!" fd = IO.sysopen client_name, File::WRONLY server.close client = IO.new fd, "w" fd = IO.sysopen “client”, File::RDONLY message = server.gets.chomp client = IO.new fd, "r" client.puts message.reverse puts client.gets client.close client.close server.close File.delete “client” File.delete “server”
  46. 46. semaphores exist independently of processes provide blocking access allowing processes to be synchronised live in the file system and have a file descriptor usable from Ruby with syscall
  47. 47. posix semaphores process 1 process 2 require ‘fcntl’ Open, Wait, TryWait, Post = 268, 271, 272, 273 Open, Wait, Post, Close = 268, 271, 273, 269 s = syscall Open, “/tmp/s” s = syscall Open, “/tmp/s”, Fcntl::O_CREAT, 1911 begin syscall Wait, s t = Time.now puts “locked at #{Time.now}” syscall TryWait, s sleep 50 puts “locked at #{t}” puts “posted at #{Time.now}” rescue Exception => e syscall Post, s puts “busy at #{t}” syscall Close, s syscall Wait, s puts “waited #{Time.now - t} seconds” end produces: produces: locked at Thu May 28 01:03:23 +0100 2009 busy at Thu May 28 01:03:36 +0100 2009 posted at Thu May 28 01:04:13 +0100 2009 waited 47.056508 seconds
  48. 48. in the works shared memory message queues
  49. 49. managed platform access memory buffers structs c runtime access
  50. 50. ruby/dl the unsung hero of MRI
  51. 51. in the standard library introduced in ruby 1.8 but poorly documented and lightly used provides access to dynamically loaded libraries supports managed C-style memory access and callback functions written in Ruby a viable alternative to the extension API
  52. 52. memory management memory pointers encapsulated by DL::PtrData garbage collected uses free() or a custom deallocator handles realloc() automatically plays nicely with ruby strings and arrays [String|Array].to_ptr PtrData.struct! and PtrData.union!
  53. 53. using native syscall require ‘dl’ libc = DL.dlopen(‘libc.dylib’) open = libc[‘syscall’, ‘IISI’] write = libc[‘syscall’, ‘IIISI’] read = libc[‘syscall’, ‘IIIsI’] close = libc[‘syscall’, ‘III’] produces: file = open.call 5, “test.dat”, 1 [3, [5, "test.dat", 1]] write.call 4, file[0], “textn”, 5 [5, [4, 3, "textn", 5]] close.call 6, file[0] [0, [6, 3]] file = open.call 5, “test.dat”, 0 [3, [5, "test.dat", 0]] buffer = DL.malloc(10) read.call(3, file[0], buffer, 10) [5, [3, 3, "textn", 10]] close.call 6, file[0] [0, [6, 3]]
  54. 54. with a procedural slant require ‘dl’ file = open “test.dat”, 0x0209 CRT = DL.dlopen ‘libc.dylib’ write file, “textn” F = ‘syscall’ close file def open file, mode file = open “test.dat”, 0x0000 CRT[F, ‘IISI’].call(5, file, mode)[0] text = read file, 10 end close file def write fd, string, bytes = string.length CRT[F, ‘IIISI’].call(4, fd, string, bytes)[0] end def read fd, bytes = 1 buffer = DL.malloc(bytes) CRT[F, ‘IIIsI’].call(3, fd, buffer, bytes)[1][2] end def close fd CRT[F, ‘III’].call(6, fd)[0] end
  55. 55. worth checking out ruby-ffi RubyInline wilson
  56. 56. the bluffer’s reading list http://slides.games-with-brains.net http://www.jbrowse.com/text/rdl_en.html http://www.kegel.com/c10k.html http://www.ecst.csuchico.edu/~beej/guide/ipc/ http://beej.us/guide/bgnet/ http://wiki.netbsd.se/kqueue_tutorial
  57. 57. QEMFD? find out at Lone Star Ruby Conf 2009
  58. 58. QEMFD? http://slides.games-with-brains.net
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×