Share and Share Alike

Share and Share AlikeUsing System V shared memory constructs inMRI Ruby projects

Who Am I?● Jeremy Holland● Senior Lead Developer at CentreSource in beautiful Nashville, TN● Math and Algorithms nerd● Scotch drinker● @awebneck, github.com/awebneck, freenode: awebneck, etc.

The Problem ● FREAKIN ● HUGE ● BINARY ● TREE

How huge?● Huge. Millions of nodes, each node holding ~500 bytes● e.g. Gigabytes of data● K-d tree of non-negligible dimension (varied, around 6-10)● No efficient existing implementation that would serve the purposes needed  Fast search  Reasonably fast consistency

Things we considered ...and discarded● Index the tree, persist to disk  Loading umpteen gigs of data from disk takes a spell.  Reload it for each query  WAY TOO SLOW

Things we considered ...and discarded● Index once and hold in memory  Issues both with maintaining index consistency and balance  Difficult to share among many processes / threads without duplicating in memory.

Things we considered ...and discarded● DRb  Simulates memory shared by multiple processes, but not really  While the interface to search the tree is available to many different processes, actually searching it takes place in the single, server-based process

Enter Shared Memory● Benefits  Shared segment actually accessible by multiple, wholly separate processes  Built-in access control and permissions  Built-in per-segment semaphore● Drawbacks  With great power comes great responsibility  Acts like a bytearray – manual serialization

Ruby-level memory paradigm vs C-level memory paradigm● Ruby:  Everything goes on the heap  Garbage collected - no explicit freeing of memory● C:  Local vars, functions, etc. on the stack  Explicit allocations on the heap (malloc)  Explicit freeing of heap – no GC

Ruby● Before start of process

Ruby● Process starts● Heap begins to grow

Ruby● Process runs● Heap continues to grow with additional allocations

Ruby● Process runs● GC frees allocated memory no longer needed...

Ruby● ...so it can be reallocated for new objects

Ruby● Process ends● Heap freed

C● Process starts● Stack grows to hold functions, local vars

C● Process runs● Memory is explicitly allocated from the heap in the form of arrays, structs, etc.

C● Process runs● A function is called, and goes on the stack

C● Process runs● The function returns, and is popped off the stack

C● Process runs● The item in the heap, no longer needed, is explicitly freed

C● Process runs● A new array is allocated from the heap

C● Process ends (untidily)● The stack and heap are reclaimed by the OS as free

TRUTHRuby itself has no concept of shared memory.

TRUTH C does.

Shared Memory● A running process (as viewed from the C level)

Shared Memory● A shared segment is created with an explicit size – like allocating an array

Shared Memory● The segment is ”attached” to the process at a virtual address

Shared Memory● Yielding to the process a pointer to the beginning of the segment

Shared Memory● A new process starts, wishing to attach to the same segment.

Shared Memory● It asks the OS for the identifier of the segment based on an integer key Are you there? Yup!

Shared Memory● ...and attaches it to itself in fashion similar to the original.

Shared Memory● Both processes can now - depending on permissions – read and write from the segment simultaneously!

Shared Memory● The first process finishes with the segment and detaches it.

Shared Memory● And thereafter, ends.

Shared Memory ● ...leaving only the second process, still attached

Shared Memory ● Now, the second process detaches...

Shared Memory ● ...and subsequently ends

Shared Memory● Note that the shared segment is still in persisted in memory● Can be reattached to another process with permission to do so

Shared Memory● Later, a new process comes along and explicitly destroys the segment, all processes being finished with it.

How its done: Configuration● Precisely how much memory can be drafted into service for sharing purposes is controlled by kernel parameters  kernel.shmall – the maximum number of memory pages available for sharing (should be at least ceil(shmmax / PAGE_SIZE))  kernel.shmmax – the maximum size in bytes of a single shared segment  kernel.shmmni – the maximum number of shared segments allowed.

How its done: Configuration● To view your current settings:

How its done: Configuration● Or...

How its done: Configuration● Setting the values temporarily can be accomplished with sysctl...

How its done: Configuration● ...or more permanently by editing /etc/sysctl.conf

How its done: Creating New and Acquiring Existing Segments● int shmget(key_t key, size_t size, int shmflag)  key_t key: integer key identifying the segment or IPC_PRIVATE  size_t size: integer size of segment in bytes (will be rounded up to next multiple of PAGE_SIZE)  int shmflag: mode flag consisting of standard o- g-w and IPC_CREAT (to create or attach to existing) and optionally IPC_EXCL (to throw an error if it already exists)

How its done: Creating New and Acquiring Existing Segments● int shmget(key_t key, size_t size, int shmflag)  Returns: valid segment identifier integer on success, or -1 on error

How its done: Attaching segments● void * shmat(int shmid, const void *shmaddr, int shmflag)  shmid: integer identifier returned by a call to shmget  shmaddr: Pointer to the address at which to attach the memory. Almost always want to leave this NULL, so that the system will address the segment wherever theres room for it.

How its done: Attaching segments● void *shmat(int shmid, const void *shmaddr, int shmflag)  shmflag: several flags for controlling the attachment – most importantly, SHM_RDONLY (what it looks like)  returns: a void pointer to the start of the attached segment, or (void *)-1 on error

How its done: Detaching segments● int shmdt(const void *shmaddr)  shmaddr: Pointer returned by the call to shmat  returns: 0 or -1 on error

How its done: Getting segment information● int shmctl(int shmid, int cmd, struct shmid_ds *buf)  shmaddr: The identifier returned by shmget  cmd: The command to execute – for this purpose, IPC_STAT  Buf: A shmid_ds struct

How its done: Getting segment informationstruct shmid_ds { struct ipc_perm; permissions/ownership size_t shm_segsz; size of segment in bytes time_t shm_atime; last attachment time time_t shm_dtime; last detachment time time_t shm_ctime; last change time pid_t shm_cpid; pid of creator pid_t shm_lpid; pid of last attached shmatt_t shm_nattch; # of attached processes}

How its done: Destroying segments● int shmctl(int shmid, int cmd, struct shmid_ds *buf)  shmaddr: The identifier returned by shmget  cmd: IPC_RMID  Buf: A shmid_ds struct (you can ignore it afterwards, but itll throw a fit if you dont provide it)

ExamplesExamples

Challenges and Caveats● Addressing  Segments are attached wherever there is room for them in the attaching process address space

Challenges and Caveats● Maybe here in one process... 0x7f195bda2000

Challenges and Caveats● ...maybe here in another 0x73f882c1f000

Challenges and Caveats● So if you store an 0x7f195bda2004 absolute pointer in the segment that points somewhere else in the segment...

Challenges and Caveats● Its not terribly likely to 0x7f195bda2004 point where you think it should when referenced in a separate process

Challenges and Caveats● Addressing  Segments are attached wherever there is room for them in the attaching process address space  Absolute pointers are effectively useless  Relative pointers – i.e. Offsets  BSTs as heaps (the data structure).  Serialization.

Challenges and Caveats● Duplication and copying  Ruby primitivesques (numerics, strings, etc) are all allocated on the heap  Shared data must be effectively copied  Diminishes the usefulness of the tool for certain applications (large data sharing)  Not everything is a nail

Challenges and Caveats● Duplication and copying  But... fantastic for certain applications  Search  Search the shared structure at c level  Copy and coerce results to ruby objects  |results| << |data to be searched|  Semaphore, interprocess messaging  Built-in to the IPC/SHM lib!

Semaphore● Tracking resource allocation  Effectively an integer checked when a process allocates some resource  If nonzero, decrement  If zero, the resource isnt available● Simple, but slightly weird API.

Message Queues● Push bytearray/string messages into the queue, shift em off● Simple, slightly less bizarre API

In closing...● Quite exciting, this computer magic● Dont just use it because its there  Have a NEED● Dont be afraid to drop to C  Dont know C?  Learn it – a pretty simple language, when alls said and done  Building ruby C extensions is actually pretty painless

Questions / Comments In which I probably get trolled

Thanks for listening!Enjoy the rest of the conference!

Share and Share Alike

by awebneck on Sep 26, 2012

Accessibility

Categories

Upload Details

Usage Rights

Statistics

Share and Share Alike Presentation Transcript