The document discusses the slab allocator, which was created two decades ago by Jeff Bonwick to efficiently manage kernel memory. The author describes loving several key aspects of the slab allocator, including its caching, debugging support, integration with tools like MDB, and porting to userspace as libumem. While libumem had worse performance than other allocators for small allocations, per-thread caching was later added to improve it. The slab allocator remains central to illumos and has stood the test of time due to its thoughtful design.

1. Down Memory Lane:
Two Decades with the Slab Allocator
CTO
bryan@joyent.com
Bryan Cantrill
@bcantrill

2. Two decades ago…

3. Aside: Me, two decades ago…

4. Aside: Me, two decades ago…

5. Slab allocator, per Vahalia

6. First <3: Allocator footprint

7. I <3: Cache-aware data structures

8. I <3: Cache-aware data structures
Hardware Antiques Roadshow!

9. I <3: Magazine layer

10. I <3: Magazine autoscaling

11. I <3: Debugging support
• For many, the most salient property of the slab allocator is its
eponymous allocation properties…
• …but for me, its debugging support is much more meaningful
• Rich support for debugging memory corruption issues:
allocation auditing + detection of double-free/use-after-free/
use-before-initialize/buffer overrun
• Emphasis was leaving the functionality compiled in (and
therefore available in production) — even if off by default
• kmem_flags often used to debug many kinds of problems!

12. I <3: Debugging support

13. I <3: Debugger support

14. I <3: MDB support
• Work in crash(1M) inspired deeper kmem debugging support
in MDB, including commands to:
• Walk the buffers from a cache (::walk kmem)
• Display buffers allocated by a thread (::allocdby)
• Verify integrity of kernel memory (::kmem_verify)
• Determine the kmem cache of a buffer (::whatis)
• Find memory leaks (::findleaks)
• And of course, for Bonwick, ::kmastat and ::kmausers!

15. I <3: libumem
• Tragic to have a gold-plated kernel facility while user-land
suffered in the squalor of malloc(3C)…
• In the (go-go!) summer of 2000, Jonathan Adams (then a Sun
Microsystems intern from CalTech) ported the kernel memory
allocator to user-land as libumem
• Jonathan returned to Sun as a full-time engineer to finish and
integrate libumem; became open source with OpenSolaris
• libumem is the allocator for illumos and derivatives like
SmartOS — and has since been ported to other systems

16. libumem performance
• The slab allocator was designed to be scalable — not
necessarily to accommodate pathological software
• At user-level, pathological software is much more common…
• Worse, the flexibility of user-level means that operations that
are quick in the kernel (e.g., grabbing an uncontended lock)
are more expensive at user-level
• Upshot: while libumem provided great scalability, its latency
was worse than other allocators for applications with small,
short-lived allocations

17. libumem performance: node.js running MD5

18. I <3: Per-thread caching libumem
• Robert Mustacchi added per-thread caching to libumem:
• free() doesn’t free buffer, but rather enqueues on a per-
size cache on ulwp_t (thread) structure
• malloc() checks the thread’s cache at given size first
• Several problems:
• How to prevent cache from growing without bound?
• How to map dynamic libumem object cache sizes to fixed
range in ulwp_t structure without slowing fast path?

19. I <3: Per-thread caching libumem
• Cache growth problem solved by having each thread track
the total memory in its cache, and allowing per-thread cache
size to be tuned (default is 1M)
• Solving problem of optimal malloc() in light of arbitrary
libumem cache sizes is a little (okay, a lot) gnarlier; from
usr/src/lib/libumem/umem.c:

20. I <3 the slab allocator!
• It shows the importance of well-designed, well-considered,
well-described core services
• We haven’t need to rewrite it — it has withstood ~4 orders of
magnitude increase in machine size
• We have been able to meaningfully enhance it over the years
• It is (for me) the canonical immaculate system: more one to
use and be inspired by than need to fix or change
• It remains at the heart of our system — and its ethos very
much remains the zeitgeist of illumos!

21. Acknowledgements
• Jeff Bonwick for creating not just an allocator, but a way of
thinking about systems problems and of implementing their
solutions — and the courage to rewrite broken software!
• Jonathan Adams for not just taking on libumem, but also
writing about it formally and productizing it
• Robert Mustacchi for per-thread caching libumem
• Ryan Zezeski for bringing the slab allocator to a new
generation with his Papers We Love talk

22. Further reading
• Jeff Bonwick, The Slab Allocator: An Object-Caching Kernel Memory
Allocator, Summer USENIX, 1994
• Jeff Bonwick and Jonathan Adams, Magazines and Vmem: Extending
the Slab Allocator to Many CPUs and Arbitrary Resources, USENIX
Annual Technical Conference, 2001
• Robert Mustacchi, Per-thread caching in libumem, blog entry on
dtrace.org, July 2012
• Ryan Zezeski, Memory by the Slab: The Tale of Bonwick’s Slab
Allocator, Papers We Love NYC, September 2015
• Uresh Vahalia, UNIX Internals, 1996