Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Linux locking mechanisms
Mark Veltzer
veltzer@gnu.org
Who am I?
● Linux kernel hacker
● Current maintainer of gnu grep(1)
● Free Source evangelist
● CTO of Hinbit
● Political p...
Why locking?
● To avoid race conditions in accessing shared memory.
● These occur because of: user space pre-emption which...
Avoiding locking - techniques
● Have each thread/CPU have it's own data.
● Use atomic operations (hardware) instead of loc...
User space vs kernel space locking
● Is completely different
● Different mechanisms, different performance
considerations,...
User space locking
User space locking mechanisms
● Are not allowed to block interrupts. Ever!
● This is derived from the definition of what a...
User space locking primitives
● pthread Spin lock
● Futex
● pthread mutex
● pthread Readers/writer lock.
● POSIX semaphore...
User space spin lock - intro
● Is implemented as a simple TAS/CAS loop with CPU
relaxing and memory barrier.
● Pure user s...
User space spin lock - issues
● The API is straight forward.
● The problem with this API is that IT DOES NOT
DISABLE INTER...
User space spin lock – when to
use?
● Use only when the two racing contexts are
running on two different cores and are the...
Futex – Fast user space locking
● The idea is to avoid trips to the kernel in the non
contended case.
● A mutex build half...
What happens when you die with a
lock held?
● Here are some suggestions:
– OS does nothing → deadlocks
– OS releases the l...
Linux has no threads
● Do you remember that Linux has no concept of a “thread”?
● Threads are just processes which happen ...
pthread_mutex
● Is now days just a wrapper for a futex.
● Could be used between processes (strange, but oh so true).
● Cou...
Pthread readers/writer lock
● Is based on the futex.
● This means good performance.
● Standard, feature poor implementatio...
POSIX semaphores
● Based on the futex.
● Again, good performance.
● Use this and not the Sys V version unless
you need the...
Sys V semaphore
● Reminder: Sys V is AT&T's version of UNIX
dating to circa 1983. In that version important
API's like thi...
Kernel locking
Kernel locking primitives
● Mutex
● Spinlock (3 types)
● Semaphore
● RW semaphores
Mutexes
● Go to sleep when finding the lock locked.
● This means they can only be used in contexts where you are
allowed t...
Spin locks
● Most common kernel locking primitive.
● Are divided into 3 types: regular, BH and IRQ.
● Regular spin locks j...
Spin lock (irq version)
● Turning of IRQs is quite fast (IF, CLI, STI are
really fast on INTEL).
● Very brutal as it incre...
When to use each?
● Passive vs Passive
– Use a mutex in interruptible mode (you are allowed to sleep in both).
– Or semaph...
Semaphores
● semaphore.h
● Usually used as a mutex and not as a semaphore.
● Up and down methods do not accept ticket numb...
RW semaphores
● rwsem.h
● Offer more performance when number of readers
outnumbers number of writers.
● Again, does not su...
RW lock
● rwlock.h
● By Ingo Molnar (author of the real time patch).
● Supports priority inheritance.
● Use this instead o...
The RT patch
● Runs all irq handlers in their own threads with
other interrupts enabled.
● Turns all spinlocks into mutexe...
Upcoming SlideShare
Loading in …5
×

Linux Locking Mechanisms

2,601 views

Published on

Agenda:
In this talk we will present various locking mechanisms implemented in the linux kernel.
From System V locks to raw spinlocks and the RT patch.

Speaker:
Mark Veltzer - CTO of Hinbit and a senior instructor at John Bryce. Mark is also a member of the Free Source Foundation and contributes to many free projects.
https://github.com/veltzer

Published in: Software
  • Be the first to comment

Linux Locking Mechanisms

  1. 1. Linux locking mechanisms Mark Veltzer veltzer@gnu.org
  2. 2. Who am I? ● Linux kernel hacker ● Current maintainer of gnu grep(1) ● Free Source evangelist ● CTO of Hinbit ● Political philosopher (checkout my book “‫ןוטלשלטון‬ ‫”ההמון‬ at book stores near you...) ● Jazz piano player
  3. 3. Why locking? ● To avoid race conditions in accessing shared memory. ● These occur because of: user space pre-emption which is based on timer interrupts (userspace), multi-core (userspace), interrupts in general (kernelspace), multi-core (kernelspace). ● Locking is not the only way to avoid such race conditions ● But this presentation is about locking and only about locking... ● In general locking is bad because it blocks your programs from executing and so slows your program ● Avoid it when you can.
  4. 4. Avoiding locking - techniques ● Have each thread/CPU have it's own data. ● Use atomic operations (hardware) instead of locking (software). ● Lock free programming. ● RCU/COW. ● Readers/Writer locks. ● Not using the shared memory model but rather the actor model for multi-processing/multi-threading. ● And many more techniques. ● Alas, we are here to talk about locking.
  5. 5. User space vs kernel space locking ● Is completely different ● Different mechanisms, different performance considerations, different API ● But ultimately they work in concert.
  6. 6. User space locking
  7. 7. User space locking mechanisms ● Are not allowed to block interrupts. Ever! ● This is derived from the definition of what a secure operating system is. ● If you have code in the kernel you can expose an API to user space to block and allow interrupts. ● This is considered a bad idea. ● First of all because it allows user space bugs to lock up your system. ● Second because it interferes with other kernel mechanisms (like watchdogs, RCU and more). ● DON'T DO IT!
  8. 8. User space locking primitives ● pthread Spin lock ● Futex ● pthread mutex ● pthread Readers/writer lock. ● POSIX semaphore ● SYS V semaphore
  9. 9. User space spin lock - intro ● Is implemented as a simple TAS/CAS loop with CPU relaxing and memory barrier. ● Pure user space implementation. ● DOES NOT DISABLE INTERRUPTS! ● Did I mention that it DOES NOT DISABLE INTERRUPTS?!? ● It is interesting to note that IT DOES NOT DISABLE INTERRUPTS. ● And finally note that NO INTERRUPTS ARE DISABLED
  10. 10. User space spin lock - issues ● The API is straight forward. ● The problem with this API is that IT DOES NOT DISABLE INTERRUPTS ● This means that you may end up spinning for a whole time slice (~1ms) if the two racing contexts are on the same core. ● This may also happen if two context are on different cores but one is pre-empted by some other context. ● This is really bad.
  11. 11. User space spin lock – when to use? ● Use only when the two racing contexts are running on two different cores and are the highest priority contexts on these two cores. ● Usually this is only fulfilled on a dedicated RT patched Linux system. ● Otherwise you get period spinning episodes. ● Kapish?!?
  12. 12. Futex – Fast user space locking ● The idea is to avoid trips to the kernel in the non contended case. ● A mutex build half in user space and half in kernel space. ● State of the lock is in user space. ● Wait list is in kernel space. ● Allows to lock/unlock without calling kernel space in the non contended case. ● A Masterpiece of Linux engineering!
  13. 13. What happens when you die with a lock held? ● Here are some suggestions: – OS does nothing → deadlocks – OS releases the lock → other contexts die because of inconsistent data – OS releases the lock and notifies the next context locking the lock that the previous owner died → This is what Linux does. ● This feature of locks is called robustness.
  14. 14. Linux has no threads ● Do you remember that Linux has no concept of a “thread”? ● Threads are just processes which happen to share a lot of memory created with the clone(2) system call. ● Don't tell this to user space developers in your company (they tend to freak out about this). ● This means that every locking mechanism in Linux can be used for multi-processing as well as for multi-threading. ● This is why futexes were made robust. ● Futexes are robust by doing postmortem on dead processes and examining the locks they leave behind in order to unlock them and mark them as suspicious.
  15. 15. pthread_mutex ● Is now days just a wrapper for a futex. ● Could be used between processes (strange, but oh so true). ● Could be made robust using the undocumented API pthread_mutexattr_setrobust(3). ● I found the documentation for this API on MSDN, of all places…:) ● Supports recursiveness, two types of priority inheritance, sharing between processes, priority ceiling and more. ● Makes lousy coffee, though...
  16. 16. Pthread readers/writer lock ● Is based on the futex. ● This means good performance. ● Standard, feature poor implementation. ● Build your own if you need more features. ● Could be used to synchronize processes and threads.
  17. 17. POSIX semaphores ● Based on the futex. ● Again, good performance. ● Use this and not the Sys V version unless you need the Sys V particular features. ● Could be used to synchronize both processes and threads.
  18. 18. Sys V semaphore ● Reminder: Sys V is AT&T's version of UNIX dating to circa 1983. In that version important API's like this one were first introduced into the UNIX world. ● Sys V semaphores are, however, crap. ● This is because they always go to the kernel. Even in the non contended case. ● Do not use. Use POSIX semaphores instead.
  19. 19. Kernel locking
  20. 20. Kernel locking primitives ● Mutex ● Spinlock (3 types) ● Semaphore ● RW semaphores
  21. 21. Mutexes ● Go to sleep when finding the lock locked. ● This means they can only be used in contexts where you are allowed to go to sleep. ● This means passive(user) context, kernel thread or workqueue context and threaded IRQ context (?!?). ● Not allowed in IRQ handlers or tasklets. ● Has 3 modes: interruptible, killable and uninterruptible. ● Try to use interruptible as much as possible as bugs in kernel code may cause non killable processes. ● Support priority inheritance under the RT patch.
  22. 22. Spin locks ● Most common kernel locking primitive. ● Are divided into 3 types: regular, BH and IRQ. ● Regular spin locks just turn off scheduling on the current CPU (in addition to being a spin lock). ● BH turn off Bottom half mechanisms (including tasklets) on the current CPU (in addition to being a spin lock). ● IRQ ones turn off interrupts on the local CPU (in addition to being a spin lock). ● Sleeping, waiting or doing heavy computation with spin locks held is considered reason for being banned from LKML. ● Turn into Mutexes under the RT patch and then support priority inheritance.
  23. 23. Spin lock (irq version) ● Turning of IRQs is quite fast (IF, CLI, STI are really fast on INTEL). ● Very brutal as it increases latency in real time implementations. ● Try not to access data structure from IRQ context so you won't have to use this. ● However, this is still one of the most common locking primitives
  24. 24. When to use each? ● Passive vs Passive – Use a mutex in interruptible mode (you are allowed to sleep in both). – Or semaphore in interruptible mode. – Or a regular spin lock. You are in no danger of spinning for long since scheduling on the current CPU is disabled. Interrupts may come in and so do tasklets but these are quick. ● Passive vs BH – spinlock_bh ● Passive vs IRQ – Use spin lock irq. – The irq part prevents races on the current CPU. – The spin lock part prevents races with other CPUs. ● BH vs BH – Spinlock ● BH vs IRQ – Spinlock irq ● IRQ vs IRQ – Spinlock irq. ● See “Rusty Russells Unreliable Guide to Locking”
  25. 25. Semaphores ● semaphore.h ● Usually used as a mutex and not as a semaphore. ● Up and down methods do not accept ticket number but always increase and decrease by 1. ● Ticket/permit count can be determined at creation time. ● Semaphores do not offer priority inheritance. Even under the RT patch. ● This means that any system call that uses this is unfit to be used in the critical path of a real time application. ● 3 modes of operation (like the mutex).
  26. 26. RW semaphores ● rwsem.h ● Offer more performance when number of readers outnumbers number of writers. ● Again, does not support priority inheritance. Even under the RT patch. ● Famous is the current->mm->mmap_sem that protects each processes virtual memory description. ● Good reason not to use malloc(3) in real time systems.
  27. 27. RW lock ● rwlock.h ● By Ingo Molnar (author of the real time patch). ● Supports priority inheritance. ● Use this instead of RW semaphores. ● Again, gives better performance when number of readers out numbers number of writers.
  28. 28. The RT patch ● Runs all irq handlers in their own threads with other interrupts enabled. ● Turns all spinlocks into mutexes to reduce latency and allow high priority tasks to get the CPU ASAP. ● If you really need a spin lock you can use the raw spinlock API which will give you a true spinlock even under the RT patch.

×