Linux Kernel Locking

Kernel locking begins with execution context. A lock that is correct between process-context callers can deadlock when the same data is reached from a softirq, hard interrupt, or nonmaskable interrupt (NMI). The primitive and its suffix must match every context that accesses the protected state.

Spin-Lock Semantics #

Declare and initialize ordinary spin locks with:

static DEFINE_SPINLOCK(queue_lock);

struct queue {
    spinlock_t lock;
};

spin_lock_init(&queue->lock);

On a non-PREEMPT_RT SMP kernel, spin_lock() disables preemption on the local CPU and spins until it acquires exclusive ownership. It does not disable local interrupts or softirqs. A holder must not sleep, schedule explicitly, perform a blocking allocation, or call code that can acquire a sleeping lock.

On a uniprocessor non-RT build, portions of the implementation can compile to preemption and compiler controls because no other CPU can contend. Correctness must not depend on that optimization; the same source must remain valid for SMP.

PREEMPT_RT changes spinlock_t and rwlock_t semantics so they use RT-mutex-based implementations and remain preemptible. Code that requires a truly spinning, non-preemptible lock across RT and non-RT kernels uses raw_spinlock_t, but raw critical sections must remain small and cannot call code that becomes sleeping on RT. The substitution is implemented inside locking headers and core code; application code should not reproduce it with configuration conditionals.

Matching Process, Softirq, and Hardirq Contexts #

Choose the narrowest variant that prevents same-CPU reentry:

spin_lock_bh() disables local bottom halves before acquiring the lock. spin_lock_irq() disables local maskable interrupts. spin_lock_irqsave(lock, flags) additionally saves the previous interrupt state, and spin_unlock_irqrestore(lock, flags) restores that exact state:

unsigned long flags;

spin_lock_irqsave(&device->lock, flags);
update_device_state(device);
spin_unlock_irqrestore(&device->lock, flags);

Use the same flags variable in one lexical acquisition/release pair and do not assume interrupts were enabled on entry. Disabling local interrupts prevents an interrupt on the holder’s CPU from waiting on its own lock; the spin lock itself excludes handlers running on other CPUs.

If data is shared only between hardirq instances, the handler may use spin_lock() when the interrupt line cannot reenter on the same CPU. The process-context accessor must still use an IRQ-disabling variant.

NMI Context Requires a Separate Design #

local_irq_disable() and spin_lock_irqsave() do not mask NMIs. If an NMI handler acquires a lock that interrupted code can hold on the same CPU, the NMI spins forever because the interrupted holder cannot resume.

NMI-shared data therefore needs a protocol designed for NMI:

• lockless or per-CPU recording with NMI-safe atomic operations;
• a sequence-counter pattern whose writer can never be preempted by the NMI reader in a deadlocking way;
• raw_spin_trylock() when dropping the NMI-side observation is acceptable;
• a raw lock only when every acquisition path proves that NMI reentry cannot occur.

“Raw” means that the lock preserves spinning semantics on PREEMPT_RT; it does not make an arbitrary critical section NMI-safe.

Acquire and Release Ordering #

Lock acquisition has acquire semantics, and unlock has release semantics. Operations inside the critical section are therefore ordered with respect to another CPU that protects the same data with the same lock:

spin_lock(&state->lock);
state->payload = payload;
state->ready = true;
spin_unlock(&state->lock);

A later successful acquirer observes the writes protected before the unlock. Locking is not a substitute for a protocol when some readers intentionally access the data without the lock. Such readers may need READ_ONCE(), release/acquire publication, RCU, or explicit barriers. Linux’s memory-barrier documentation also distinguishes lock acquire/release ordering from a general full memory barrier; code that needs ordering outside the critical section must use the documented barrier primitive.

Lock order remains a separate requirement. If one path takes A then B, another path must not take B then A. IRQ-state variants prevent local interrupt inversion, not cross-lock cycles.

Architecture Implementation #

The public spinlock_t API is defined in generic locking headers and maps through raw locking code to architecture operations. On SMP, raw_spinlock_t contains an arch_spinlock_t plus optional debugging and lockdep state. Architectures select an implementation appropriate to their atomic instructions and virtualization environment.

x86 uses the queued-spinlock infrastructure. The uncontended path operates on a compact lock word; contended callers form a queue rather than all repeatedly invalidating one cache line. Paravirtualized hooks can replace portions of the wait and wake behavior. Architecture code provides ordering and atomic primitives, while generic code provides API consistency, instrumentation, and configuration-dependent transformations.

Source layout is not an ABI. Verify a specific kernel through include/linux/spinlock*.h, kernel/locking/, and arch/<arch>/include/asm/ from the same revision.

Contention and Debugging #

Contention wastes CPU time and moves the lock’s cache line among processors. Keep spin-locked sections bounded, move allocations and slow I/O outside them, partition unrelated state, and avoid taking one global lock for per-CPU data. Replacing one lock with several can reduce contention but adds ordering and lifetime complexity, so measure before changing the design.

Kernel facilities help establish facts:

• CONFIG_LOCKDEP and lockdep annotations detect observed and potential dependency cycles, incorrect IRQ usage, and context mismatches.
• CONFIG_DEBUG_SPINLOCK adds ownership and state checks.
• CONFIG_PROVE_LOCKING enables broader locking validation.
• lockstat and perf lock report contention and acquisition behavior.

Debug builds alter timing and overhead. A clean lockdep run increases confidence but cannot prove that unexecuted paths have a correct lifetime or memory-ordering protocol.

Reader-Writer Spin Locks #

rwlock_t permits concurrent readers or one writer:

static DEFINE_RWLOCK(routes_lock);
unsigned long flags;

read_lock_irqsave(&routes_lock, flags);
route = lookup_route(key);
read_unlock_irqrestore(&routes_lock, flags);

write_lock_irqsave(&routes_lock, flags);
replace_route(key, replacement);
write_unlock_irqrestore(&routes_lock, flags);

Reader-writer spin locks require more atomic operations than a plain spin lock. They provide no portable fairness guarantee; a workload and implementation can delay a writer behind readers or delay readers behind a writer. A read lock cannot be upgraded safely to a write lock. If an operation might modify the structure, release and retry under the write lock or acquire the write lock from the start.

New uses deserve scrutiny:

• Use a mutex when all access is in sleepable process context, especially when a holder may block.
• Use RCU when reads dominate, readers can tolerate its access model, and updates can defer reclamation until after a grace period.
• Use a sequence lock/count for small snapshots when readers can retry and writers can be serialized.
• Use a plain spin lock when critical sections are short; its lower reader-side complexity can outperform rwlock_t.

Selection depends on context, hold time, contention, read/write ratio, and object lifetime rather than on the abstract appeal of parallel readers.

References #

• Kernel Documentation: Lock Types and Their Rules
• Kernel Documentation: Spin-Lock Lessons
• Kernel Documentation: Locking
• Kernel Documentation: Memory Barriers
• Kernel Documentation: Lockdep Design