x86 Protected Mode and Privilege

x86 protected mode combines descriptor-based segmentation, controlled control transfers, paging, and privileged instructions. Long mode reduces ordinary segmentation but retains descriptor tables, privilege levels, interrupt gates, and a task-state segment (TSS). Operating systems use those mechanisms to enter the kernel and isolate user execution, while page tables provide the primary memory-protection boundary on current x86-64 systems.

Table Registers #

The global descriptor table (GDT) contains code, data, and system descriptors shared by a logical processor. GDTR stores the GDT’s linear base and byte limit. GDT entry zero is conventionally a null descriptor, which makes a selector with index zero unusable for ordinary code or data access.

The local descriptor table (LDT) is an optional table selected through LDTR. LDTR contains a selector for an LDT system descriptor in the GDT and caches the resolved base, limit, and attributes. A task can have its own LDT in the legacy architecture, although current 64-bit operating systems rarely use one.

IDTR stores the linear base and byte limit of the interrupt descriptor table (IDT). The IDT is indexed by an exception or interrupt vector, not by an ordinary segment selector. Its entries are gates that identify handler code and control entry semantics.

The task register (TR) contains a selector for a TSS descriptor in the GDT and caches that descriptor. LGDT, LIDT, LLDT, and LTR load the table state under architectural privilege and type checks.

Selectors and Hidden Caches #

A 16-bit segment selector contains an index, table indicator (TI), and requested privilege level (RPL):

text

15                              3 2   1 0
+--------------------------------+---+---+
|             index              | TI|RPL|
+--------------------------------+---+---+

TI=0 selects the GDT, while TI=1 selects the current LDT. The processor multiplies the index by the descriptor size, checks the table limit, fetches the descriptor, and validates its type, presence, and privilege for the operation.

CS, SS, DS, ES, FS, and GS expose the selector but also contain a hidden cache of base, limit, type, and access rights. Memory references use the cache rather than reading the descriptor table again. Rewriting a GDT or LDT entry therefore does not change a loaded segment; software must reload the selector or perform the architecturally required transition.

LDTR and TR also cache resolved descriptors. A context switch that changes a GDT, LDT, or TSS must account for those caches rather than treating the visible selector as the complete state.

Segment Descriptors #

Legacy protected-mode code and data descriptors are eight bytes. They encode a 32-bit base, a 20-bit limit with optional 4 KiB granularity, a descriptor privilege level (DPL), present bit, descriptor class, and type-specific flags.

S=1 identifies a code or data descriptor; S=0 identifies a system descriptor or gate.
A code descriptor selects executable, readable, and conforming behavior.
A data descriptor selects writable and expand-down behavior.
G scales the limit, while D/B controls the default operand or stack size according to descriptor type.
In long mode, L identifies a 64-bit code segment; invalid combinations with D are rejected.

Segmentation adds the descriptor base to an instruction’s effective offset and checks the limit and access rights before paging. A flat 32-bit model uses base-zero, large-limit segments while retaining the checks.

In 64-bit mode, normal address generation treats CS, SS, DS, and ES bases as zero and does not apply their legacy limits. FS and GS retain separately controlled 64-bit bases. Code-segment type, CS privilege, and selected SS rules remain architecturally significant.

System Descriptors and Gates #

System descriptors identify LDTs, available or busy TSSs, and legacy gates. IA-32 uses eight-byte TSS and LDT descriptors. In 64-bit mode, TSS and LDT descriptors occupy sixteen bytes so they can contain a 64-bit base; the upper eight bytes consume the next GDT slot.

A call gate names a destination code selector and offset and provides a checked path for a far call to the same or a more privileged level. Legacy call gates can copy a configured number of parameters to the new stack. A far return reverses the privilege transition under another set of checks.

An IDT entry can be an interrupt gate, trap gate, or, in legacy mode, task gate. Interrupt and trap gates contain a handler selector and offset. An interrupt gate clears IF on entry; a trap gate preserves it. A 64-bit interrupt or trap gate also contains an IST selector used for emergency or dedicated stacks.

Privilege Values and Effective Checks #

x86 defines four privilege levels numbered 0 through 3. A smaller number is more privileged. The current privilege level (CPL) is represented by the low two bits of CS; kernels commonly run at CPL 0 and applications at CPL 3. Rings 1 and 2 exist but are not required by an operating-system design.

Each descriptor has a DPL, and each selector carries an RPL. For many data accesses, the processor forms an effective privilege level:

text

EPL = max(CPL, RPL)

Access to a normal data segment requires EPL <= DPL, along with type and presence checks. Because larger numbers are less privileged, an RPL can deliberately reduce the authority of a selector. SS loads use stricter rules: the selector RPL and writable data-segment DPL must match CPL.

Code transfers use rules specific to their mechanism:

A direct far transfer to a nonconforming code segment does not arbitrarily lower CPL; its DPL must match the required current privilege relationship.
A conforming code segment permits execution from numerically equal or less privileged CPLs when its DPL allows it, while CPL remains unchanged. Such a transfer does not grant access to the target DPL’s data.
A call-gate selector is checked against the gate DPL using CPL and RPL. The gate then controls entry to its destination code segment.
A software INT n checks the IDT gate DPL against CPL. Processor exceptions and external interrupts do not use the gate DPL as a caller authorization check because software did not invoke them through INT n.

No single CPL/DPL comparison describes every operation. Segment loads, conforming code, gates, task switches, IRET, port I/O, and privileged instructions each define their own checks.

Controlled Transitions #

User code cannot load a ring-0 code selector and jump to an arbitrary kernel address. A gate, exception, interrupt, task mechanism, or configured syscall instruction establishes a controlled entry point and saved return state.

For an interrupt or exception gate that enters a numerically lower CPL, the processor obtains a stack pointer for the destination privilege level from the current TSS. It switches stacks, saves the old stack context and return state, and begins at the gate’s handler. The exact frame, optional error code, and saved values depend on execution mode and vector.

A same-privilege gate normally keeps the current stack. In 64-bit mode, a nonzero IDT IST field selects its configured IST stack even without a CPL change. Kernel entry code must then save remaining registers, establish ABI invariants, validate user-controlled state, and handle nested entry safely.

Returning with IRET validates and restores the saved destination. A return to a less privileged CPL also restores the outer stack. Software must not place unchecked selectors, flags, or instruction pointers into an interrupt-return frame because return-time validation alone does not express kernel policy.

Interrupts and System Calls #

Exceptions are synchronous consequences of instruction execution. External interrupts arrive through an interrupt controller and are delivered when priority, masking, and processor state permit. Both ultimately select IDT vectors, but only selected exceptions push an architectural error code.

SYSCALL/SYSRET and SYSENTER/SYSEXIT avoid general IDT gate lookup and use model-specific registers to establish entry and return state. SYSCALL in 64-bit mode saves the next RIP in RCX, saves RFLAGS in R11, loads configured code selectors and entry RIP, and masks selected flags. It does not switch RSP; a kernel entry stub must move to a trusted stack without exposing unsafe intermediate state. SYSENTER has different configured stack semantics. Register arguments and error conventions belong to the operating-system syscall ABI, not to x86 itself.

Call gates remain architectural but current kernels generally use syscall instructions and interrupt gates because those paths fit flat address spaces and software-defined entry frames. Paging, SMEP, SMAP, protection keys, and other controls complement descriptor checks.

IA-32 Hardware Tasks #

The 80286 and IA-32 protected-mode architecture defines a hardware task as state represented by a TSS and TSS descriptor. An IA-32 TSS stores EIP, EFLAGS, general-purpose registers, segment selectors, CR3, an LDT selector, stack pointers for privilege levels 0 through 2, and an I/O-permission bitmap offset.

A hardware task switch can be initiated by:

A far CALL or JMP to a TSS descriptor.
A far transfer through a task gate.
An interrupt or exception whose IDT entry is a task gate.
IRET when EFLAGS.NT indicates a nested task.

The processor validates the destination, saves selected outgoing state in the current TSS, loads the destination TSS, marks descriptor busy state as required, loads TR and other task state, and sets the task-switched flag in CR0. Exact busy-bit, link, and exception behavior depends on the initiating mechanism.

A nested task switch initiated by CALL, a task gate, or an interrupt task gate can write the old TSS selector into the destination TSS backlink and set the nested-task (NT) flag. IRET with NT=1 follows the backlink to resume the previous task. A backlink is architectural control state rather than a scheduler-maintained process list; malformed or inconsistent task state can produce task-segment exceptions.

Current operating systems switch scheduled threads in software because hardware task switching has fixed state layout and side effects, while a kernel must also manage scheduler metadata, address-space policy, per-CPU state, floating-point and vector state, debug state, and implementation-specific mitigations. A software switch can save exactly the state required by the kernel’s context and ABI. IA-32 kernels still use a TSS for privilege-stack and I/O-bitmap functions without using hardware task switching for scheduling.

Long-Mode TSS Usage #

64-bit mode does not support hardware task switching, but every logical processor still needs a valid 64-bit TSS for normal kernel operation. The TSS contains:

RSP0, RSP1, and RSP2, used when an interrupt or exception enters the corresponding more-privileged CPL without selecting IST.
Seven interrupt-stack-table pointers, IST1 through IST7.
An I/O-permission-bitmap offset.

An IDT gate’s three-bit IST field selects one of the seven IST pointers. IST is useful when the current stack may be unusable or when a handler requires a separate stack, such as a double-fault path. IST hardware loads a new stack pointer; it does not provide a complete software nesting policy. Entry code must still prevent uncontrolled reuse or overflow of a finite emergency stack.

The I/O-permission bitmap can allow or deny IN and OUT by port for less-privileged code, subject to IOPL and mode rules. Its location and bounds are derived from the TSS descriptor limit, so the descriptor, offset, bitmap, and terminating byte must be constructed consistently.

The TSS descriptor remains a sixteen-byte system descriptor in the GDT. LTR loads it into TR, sets or observes the defined busy type, and caches its base and limit. Each logical processor normally has its own TSS because privilege entry needs a processor-local trusted stack. Loading TR is initialization for entry handling, not a scheduler context switch.

Long-Mode Considerations #

Long mode retains the standard CPL 0/CPL 3 kernel-user division, code descriptors, IDT gates, the GDT, TR, and a TSS. It removes most segment-base and limit effects from ordinary 64-bit addressing and omits hardware task switches. Page-table user/supervisor, writable, and execute-disable permissions therefore form the primary address-space boundary.

Descriptor state still cannot be omitted. A 64-bit code segment must have a valid descriptor, interrupt gates contain a code selector, privilege entry uses TSS stack state, and compatibility mode applies additional legacy descriptor semantics. Boot and kernel code should follow the processor vendor’s current system-programming manual because reserved fields and mode-transition ordering are architectural requirements.

Primary References #

Intel 64 and IA-32 Architectures Software Developer Manuals, especially Volume 3A
AMD64 Architecture Programmer’s Manual, especially Volume 2