x86 and x86-64 Instruction Encoding

x86 uses variable-length instructions. An instruction can contain legacy prefixes, an opcode, a ModR/M byte, a scale-index-base (SIB) byte, a displacement, and an immediate. Only the opcode is always present:

Long mode can add a REX prefix between the legacy prefixes and opcode. VEX, XOP, and EVEX encodings replace parts of this layout for newer instruction families. Regardless of the form, an architectural x86 instruction cannot exceed 15 bytes.

The fields are not a direct serialization of assembly-language tokens. An opcode can encode part of an operand, a ModR/M field can extend an opcode, and an address expression selects one of the processor’s addressing forms rather than requesting a sequence of additions and multiplications.

Operations and Operands Do Not Have a Fixed Boundary #

Assembly syntax presents an instruction mnemonic followed by zero or more operands:

xchg eax, ecx
inc  dword [eax]
mov  edx, [ecx + 0x32251212]

The binary encoding does not reserve one field for the mnemonic and separate fields for every operand. x86 uses several compact encoding strategies.

ModR/M Opcode Extensions #

Many instructions use the three-bit ModR/M reg field as additional opcode bits. Intel and AMD syntax describes these forms as /0 through /7. For example, FF /0 is INC r/m32, while FF /1 is DEC r/m32.

In 32-bit mode, these instructions encode as:

ff 05 00 00 00 00    inc dword [0]
ff 0d 00 00 00 00    dec dword [0]

Both use opcode byte FF. Their ModR/M bytes differ:

05 = 00 000 101    mod=00, opcode extension=/0, r/m=101
0d = 00 001 101    mod=00, opcode extension=/1, r/m=101

With mod=00, r/m=101 denotes a displacement-only address in 32-bit address size. The four zero bytes are that absolute disp32. The reg field cannot name a register in these encodings because it selects the operation.

Registers Embedded in Opcodes #

Some opcode families embed a register number in the low three bits of the final opcode byte. Instruction tables describe this as +rb, +rw, +rd, or +rq, depending on the register width.

The one-byte opcodes 90 through 97, for example, exchange EAX with one of the eight 32-bit general-purpose registers in 32-bit mode:

An instruction table writes this family as 90+rd. By contrast, xchg ebx, ecx uses opcode 87 plus a ModR/M byte:

87 d9    xchg ebx, ecx

Other +r families include PUSH, POP, MOV with an immediate, and BSWAP. The one-byte INC and DEC register forms use bytes 40 through 4F only outside 64-bit mode. Long mode assigns those byte values to REX prefixes.

Assembly Address Expressions Select Encoding Fields #

An expression such as [eax + ebx*4 + 0x32] does not encode arithmetic instructions. It asks the assembler to choose ModR/M, SIB, and displacement fields that make the address-generation hardware compute the expression.

For example:

ff 44 98 32    inc dword [eax + ebx*4 + 0x32]

The bytes decompose as follows:

• FF /0 selects INC r/m32.
• ModR/M 44 is 01 000 100: memory addressing, /0, SIB follows.
• SIB 98 is 10 011 000: scale 4, index EBX, base EAX.
• 32 is an 8-bit displacement.

The same written mnemonic can therefore use different lengths and field layouts depending on its operands.

Legacy Prefixes #

Legacy prefixes occupy four semantic groups. An instruction can use at most one effective prefix from each group. Encoders should not emit duplicate prefixes from one group: behavior for redundant or conflicting prefixes has varied across processors.

Assemblers use a conventional group order, although legacy decoding does not require all groups to appear in numerical order. A REX prefix is different: when present, it must follow the legacy prefixes and immediately precede the opcode or opcode-map escape.

Lock and Repeat Prefixes #

F0 is the LOCK prefix. It is valid only for specified read-modify-write instructions whose destination is memory. A valid locked operation is atomic with respect to other logical processors. Modern processors normally obtain this property through cache coherence while retaining exclusive ownership of the affected cache line. They do not assert an external bus lock for every LOCK instruction. Split accesses and uncacheable memory can require stronger handling. Using LOCK with an unsupported instruction or register destination raises #UD.

F3 represents REP or REPE/REPZ, while F2 represents REPNE/REPNZ for string operations. The count comes from CX, ECX, or RCX, selected by address size. The exact stop condition depends on the instruction.

For several SSE and later instruction families, F2, F3, or 66 is a mandatory opcode-selection prefix rather than a repeat or size operation. The same opcode-map bytes can identify different instructions according to that prefix. VEX and EVEX encode the equivalent selection in their pp field.

Segment Overrides and Branch Hints #

The segment override bytes are:

In 16- and 32-bit modes, an override changes the segment used for a memory operand. In 64-bit mode, CS, DS, ES, and SS bases are treated as zero, so only FS and GS overrides normally affect address formation.

Some older processors interpreted 2E and 3E on conditional branches as static not-taken and taken hints. Current software should not depend on those hints.

Operand-Size Override #

66 selects the non-default operand size for instructions to which it applies. In a 32-bit code segment, it commonly switches between 32- and 16-bit operands:

ff 05 00 00 00 00       inc dword [0]
66 ff 05 00 00 00 00    inc word  [0]

The prefix changes the width of the integer loaded and stored by INC. It does not change this example’s 32-bit address displacement. Operand size and address size are independent.

In 64-bit mode, the default integer operand size is commonly 32 bits. 66 selects 16 bits, while REX.W selects 64 bits for instructions that support those sizes. Individual opcode definitions can establish different rules.

Address-Size Override #

67 selects a non-default address size. In a 32-bit code segment it can select 16-bit addressing. In 64-bit mode it selects 32-bit rather than 64-bit addressing; long mode does not provide 16-bit address-size encodings.

Address size determines which registers and addressing table ModR/M and SIB use. It also selects the count register for string instructions. It does not by itself change the width of the value being loaded, stored, or computed.

Opcode Maps #

Legacy encodings use several opcode maps:

• The one-byte map contains opcodes other than the 0F escape.
• 0F xx selects the two-byte map.
• 0F 38 xx and 0F 3A xx select additional three-byte maps.

An opcode can contain fields for direction, width, condition code, or a register. The instruction manual, not the number of opcode bytes alone, determines whether ModR/M, displacement, and immediate fields follow.

VEX and EVEX prefixes encode an opcode-map selector instead of spelling the legacy 0F, 0F 38, or 0F 3A bytes. They also encode mandatory-prefix state, vector length, additional register bits, and source operands. Their apparent prefix bytes are therefore part of a distinct encoding form, not a sequence of independent legacy modifiers.

The ModR/M Byte #

ModR/M divides into mod, reg, and r/m:

The opcode defines the meaning of reg. It can name an operand register or provide a /0 through /7 opcode extension. The mod and r/m fields select a register or memory addressing form.

Special r/m and SIB values modify the mod=00 rule.

Register Form #

With mod=11, r/m names a register. If reg is an operand field, it names another register. Without extension bits, the three-bit codes select:

Operand size and instruction class determine whether the code identifies a general-purpose, MMX, vector, control, debug, or other register class. REX, VEX, and EVEX extension bits can add higher-numbered registers.

Memory Form #

With mod other than 11, r/m selects a memory-addressing form. In 32-bit address size:

• Most r/m values select a base register.
• r/m=100 selects a following SIB byte.
• mod=00, r/m=101 selects an address consisting of disp32 with no base.
• mod=01 adds a sign-extended disp8.
• mod=10 adds a disp32.

In 64-bit address size, mod=00, r/m=101 instead selects RIP + sign-extended disp32. RIP-relative addressing supports position-independent references to nearby code and data. An absolute address can require another encoding strategy, including a SIB no-base form or loading the address into a register.

The decoder selects an effective-address form in stages:

Sixteen-bit address size uses a separate table composed from BX, BP, SI, and DI; it has no SIB byte. The displacement selected by mod=10 is 16 bits in that address size.

The SIB Byte #

In 32- and 64-bit address-size forms, a memory ModR/M value of r/m=100 selects a SIB byte:

The effective address is:

base + index * scale + displacement

The scale field encodes:

Without an extension bit, index=100 means no index. In long mode, extension bits permit additional register selections. For mod=00, base=101 means no base and requires a disp32; with mod=01 or 10, the same base bits select EBP or RBP and use the displacement implied by mod.

ESP or RSP as a base therefore requires a SIB byte even when no scaled index is needed. Assemblers hide that representational detail.

Displacements and Immediates #

A displacement participates in effective-address calculation. Its presence and encoded width come from the addressing form:

• disp8 is sign-extended to the active address size.
• disp16 occurs in selected 16-bit addressing forms.
• disp32 is used by 32-bit forms and by base-relative or RIP-relative long-mode forms.

An immediate is an operand value embedded in the instruction. Its width is opcode-specific rather than a direct consequence of the destination width. Examples include:

• arithmetic on a 16- or 32-bit operand with a sign-extended imm8;
• arithmetic on a 64-bit operand with a sign-extended imm32;
• MOV r64, imm64, which can carry a full 64-bit immediate;
• shift instructions whose count is the implicit value 1, register CL, or an encoded imm8.

Displacements and immediates are stored little-endian. A disassembler needs the opcode and mode before it can decide where one field ends and the next begins.

Long-Mode REX Encoding #

The one-byte REX prefix has the form 0100WRXB and occupies values 40 through 4F:

A REX prefix is needed when an instruction uses a supporting 64-bit operand, references R8 through R15, or needs the byte registers SPL, BPL, SIL, or DIL. It must immediately precede the opcode. A legacy prefix placed after a REX byte prevents that byte from serving as the instruction’s effective REX prefix.

Any REX prefix makes AH, BH, CH, and DH unavailable in that instruction. Their legacy encodings overlap the low-byte register codes introduced by REX.

Long mode defaults are not uniformly 64-bit:

• Address size defaults to 64 bits and 67 selects 32 bits.
• Most integer instructions default to 32-bit operands.
• 66 commonly selects 16-bit operands.
• REX.W selects 64-bit operands where defined.
• Stack, branch, and fixed-width instructions have opcode-specific rules.

For example:

48 8b 44 8b 20    mov rax, [rbx + rcx*4 + 0x20]

• 48 is REX.W.
• 8B is MOV r64, r/m64.
• ModR/M 44 selects a memory operand, destination register RAX, a SIB byte, and disp8.
• SIB 8B selects scale 4, index RCX, and base RBX.
• 20 is the displacement.

VEX and EVEX #

REX extends the legacy general-purpose encoding, but vector instruction families need more state. VEX encodes an opcode map, mandatory-prefix selection, vector length, register extensions, and an additional source-register field. Common VEX forms use two or three bytes.

EVEX adds further register bits and controls for masking, zeroing, broadcast, rounding, and larger vector lengths. Some EVEX memory operands use compressed displacement scaling, so the encoded disp8 is multiplied by a tuple-dependent factor before address calculation.

VEX and EVEX reuse byte values that had other meanings in older modes. Mode and prefix-validity rules let processors distinguish the forms. Decode them as complete prefix structures rather than applying legacy-prefix rules byte by byte.

A Practical Decoding Procedure #

Decode an instruction in dependency order:

1. Establish execution mode, code-segment defaults, and enabled architectural extensions.
2. Consume valid legacy prefixes.
3. Identify REX, VEX, XOP, or EVEX form when applicable.
4. Select the opcode map and opcode.
5. Consult the opcode definition to determine whether ModR/M is required.
6. If ModR/M selects memory and r/m=100 in a SIB-capable address size, decode SIB.
7. Determine displacement width from address size, mod, and special field values.
8. Determine immediate width from the opcode and effective operand size.
9. Apply register-extension fields and instruction-specific semantic rules.

No byte is self-describing. 66, for example, can be an operand-size override, a mandatory opcode selector, or encoded indirectly in VEX/EVEX state. Reliable decoding therefore requires the instruction tables from the processor manuals.

References #

• Intel 64 and IA-32 Architectures Software Developer’s Manual
• AMD64 Architecture Programmer’s Manual