Linux x86 Calling Conventions

A calling convention is a binary contract for argument placement, results, preserved state, and stack layout. Linux x86-64 programs use the System V AMD64 function ABI; 32-bit programs use the System V i386 ABI. Linux system calls use separate entry ABIs and must not be treated as ordinary function calls.

AMD64 Function Calls #

The AMD64 ABI first classifies each argument. Integer-class values, including ordinary integers and pointers, use rdi, rsi, rdx, rcx, r8, and r9. SSE-class values use xmm0 through xmm7. Exhausting one class does not by itself exhaust the other; arguments that cannot be assigned according to the classification rules go in the stack input area.

Aggregates are divided into eightbyte classes. A qualifying small structure can therefore occupy multiple integer or vector registers, while a memory-class structure is passed on the stack. For a memory-class return value, the caller allocates storage and passes its address in hidden argument register rdi; the callee returns that address in rax.

Integer-class return values use rax and then rdx; SSE-class results use xmm0 and then xmm1. Floating-point float and double results use xmm0, while x87 classes such as long double use the x87 stack as specified by the ABI.

rbx, rbp, r12 through r15, and rsp are callee-saved. A callee that changes one must restore it before ret. Other general-purpose registers and all vector registers are caller-saved under the base System V ABI, so a caller preserves any live values it needs after a call.

Immediately before call, rsp is 16-byte aligned. The pushed eight-byte return address means an ordinary callee enters with rsp % 16 == 8. Stack arguments are laid out with the padding required by their alignment; they are not simply pushed one at a time. The ABI also reserves a 128-byte red zone below the current rsp. Leaf userspace functions can use it without adjusting the stack because normal signal delivery preserves it. Kernel code and code built with -mno-red-zone must not rely on that area.

Variadic Functions #

Fixed arguments to a variadic function use the normal classification algorithm. A caller additionally sets %al to an upper bound, from 0 through 8, on the number of vector argument registers used. Only the low byte is defined for this purpose. A handwritten call to printf that leaves %al unspecified therefore violates the ABI even if every visible argument appears correct.

A variadic callee that uses va_start provides a register-save area in its own frame and saves the argument registers it may need. AMD64 va_list tracks:

• gp_offset and fp_offset within that save area;
• overflow_arg_area for arguments passed in memory; and
• reg_save_area, the base of the saved register arguments.

va_arg applies type classification and alignment while advancing the relevant location. Because default argument promotions apply, a caller passes float as double and narrow integer types as int or unsigned int where required.

i386 Function Calls #

The i386 System V ABI passes ordinary arguments in the stack input area in right-to-left order. At entry, 0(%esp) contains the return address and 4(%esp) contains the first argument. The caller removes arguments after return under the standard C convention.

ebx, esi, edi, ebp, and esp are callee-saved; eax, ecx, and edx are caller-saved. An integer or pointer result uses eax, and a two-word integer can use edx:eax; floating-point results use st(0). Structure return conventions depend on classification and can use a hidden destination pointer.

A conventional frame-pointer prologue is:

push    ebp
mov     ebp, esp
sub     esp, local_size

ebp is not intrinsically a frame pointer, so optimized code can omit this sequence. Historical i386 code assumed four-byte stack alignment. The current GNU/Linux i386 ABI requires 16-byte alignment at a call boundary, and compilers can realign entry when interoperating with legacy callers. Handwritten assembly must follow the ABI selected by its toolchain rather than infer alignment from old examples.

Linux System Calls #

The x86-64 Linux syscall ABI is:

Hardware saves the user return address in rcx and user flags in r11; the fourth argument therefore uses r10, unlike an AMD64 function call. The kernel returns either a successful value or a negative error in rax; values from -1 through -4095 encode -errno. A libc wrapper converts such an error to -1 and sets thread-local errno.

; x86-64 Linux: write(1, message, message_length)
mov     eax, 1
mov     edi, 1
lea     rsi, [rel message]
mov     edx, message_length
syscall

For an i386 process using int $0x80, eax holds the number, arguments use ebx, ecx, edx, esi, edi, and ebp, and eax holds the result. These registers are syscall operands even though ebx, esi, edi, and ebp are callee-saved in the i386 function ABI; a function containing inline entry code must still preserve the function ABI for its caller.

Function ABI versus Syscall ABI #

An AMD64 function call uses rcx for argument four and preserves rbx, rbp, and r12–r15; an x86-64 syscall uses r10 for argument four and has explicit rcx/r11 clobbers. A function returns errors according to its declared API, whereas the raw kernel boundary uses negative error codes. The syscall instruction also changes privilege level and does not push a return address or switch stacks by itself; Linux entry code performs the required kernel-stack transition.

Application code should prefer libc wrappers because they implement both contracts, error translation, cancellation behavior where applicable, and vDSO selection. Inline assembly must describe register, condition-code, and memory effects to the compiler and is appropriate primarily for C libraries and runtimes.

The System V AMD64 psABI, syscall(2), and Linux entry_64.S are the primary references for these contracts.