Linux System Calls

Linux system calls form an architecture-specific binary interface between a process and the kernel. Application code normally targets a C library API instead, because the library supplies types, feature-test macros, compatibility behavior, and the machine-specific entry sequence.

Library API and Kernel ABI #

A source-level call such as open(path, flags) is not a promise to execute a kernel system call named open. glibc can implement it with openat, add cancellation checks, or select a time-width-compatible entry point. Functions such as clock_gettime can execute in the vDSO without entering the kernel. Conversely, Linux exposes system calls such as gettid for which older C libraries did not provide a wrapper.

The kernel ABI consists of:

• a system-call number for the process’s architecture and ABI;
• arguments represented in registers, and on some ABIs partly on the stack;
• an entry instruction such as syscall, int $0x80, svc, or ecall; and
• a raw return convention for success and errors.

SYS_name constants from <sys/syscall.h> identify numbers for the target build ABI. They are not stable across architectures: x86-64 write is 1, while i386 write is 4. Compat and x32 processes can use yet another table on the same kernel. The kernel dispatches the number through that ABI’s syscall table to an implementation commonly declared with a SYSCALL_DEFINE* macro.

Registers, Results, and errno #

Two common Linux x86 entry ABIs illustrate the mapping:

The x86-64 instruction overwrites rcx and r11, which is why argument four uses r10 rather than the C function ABI’s rcx. Other architectures have different alignment and register-pair rules; for example, some 32-bit ABIs split and align 64-bit arguments.

On x86, the kernel reports an error as a negative value from -1 through -4095 in the result register. A libc wrapper changes that raw result to -1 and stores the positive error number in thread-local errno. Architectures with a separate error indicator are normalized by their libc wrappers. Code must therefore test a documented wrapper return value rather than read errno after a successful call.

Direct Use of syscall #

glibc provides an indirect wrapper for an entry that has no normal wrapper:

#define _GNU_SOURCE
#include <sys/syscall.h>
#include <unistd.h>

long tid = syscall(SYS_gettid);
if (tid == -1) {
    /* errno describes the failure. */
}

The prototype is long syscall(long number, ...);. The function performs the architecture entry sequence and translates kernel errors into errno, but its variadic arguments provide no type checking. Cast values to the exact ABI width and follow architecture-specific rules for split 64-bit arguments. A new call can also return ENOSYS when the running kernel does not implement its number.

Prefer a libc wrapper when one exists because direct use can bypass:

• structure and time-width compatibility implemented by libc;
• wrapper-defined fallback to an older system call;
• POSIX thread cancellation behavior; and
• vDSO acceleration.

Raw inline assembly adds register constraints, clobbers, memory ordering, and error translation to those hazards. It is justified only when implementing a C library or another ABI-level runtime.

Architecture and Version Portability #

A kernel system call is Linux-specific unless a higher-level standard also defines the library behavior. Compile-time SYS_name availability only proves that headers describe an ABI; it does not prove that the running kernel implements the call. Probe by calling it and handling ENOSYS, then use a semantically correct fallback where one exists.

Do not copy numbers from another architecture or hard-code a host table into a cross-compiled program. Structures passed through the boundary also depend on the selected ABI’s field widths, alignment, endianness, and UAPI definition. Seccomp and audit identify both an architecture token and a number for the same reason.

Restart, Cancellation, Tracing, and Seccomp #

A signal can interrupt a blocking call. During return to userspace, the kernel uses internal -ERESTART* results to decide whether to reissue the call; userspace normally sees either a completed call or -1/EINTR. SA_RESTART, the particular system call, and whether data was already transferred determine the outcome. An application should retry EINTR only when the operation’s API makes retry safe and must process a positive short result before retrying.

POSIX thread cancellation is a libc property, not a kernel syscall property. glibc wrappers for cancellation points coordinate cancellation around blocking operations; calling syscall() directly can bypass that coordination.

strace and ptrace observe kernel entry and exit, arguments, and raw results. Calls satisfied entirely by the vDSO do not generate a syscall stop. Tracers can also alter numbers, arguments, or results, so traced behavior is not an independent security boundary.

Seccomp BPF evaluates the syscall architecture, number, and raw argument values before kernel execution. A robust allowlist checks the architecture token as well as the number because number spaces overlap. Filters cannot dereference pointer arguments, and libc upgrades can select a different underlying syscall than the one suggested by an API name. Seccomp therefore minimizes kernel attack surface but does not constitute a complete sandbox.

The upstream syscall(2) page lists entry and register conventions, while the kernel seccomp documentation defines filtering semantics.