CUDA Fundamentals

CUDA is NVIDIA’s platform for general-purpose GPU computing. CUDA C++ extends C++ with constructs for launching work on a GPU, while the nvcc compiler coordinates host compilation and device-code compilation. It is not a separate replacement for C++; it is a programming model and toolchain built around NVIDIA GPUs.

The Execution Model #

A CPU program, called host code, launches a kernel on the GPU. The kernel runs many lightweight threads. Threads are grouped into blocks, and blocks form a grid. Threads within a block can cooperate through fast shared memory and synchronization; blocks are designed to run independently so the GPU can schedule them across multiprocessors.

vector_add<<<grid_size, block_size>>>(a, b, result, count);

Correct launch dimensions, memory access patterns, and transfer costs matter as much as the arithmetic. Moving small amounts of data repeatedly between host and device can erase the benefit of GPU parallelism.

Memory and Synchronization #

Global memory is visible to all threads but has comparatively high latency. Shared memory is local to a block and is useful for reusing data. Registers are private to a thread. Coalesced global-memory accesses, where neighboring threads access nearby addresses, usually improve throughput.

Kernel launches are asynchronous with respect to the host by default. Check CUDA errors and synchronize only when a result is needed; unnecessary synchronization reduces overlap between CPU work, transfers, and GPU execution.

Multi-Instance GPU #

Multi-Instance GPU (MIG) is available on selected NVIDIA data-center GPU architectures. It partitions one physical GPU into isolated GPU instances with dedicated portions of compute and memory resources. CUDA applications see an assigned instance as a separate device, allowing stronger workload isolation than ordinary process sharing. MIG support and available profiles depend on the exact GPU model and driver stack.