NVIDIA Triton Inference Server

NVIDIA Triton Inference Server serves machine-learning models over HTTP and gRPC. It supports several backends, including TensorRT, PyTorch, ONNX Runtime, TensorFlow, Python, and ensembles, while providing batching, metrics, model versioning, and concurrency controls.

Start with a Container #

The supported Triton containers are published through NVIDIA NGC:

docker pull nvcr.io/nvidia/tritonserver:<tag>
docker run --gpus all --rm -p 8000:8000 -p 8001:8001 \
  -v "$PWD/models:/models" \
  nvcr.io/nvidia/tritonserver:<tag> tritonserver \
  --model-repository=/models

Select a tag whose backend support matches the model. Minimal images contain the server only, while other images add selected framework or SDK components. Check the release notes for compatibility among the server, CUDA driver, and model runtime.

Model Repository Layout #

Triton discovers models from a repository with one directory per model and one or more numeric version directories:

models/
  classifier/
    config.pbtxt
    1/
      model.onnx

config.pbtxt declares model inputs, outputs, batching behavior, and backend-specific settings. Triton can infer some configuration for supported formats, but explicit configuration is clearer and is required for many features. A Python backend model normally provides model.py in its version directory and implements Triton’s backend interface.

Requests and Operations #

Clients send named tensors in an inference request and receive named output tensors in the response. Dynamic batching can combine compatible requests to improve throughput, but its queue delay and batch size must be tuned against latency targets.

Expose only the endpoints and model repository access appropriate for the environment. Monitor the Prometheus metrics endpoint, test with representative input shapes, and version models so a rollback is operationally simple.