Inference with vLLM

• generative models: llm = LLM(model=..., task="generate") (task=generate) only
• pooling models: llm = LLM(model=..., task="embed") (task={embed,classify,reward}) only
• upon success, generative models returns text, pooling models return hidden states.

Custom model integration #

1. git clone --depth 1 https://github.com/vllm-project/vllm/
2. Add a file in Transformers repository and put it into the vllm/model_executor/models directory

Existing Model Executors #

https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/models

hugging face transformer repo #

• .gitattributes

  Configures Git behavior, usually Git LFS tracking for large weight files. Not used during inference.

• config.json

  Defines model architecture and hyperparameters: model type, layer count, hidden size, attention heads, vocabulary size, positional encoding, dtype, and architecture class.

• generation_config.json

  Default generation behavior such as EOS/PAD token IDs, sampling settings, and maximum output length. Call-time parameters can override it.

• model.safetensors or model-x-of-y.safetensors pieces of the model’s weights. Sharding avoids one enormous file. Shard boundaries do not necessarily correspond to layers.

• model.safetensors.index.json

  Maps every parameter name to the shard containing it, for example model.layers.0… → model-00001-of-00024.safetensors. It usually also records total weight size.

• tokenizer_config.json

  Tokenizer metadata: tokenizer class, maximum length, special tokens, normalization options, and often the chat template.

• tokenizer.json

  Complete serialized “fast tokenizer” pipeline, including vocabulary, splitting, normalization, BPE rules, and post-processing.

• vocab.json

  Maps textual tokens or byte representations to integer token IDs.

• merges.txt

  Ordered BPE merge rules describing how smaller symbols combine into larger tokens.