Note 02 / Inference · Operations
Llama on GPU: budget memory, then measure
Model weights are only part of memory use. Check KV cache, context and runtime before selecting a serving configuration.
Does the model really fit the GPU?
A model name and quantisation bit width are insufficient to plan reliable serving. Quantisation metadata, activations, runtime buffers and KV cache consume memory in addition to weights. Context length and simultaneous requests change the requirement. L40S is a possible target device, not a guarantee for a particular model configuration.
Calculate a lower bound
The calculation considers only ideally stored weights. It helps identify implausible assumptions early. It models neither the actual checkpoint nor vLLM’s allocation. Do not confuse GB and GiB when planning further.
from decimal import Decimal
def weight_lower_bound_gib(parameters, bits):
if parameters <= 0 or bits <= 0:
raise ValueError("parameters and bits must be positive")
# Weights only: excludes KV cache, buffers and metadata.
weight_bytes = Decimal(parameters) * Decimal(bits) / 8
return weight_bytes / (Decimal(1024) ** 3)
# Arithmetic example, not a GPU memory measurement.
estimate = weight_lower_bound_gib(70_000_000_000, 4)
assert Decimal("32") < estimate < Decimal("33")
Check checkpoint and runtime together
Record model revision, tokenizer, quantisation format, vLLM version, GPU and driver. Check compatibility in the documentation for that version. GGUF, AWQ and GPTQ are different formats; a llama.cpp designation is not automatically a valid vLLM configuration. Current vLLM documentation describes GGUF support as experimental and moved to a separate plugin.
Start with bounded context length and controlled concurrency. Record memory at startup, during prefill and during token generation. Test long inputs and cancellations too. Successful startup does not show that traffic bursts or long contexts remain stable.
Prepare a reproducible measurement plan
Define measurement boundaries: time to first visible token, total duration to the final token, and error rate separately. Record prompt lengths, output limit, concurrency, warm-up and repetitions. Compare quantisations against the same evaluation set; lower memory requirements may change answer quality.
What was actually checked here
The local Python test checks the arithmetic function and invalid inputs. No GPU benchmark was run. This note therefore reports no latency, throughput or measured quality values and claims no optimal vLLM parameter. A second GPU does not automatically provide high availability: a shared host remains a shared failure point.