Choose GPU Server VRAM for Model Loading and Inference

You deploy a large language model on a US or Hong Kong server and hit out-of-memory errors during inference. Long context windows make it worse. You wonder, “How much GPU memory do I really need for my model, batch size, and concurrent requests?”

The answer involves calculating model weights, activation memory, and concurrency overhead. For example, a 7B parameter model in FP16 needs exactly 14 GB for weights alone. That’s 7 billion parameters multiplied by 2 bytes each.

This guide walks you through a step-by-step estimation process. You’ll learn about model loading, batch size impact, and concurrent inference. You can choose GPU server video memory for your US or Hong Kong server wisely without overspending or under-provisioning. Focus on practical solutions, not theory.

Key Takeaways

• Calculate weight memory by multiplying parameters by bytes per parameter. A 7B model in FP16 needs 14 GB.
• Add 1-2 GB for CUDA context and framework overhead. This gives a safe memory baseline.
• Activation memory grows linearly with batch size. KV cache grows quadratically with sequence length.
• Weights are shared across concurrent requests. Each request adds its own activation and KV cache.
• Use profiling tools like vLLM or nvidia-smi to verify estimates. Test on a small GPU before buying.

How to Choose GPU Server Video Memory for Model Loading

Estimating Weight Memory from Parameters

You start with a simple calculation. The memory for model weights equals the number of parameters multiplied by the bytes per parameter. Each data type uses a different amount of space.

A 7B parameter model in FP16 requires 14 GB for weights alone. That’s 7 billion multiplied by 2 bytes. If you switch to INT8, the same model needs only 7 GB. INT4 drops it further to 3.5 GB. This reduction happens because each weight uses fewer bits. INT4 stores 4 bits per weight compared to 32 bits for FP32, giving you an 8x reduction.

The standard memory formula for model states during training, derived from the ZeRO paper, is (p + p + 12) * model_size. Here, p is the precision in bytes per parameter (2 for FP16, 4 for FP32), and model_size is the number of parameters in billions. A 10B model in mixed precision requires 16 bytes per parameter, totaling 160 GB.

This formula applies to training, not inference. For inference, you skip the optimizer states. You only need the weights themselves.

Accounting for Overhead and CUDA Context

Weights alone don’t tell the full story. Your GPU also needs memory for the CUDA context, cuDNN and cuBLAS workspaces, and kernel launch overhead. These components consume space before your model even runs.

Inference runtimes and serving frameworks like vLLM and TensorRT-LLM add 0.5–2 GB of CUDA context and framework overhead on top of model weights and KV cache. This range applies across popular GPU models, including A100 and A6000. The exact value depends on your specific framework and GPU configuration.

You should add 1–2 GB to your weight memory for a safe baseline. This buffer covers the overhead without overestimating. For a 7B FP16 model, you’d plan for 15–16 GB total. That means a single 24 GB GPU like the RTX 4090 gives you comfortable headroom.

When you choose GPU server video memory, remember that optimizer states don’t matter for inference. They only matter for fine-tuning. Your calculation for deployment focuses on weights plus overhead. This approach keeps your estimate accurate and your budget in check.

Batch Size and Activation Memory

How Activations Scale with Batch Size

Activation memory holds intermediate tensors during the forward pass. Unlike weights, these tensors change with every input. They grow linearly with batch size. You can express this relationship simply: activation memory per sample multiplied by batch size equals total activation memory.

Consider a concrete example. A single layer processing 32 images at 224×224 resolution with 64 channels stores 224 × 224 × 64 × 32 values. That equals 102,760,448 values, or 401.40 MB in 4-byte floats. Double the batch size to 64, and the memory doubles to 802.80 MB. This strictly linear relationship holds because batch size acts as a simple multiplier.

Activation Memory = 2 × (sequence length) × (batch size) × (hidden size) × (number of layers)

This formula reveals the same pattern. You can factor out batch size, leaving per-sample activation memory multiplied by batch size.

Sequence length complicates the picture. The attention matrix grows with the square of sequence length. Doubling sequence length quadruples memory. A 2048-token sequence produces an attention matrix of 536,870,912 elements, consuming 2.0 GB in float32. At 4096 tokens, that matrix jumps to 2,147,483,648 elements and 8.0 GB. This quadratic scaling explains why long-context inference exhausts memory so quickly.

The Role of KV Cache in Memory Estimation

For LLM inference, the KV cache often causes out-of-memory errors. This cache stores key and value tensors for every token the model has processed. It scales with batch size, sequence length, and layer count.

You can estimate KV cache per token per layer with a straightforward calculation. Multiply 2 by the number of attention heads, then by the head dimension, then by bytes per element. For a model with 32 heads and head dimension 128, using FP16 (2 bytes), each token per layer needs 2 × 32 × 128 × 2 = 16,384 bytes, or 16 KB. Multiply that by sequence length, batch size, and layer count to get total KV cache memory.

Profiling tools help you verify these estimates. The vLLM profiler measures GPU memory usage before deployment, showing exact scaling with concurrency. For example, Llama 8B goes from 16 GB to 23 GB with 4 concurrent requests. You can also monitor real-time usage with nvidia-smi, comparing overall GPU memory against initial allocation to infer KV cache growth.

Estimating Memory for Concurrent Inference

Sharing Weights and Adding Activation Overhead

When multiple users send requests to your model simultaneously, the weights stay in memory once. You don’t duplicate the 14 GB of weights for each request. Instead, each concurrent request adds its own activation memory and KV cache on top of the shared weights.

The total memory calculation follows this pattern: weight memory plus activation memory per request multiplied by the number of concurrent requests, plus overhead. This formula gives you a clear picture of what your GPU must handle.

Dynamic batching helps you use this memory more efficiently. The technique monitors GPU memory in real-time and adjusts batch sizes to prevent out-of-memory errors while maximizing throughput. Variable-length batch processing groups similar-length sequences together, reducing padding overhead. These approaches adapt to current memory constraints, avoiding wasted allocation.

You also need to understand how memory partitions on your GPU. The memory visible to vLLM equals total GPU memory multiplied by the gpu_memory_utilization setting. From that usable memory, you subtract model weights, peak activations, and framework overhead. What remains becomes the KV cache. This partitioning shows you that not all GPU memory is available for inference.

Practical Example and Calculation Walkthrough

Let’s walk through a concrete scenario. You have a 13B parameter model in FP16, which requires 26 GB for weights. Each concurrent request needs about 2 GB of activation memory, including its KV cache. You expect 10 concurrent requests.

Your calculation: 26 GB for weights plus 2 GB multiplied by 10 requests equals 20 GB, plus 1 GB overhead. The total comes to 47 GB. This suggests you need a 48 GB GPU like the A6000, or you could split the model across multiple GPUs.

Now consider the trade-offs between batch size and concurrency. Large batch sizes consume substantial GPU memory, especially KV cache, without proportional throughput gains. They also degrade latency due to DRAM bandwidth saturation. A profiling-driven method called the Batching Configuration Advisor determines an optimal batch size that balances throughput and latency. The memory you free by using this optimal batch size allows you to run multiple model replicas on the same GPU. Model replication increases throughput by 33.7% for OPT-1.3B with 4 replicas compared to a single replica with maximum memory allocation.

Memory sensitivity to output length matters too. With OPT-1.3B, a batch of 520 requests uses only 20% of KV cache when generating 130 output tokens, but over 80% when generating 520 tokens. Gains from concurrency diminish for unusually long outputs.

GPU memory sharding offers another path. Pipeline parallelism splits the model vertically into layer chunks across GPUs, quartering weight memory per device with a 4-way split. Tensor parallelism shards individual layers horizontally across GPUs, reducing per-device memory for weights and activations. Both techniques distribute the memory footprint, freeing space on each GPU for larger batches or more concurrent requests.

When you choose GPU server video memory, remember that total GPU count scales with concurrency and model parallelism. The formula Total GPUs = concurrency × (tensor_parallel_size × pipeline_parallel_size) shows this relationship directly. This insight helps you choose GPU server video memory wisely. Start with these calculations, test with your workload, and you’ll avoid both out-of-memory errors and wasted budget.

You now have a clear three-step process. First, calculate weight memory from parameters and precision. Second, estimate activation memory per sample, including the KV cache. Third, multiply by your expected concurrency and add overhead.

This method requires iteration. Profiling tools like vLLM’s profiler or nvidia-smi verify your estimates against real workloads. Run a test on a small GPU before committing to hardware. Online GPU memory calculators offer a quick starting point.

Start with these formulas, test with your workload, and you’ll avoid both out-of-memory errors and wasted budget. Remember to choose GPU server video memory based on measured data, not guesswork. Dynamic batching and KV cache awareness keep your deployment efficient.

FAQ

What is the minimum GPU memory for a 7B model in FP16?

You need at least 15–16 GB. The weights alone require 14 GB. Add 1–2 GB for CUDA context and framework overhead. A 24 GB GPU like the RTX 4090 gives you comfortable headroom for inference.

Why does long context cause out-of-memory errors?

The KV cache scales quadratically with sequence length. Doubling the sequence length quadruples the memory needed for attention. For example, a 4096-token sequence uses 8 GB in float32, versus 2 GB for 2048 tokens.

Can I run multiple users on one GPU?

Yes, but each user adds their own activation memory and KV cache. Weights are shared across all users. You calculate total memory as weight memory plus (activation per user × number of users) plus overhead.

Should I use quantization for deployment?

Quantization reduces memory and speeds up inference. INT8 uses 1 byte per parameter versus 2 bytes for FP16. A 7B model drops from 14 GB to 7 GB. This lets you fit larger models on smaller GPUs.