Memory bandwidth (in inference context)
The speed at which data can be read from and written to a GPU's memory. It is the primary bottleneck for how fast an AI model can generate text.
What it is
During AI inference, generating a single token requires pushing the entire model's weights through the processor. Therefore, the speed of text generation is rarely limited by pure compute power (FLOPs); it is bottlenecked by how fast the hardware can move the model's massive data from VRAM into the compute cores.
When you would use it
You consider memory bandwidth when designing deployment infrastructure, understanding that buying a GPU with faster memory will yield higher tokens-per-second than simply buying a GPU with more compute cores.
Common operations
- Evaluating which hardware to purchase for a self-hosted AI cluster.
- Using quantization to reduce memory size, thereby bypassing bandwidth limits.
Related terms
Where this is taught
No learning path uses this term yet. Browse the Learning Atlas for guided sequences through related ideas.