Beyond attention · Part 3
Why hybrid AI costs less to serve
September 25, 2026
Every time a chatbot answers you, a data center pays for two things, the chips and the time you hold them. Part 1 looked at the energy and water side of that bill, and Part 2 explained how hybrid models mix a few attention layers with many state layers. This post follows the money. It walks through why serving a large language model is mostly a memory problem, why memory decides how many people one server can help at once, and why hybrid models change that arithmetic most when prompts get long.
Beyond attention · Part 3 · At a glance
Less memory per conversation means more users per GPU, and that sets the cost per token
128K token conversations that fit on one 8 × H100 node
Our arithmetic from published configs, 16 bit weights and cache, activations and overhead ignored.
320 KiB
KV cache per token for Llama 3 70B (our arithmetic from its config)
12.8 ms
To read one 128K cache for every generated token on an H100 (our arithmetic)
2×
Gemini's input price above 200K tokens, a sign that long context costs more to serve
Weights are shared across users, the cache is not. A model that keeps less per conversation fits more conversations in the same memory.
Almost every number below is our arithmetic from published model configurations, not a measurement. Where a vendor or a research team reports a result, we say so and link to it.
Two phases, two bottlenecks
A language model reads and writes in tokens, small chunks of text of roughly a word or part of a word. Serving a request happens in two phases.
Prefill is reading the prompt. The model processes all of the prompt’s tokens together, in parallel, which keeps the chip’s math units busy. Prefill is limited mostly by raw compute.
Decode is writing the answer, one token at a time. Each new token depends on the one before it, so the model runs its whole network once per generated token. For each of those steps the chip has to pull the model’s weights, and a record of the conversation so far, out of its main memory. On a GPU that main memory is HBM (high bandwidth memory), a stack of memory chips sitting next to the processor. It is large, but moving data out of it is slow compared with doing math on the chip itself.
A useful way to see which limit applies is the roofline, a plot that compares how many math operations a task does per byte it reads with how many the chip can do per byte it can fetch. The NVIDIA H100 lists 3.35 TB/s of memory bandwidth and “1,979 teraFLOPS” of BF16 math, a figure that NVIDIA’s footnote says assumes sparsity. Taking the dense rate as half of that, the chip can do roughly 295 operations for every byte it reads before math becomes the limit (our arithmetic). Decode for a single user does about 1 operation per byte of weights. That is far below the ridge, so decode spends most of its time waiting on memory. The survey by Yuan and colleagues builds this roofline argument in detail, and a simulation study of accelerator design puts it bluntly, “decode is heavily memory-bound.”
The roofline
Decode reads far more bytes than it can use
Our arithmetic from NVIDIA's published H100 SXM figures and the Llama 3 70B configuration. Not a measurement.
~295 FLOPs per byte
H100 ridge point, where math starts to be the limit
~1 FLOP per byte
Single user decode with 16 bit weights
12.8 ms
Just to read one 128K token Llama 3 70B cache, per generated token
Batching is the standard fix. If a server decodes for many users at once, it reads the weights once per step and shares them across the whole batch. More users per step means more tokens per second for the same hardware. The catch is that each user also carries a private record that cannot be shared, and that record is where hybrids come in.
The KV cache, and how big it gets
A transformer’s attention layers let each new token look back at every earlier token. To avoid recomputing the past at every step, the model stores a key and a value vector for every earlier token in every attention layer. That store is the KV cache. It grows by a fixed amount with every token, for every user. The vLLM paper describes this memory as “huge,” growing and shrinking with each request, and says poor handling of it ends up “limiting the batch size.”
The size per sequence follows directly from the model’s configuration.
KV bytes = 2 (keys and values) x attention layers x KV heads x head dimension x tokens x bytes per numberTake Llama 3 70B. Meta’s paper lists 80 layers and 8 key value heads, and the head dimension works out to 128 (a model width of 8192 split over 64 query heads). Llama 3 already uses grouped query attention, which shares each key value head among several query heads to shrink the cache. Stored in BF16, a 2 byte number format, each token costs 2 x 80 x 8 x 128 x 2 bytes, which is 320 KiB (our arithmetic).
That sounds small until you multiply. At 8,192 tokens the cache is 2.5 GiB for one conversation. At 131,072 tokens, the 128K context length that many products advertise, it is 40 GiB, or about 42.9 GB. One long conversation needs a cache larger than half of an 80 GB H100.
It also has to be read. Every decode step reads that sequence’s entire cache, and unlike the weights, the cache cannot be shared with anyone else in the batch. On one H100’s bandwidth, moving 42.9 GB takes about 12.8 ms, for every token of the answer (our arithmetic, ignoring everything else the chip is doing).
What a hybrid keeps instead
A hybrid model replaces most attention layers with state space layers such as Mamba. A state layer does not keep a record per token. It keeps one fixed size state, a compact summary it updates with each new token. The Mamba authors write that inference “requires only constant time per step since it does not require a cache of previous elements.” Part 2 covers what that summary can and cannot remember.
IBM’s Granite 4.0 family is a clean public example. The H-Small model card and config list 40 layers, of which 4 are attention and 36 are Mamba2, with 8 key value heads of size 128, the same attention shape as Llama 3 70B. It has 32B parameters in total, 9B of them active per token.
Only 4 layers keep a KV cache, so a token costs 16 KiB of cache instead of 320 KiB. The 36 Mamba2 layers each hold a state of 128 heads by 64 channels by 128 state entries, plus a small convolution buffer. In BF16 that comes to about 77 MB per sequence for all 36 layers, and it is the same at 1,000 tokens or a million (our arithmetic from the config). Some serving stacks keep this state in 32 bit numbers, which would double it. We have not checked what any particular stack does for this model.
Memory per conversation
The transformer cache climbs with every token, the hybrid's barely moves
KV cache plus recurrent state for one sequence, in GiB, at 16 bit precision. Our arithmetic from published configs, not a measurement.
Show as table
| Context length (tokens, log scale) | Llama 3 70B | Granite 4.0 H-Small |
|---|---|---|
| 4,096 | 1.25 GiB | 0.14 GiB |
| 16,384 | 5 GiB | 0.32 GiB |
| 32,768 | 10 GiB | 0.57 GiB |
| 65,536 | 20 GiB | 1.07 GiB |
| 131,072 | 40 GiB | 2.07 GiB |
| 262,144 | 80 GiB | 4.07 GiB |
At 128K tokens the hybrid needs about 2.22 GB per sequence against 42.9 GB for the transformer. The two models are different sizes, so a fairer test holds the model fixed. If all 40 of Granite’s layers were attention with the same heads, the cache at 128K would be 20 GiB. The real hybrid keeps about 10 percent of that, roughly 90 percent less memory per sequence (our arithmetic).
The authors of other hybrids report the same direction. AI21’s Jamba, which interleaves one attention layer for every seven Mamba layers, publishes a table of cache sizes at a 256K context.
Reported by the authors
Jamba's authors report a 4 GB cache where comparable transformers need 32 to 128 GB
KV cache at 256K tokens, 16 bit, as published in Table 1 of the Jamba paper (March 2024). The authors' numbers, not ours.
Show as table
| Item | Value |
|---|---|
| Llama-2 (6.7B parameters) | 128 GB |
| Mistral (7.2B parameters) | 32 GB |
| Mixtral (46.7B total, 12.9B active) | 32 GB |
| Jamba (52B total, 12B active) | 4 GB |
IBM makes a broader claim for Granite 4.0, “over 70% reduction in RAM needed to handle long inputs and multiple concurrent batches” compared with conventional transformers. IBM’s figure counts total memory including weights, so a smaller percentage than the cache alone is what you would expect.
From memory to sequences per node
Here is where memory turns into money. A server’s memory holds the weights once, and whatever is left is the budget for everyone’s caches. The number of conversations that fit in that budget caps the batch size.
Take a common unit, a node of 8 H100 GPUs with 80 GB each. Llama 3 70B in BF16 takes about 140 GB for weights, leaving about 466 GiB. That fits 186 sequences at 8K tokens, but only 11 at 128K. Granite 4.0 H-Small’s 32B parameters take about 64 GB, leaving about 536 GiB, and at 128K that holds 258 sequences. The same model with every layer as attention would hold 26 (all our arithmetic).
Concurrency
At 128K tokens, one node holds 11 transformer conversations or 258 hybrid ones
How many 131,072 token sequences fit in an 8 x H100 node's memory after weights. Our arithmetic from published configs, not a measurement.
Show as table
| Item | Value |
|---|---|
| Llama 3 70B style (80 attention layers) | 11 sequences |
| Granite H-Small, all attention (counterfactual, 40 attention layers) | 26 sequences |
| Granite 4.0 H-Small style (4 attention layers plus Mamba2 state) | 258 sequences |
From sequences to cost per token
The last links in the chain are simple. A node costs about the same per hour whether it serves 11 users or 258. Cost per token is that hourly cost divided by the tokens the node produces in an hour, and during decode each step produces one token for every sequence in the batch.
The chain
Smaller memory per conversation leads to cheaper tokens
How the pieces above connect. A reasoning chain, not a measured result.
1Cache size per sequence
Set by attention layers, KV heads, head size, precision, and context length. Hybrids keep a cache in only a few layers plus a fixed state.
2Sequences that fit
Memory left after weights, divided by memory per sequence. This caps the batch size.
3Batch size
Weights are read once per step and shared across the batch, so larger batches make better use of each trip to memory.
4Tokens per second per GPU
Each decode step yields one token per sequence in the batch, limited by how many bytes must move per step.
5Cost per token
The hourly cost of the hardware divided by the tokens it produces. More tokens from the same node means a lower cost for each.
Memory also shapes time per step, not only the count. At 128K tokens a Llama 3 70B style sequence moves about 42.9 GB of cache per generated token, and the hybrid about 2.22 GB (our arithmetic). Real throughput depends on kernels, scheduling, and hardware, so we lean on what builders report. NVIDIA reports that its Nemotron-H 56B hybrid “can generate 2.4× more output tokens per second per GPU compared to Qwen-2.5-72B and Llama-3.1-70B” with 65,536 input tokens on H100s. The Jamba authors report that “with 128K tokens its throughput is 3x that of Mixtral.” Those are the vendors’ own results on setups they chose.
This matters beyond one server. The International Energy Agency reports that memory prices rose “by an order of magnitude between late 2024 and early 2026” and expects a shortage of high bandwidth memory “to persist through at least the end of 2027” (IEA, Key Questions on Energy and AI). When HBM is the scarce part, a design that needs less of it per user changes the economics directly. Part 4 takes that thread into chip design.
Where the savings shrink
The savings are real in the arithmetic, but they are not universal.
Short prompts see little benefit. At 4K tokens a Llama 3 70B style cache is only 1.25 GiB, and the weights dominate memory. The hybrid’s lead mostly shows up with long documents, long agent traces, and many users at once. The Falcon-H1 team, who build hybrids, write that “Transformers are slightly faster at shorter context lengths.”
Prefill still costs compute. Reading a 128K token prompt means running every token through all the model’s weights, whatever the layer type. State layers avoid the attention cost that grows with the square of the prompt length, but the feed forward and expert layers do the same work either way.
Prefix caching is harder. Serving systems often save the cache for a shared prefix, such as a long system prompt, and reuse it across requests. An attention cache can be cut at any token. A recurrent state is overwritten in place, so it cannot be rolled back to an earlier point. The Marconi paper (MLSys 2025) shows this “mandates only exact-match cache hits” for hybrids, and proposes a caching policy that its authors report reaches “up to 34.4× higher token hit rates.”
Quality and maturity are open questions. Fixed size states are lossy, and Part 2 covers the recall research. MiniMax, which shipped a hybrid in MiniMax-01, returned to full attention for MiniMax M2. Its pretraining lead wrote that hybrids showed “clear deficits in complex, multi-hop reasoning tasks” at larger scale and that “the infrastructure for linear and sparse attention is much less mature.” Lower serving cost only matters if the answers are good enough for the job.
Global attention still grows. Granite, Jamba, and Nemotron-H keep a few full attention layers, so their cache still climbs with context, just more slowly. Samba style designs that use a sliding window are the case where attention memory is also capped, a topic for Part 4.
Providers already price the cache
One sign that long contexts cost more to serve is in public price lists. Google’s Gemini API pricing (fetched September 25, 2026) charges Gemini 3.1 Pro Preview $2.00 per million input tokens for prompts up to 200K tokens and $4.00 above that, with output rising from $12.00 to $18.00. Google does not say why. Our reading is that longer prompts hold more cache for longer, which fits the arithmetic above, but that link is our interpretation.
If that reading is right, the pricing points at the same lever this post has followed. The cost of a long conversation is mostly the memory it occupies. Hybrids shrink that memory by keeping exact attention in a few layers and a fixed summary everywhere else. Whether that trade is worth it depends on the workload, and on whether the model still answers well.
Sources
- The Llama 3 Herd of Models, Meta, July 31, 2024.
- granite-4.0-h-small model card and config, IBM, October 2025.
- IBM Granite 4.0 announcement, IBM, October 2, 2025.
- Jamba: A Hybrid Transformer-Mamba Language Model, Lieber et al., AI21, March 28, 2024.
- NVIDIA H100 product page, NVIDIA, fetched September 25, 2026.
- LLM Inference Unveiled: Survey and Roofline Model Insights, Yuan et al., arXiv, February 26, 2024.
- Prefill vs. Decode Bottlenecks: SRAM-Frequency Tradeoffs and the Memory-Bandwidth Ceiling, Uppsala University, arXiv, December 26, 2025.
- Efficient Memory Management for Large Language Model Serving with PagedAttention, Kwon et al., SOSP 2023, September 12, 2023.
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, Ainslie et al., Google, May 22, 2023.
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces, Gu and Dao, December 1, 2023.
- Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models, NVIDIA, April 4, 2025.
- Falcon-H1 blog, Technology Innovation Institute, May 20, 2025.
- Marconi: Prefix Caching for the Era of Hybrid LLMs, Pan et al., MLSys 2025, November 28, 2024.
- Why Did M2 End Up as a Full Attention Model?, Haohai Sun, MiniMax, October 29, 2025.
- Key Questions on Energy and AI, International Energy Agency, April 2026.
- Gemini Developer API pricing, Google, fetched September 25, 2026.