The question buyers keep asking is which accelerator delivers the most tokens per watt. For a decode-stage agent holding several hundred thousand tokens of context, that question no longer has a useful answer, because the thing that runs out first is resident memory capacity, and the price of resident memory tripled over three quarters while accelerator list prices did not.
For a decode-stage agent holding several hundred thousand tokens of context, the thing that runs out first is resident memory capacity, not arithmetic. A Llama-3-70B-class model with eighty layers, eight KV heads and 128-dimension heads stores 320 KiB of key-value cache per token at BF16. One million tokens of that is 328 GB, against 140 GB of BF16 weights. The cache is 2.3 times the model, and unlike the weights it is paid per session and per token rather than once per instance. Sizing a deployment on throughput hides that entirely.
The cost floor under that memory moved, and accelerators had nothing to do with it. Micron reported fiscal Q3 2026 revenue of $41.46 billion, up 73.7 percent sequentially, on a non-GAAP gross margin of 84.9 percent, with DRAM average selling prices up in the low-60s percent against bit shipments up low single digits. Essentially none of the growth was volume. At that margin cost of goods is 15.1 cents of every revenue dollar, which puts today's DRAM price roughly four and a third times the level that would produce an ordinary 35 percent supplier margin at today's cost base. There is no tier in the memory hierarchy that is not repricing, so tiering cache out to host DRAM to escape HBM pricing buys into the same move one level down.
This report covers the memory hierarchy serving decode-stage inference at 100,000 to 1,000,000 tokens of context: HBM3E and HBM4 on package, LPDDR5X and GDDR7 as capacity tiers, conventional server DRAM, CXL-attached pools, and SSD-backed context storage. Three things about the category's published numbers get discounted before anything else is built on them. Accelerator vendors publish throughput at operating points they do not fully specify. Intel declined to publish Crescent Island's memory bandwidth, which is the single number that decides whether that product's economics work. And per-gigabyte memory prices from commercial data sites contradict each other by a factor of six inside the same quarter, so the dollar restatement this report set out to write cannot be completed honestly, and it says so rather than inventing one.
Four questions frame the assessment: how large the key-value cache actually is for the models being marketed for agent work, what Intel gave up by choosing LPDDR5X for Crescent Island, what memory repricing did to the cost floor, and whether vendor efficiency claims survive a restatement that charges for memory. Every claim below carries an evidence label in the report itself. No bench work was performed, and every session-count figure is an optimistic ceiling that ignores fragmentation, paged-attention block overhead and the serving runtime's own working set.
Eighty layers, eight KV heads, 128 dimensions, BF16: two tensors at 2 x 80 x 8 x 128 x 2 bytes is 327,680 bytes per token, or 320 KiB. One million tokens is 328 GB against 140 GB of weights, a ratio of 2.3. That ratio is the wrong comparison and it is the one everyone makes. Weights are a fixed cost per served instance. Cache is variable in two dimensions at once, sessions and tokens, so the question has no fixed answer, only a crossover concurrency.
GLM-5.3-Flash runs 34 linear-attention layers against 11 full-attention layers across 45 total. Qwen3.8-Flash-Next runs 36 Gated DeltaNet layers against 12 sparse-attention layers across 48. Both are 3:1. Z.ai states the architecture cuts KV cache size 4.44 times against the previous GLM-5.3 and attention compute 3.01 times. Independent convergence on the same ratio by two labs one week apart is a design constraint asserting itself, not a coincidence.
On the estimated 1.5 TB/s and 1.3 PFLOP/s FP8, Crescent Island needs a decode batch near 433 to balance its matrix engines. After a 329 GB FP8 checkpoint, 151 GB remains, which is 238 concurrent 100k-token sessions on GLM-5.3-Flash and about ten on a BF16 grouped-query model of half the size. The XMX array is provisioned for a batch the memory cannot hold, and the shortfall widens from 1.8x to 18x between 100k and 1M context. DeepSeek-V3 at FP8 does not fit at all; its weights alone exceed the card.
Asked directly at Hot Chips, Intel said it was not disclosing bandwidth at this time. A figure near 1.5 TB/s sits beside 8 TB/s on a B300 and roughly 2 TB/s on a single HBM4 stack under the JEDEC maximum, so the entire 1280-bit LPDDR5X array delivers about what one HBM4 stack delivers. Confirmation of the supply reading would be a general-availability sheet quoting bandwidth as a per-SKU range. Refutation would be Intel publishing one figure above 2 TB/s before shipping, which would make the compute-balance finding wrong in Intel's favour.
Fiscal Q3 2026, ended 28 May 2026: revenue $41.46 billion, up 73.7 percent sequentially and 345.6 percent year over year; non-GAAP gross margin 84.9 percent, up ten points; DRAM revenue $31.3 billion with average selling prices up low-60s percent against bits up low single digits. Guidance for Q4 is $50.0 billion at roughly 86 percent. The decomposition back-solves to about $18.8 billion of prior-quarter DRAM revenue against an actual $18.1 billion, so it holds: essentially none of the growth was volume.
NVIDIA reports over 16,000 tokens per second per GPU and over 200 tokens per second per user for Qwen3.8-Flash-Next on GB300 NVL72, at FP8, with no batch size, context length or power figure stated. Sixteen thousand divided by two hundred is eighty concurrent sessions. On the assumed head geometry, eighty sessions at 1M tokens is roughly 983 GB of cache per GPU against 288 GB of HBM3E. The headline throughput therefore cannot be a 1M-token number, and NVIDIA does not claim it is.
The 7.6x prefill and 4.9x decode figures are stated at 1M-token context with 90 percent of the prefix already cached. That is the warm path. An agent reopening the same repository hits it; a fresh session on a new repository or document set gets none of it. The benchmark is not wrong, it measures a real production condition. It is the condition a buyer sizing worst-case capacity must not plan against.
One widely cited commercial source puts HBM3E near $8.33 per GB in August 2026, an LPDDR5X contract figure works out to about $12.16 per GB in the preceding quarter, and an LPDDR5X spot figure to about $1.87. Those three cannot all be right, and two of the three would invert Intel's cost argument by making LPDDR5X dearer per gigabyte than HBM. None of the sites publishes a methodology or settlement source. The absence is the finding: the central premise behind capacity-tier accelerators is not verifiable from public 2026 data, and buyers should demand it in writing at the configuration they intend to buy.
Bytes of cache per token is the useful unit, because it multiplies cleanly by context length and by concurrency and it is the only quantity that separates the architectures cleanly. The first two figures follow directly from published layer counts and head geometry. The last two combine disclosed layer splits with an assumed latent rank and head count, and they mix precisions, so the FP8 figures would double at BF16. Read this as an ordering with roughly correct spacing, not as four measurements.
Eighty layers, eight KV heads, 128 dimensions, BF16. Keys and values stored for every layer and every KV head: 2 x 80 x 8 x 128 x 2 bytes, or 327,680 bytes. This is the baseline every other number here is measured against, and it is what most deployed long-context serving still runs.
328 GB for one 1M-token session
DeepSeek-V3 replaces per-head key and value tensors with a single compressed latent plus a small rotary component: 512 latent dimensions and 64 rotary across 61 layers, which at BF16 is 70,272 bytes. A 4.7-fold reduction against grouped-query at the same precision, achieved inside a full-attention stack.
4.7x better than grouped-query
A growing cache in only 11 of 45 layers, the rest being Kimi Delta Attention whose state is fixed per sequence and does not grow with context. Assuming DeepSeek-comparable latent geometry at FP8, that is 11 x 576 x 1, or 6,336 bytes. The latent rank is the assumption; the layer split is disclosed.
52x better than grouped-query
A growing cache in 12 of 48 layers. Assuming four KV heads at 128 dimensions in FP8, that is 2 x 12 x 4 x 128, or 12,288 bytes. The head geometry is the assumption. What is not an assumption is that three quarters of the layers are Gated DeltaNet and compress history into a fixed recurrent state.
27x better than grouped-query
Sparse attention does not shrink the cache, and this is the point most often got wrong. GLM-5.3-Flash's full-attention layers use a top-2048 IndexPool and Qwen3.8-Flash-Next's use a top-512-block budget that works out to the same 2048 tokens. Both reduce how many cached tokens each query attends to, which cuts attention arithmetic. Neither reduces how many cached tokens must remain addressable, because the indexer has to be able to select any of them. Sparse attention is a compute optimisation with a memory bandwidth benefit and no capacity benefit; the capacity benefit in these models comes entirely from the linear layers. That distinction decides which memory tier a workload can tolerate. A compute optimisation lets you buy fewer FLOPS. A capacity optimisation lets you buy cheaper bytes, and only the second one makes an LPDDR5X part viable.
Take NVIDIA's implied eighty sessions per GPU and test it against memory. At the assumed 12,288 bytes per token, eighty sessions holding 100k tokens is about 98 GB of cache, which fits inside 288 GB of HBM3E alongside a sharded checkpoint. The same eighty sessions at 1M tokens is about 983 GB, more than three times the package. Capacity-constrain the same GPU at 1M and concurrency falls to roughly 23 sessions; holding 200 tokens per second per user, per-GPU throughput falls to about 4,600 tokens per second. A 3.5-fold reduction with no change to the silicon, the model, or the power draw.
Intel is not giving up much bandwidth efficiency. At an estimated 1.5 TB/s and 350 W that is about 4.3 GB/s per watt, against about 5.7 for a B300 at 8 TB/s and 1,400 W. What it gives up is bandwidth density per socket, which converts into more cards, more PCIe fabric and more failure domains for the same aggregate. That is a scale-out cost landing on the rack budget, not the electricity bill. On the arithmetic here a 350 W air-cooled part plausibly beats a 1,400 W liquid-cooled one on tokens per joule at 100k context, which is the entire case for the capacity tier. But Intel's advantage erodes tenfold between 100k and 1M while NVIDIA's erodes three and a half fold, because the constraint that binds Intel is capacity and capacity is what runs out first.
A result that large should make a reader suspicious, so the report names the falsifiers rather than defending it. It fails if Intel's real bandwidth is below about 1.2 TB/s, if the LPDDR5X array cannot sustain its rated figure inside a 350 W board budget while the XMX engines are also drawing, if FP8 KV cache proves unusable in production and the concurrency halves, or if GLM-5.3-Flash's latent rank is materially larger than DeepSeek's. Any one of those moves the capacity-tier figures down by a third or more.
The calculation is free capacity after weights, divided by cache per session, and it is what determines whether a deployment serves ten agents or two hundred and forty. Ask every vendor for throughput at a stated batch size and a stated context length, and treat a benchmark that omits either as unusable. If your agents run near 100k tokens, price a capacity-tier part seriously. If they run near 1M, the capacity tier's advantage has largely gone and you are back to buying HBM. Test the FP8 KV cache path before committing to a plan built on it, because every favourable number here doubles in memory terms on a BF16 fallback, and the fallback is a runtime and hardware-generation question rather than a choice.
For a CFO the framing is different. You are being asked to underwrite a memory price sitting roughly four times above the level consistent with an ordinary supplier margin. That is not an argument against signing, since supply is genuinely constrained and about $100 billion of it has already been contracted with roughly $22 billion of cash deposits and letters of credit behind it. It is an argument for structuring the term so the memory component is not fixed for its whole length. Ask specifically whether the quoted price embeds a memory pass-through, and whether it reprices downward as well as upward. Then ask internally what context length your agent workloads actually use, because the answer moves cost per token by three to ten times and nobody has priced it into the business case.
For a platform architect, a context window limit is a memory budget wearing different clothes and it is the highest-leverage cost lever available. Moving a service's cap from 1M to 100k tokens raises concurrent sessions per accelerator roughly tenfold, a bigger effect than any accelerator choice in this report. If you are choosing a model rather than a cap, attention architecture matters more than parameter count for serving cost: a 320-billion-parameter linear hybrid holds fifty times less cache per token than a 70-billion-parameter grouped-query model, which makes the larger model the cheaper one to serve at long context. That inversion is new and most capacity planning has not caught up with it.
The report closes with four scenarios to 2028 and their earliest visible signs: price plateau at 40 percent, architecture absorbing the shock at 30, the move continuing to compound at 20, and the capacity tier failing commercially at 10. The probabilities are the least reliable content in it. The indicators attached to each are the useful part, and they are all observable without private information: a Micron quarter where bit growth exceeds average selling price growth, a frontier Western lab shipping a model with over half its layers linear-attention, a hyperscaler itemising memory as a separate capital expenditure line, or Intel reaching general availability without ever publishing a bandwidth figure.
This isn't a vendor summary. Every sentence is labeled by what stands behind it: verified fact, vendor claim, third-party estimate, my assessment, hypothesis, or scenario. Sources are numbered and clickable. Forward-looking sections use scenarios with observable tripwires, not forecasts. It's the same method behind every market assessment I write.
Twenty-three pages, built from public sources with no client brief and no interviews. Read it in the browser or take the PDF.
Each report here answers a real question, directed and researched against public sources and evaluated against a stated assumption, then delivered as Word and PDF. If you're weighing a platform, sizing a category, or defending a number to a board, tell me the decision behind it and I'll tell you honestly whether a report is the right tool.
Commission an assessment