A Singapore bank has invested in eight H100 GPUs housed in a local data center to avoid moving sensitive customer documents to external cloud providers, reflecting enterprise concerns about data sovereignty and vendor lock-in. The expensive hardware sits underutilized during off-peak hours, illustrating the tension between compliance requirements and computational efficiency in enterprise AI infrastructure decisions.
Text diffusion, the GPU hangover, and the one place parallel generation actually pays off
By Shreshta Shyamsundar and Anmol Jain July 20, 2026 • 12 minute read
Somewhere in a Singapore data center, a bank is paying for eight H100s that spend most of the night waiting. The cluster was bought for good reasons (discomfort with customer documents leaving the building, a strategy team’s aversion to lock-in), so the bank secured its own sovereign compute. Now the finance team is asking why a machine that costs more per hour than a senior engineer runs at a fraction of its capacity. This is the GPU hangover. Over the last two years, enterprises rushed to lock in private clusters and reserved cloud nodes to build AI they could control. The hardware arrived; the utilization did not. The reason isn’t bad planning. It’s a mismatch between how standard models generate text and how enterprises actually use them, and text diffusion is the most interesting candidate for closing the gap. It’s also the most oversold, and the oversell hides in which workloads it actually helps.
Start with the physics. A standard autoregressive model, from the Llama, Mistral, or GPT families, for instance, generates one token at a time. The weights never change and never leave the card; they sit in the GPU’s high-bandwidth memory the whole time. The bottleneck is one level down. Arithmetic happens only in the chip’s tiny pool of on-chip memory, which is nowhere near big enough to hold a multibillion-parameter model. So for every single token, the full set of weights has to be streamed out of that main memory and through the compute units again—rereading the model from the card’s own memory into the card’s calculators, once per token, because the calculators cannot keep it resident. The math finishes almost instantly and the units then idle, waiting for the next slice of weights. Measured as arithmetic intensity, operations per byte moved, this sits near 1 at batch size one, while modern GPUs are built for intensities in the hundreds. The chip is starved, bottlenecked not by a shortage of compute but by the speed of the feed. The escape hatch is batching: Read the weights once and use them to compute the next token for hundreds of requests at the same time, amortizing that one expensive read across hundreds of tokens of useful work. On the same hardware, small versus large batches can swing cost per token 10- to 30-fold, which is why public APIs, running enormous batches across thousands of users, are cheap.
Full story reconstructed from Oreilly.com. Formatting and media may differ from the original.
Everything hinges on whether you can accumulate concurrent work. An overnight queue of a million documents is trivially batchable, because nobody’s waiting. But when a single request must return in under a second, say a developer’s code completion or an onboarding check while the customer stands at the counter, you’ve spent your latency budget and can’t wait to fill a batch. The first kind of workload is not really memory-bound; you batch your way out of it. The second kind is, and no amount of total volume rescues it. And there’s a further subtlety: Generating tokens is memory-bound, but reading the prompt is already compute-bound, since the input is processed in parallel. Document extraction is mostly reading, long input and short output, so even a standard model spends much of that job in the regime where it was never starved in the first place.
Diffusion attacks exactly the part that is starved. Borrowing its mechanism from image generation, it starts with a block of masked or noisy tokens and refines the whole block in parallel over a few denoising passes, less like a typewriter and more like an editor revising a full draft at once. Because each pass does real arithmetic across the whole block, it’s compute-bound even at batch size one. Where autoregressive intensity sits near 1, a comparable diffusion model’s lands in the hundreds. It saturates the compute you already pay for without the concurrency you don’t have. The numbers are real. Inception Labs’ Mercury reported over 1,100 tokens per second on H100s for code generation, and the 2026 Mercury 2 release reported roughly 1,000 tokens per second on Blackwell at low latency. Google showed the paradigm at frontier scale with Gemini Diffusion, and open source LLaDA showed diffusion models follow autoregressive-like scaling laws. These are early but real: Mercury 2 is commercially available, Gemini Diffusion is in enterprise preview with general availability expected later in 2026, and the open models are maturing fast, even as autoregressive systems still dominate on tooling and ecosystem rather than any theoretical ceiling. So the headline is true in one specific place: for a latency-bound, single-stream request, diffusion can run an order of magnitude faster, because the autoregressive model is stuck memory-bound and cannot be batched out of it. But saturating the GPU is an engineering metric, and you can saturate a chip doing useless work. The real question is what it costs to produce a useful token, and on which workloads.
Before declaring a winner, a fair comparison has to account for what autoregressive serving can already do. Speculative decoding and its descendants, Medusa and EAGLE, use a small draft model to propose several tokens that the main model verifies in a single pass, giving roughly two- to four-fold single-stream speedups with no change in quality. Mixture-of-experts models attack the same wall from another direction, activating only a fraction of their weights per token and so moving less memory per token generated. The question is therefore not autoregressive versus diffusion in the abstract; it’s whether diffusion’s structural parallelism beats a speculatively decoded model’s incremental gain on the workload you actually have. For a tight single-stream latency target, diffusion’s edge is large and durable. For offline batch, neither trick matters much, because batching already pushes both architectures into compute-bound territory. Any framing that ignores speculative decoding is selling a false binary.
Whichever trick you reach for, the economics reduce to a single identity:
Effective cost per token = node cost per hour ÷ (throughput × utilization)
A public API is priced per token, concurrency independent, with no idle penalty. Owned compute is priced per hour, and its per-token cost is derived from how much you push through, so throughput and utilization are the only levers, and diffusion moves the first one decisively but only where batching is unavailable. The prices make the stakes concrete. A reserved AWS p5.48xlarge, eight H100s, lists near $55 an hour on demand, and one-year savings plans cut that by roughly 40 percent, to about $33 an hour. Against a cheap commodity API, a small model under a dollar per million tokens, owned compute loses on pure cost regardless of architecture; a $33-an-hour box, however well used, can’t beat a token you can rent for 40 cents. Diffusion’s economic win appears in only two situations: when the token you would otherwise buy is expensive, frontier or reasoning output at $5 to $15 per million, where a saturated owned node comfortably undercuts the API, or when the data can’t go to an external API at all, so the comparison becomes owned diffusion versus owned autoregressive. Most regulated enterprises live in that second case.
Nowhere is the distinction clearer than in the bank’s own document operation, which has two faces that look alike and behave like opposites. The overnight batch, millions of KYC packets, letters of credit, and loan files parsed into JSON while no one waits, is the easiest possible workload to batch. With continuous batching, a standard model runs at several thousand tokens per second and clears the queue on a single node; diffusion is somewhat faster and finishes the window sooner, but both fit on one box at a similar cost. If this were the whole workload, switching architectures would be hard to justify, because autoregressive batching has already solved most of the problem, and this job is mostly prefill anyway, its input tokens dwarfing the JSON output an API would bill for. The real-time path inverts the conclusion entirely. A relationship manager onboarding a customer needs the documents parsed in under a second while the customer waits; an officer clearing a letter of credit needs the answer now; an agentic flow is blocked on a single document before it can proceed. These requests arrive one at a time, each with a hard latency budget, so you can’t batch them, because batching trades latency for throughput and there is none to trade. A large autoregressive model in single-stream decode emits only tens of tokens per second, so a few hundred tokens of output take several seconds, and speculative decoding helps but does not reach interactive speed, while diffusion returns the same record in well under a second. The cost shows up as node count, and now it’s correctly attributed: to hold a subsecond target with the autoregressive model you must keep batches tiny, so each node serves only a handful of concurrent real-time requests and meeting peak demand means overprovisioning across many nodes, whereas diffusion clears each request fast enough that one node absorbs far more low-latency traffic and fits the same service level on a fraction of the fleet. The savings are real, and they come from the latency constraint defeating batching, not from low concurrency in the abstract.
The lesson of those two jobs generalizes into a routing rule sharper than the usual advice of customer-facing on APIs and internal on owned compute. The real test has two axes: whether the work can be batched, meaning it’s offline-tolerant rather than latency-bound and serial, and what each token is worth. Latency-bound, decode-heavy, low-value generation such as code completion, real-time extraction, and the chatter of agentic workflows is the diffusion sweet spot, where batching is unavailable, the quality gap is tolerable, and a fast owned node beats both an overprovisioned autoregressive fleet and an expensive API. High-value reasoning, where a wrong answer is costly, stays on frontier autoregressive models. And offline batch of any value density goes to whatever you already run well, because batching has already made it efficient.
That discipline matters because diffusion carries real constraints. Quality isn’t free: Diffusion trades some accuracy for speed, landing around 85% to 95% of strong autoregressive baselines, competitive on structured output but trailing by 5% to 15% on hard reasoning, on vendor and secondary figures that deserve independent verification against your own data. That’s fine for field extraction and not fine for credit decisions, so any serious deployment budgets a fallback for outputs that miss a confidence threshold and folds its cost back into the effective rate. Being compute-bound is itself a cost, since diffusion earns its high intensity partly by doing more total work per useful token, which is why the metric that matters is always tokens per dollar at an acceptable quality bar and never utilization on its own. The baseline is also moving: speculative decoding, better schedulers, and mixture-of-experts models keep narrowing the gap without a model swap, so diffusion has to beat a moving target rather than the naive one. And the tooling is early, with open-source diffusion serving in 2026 sitting roughly where open-source autoregressive serving did in early 2024, functional and improving fast but short on the mature inference stacks teams take for granted with vLLM or TensorRT-LLM. Every conclusion here also moves with two prices you don’t fully control, the API rate you compare against and the hardware rate you negotiated, so it is worth dating your assumptions and revisiting them.
The hangover, in the end, is not that enterprises bought the wrong hardware. Many bought it for reasons like sovereignty, data control, the avoidance of lock-in that have nothing to do with token economics and won’t go away. They bought it expecting it to behave like a public cloud, then ran it at a concurrency that cloud economics depend on and that their most valuable internal workloads, the latency-bound ones, can never reach. Text diffusion is not a way to beat the API, nor a blanket upgrade for everything an enterprise runs. It’s a precise tool for a precise gap, the latency-bound, decode-heavy, sovereignty-constrained work where batching is impossible and an autoregressive model leaves a node both starved and overprovisioned. For the copilots, the real-time checks, and the agentic steps that have to answer now, it turns that node from a guilty line item into a saturated asset, on a fraction of the boxes the alternative would need. That’s a narrower claim than rescuing your hardware ROI, and a far more durable one. The future of enterprise AI is the right architecture, on the right hardware, carrying the right tokens, and knowing which tokens those are is the part no vendor will sell you.
Inception Labs, “Mercury: Ultra-Fast Language Models Based on Diffusion” (arXiv:2506.17298) and Mercury 2 launch coverage, February 2026
“Consistency Diffusion Language Models” (arXiv:2511.19269) on the arithmetic intensity of autoregressive versus diffusion decoding across batch sizes
Baseten’s “A guide to LLM inference and performance” on the memory wall, batching, and the prefill versus decode distinction
Leviathan et al., “Fast Inference from Transformers via Speculative Decoding” (2023), with Medusa and EAGLE; AWS EC2 P5 pricing pages and 2025 P5 savings-plan announcements
LLaDA2.0 (Bie et al., 2025) on the scaling behavior of diffusion language models.
Note: Throughput figures are engineering approximations for a 70B-class model; substitute your own measured numbers, at your own batch sizes and sequence lengths, before any procurement decision.
