GPU Starvation Is a Data-Path Problem, Not Just a Chip Problem

GPU starvation is your accelerator waiting on input instead of computing. What causes it, which layer actually fixes it, and where storage fits.

TL;DR

• GPU starvation is an accelerator waiting on input or upstream work instead of computing. Three things cause it: upstream compute, coordination between nodes, and moving the data.
• There is no single fix. Solutions are shipping from storage, memory, silicon and software, which tells you it is a multi-layer problem.
• Storage is one layer of that fix, not the fix. It enters at four specific moments rather than on every token.
• Nirvana's layer is high-performance block storage. ABS ships 20,000 baseline IOPS against AWS gp3's 3,000 default, bursting to 600,000, and in our own benchmarks runs 10.5x faster than gp3 on cold-read analytical queries.

What is GPU starvation?

GPU starvation is when an accelerator sits waiting on input or upstream work instead of computing. You are paying accelerator rates for a component doing nothing.

It is easy to miss because it does not look like a failure. Nothing has to error, and the job may still complete. Utilization simply sits lower than expected.

For two years, the AI performance conversation was dominated by silicon: which GPU, how many, at what price. That focus is now moving down the compute stack. As organizations deployed GPU fleets at scale, the layers beneath them, including memory, networking, caching and storage, became more obvious.

What causes GPU starvation?

Three things, and they compete with each other:

1. Upstream compute. CPU preprocessing and scheduling that has to finish before the GPU has anything to work on.
2. Coordination between nodes. Synchronization and collective communication across a distributed job.
3. Moving the data. Transfer across network, memory and storage.

All three leave expensive compute idle. The third gets far less attention than GPU selection, and it is the one this piece is about.

Where storage enters the loop

Storage is not in the loop for every token, since active weights, activations and KV cache normally sit in GPU HBM or host memory. It enters at four moments:

• Model and index loading, where large sequential reads set cold-start time
• Checkpointing, which creates large sequential writes
• KV cache offload to NVMe, which is bandwidth and latency sensitive
• Retrieval, when a query reaches a disk-backed index or a database

Those four are expensive moments, which is why the storage tier is getting attention it did not get a year ago.

Who has been working on the data path

Silicon supply is no longer the only variable teams can act on. With fleets deployed, attention has moved to utilization: how much of the compute you already pay for is actually doing work. A quick recap of teams optimizing the data path.

July 1. Meta rebuilt its storage layer. Meta rebuilt the BLOB storage layer on its Tectonic fabric, cutting average cross-region ingestion from 150 minutes to 10 and the worst case from 89 hours to roughly 3. Those figures measure dataset ingestion and research wait time rather than GPU utilization, and the fix combined metadata redesign, caching, prefetching and regional placement.

July 13 to 14. Quobyte extended toward the cache tier. The parallel file-system vendor expanded its GPU-converged storage toward KV-cache and context loading. Its CEO, quoted in Blocks & Files, put the demand shift in three words: inference, inference, inference.

July 14. Cloudera and VAST Data announced a joint solution whose launch claim was eliminating GPU starvation, their phrase rather than ours. A launch claim is not proof the solution works, but it does show both vendors expected buyers to recognize the problem by name.

July 23. AMD and Cerebras split inference across two kinds of hardware, with AMD's GPU-based Helios system handling prompts and large context windows and the Cerebras Wafer-Scale Engine handling memory-bandwidth-intensive token generation. This is processor and memory architecture rather than storage, and it is not evidence the chip stopped being the constraint; Cerebras is addressing that constraint with a different kind of silicon.

July 29. MaxLinear showcased Panther, a storage-offload processor that handles compression and data movement separately from the CPU, claiming up to 2.2x greater effective KV-cache capacity. That figure is vendor-reported, and the announcement was a showcase rather than a shipping milestone.

July 31. XCENA announced its production MX1 lineup, targeting CXL memory capacity, pooling and data movement. This is a memory-tier answer rather than a block-storage one.

August 14. Google made adaptive concurrent prefetching the default in GCSFS 2026.8.0, the layer that lets Python tools read from Google Cloud Storage. Google reports 5x single-file throughput and up to 21 GiB/s, but only for GCSFS paired with Rapid Bucket, so an ordinary workload should not be assumed to get them.

The fix is coming from the full data path: storage, metadata, caching, memory, interconnects and the software that keeps those layers working ahead of the GPU. Meta's own solution spanned several of those layers. AMD and Cerebras addressed silicon. XCENA addressed memory pooling.

Vendors are offering different solutions because this is a multi-layer problem.

Nirvana Accelerated Block Storage (ABS)

The layer Nirvana works on is high-performance storage. Most companies build on object storage such as S3. Nirvana purpose-built high-performance block storage, which covers a narrower set of jobs: model and index loading, checkpointing, KV-cache offload to NVMe, disk-backed vector indexes, and the Postgres and ClickHouse instances underneath the pipeline.

We recently partnered with Altinity, who run their managed service for ClickHouse® on Nirvana as a BYOC substrate. BYOC, or bring your own cloud, is where ABS is best positioned: high-performance storage without upgrading your instances, paying more, or over-provisioning. It works with what you already run, and Nirvana Kubernetes Service (NKS) lets it integrate into your architecture easily.

ABS ships 20,000 baseline IOPS rather than AWS gp3's 3,000 default, bursting to 600,000, at a monthly storage price closer to gp3 than io2, on infrastructure you keep and can leave. On top of that, your workload also sits on the same rack and private network as its data, so a round trip is microseconds rather than milliseconds.

Our own benchmarks, not independent testing:

• ClickBench: 10.5x faster on cold-read analytical queries than AWS gp3
• LangChain agents: read-heavy tasks completed 14 to 17% faster than AWS io2, and checkpoint-heavy workloads finished 24 to 28% sooner at 100,000 checkpoints
• SSB: 34 ms against a three-billion-row table on a single node

Persistent sandboxes with instant pause and resume, backed by ABS

Apart from cloud, Nirvana also runs sandboxes backed by its very own ABS. Sub second pause and resume: the volume at /workspace sits on a live disk with no expiry window, so an agent resumes instead of re-cloning the repo and reinstalling its dependencies, and the compute charge stops while it is paused. Persisting on a live disk rather than a snapshot-and-restore pipeline is a storage-architecture decision, which is why it comes from a storage company.

If you are optimizing your cloud with high-performance storage so your agents don't wait, or you are building agents and want to compare substrates, get in touch. We have several proofs of concept running now.

Get in touch, read the benchmark methodology and full ClickBench results, or compare pricing for your workload profile.

About Nirvana Labs

Nirvana Labs is a high-performance storage cloud purpose built for blockchain, AI and databases i.e. the most demanding, real-time, stateful workloads. Accelerated Block Storage (ABS) offers 20K baseline IOPS included, no over provisioning. Nirvana Kubernetes Service (NKS) with Karpenter auto-scaling, high clock-speed compute and private networking. Backed by Jump Trading, Crucible, etc with 50+ customers live in production today.

Learn more at Nirvana Labs

Nirvana Cloud | Pricing | Blog | Docs | Changelog | LinkedIn |Twitter | Telegram| YouTube