PCIe Gen3 x8 runs at its limit when a MoE streams experts on an RTX 5060 Ti

The specification sheet of the RTX 5060 Ti says PCIe 5.0, and most buyers stop reading there. Two details sit underneath that line and they decide how the card behaves once a model no longer fits in its 16 GB. One: the card is electrically x8, with the slot of an x16 card and half of the lanes wired. Two: the link generation is negotiated with the host, and a mid-range desktop CPU from a few years ago will happily run a Gen5 card at Gen3. On my measurement bench the card and the CPU settled on a link of about 7.9 GB/s in each direction, a quarter of what the card could do in a recent platform, and for months it made no difference at all. Then I started testing an inference engine that streams mixture-of-experts weights from system RAM, and the bus went from idle to saturated. This article covers that link: how to read it, what the measurements showed, why the obvious explanation isn't proven, and what it means if you're buying a 16GB card for local inference. It's a companion to the hands-on test of Strata on the same machine, which is where these numbers come from.

What does an x8 card on a Gen3 host actually get?

About 7.9 GB/s per direction, one quarter of the same card in a Gen5 slot. PCIe 3.0 runs at 8 gigatransfers per second per lane with 128b/130b encoding, so each lane moves about 0.985 GB/s of payload. Eight lanes give about 7.88 GB/s. A Gen4 host would double that, a Gen5 host would double it again to about 31.5 GB/s on the same eight lanes.

The RTX 5060 Ti is specified by NVIDIA as a PCIe 5.0 card with an x8 interface, and with no spare lanes to compensate, a slower link generation hits it harder than an x16 card. My host is a Ryzen 5 4500, a six-core desktop part whose integrated PCIe controller is limited to Gen3. The two together negotiate the lowest common denominator: Gen3 x8.

You can read the negotiated link on any Linux machine with the NVIDIA driver loaded:

$ nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.gen.max,pcie.link.gen.hostmax,pcie.link.gen.gpumax,pcie.link.width.current,pcie.link.width.max --format=csv
pcie.link.gen.current, pcie.link.gen.max, pcie.link.gen.hostmax, pcie.link.gen.gpumax, pcie.link.width.current, pcie.link.width.max
1, 3, 3, 5, 8, 16

The current generation reads 1 at idle because the link downshifts to save power; under load it climbs to the maximum the host allows. Look at gpumax, the card's own limit, and hostmax, the platform's. On my box they read 5 and 3. The width needs one more look: the maximum reads 16 because the connector is a full x16 slot, but only eight lanes are wired on this card, and the current width, 8, is what you actually get. For expert offloading on your machine, that one line beats any benchmark you'll find online.

The generation printed on the box is a ceiling. The generation your link runs at is a negotiation, and the host usually wins it.

When the bus sits idle

When the whole model sits in VRAM, the PCIe link is used once, to load the weights, and then almost never. During five minutes of continuous generation, the two models I serve in production on this card moved 0.03 to 0.04 GB/s from host to GPU on average. While reading a 28,313-token prompt they reached 0.07 to 0.09 GB/s. Those are the bytes of the requests and the responses, nothing more.

Hence the many reviews concluding that PCIe generation doesn't matter for inference. For a dense 27B model at 4 bits or a 35B MoE whose experts all sit in VRAM, that conclusion is correct. The link could be Gen1 and the decode speed would barely move. It only changes the loading time, and a 15 GB model loads in seconds on any recent NVMe and link.

Planning a local inference box for a team, with a platform check like this done before buying hardware? Then my hub on AI infrastructure for companies collects the method I use for self-hosted setups, starting from the measured perimeter.

When the bus becomes the busiest component in the box

Strata works differently. Its most frequently used experts stay resident in VRAM, about 45% of them in the configuration I tested, and the rest wait in system RAM. When a token needs a cold expert, the engine either moves it over the bus or computes it on the CPU. Over the whole session, 77% of the expert computations hit the GPU cache; the rest went elsewhere.

The bus ran at its limit. I sampled the link with nvidia-smi dmon every half second during two phases of the same task: a long prompt of 28,313 tokens, and five minutes of continuous generation.

For Strata, the 95th percentile of host to GPU traffic was 7.8 GB/s during prefill and 7.7 GB/s during generation; it was not recorded for the in-VRAM models.

The peaks are what matter. The highest single samples were 7.75 GB/s during generation and 7.95 GB/s during prefill, on a link whose theoretical payload ceiling is about 7.88 GB/s. The small excess over the theoretical figure is sampling granularity; what counts is that the traffic came in bursts that filled the link. During prefill the engine also read the model files from disk at up to 400 MB/s, because part of the expert set was paged in for the long prompt.

The mean during generation, 1.7 GB/s, looks harmless, but bursty workloads need a different statistic. A token that waits for an expert waits for the whole transfer, and the transfers ran at the ceiling.

Is the bus the bottleneck? The data says maybe

That's the obvious reading, and I can't prove it. During the same generation phase, the CPU was not idle: total load across twelve threads was 44%, six cores ran above 50%, and the CPU package reached 81 °C. Cold experts Strata leaves on the host are computed on the CPU with AVX2, and that work competes with the transfers for the same memory controller.

The data fits two readings. In one, the bus is the limit: every cache miss stalls on a full link, and a Gen4 or Gen5 host would decode faster. In the other, the CPU is the limit: the transfers saturate the link because they happen in bursts, but the decode speed is set by how fast six Zen 2 cores compute the experts that stay on the host. Both are compatible with a saturated link and a busy CPU at the same time.

A saturated link shows the bus working at its limit. Whether you're waiting on the bus is another matter.

The two hypotheses make different predictions, and that is how they could be separated. If the bus is the limit, forcing the link down to Gen2 should roughly halve the cold-expert throughput and slow the decode visibly. If the CPU is the limit, the same downgrade should change little. A second machine with a Gen4 host and the same card would give the opposite test. I've done neither, and I won't state a cause I haven't measured.

There is one indirect hint. The project documents its own numbers on a Ryzen 5 7600 with an RTX 5070, a platform with PCIe Gen5 and an x16 card: up to eight times the link bandwidth of my box. Its stated decode speed for the same Coder model is 55 tokens per second; mine was 38 on mixed content and 46 on code, with half the system RAM and a lower-RAM mode. A link-bound engine would explain the gap; so would a slower CPU and less memory. I'd call it a hint and nothing more.

What the RAM speed changed

By accident, the box also gave me a second, cleaner experiment. Halfway through the testing I enabled the memory profile that runs the DDR4 at its rated 3600 MT/s instead of the 2133 MT/s it had been running at. I ran part of the Strata measurements again at the new speed.

On the fifty Python problems, the decode median went from 46.1 to 50.6 tokens per second, a paired ratio of 1.10, with 47 of the 50 problems faster. Wall time for the whole set went from 14.1 to 13.7 minutes. The general questions answered word for word the same, only faster.

A 69% increase in memory transfer rate bought a 10% increase in decode speed. That suggests a pipeline where system RAM is one of several limits without dominating, and it leans slightly toward the CPU-bound reading over the bus-bound one, because the transfers from RAM to the GPU still cross the same Gen3 link. A bus limit stays possible too. The clear part is that on this platform no one component explains the speed; the engine is balanced across RAM, CPU and link, and each one costs a fraction.

How to check your own link before you buy

Considering a 16GB card for local inference? Three steps are enough.

1. Read the negotiated generation and width under load.

   $ nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.gen.hostmax,pcie.link.width.current --format=csv

   Verify: run it while the GPU is busy, otherwise the current generation reads 1.

2. Watch the bus while the model generates.

   $ nvidia-smi dmon -s t -d 1

   Verify: the rxpci column, in MB/s, stays near zero for a model that fits in VRAM.

3. Compare the peak with the theoretical ceiling of your link: about 0.985 GB/s per lane at Gen3, twice that at Gen4, four times at Gen5. Multiply by the width.

If your workload peaks well below the ceiling, a faster host won't help. If they touch it, as they did here, a newer platform might help, and the only way to know is to test the same model on both.

What this means for a 16GB setup

For the common case, a model chosen to fit in 16 GB of VRAM, the PCIe generation of your host does not matter for inference speed. An older desktop with a Gen3 controller is a fine home for an RTX 5060 Ti that keeps a dense 27B or a 35B MoE in its own memory. The general strategies for local models on this card apply unchanged, and none of them depends on the bus.

Things change once you plan to run models larger than the card through expert offloading, whether with Strata, with llama.cpp's --n-cpu-moe, or with any engine that keeps weights in system RAM. Then three components share the work: the link, the CPU that computes cold experts, and the RAM that holds them. On my box all three were busy at once, the link at its ceiling, the CPU at 44%, the RAM at 25 GB of 31. Upgrading one of them alone would buy a fraction of the speed, as the RAM experiment shows.

One more word on the card's x8 interface. On a Gen5 platform an x8 card still has 31.5 GB/s, more than any consumer offloading engine I've seen needs. On a Gen3 platform it has a quarter of that, and an x16 card on the same host would have twice as much. If you buy for offloading on an older platform, count the lanes.

Why llama.cpp offloading behaves differently

Strata's offloading and the one llama.cpp itself offers load the bus differently, so keep them apart. With --n-cpu-moe N, llama.cpp leaves the expert weights of the first N layers in system RAM and computes those experts on the CPU; it doesn't move them to the GPU per token. The PCIe traffic is then the activations going back and forth between the CPU-computed layers and the GPU-computed ones, which is small compared to moving weights. You pay in CPU time and memory bandwidth instead of link bandwidth.

I measured that path on the same card earlier, on a 35B mixture of experts at Q4_K_M with the RAM still at 2133 MT/s. At the best offload step the cost was 43% of the decode speed and 84% of the prefill speed compared with the model fully in VRAM, and a memory cap sweep showed that 12 GiB of system RAM was enough to keep the offloaded configuration at full speed, while 10 GiB cut prefill to 13% of its maximum. Those numbers describe a CPU-bound offload, and they are a lower bound for a platform with faster memory.

So a blanket "offloading is slow on PCIe Gen3" claim is too coarse. Which offload you use decides which component you saturate, and a platform upgrade can speed up one strategy and leave the other untouched.

What I take from this test, and from the first article in this series on 16GB cards, is that a saturated resource points to where to look next. The link was full, the CPU was busy and the memory was nearly exhausted, and the decode speed was the product of all three. If you are designing a local inference box for a team and want the platform tested against the models you intend to run before anyone orders hardware, you can describe the case in the free quote form: it takes two minutes, and I tell you if it's something I can help with.