Two cards give you two pools of memory and a slow wall between them, not one bigger card.
A 16GB card next to a 24GB card gives you 40 gigabytes of capacity and not one gigabyte of a 40GB card.
The fantasy is understandable. The two numbers add up on a spec sheet, the model loader reports both devices, and a checkpoint that would not fit on either card alone loads and runs. Both fans spin. The log shows weights resident on CUDA0 and CUDA1. It looks like the wall came down. It did not. What you built is two memory pools connected by a link that is far slower than the memory inside either card, and every design decision from here follows from that one fact.
The physics settle the argument before the benchmark starts. A 3090 moves data across its own memory at roughly 940 gigabytes per second. The PCIe 4.0 x16 slot connecting it to a second card tops out near 32 gigabytes per second in one direction, and a consumer board rarely gives both cards a full x16 link at the same time. NVLink, where an older pair of 3090s can still use it, raises that ceiling without erasing the gap. The number that governs token generation is the bandwidth inside a card. The number that governs a split is the link between cards. Those two differ by more than an order of magnitude, and no amount of added capacity closes it.
llama.cpp will split the model anyway, which is exactly why the fantasy survives contact with reality. It offers three real placement modes, and each one prices the boundary differently. Understanding what you actually bought means understanding what each mode does with that slow link.
The default is pipeline parallelism, what the flag calls layer split. Each card holds a contiguous block of the model’s layers, and the KV cache for a layer lives on the card that owns that layer. A token runs through the first card’s layers, crosses the link once to reach the second card’s layers, and finishes there. The transfer is cheap because only the activations cross, not the weights, which is why layer split tolerates a slow interconnect. What it does not do is make the two cards compute in parallel. The token is processed sequentially, one card then the other. You gain capacity and pay a modest transfer tax. You do not gain speed. Generation runs at roughly the pace of the pipeline, not double it.
The mode that would actually parallelize the compute is tensor split, and llama.cpp marks it experimental for reasons that matter on a consumer machine. It divides both weights and KV cache across the cards and performs several cross-GPU reductions inside every layer, which is what lets it cut latency instead of merely adding memory. That design is the one the consumer interconnect punishes hardest. Tensor split requires flash attention enabled or it refuses to start. It cannot use a quantized KV cache at all: ask for Q8 K and V and it errors out, so the memory you saved with an 8-bit cache on a single card is gone. It is not implemented for a long list of architectures, including most of the MoE and hybrid models people buy a second card to run. DeepSeek2, OLMoE, Grok, the Nemotron hybrids, GLM-DSA, and the Mamba and Mamba2 state-space families all fail with an explicit not-implemented error and send you back to layer split. Even when it does run, it wants NCCL compiled in and benefits from CUDA peer-to-peer, which is usually restricted to workstation and datacenter cards and can crash or corrupt output on some consumer motherboards with IOMMU enabled.
The older row-split mode that once served as the tensor-parallel path is now deprecated in the current documentation, superseded by tensor mode and described as comparatively poor performance. Community runs on mismatched pairs have reported layer split edging out row split even with NVLink present. The pattern across all of it is consistent. The mode that is cheap to run does not parallelize compute, and the mode that parallelizes compute is fragile, interconnect-bound, and unavailable for the exact models that motivated the second card. Two cards are two pools. The software can span them. It cannot dissolve the wall between them.
The better design starts by refusing the 40GB framing. Ask what two independent pools are good at, and the answer is running two things at once. Put the text model on the 24GB card where its KV cache has room to grow with context, and give the 16GB card a job of its own: an embedding model for retrieval, a vision model for a multimodal agent, a draft model feeding speculative decoding to the primary, or a second agent that should not share a queue with the first. Each card holds a model that fits it, each runs at its own full bandwidth, and no token crosses the boundary during generation. That is not a consolation prize for missing the 40GB dream. It is usually more total useful work than one split model delivers.
The concurrency case makes the split design look even stronger. A single person watching one terminal wants one model as fast as possible, and there the split’s transfer tax is pure cost with nothing bought back. The moment you run more than one workload, two independent endpoints stop competing for the same card’s memory and scheduler. A coding agent hammering the 24GB card with long-prompt prefills does not stall the retrieval service on the 16GB card, because they never share a pool. Isolation is a property you get for free from two cards and lose the instant you fuse them into one split model.
There is a legitimate case for spanning one model across both cards, and it is the honest one. The model does not fit on either card, and the alternative is spilling into system RAM, which is slower than the PCIe link between two GPUs. Layer split across two cards beats CPU offload. When that is the situation, run it, then measure it against the number that matters instead of the capacity you unlocked. On my own bench a 24GB 3090 runs a 15.33GB Qwen3.8-27B Q4 file at 99 tokens per second in decode with a Q8 KV cache and a 128K context allocation, and processes a 99,000-token prompt at 761 tokens per second before generation begins. A split has to clear that bar or explain why it cannot. If the pain I feel is slow prefill on a coding repository, adding a boundary crossing does not fix it and may add to it. Capacity was never the complaint.
This is the mixed-GPU version of the month’s recurring test. Two cards establish fit, and fit is worth having. They do not establish speed, and they say nothing about quality. Treating 16GB plus 24GB as a 40GB card fails all three claims at once by assuming the capacity number drags the other two along with it. It does not. The wall between the pools is real, the software that spans it charges for every crossing, and the fastest machine is usually the one where each card has a reason to exist. Design for two tools, not one imaginary bigger one.
If this was useful, forward it to one engineer who needs less noise in their feed.