Fit the quantization to the workload’s memory budget, then make it earn the extra bytes.
A quantization preset cannot know how much conversation you need to keep in memory. It cannot know what your desktop already took, whether another process shares the GPU, or which tensors in this checkpoint suffer most when compressed. Picking a smaller file until the model loads gets you running. It can also leave you stuck between a quantization that wastes available memory and one that spills out of it.
Shoehorn works backward from that gap. Give it a model, a context length, and a memory envelope. It allocates different precisions to different weight tensors until they fit the space left after inference overhead. Last week’s runtime arc asked which engine belongs on your machine. This week’s optimization arc opens one step earlier, with the file you hand that engine.
The current tagged release is v0.3.0, published August 19, unchanged when checked September 19. This is a young MIT-licensed Rust project, not a replacement inference runtime. Shoehorn implements its own quantizer and writes standard GGUF v3 files; llama.cpp supplies inference and importance-matrix generation. This assessment comes from the tagged source, documentation, and open contributions. I have not reproduced its performance or quality results.
The distinction from a stock preset needs care. A name such as Q4_K_M does not mean every tensor uses exactly four bits. llama.cpp already supports quantization mixtures and per-tensor overrides. Shoehorn’s contribution is choosing the mixture against a particular byte budget, rather than asking the operator to select a preset or hand-build the assignment. That changes the starting question from which file to download to how much memory the weights are allowed to occupy.
On NVIDIA, its probe asks NVML for the first device’s free VRAM. On Apple Silicon, it asks Metal for the recommended maximum GPU working-set size, which is not the same measurement as currently free memory. AMD discovery uses a rocm-smi fallback that the project describes as not yet exercised on real hardware. Intel requires an explicit budget, and two GPUs still count as one device for this planner. Wednesday’s warning about mismatched cards survives intact.
The budget calculation subtracts a KV-cache term, an estimated compute allocation, and a safety reserve before assigning any remaining bytes to weights. KV cache stores attention state for the conversation; more context consumes more of it. The default reserve is 512 MiB. A fit designed around a short context can therefore spend memory on higher-precision weights that a longer conversation needs for its cache. Neither file is inherently the better choice.
I would resist the README’s use of the word “exact” as an operational guarantee. The cache arithmetic follows the model metadata and selected cache type, while actual allocation also depends on the runtime and architecture. Compute buffers are explicitly an estimate. Shoehorn’s optional calibration pass loads the fitted model through llama.cpp, measures allocations, solves again, and rewrites the affected tensors. That improves the estimate for that configuration; it does not promise headroom for a different serving workload.
Once the weight budget exists, Shoehorn needs a reason to spend more bits on one tensor than another. The importance matrix, or imatrix, supplies activation statistics collected while the source model processes calibration text. Those statistics weight the damage caused by compressing different input columns. Shoehorn encodes and decodes candidate tensor formats, then scores their reconstruction error with that weighting. The default samples rows; an optional exact-errors mode scores every row, at additional cost.
The solver treats the assignment as a multiple-choice knapsack problem: select one format per tensor without exceeding the available bytes. It adjusts a penalty on memory use until a mixture fits, then spends leftover space on upgrades with the best error reduction per extra byte. This is a practical optimization method, not a proof that it found the best possible model. It optimizes the error measure it was given.
That last qualification matters more than the utilization percentage. Reconstruction error is not a coding test, a tool-call success rate, or a measure of whether the model follows a long instruction. Calibration text also determines which activations the imatrix sees. The source itself includes protective rules that keep embeddings and the output tensor above an aggressive low-bit floor. Even this automated optimizer contains judgments about where its objective can mislead it.
An open September contribution makes that limitation concrete. Its author describes a Muse Glimmer 30B plan that spends substantial memory keeping embeddings and the output tensor at Q8_0 while pushing many other tensors down to IQ2_XXS. The contributor argues that checkpoint-specific evaluation favors spending those bytes elsewhere and proposes exact per-tensor constraints. That is a contributor report, not a result I independently verified, and the feature remains unmerged. It is still a useful warning: a solver can allocate every byte correctly according to the wrong objective.
The project’s own results show both why this is worth investigating and where the evidence stops. Its Qwen3-14B example targets an artificial 8 GiB total envelope on an M4 Pro machine. At 8,192 context, the remaining weight budget is 5.88 GiB, filled by a mixture averaging 3.42 bits per weight. The project reports roughly six minutes to encode, 23 generated tokens per second, and held-out perplexity of 6.85. Those measurements belong to that machine and test, not to every computer with 8 GiB available.
Its smaller-model experiment is less flattering and more instructive. On Qwen3-0.6B, the published held-out perplexity rises from 14.53 for the BF16 source to 212.7 for a tightly compressed 2.84-bit mixture. Lower perplexity is better. The file fits, but the language-modeling loss deteriorates sharply. A nearly full memory gauge cannot tell you whether the model remains useful, and the same README supplies the evidence against treating it as a quality score.
Standard GGUF output does reduce one adoption cost. You do not need Shoehorn running alongside the model after quantization; the file goes to a runtime that supports its architecture and tensor formats.
Start from high-precision weights rather than stacking another quantization pass onto an already compressed file. Budget for source and output storage, CPU work, and an imatrix or an explicit decision to proceed without one. A model too large to run in its original precision may also make local calibration awkward. Downloading a known-good community quant is cheaper in setup time.
There are smaller signs of a project still settling. An open Linux compatibility fix reports that the v0.3.0 prebuilt binary requires a newer glibc than Ubuntu 22.04 supplies. Windows support is described as compile-tested rather than field-tested. The current branch also differs from the release in how it interprets bare numeric budgets: use an explicit unit such as 16GiB, not 16. Pinning the version matters even before you load a model.
I would use Shoehorn when a stable, repeated workload falls awkwardly between available quants and the machine has a predictable memory allowance. Preview the per-tensor plan before spending time encoding. Keep the source revision, imatrix, fitted-file hash, context length, cache precision, and runtime build together. That record makes the custom file something you can reproduce rather than another download whose filename tells half the story.
The test that decides whether to keep it is a comparison against the best existing quant of the same checkpoint on the same workload. Hold context, cache precision, and serving conditions constant. Measure time to first token and decode speed alongside peak memory, then run tasks that can expose damaged behavior. Perplexity is a useful warning light, but an agent needs to produce valid tool calls and complete its work. A custom mixture earns the maintenance cost only when those results improve enough to matter.
I would keep the existing quant when it already clears that bar. Unused VRAM may be deliberate room for a longer conversation, another request, or a draft model for speculative decoding. Shoehorn is useful because it lets you decide where the spare capacity goes. It becomes counterproductive when filling the budget becomes the goal.
Tomorrow’s cache discussion starts at precisely that boundary: the memory left after loading weights still has a job. Before giving the next spare gigabyte to quantization, decide what you are willing to take it away from.
If this was useful, forward it to one engineer who needs less noise in their feed.