A useful benchmark is a reproducible workload record, not the fastest number the terminal printed.
A benchmark result without enough evidence to rerun it is a story about a machine. The number may be accurate. It may even repeat on the same afternoon. It still cannot tell you whether another quant, runtime build, cache state, or prompt produced the improvement you think you measured.
The first job is freezing the thing under test. Record the model repository and exact file, then hash the local weights. Pin the source revision too. Hugging Face supports downloading a model snapshot by a full commit hash, which is more useful than writing “latest” in a notebook and hoping latest stands still. Record the runtime commit, build flags, operating system, accelerator driver, and the complete launch command. If one of those changes, you have a new experimental condition rather than another sample from the old one.
Hardware descriptions deserve the same precision. “RTX 3090” leaves out the CPU feeding it, system RAM, power limit, cooling state, and other processes occupying the card. A mixed-GPU run also needs device order, layer or tensor split, and interconnect state. Two machines with the same GPU label can reach different numbers for honest reasons. The record has to make those reasons visible before the comparison begins.
The workload comes next, and this is where most local benchmarks become misleading. Save the exact prompt or its hash, the chat template, sampling settings, seed when applicable, requested output length, and concurrency. Count tokens after templating, not from the source document. The current llama.cpp server response separates newly processed prompt tokens, reused cache tokens, and predicted tokens, which is the distinction a warm agent turn needs. A 60,000-token request with 59,000 cached tokens is not the same prefill workload as a cold 60,000-token request.
Time to first token, prefill throughput, and decode throughput answer different questions. Time to first token includes the delay before useful output reaches the client. Prefill tokens per second describes processing the input. Decode tokens per second describes generating the continuation after that input is ready. End-to-end task time includes the costs your user actually feels, including tokenization, sampling, queueing, network transport, and any prompt assembly outside the engine. The llama-bench documentation explicitly says its measurements exclude tokenization and sampling. That makes it useful for engine comparisons and incomplete as a user experience measurement.
Short completions are especially easy to abuse. A two-token probe can spend more of its run on startup, scheduling, or speculative-decoding overhead than on sustained generation. One unusually fast sample has the opposite problem: it may capture a lucky cache, temperature, or background-load state. Use enough generated tokens for the decoder to settle, repeat the run, and keep every result. Report the median and the spread, not the best line from the terminal.
Cache state needs to be an explicit test dimension. A cold run starts after the process and relevant caches have been reset according to the protocol. A warm run repeats with the intended model state resident. A cache-reuse run preserves a known prompt prefix and records how many tokens the engine actually reused. Calling all three “warm” hides the mechanism that produced the speedup. Restarting the runtime between variants is often the cleanest boundary, but the rule matters more than the ritual: define the state another person must recreate.
Resource measurements need a time series, not a screenshot after completion. Sample peak GPU memory, peak system RAM, and power across model load, prefill, and decode. Record the sampling interval and whether the reported power value is instantaneous or averaged. NVIDIA documents queryable memory and power fields in nvidia-smi, including separate instantaneous and average power reporting in current tooling. A single idle reading beside a throughput number does not establish the power cost of the run.
I would keep one machine-readable record beside every result set. The exact field names matter less than refusing to leave the conditions in somebody’s shell history:
model_file: exact-name.gguf
model_sha256: full-hash
model_revision: full-commit
runtime_commit: full-commit
driver: exact-version
command: complete-launch-command
prompt_sha256: full-hash
prompt_tokens_processed: 0
prompt_tokens_reused: 0
generated_tokens: 0
state: cold | warm | prefix-reuse
ttft_ms: 0
prefill_tokens_per_second: 0
decode_tokens_per_second: 0
peak_vram_mib: 0
peak_ram_mib: 0
average_power_watts: 0
quality_gate: pass | failThat record protects the comparison from a common category error. Changing from Q4 to Q6, moving from 32K to 128K context, enabling MTP, and upgrading the runtime in one step may produce a faster result. It cannot tell you which change caused it. Hold the model revision, prompt, occupied context, generated length, cache precision, sampling, and hardware state constant while changing one variable. If the new setting needs another variable to work, report the pair as a configuration change instead of crediting one flag.
Performance still proves only performance. Every run needs at least one quality gate that can fail. A long-context setup should place a known fact at multiple positions and require exact retrieval after the intended context length is occupied. The gate should also test the cache precision and runtime path you plan to ship. Passing one needle is not a certificate of long-context reasoning. The RULER paper found that simple needle retrieval can look strong while harder multi-hop and aggregation tasks deteriorate as context grows.
The second quality gate should be the work that justified the machine. Give a coding model a real repository task with tests, a document model a real extraction with an answer key, or an agent a multi-turn tool sequence with expected arguments and stopping conditions. Preserve inputs and score outputs with a rule decided before the run. A synthetic needle tells you whether a fact survived the path. The real workload tells you whether the model can use what survived.
This is also why cross-model throughput charts need restraint. Tokens are produced by different tokenizers, and two models may emit different numbers of tokens for the same answer. Prompt templates, reasoning modes, output limits, and quality can differ too. Compare the same checkpoint and workload when isolating runtime or quantization changes. When comparing models, lead with completed task quality and end-to-end time, then show token rates as supporting diagnostics rather than a universal leaderboard.
Formal benchmark suites take this discipline further. The MLCommons inference rules require consistent systems and frameworks and treat replicability as mandatory. A local benchmark does not need their entire submission machinery. It does need the underlying honesty: another operator should be able to identify the system under test, reproduce the run, and explain any material difference.
End each comparison with three separate verdicts. Fit says the complete workload stayed within memory at the claimed context and concurrency. Speed says the measured latency and throughput meet a declared threshold under the recorded state. Quality says the synthetic gate and real task met their acceptance rules. A model can pass one, two, or all three. Collapsing them into “runs great” is how a benchmark turns into advertising.
The number worth keeping is not the fastest one. It is the one that survives the rerun after you have written down everything that could make it disappear.
If this was useful, forward it to one engineer who needs less noise in their feed.