Useful speed belongs to the task, not the model screenshot.
One token per second can be a systems breakthrough and a miserable product. Getting a 744-billion-parameter model to answer on hardware that could never hold its weights in fast memory proves something important about inference architecture. It does not prove that anybody should wait for the answer during ordinary work.
The arithmetic is not subtle. A 250-token response takes four minutes and ten seconds to decode at one token per second, after the first-token wait. An 800-token review takes more than thirteen minutes. Token counts vary by model and text, so neither number predicts a particular paragraph exactly. They do expose the scale of the decision before optimism turns a feasibility result into a daily-driver recommendation.
Colibrì makes the case concrete because the project publishes the uncomfortable numbers beside the clever architecture. Its v1.12.0 benchmark documentation describes a GLM-5.2 int4 container around 370 GB. The original 25 GB machine keeps 9.9 GB resident and reads roughly 11 GB from disk for each cold decoded token. The reported cold rate is 0.05 to 0.1 tok/s. That machine answering at all is remarkable. A 250-token answer at the faster end still consumes about forty-two minutes of decode.
More capable hardware changes the rate without changing the test. Colibrì’s v1.12.0 Vulkan documentation reports 1.58 tok/s sustained for a 512-token GLM-5.2 run on a 12-core Zen 2 system with an RX 9070. That is a project measurement for one disclosed system and path, not a result reproduced here. At 1.58 tok/s, the 512-token run still takes more than five minutes to decode. The improvement is real, and the interaction is still slow.
Tokens per second cannot carry the whole verdict. Queueing and prefill contribute to the wait for the first token. Inter-token gaps follow once output begins, and the task may continue after generation ends. The vLLM metrics documentation separates time to first token, inter-token latency, time per output token, prefill time, decode time, and end-to-end request latency for this reason. A single decode rate hides the wait before generation. It also hides whether the useful result appeared in token twenty or token five hundred.
The first useful speed band is batch work. A document extraction job that enters a queue at midnight and has to finish before morning can tolerate a slow stream if the batch meets its deadline, produces better answers, stays private, or costs less than the available alternatives. Nobody is reading the tokens as they arrive. Set the batch budget in completed items per hour at an accepted error rate, including failures and reruns. One tok/s may clear that bar when each item needs a short output and the machine can work unattended.
Interactive chat sits on the other side of the line. At one tok/s, every token creates roughly a one-second gap in the stream. The user can read faster than the model only until the buffered text runs out, then attention turns into waiting. A stronger answer can justify some delay, especially for a hard question asked once. Requiring several clarifications converts one wait into a sequence of waits, which is where the daily workflow starts to collapse.
An agent can fail that test before decode becomes the largest number. Consider an illustrative six-step repository task where each model turn emits 150 tokens. At one tok/s, decoding alone consumes fifteen minutes. Add six prefills, tool execution, validation, and any retry caused by a malformed call. If the client rebuilds or edits early context between steps, cache reuse may disappear and repeated prefill can dominate. The agent’s speed band is therefore full successful loop time, not the rate printed during its best continuation.
These bands are not universal tok/s cutoffs. Batch work has a completion deadline, chat has a first-useful-output and full-response budget, and an agent has a full-loop budget. The rate matters only through the clock attached to the person or system waiting on the result.
The strongest objection to a latency cutoff is answer quality. A colossal model at one tok/s may solve a task that a smaller model at thirty tok/s gets wrong. Speed does not compensate for failure, and a quick wrong answer is not useful. The reverse shortcut fails too: more total parameters do not guarantee a better result after quantization, routing, prompt changes, or an unfamiliar task. Compare accepted outcomes and elapsed time together. Quality decides whether a candidate remains in the test; latency decides which qualifying candidate belongs in the workflow.
Output length matters inside that comparison. A model that reaches the correct patch in 120 tokens can beat one that narrates its way through 600, even at a lower raw decode rate. Constraining the answer, using structured outputs, or asking for a closed decision can rescue a workload without making the engine faster. Those changes count only when they preserve the information and correctness the job requires. Truncating an explanation before the decisive caveat is a latency improvement on paper and a quality failure in use.
Bounded decisions show how far that redesign can go. Colibrì v1.12.0 added Brio mode, which scores a supplied set of allowed answers instead of generating a free-form response. The release notes say those requests can report zero completion tokens. That does not make prompt processing free, and I have not reproduced the feature, but it changes the workload. A classifier that chooses among four known labels should not be judged by the time needed to write 500 tokens it never has to produce.
I would define the threshold before choosing the model. Give a batch job a completion deadline and minimum accepted items per hour. Interactive work needs a maximum cold time to first useful output and a full-response budget at the length the task actually needs. An agent needs a maximum end-to-end time and retry allowance for the whole tool loop. Run the model against those budgets with a real prompt distribution rather than one short greeting.
An example acceptance sheet might require 100 documents before 6:00 a.m., a useful first chat response within fifteen seconds and completion within ninety, or an agent task finished within ten minutes including one retry. Those are sample budgets, not universal thresholds. Their value is forcing the team to name the wait it is willing to buy.
Cold and warm paths need separate results. Warm cached experts can make the second identical prompt look like a different machine, while a changed document or new subject can bring storage misses back. Record context length, prefill time, first-token time, output tokens, decode time, cache state, and the final task score. A median without the slow path can hide the exact turn that makes someone abandon the tool.
Colibrì has proved that these model classes can run on consumer hardware that conventional runtimes could not use for them. Its project measurements establish speed on disclosed configurations, from a research demonstration to something closer to deliberate use. Task quality still belongs to the exact model, quant, prompt, and evaluation. Feasibility, measured latency, and accepted outcomes remain separate verdicts.
Keep the colossal slow model when its unique answer quality clears a batch or bounded-decision deadline. Keep a faster smaller model for chat and iterative agents when both pass the same quality gate. The largest model that eventually prints a correct answer is an achievement. The model that finishes the working loop before the operator leaves is the workflow.
If this was useful, forward it to one engineer who needs less noise in their feed.