Fit is a storage result. A token is still an I/O result.
The SSD is not a workaround in Colibrì. It is the expert store.
GLM-5.2 is 744 billion parameters. The int4 container is about 372 GB on disk. The dense slice, shared experts, embeddings, sits around 9.9 GB resident. The 19,456 routed experts live on NVMe and page in when the router names them. The project’s own 25 GB box, the 12-core laptop this started on, still decodes at 0.05 to 0.1 tok/s cold. That number is in the README. They published the floor on purpose.
Yesterday I said the product is the placement policy. This is one policy, written in C, with the disk as the warehouse.
The current tagged release is v1.10.1, published August 31. I planned this post against v1.8.0 on August 24. Two family drops landed in that week. v1.9.0 added GLM-5.3-Flash with vision on 25 GB of RAM. v1.10.0 added Qwen3.8-Flash-Next as a complete engine: tool calling and vision in the same tag, not as follow-ups. v1.10.1 is a packaging repair for the prebuilt archives. If you build from source, it is v1.10.0. This morning the repo sits at 26,879 stars, Apache-2.0, last push a few hours ago. I am writing against the tag. I have not run this engine on the 3090. Every speed number below is project-reported or community-logged unless I say otherwise.
Colibrì is not a general local runtime. It is a family of tiny C engines and one launcher. coli reads config.json, picks c/colibri or c/glm53 or c/qwen38 or whichever sibling matches, and renders that family’s chat template. No BLAS. No Python in the forward pass. Python is the converter, the launcher, and the optional gateway. The engine at runtime is a few hundred kilobytes of C. CUDA, Metal, and Vulkan exist as tiers. None of them is required. The README is blunt: a GPU only ever makes it faster. Speed is set by the disk, because the experts are streamed from it.
COLI_MODEL=/nvme/glm52_i4 ./coli plan
COLI_MODEL=/nvme/glm52_i4 ./coli chatEight families sit in the roster as of this writing. GLM-5.2 at 744B. GLM-5.3-Flash at 321B with a vision tower. Inkling at 975B. Kimi K3 at 2.8T. DeepSeek V4 Flash at 284B. Qwen3.8-Flash-Next at 125B plus a 51B hashed n-gram. Qwen3.6-35B-A3B. OLMoE at 7B. That list is a support matrix, not a promise that every family is a daily driver. Kimi K3 still wants about 1.6 TB on disk. Qwen3.8 is CPU only and reads the official FP8 checkpoint with no conversion. Qwen3.6 is the small one that can actually live in RAM.
The architecture is the month’s thesis with the costume off. Dense weights stay hot. Routed experts are mmap’d. A hit is one probe as of v1.8.0. A miss is a disk read. Prefill on DeepSeek V4 was rewritten expert-major so each distinct expert is read once per chunk rather than once per layer. Recurrent-state checkpoints exist because an agent that re-prefills the whole conversation through SSM layers is how you wait minutes for turn two. Kimi K3 parks those checkpoints in RAM or on disk with COLI_K3_CKPT. DeepSeek V4 measured the same idea on an RTX 5080 plus two NVMe drives: 90 seconds for a 3,324-token prefill once, then 6 to 9 seconds on later turns, decode around 1.6 tok/s at 3k context. Those are their numbers. They are also the first numbers in this stack that treat the second turn as the product.
Vulkan is the backend for cards the vendor stack abandoned. An RX 580 still has a path. Their notes put it 19 to 24 percent ahead of ROCm on an RX 9070 XT. Metal exists. Kimi K3’s Metal path accelerates KDA and projections; MoE experts stay on CPU. DeepSeek V4’s CUDA tier now reaches Pascal and Turing if you build with CUDA_ARCH=portable-pre-ampere NO_TC=1. A GTX 10 card is not a 5090. It is also not “no GPU path.” The interesting claim is still the cold end of the ladder, not the fully resident one.
The complication is the screenshot.
“Runs on 25 GB” is true. “Chat” is the word people glue onto it. On that original laptop, GLM-5.2 cold decode is 0.05 to 0.1 tok/s. Peak RSS during chat sits around 20 GB, auto-capped, on a box with 25 GB installed. A 128 GB CPU-only desktop is logged around 1.8 tok/s warm. A single 5070 Ti laptop-class box is 1.07 tok/s with the GPU-resident pipeline. Full expert residency on 6× RTX 5090 is 5.8 to 6.8 tok/s decode with about 13 seconds to first token in their July 12 experiment log. The website currently prints 9.0 to 9.2 tok/s for a 6× 5090 full-residency NUMA row. I am not reconciling those two rows from this desk. Both describe experts that no longer commute. Neither describes a 25 GB box.
GLM-5.3-Flash is the cleaner physics lesson. 321B total. About 12 GB resident at GLM53_BITS=4. MLA folded so KV is 33 KB per token instead of 1.39 MB. One decode token touches 42 sparse layers × 8 experts × 14.2 MB, which is 4.8 GB. Their reference SSD, measured with O_DIRECT, gave 72 MB/s at queue depth 1 and saturated near 200 MB/s at QD4. That puts a floor of 24 seconds per token on that hardware: what the disk takes to deliver the bytes, with any CPU and any GPU. Faster silicon does not move it. Fewer bytes would. v1.9.0 published that table on purpose. Cold decode around 44 seconds per token, warm around 20. I already used those numbers yesterday. They have not gotten kinder overnight.
Qwen3.6 is the exception that proves the placement rule. 35B-A3B, about 20 GB on disk, wants full RAM residency. The CUDA VRAM expert tier measured 1.44 to 10.05 tok/s, 7.0×, on two 8 GB cards, bit-identical to CPU. That is an interactive number. It is also a 35B working set sitting in the hot pool, not a 744B warehouse paging from NVMe. If your box can keep the experts resident, Colibrì stops being a streaming engine and starts being a small C runtime with a cache. The interesting claim is the other end of the ladder.
I would rather run the dense 27B I already measured at 99 tok/s on the 3090 than wait on a 744B whose experts commute 4.8 GB per token. That is opinion. It is also the month’s test applied to this engine: fit, speed, quality. Fit is proved. Speed is published, and it is honest. Quality is their oracle work and the gs64 versus per-row ablation, not my needle at 128k. The older per-row int4 mirrors measured about 9 points worse and were the root of think-mode loops. Use the gs64 container with the int8 MTP head, or you are debugging a quant, not an engine.
Use this when you have 24 to 32 GB of RAM, fast NVMe, and a reason to address a frontier MoE you cannot house. Batch extraction. A systems experiment. A vision pass on GLM-5.3-Flash or Qwen3.8 that you do not want to send to a server. You will tolerate seconds to tens of seconds per token, or you have enough RAM and VRAM that the experts stop hitting disk. Skip it when the job is an agent loop on a model that already fits. llama.cpp on Qwen3.8-27B already answered that on this desk: 99 tok/s decode, 761 tok/s on a 99k prefill, ORANGE-PLUM-917 at the midpoint. Colibrì is allowed to win the sparse-pool argument. It is not allowed to replace that measurement with a load screen.
The launcher is the usable surface. coli chat, coli serve, coli web, coli plan, coli doctor, coli tune. OpenAI-compatible server on the families that wired tool calling. Vision through a path paste, a drag-and-drop, or an image_url, with the turn refused when the attach fails instead of generating about nothing. Known issue #1278: GLM-5.3-Flash chat can still end on thinking with --no-think. That is open. It is not a reason to skip the engine. It is a reason not to file “reasoning models work” as a workflow until you have a finished answer.
Monday is FreeToken. Bandwidth-adaptive CPU and GPU co-execution. A global LRU expert cache. VRAM that can move between experts and KV without a reload. A different opinion about the same warehouse. Colibrì’s opinion is simpler: the SSD is the expert store, the hot pool holds the dense slice plus whatever the LRU and the learned pins can keep, and a miss costs the disk. That is a legitimate systems breakthrough. On a particular machine it still means 1 tok/s, or 0.05.
Loading is not success. A token that moves 4.8 GB is the working set with the costume off.
If this was useful, forward it to one engineer who needs less noise in their feed.