Context length, cache precision, and concurrency belong in the budget before the weights do.
The KV Cache Is the Model You Forgot to Budget For
Substack is the home for great culture
Context length, cache precision, and concurrency belong in the budget before the weights do.