AI Slop
Coding assistants feel effortless. The evidence says they make us write more code and work slower, and the dependency that builds only benefits one party.
ai
2 posts
KV Store: paying memory to avoid recompute
Without a KV cache, every new token re-attends over the whole prefix. With one, you trade that compute for GPU memory — about 40 GB of it on Llama 3 70B at 128K context, and PagedAttention decides how much of that you waste.