I run local language models on a Mac and keep a small, changing roster of which build earns its disk space. That week I pulled down two new candidates for the vision-capable slot: an ordinary compressed 9-billion-parameter build, and a more aggressively compressed 4-bit variant of a larger 27-billion model, in on the promise that the smaller footprint would still see.
The benchmark that returned nothing
I ran the same four-way test I run on every new vision candidate: two photos, the same code path, the same questions. Three of the four models answered normally — expressions, captions, fine detail, all read correctly. The 4-bit build answered, both times, that no image had been provided.
Nothing else about the request had changed. Same code, same two images, same call shape. Whatever lets a model read pixels — the vision projector, the small bridge between image and token space — had not survived that compression step, even though the surrounding language model clearly had, since it could still form a fluent sentence about having received nothing.
The pattern I'd already seen once
This wasn't a surprise so much as a confirmation. An earlier attempt at a different alternative build had failed the same way for a different reason, and this made it the second such failure. A build can look complete on paper — right parameter count, right file size, loads without error, answers text questions competently — and be silently missing the one capability that never gets checked unless you check it on purpose. Nobody benchmarks “did it notice the image” because a working model would never need to be asked.
The 4-bit build was deleted, and the 16GB it held came back.
What the failure obscured
Buried in the same round of testing was a real, useful result. Pure decoding speed on the same quantization scheme was faster than the standard build — 14.8 tokens per second against 12.6 — and the smaller 9-billion model turned out to be the actual value pick of the round: the smallest file, the fastest cold start at 2.3 seconds, the fastest decode at 37.9 tokens per second, and vision output on par with a much larger model, just wordier, which made its wall-clock time look worse than the work it was actually doing.
Both findings are true at once. Compression can buy real, measurable speed, and compression can also silently amputate a capability that only shows up as a confident, wrong answer — “no image was provided” is a much easier failure to miss than a garbled one. The fix wasn't to avoid compressed builds; it was to add a rule: never trust a new build on a text benchmark alone if it claims to handle images, because the one that quietly can't will tell you so as fluently as the one that can't tell you anything at all.