An image-to-video pipeline that had been sitting idle for over a week got its first real check after being restarted: four clips, same fixed resolution, same opening sentence describing the motion, each generated from a different starting image — a photorealistic crowd scene, an anime close-up, a stylized illustration, a rendered scene. All four finished in about two minutes each, no crashes, no dropped frames.
The automated checks I run before ever looking at footage — a frame-strip comparison and a motion score that measures how much actually changes between frames — passed clean on every clip. Nothing flagged as frozen, nothing flagged as collapsed into noise. By every number the pipeline records automatically, it was four clean successes in a row.
Only one of the four actually looked good
Then I watched the actual playback and judged it by eye, clip by clip. One looked genuinely smooth — natural, fluid motion that held up on repeat viewing. The other three didn't; watched as video rather than as a frame strip, they had a quality I can only describe as "not good" — motion that felt off in a way none of the automated numbers caught. All four had passed the same collapse checks. Only one passed the actual question, which is whether it's watchable.
I have two candidate explanations and no way yet to tell which, if either, is right. The clip that looked good was the anime-style one, and flat cel-shaded surfaces tend to hide small motion artifacts that would be obvious on a photorealistic surface — that's one option. The other is that its starting image happened to come from an actual video frame rather than a still image, giving the model a head start on plausible motion the other three didn't have. Untangling that needs a proper comparison: hold the style fixed and vary the source type, or the reverse, one variable at a time — not four clips that differ on both axes at once.
Passing and being good are different axes
What this settles, at least for now, is what the automated check is actually for. It answers "did this collapse into visible noise," and it answers that reliably. It does not answer "is this good," and trying to build an automatic accept-or-reject gate on top of a metric that can't tell those two questions apart would mean quietly shipping three clips out of four that a human would reject on sight. The gate stays a person watching playback, not a number in a log — not because the number is wrong, but because it was never measuring the thing being asked of it.