← all results

Reproduction report № 5

The resolution-match effect: when the reference harness underrates its own checkpoints

pera engine · July 2026 · four models, two suites, one consistent direction

The finding: four different models — SmolVLA, π0.5, PulseVLA, and MolmoAct2, all trained on 256×256 LIBERO images — score higher through pera, which serves the simulator's native 256×256 render, than through their own reference harness, which renders the same scenes at 360×360. The gain concentrates almost entirely on two precision tasks and reaches +26 episodes out of 50 in the strongest case (p < 10⁻⁴). Everywhere else, the two paths agree. The most economical explanation: evaluation resolution is part of the training contract, and the reference harness is the one breaking it.

"Eligible" = checkpoints trained on 256×256 data whose reference eval renders at 360×360. Models outside this pattern (OFT, GR00T, X-VLA, Octo — different training pipelines or native-resolution references) show no such asymmetry.

The data

Same machine, same checkpoints, same episode counts (n = 50/task). "Reference" is each model's own eval stack; "pera" serves the native 256×256 render with the model's own preprocessing unchanged:

Three properties make this a finding rather than noise. Direction: five significant-or-borderline task-level divergences across four models, every one favoring the native-resolution path; zero reversals anywhere in ~40 task-level comparisons. Localization: the same two tasks recur — picking a bowl off a small ramekin pedestal, and seating a narrow bottle on a cabinet top. Both demand fine visual discrimination of small supports. On non-precision tasks the paths agree (PulseVLA: 9 of 10 spatial tasks at p ≥ 0.20; π0.5 spatial overall: identical 487/500 totals). Selectivity: models outside the 256-trained/360-evaluated pattern show nothing like it.

The mechanism, most plausibly

These checkpoints were trained on the standard LIBERO datasets, whose frames are 256×256 simulator renders. Their reference eval stack renders episodes at 360×360; the policy's preprocessing then resizes (with padding, in this family) to the network's input size. A 360-render downscaled and a 256-render upscaled do not produce the same pixel statistics: edge sharpness, aliasing on thin structures, and the few-pixel-wide cues that distinguish "gripper aligned with the ramekin rim" from "two pixels off" all shift. For most tasks the policy is robust to this. For tasks whose success hinges on exactly those cues, the distribution shift costs real episodes — and evaluating at the training resolution gives them back.

Why it matters

• For model authors: your released checkpoint may be better than your own eval says. SmolVLA's released model reads 4–5 points higher at its training resolution; its card's headline number was depressed by its own harness default.
• For benchmark maintainers: a single default render resolution silently taxes every checkpoint trained at a different one — and the tax is invisible in aggregate, surfacing only on precision tasks.
• For the field's numbers: this is report № 4's thesis with the roles reversed. There, integrations broke models via convention mismatches; here, the reference harness itself carries the mismatch. Same-machine, same-checkpoint anchoring is what makes the difference attributable at all — without it, a +5-point platform difference is just an argument.

Evidence