← all results

Reproduction report № 4

Three convention bugs, or: why robot evaluation numbers can't be trusted without anchors

pera engine · July 2026 · findings from onboarding eight VLA model families

The claim of this report: the biggest threat to a robot-policy score is not variance, not cherry-picking, not even step budgets — it is the silent convention mismatch. In one week of onboarding eight model families onto identical benchmark protocols, we hit three of them. Each produced clean-looking episodes, healthy infrastructure, zero errors — and a catastrophically wrong number. Each was invisible without running the model authors' own harness on the same machine and comparing. And each is now a rule enforced by pera's contract system.

All three bugs were caught before publication — by per-task anchor comparison, by a two-episode smoke test, and by watching failure videos. None would have been caught by aggregate statistics alone.

Bug 1 — the θ=π quaternion branch (OpenVLA-OFT)

Symptom. Nine of ten LIBERO-Spatial tasks matched the authors' harness exactly; task 5 collapsed from 98% to 58%.

Diagnosis. Byte-level diffing of observations between the two paths at the same seed. The only difference: our adapter canonicalized the end-effector quaternion before serving it. Mathematically a no-op — a quaternion and its negation are the same rotation. But OFT consumes orientation as axis-angle via its own quat2axisangle, and axis-angle is double-valued at θ=180°: (axis, π) and (−axis, π) describe one rotation two ways. Canonicalization flipped which branch the conversion landed on — and the model was trained on the other one. Task 5 happens to operate near θ=π; only it paid.

The rule it became. The platform serves the native quaternion, bit-for-bit as the simulator reports it; consumers apply their own training-era conversion. An API's job is fidelity, not hygiene — "cleaning up" data crosses into changing what the model was trained on. With the fix, the pera path reproduced the authors' harness per-task identically, all ten tasks (report № 1).

Bug 2 — the rotated proprioception frame (X-VLA)

Symptom. A two-episode smoke test scored 0/2. The failure video showed something stranger than incompetence: a confident approach, then an endless grasp-retry loop — the gripper oscillating open/closed at the goal for 200 steps.

Diagnosis. X-VLA reads its end-effector orientation from the OSC controller's grip-site frame. The standard observable most stacks serve — and what pera's state field carried — is the hand-body frame. On the Franka Panda these differ by a constant 90° rotation about Z. We measured it on-box across tasks and poses: constant to 0.05°. The model believed its wrist was rotated 90° from what its cameras showed, and every grasp it planned disagreed with every grasp it saw.

The rule it became. "EEF pose" is not a format — it's a family of frames. The integration checklist now demands the frame be verified against the training controller, not assumed from the observable. One constant rotation matrix in the client (R_OBS2CTRL) took X-VLA from 0/2 to a 98.4% published row, statistically inseparable from the authors' own server evaluated on the same machine.

Bug 3 — upside-down training worlds (SmolVLA, π0.5)

Symptom. SmolVLA scored 20.2% on a suite where the authors' own harness, same machine, scored 73.4%. Videos showed the arm hovering, lost — purposeful motion, wrong universe. π0.5 failed its smoke 0/2 with the same look.

Diagnosis. The LIBERO simulator renders frames upside-down. One version of a popular eval stack passes them to the policy raw (inverted) and flips them only for video export; the next minor version flips them upright for the policy too. Two checkpoints trained months apart therefore expect opposite orientations — and nothing in either checkpoint's config says which. Serve the canonical upright image to a raw-trained model and it navigates a mirrored world.

The rule it became. Image orientation is part of the training contract and must be pinned per checkpoint against the exact eval-stack version that trained it. With the un-flip in place, SmolVLA scored 77.6% — above its own harness — and π0.5 went 0/2 → 2/2 on the spot, finishing at 97.4%, equal to its same-machine anchor.

What the three have in common

• They are silent. No exception, no NaN, no infra error. The simulator runs, the videos render, the bundle checksums. Only the number is wrong.
• They are catastrophic and weirdly shaped. One task loses 40 points while nine are untouched; a model retries grasps forever; a score lands at a quarter of truth. Aggregate variance doesn't look like this — convention bugs have structure.
• They are invisible without an anchor. If we had published 20.2% for SmolVLA, who would have known? The number is plausible for a small model. Only the same-machine authors'-harness run said otherwise. This is why every pera row ships with one.
• The videos diagnose them faster than statistics. Each bug was characterized in minutes of watching behavior — mirror-world hovering, grasp-retry oscillation — after hours of numeric analysis had only localized it.

Why this is an argument for contracts

pera's manifest system exists because of exactly this failure class. A policy declares its conventions — camera roles and resolutions, state format and rotation representation, action formats, gripper semantics — from a closed registry where free text is a schema error. The platform resolves the declaration against the benchmark's and returns direct (untouched), adapted (explicit, recorded transforms), or refused (incompatible, before any simulation runs). The registry cannot make anyone read their training code — but it forces the questions to be answered somewhere auditable, and when a score looks wrong, the resolved contract in the evidence bundle says precisely what every byte passed through.

A postscript on what honest evaluation surfaces once conventions are right: with all three bugs fixed, four of our eight models scored higher through pera than through their own reference harness — consistently on precision tasks, consistently for checkpoints trained on 256×256 images that the reference harness evaluates at a different render resolution. That finding (four models, two suites, p as low as 10⁻⁴) is the subject of report № 5: sometimes the reference harness is the one with the convention mismatch.

Evidence