Reproduction report № 1
OpenVLA-OFT on LIBERO-Spatial: an episode-exact replication, and the bug between us and it
Claim under test: OpenVLA-OFT reports 97.6% on the LIBERO-Spatial suite. We reproduced it three ways: the authors' unmodified harness on our machine (98.8%), the same checkpoint driven through pera's remote Environment API in author-protocol mode (98.8% — per-task identical, all ten tasks), and pera's canonical per-episode-seeded protocol (98.6%, the published row). The result replicates. Getting the API path to agree episode-for-episode required finding an observation-encoding bug subtle enough that it cost one task 40 points while leaving the other nine untouched — which is the point of this report.
All runs: 10 tasks × 50 episodes = 500 scored episodes, zero infrastructure errors, single RTX 5090 machine, authors' released LIBERO-Spatial checkpoint.
Why reproduce a number nobody disputes
Published VLA results are produced by one harness, on one machine, with settings scattered across the evaluation script, utility files, and model constants. Anyone who has tried to re-run one knows the score you get first is rarely the score in the paper — and nothing tells you whether the gap is your environment, their protocol, or a real disagreement. pera's premise is that an evaluation number is only worth publishing with a runnable upstream anchor next to it. This report is the template: reproduce the published number with the authors' own code on our hardware, then show that the same checkpoint scores identically through our protocolized, remote API — and document every deviation we had to understand along the way.
Setup: three anchors, one machine
- Published: 97.6% on LIBERO-Spatial, from the OpenVLA-OFT paper.
- Same-machine ground truth: the authors'
run_libero_eval.py, unmodified, from theiropenvla-oftrepository with their pinned transformers fork — 494/500 = 98.8% (Wilson 95% CI ≈ 97.5–99.5%, so the paper's 97.6% sits inside our run's noise band). - pera API path: the model runs client-side in its own environment and drives the simulator over a WebSocket Environment API — observations one way, action chunks the other, with the engine enforcing the protocol, recording every step, and exporting a checksummed bundle. The client is ~130 lines; the model code is the authors' own.
The protocol archaeology that preceded the runs pinned every setting the authors'
harness actually uses — 8-step action chunks executed open-loop, 224×224 center-crop
pipeline, seed-once-per-task environment seeding, the libero_spatial
unnormalization key — so that our protocol either matches each one or deviates on
purpose, in writing.
Result: per-task identical, not statistically similar
In author-protocol mode (seed_mode=seed_once), the pera API path and
the authors' harness converge to the same success count on every one of the ten
tasks — 494/500 overall, episode-count-exact:
| path | t0 | t1 | t2 | t3 | t4 | t5 | t6 | t7 | t8 | t9 | overall |
|---|---|---|---|---|---|---|---|---|---|---|---|
| authors' harness | 1.00 | 1.00 | 1.00 | .96 | .96 | .98 | .98 | 1.00 | 1.00 | 1.00 | 98.8% |
| pera API | 1.00 | 1.00 | 1.00 | .96 | .96 | .98 | .98 | 1.00 | 1.00 | 1.00 | 98.8% |
Agreement this exact is a stronger statement than a matching average: it means the remote API is not distorting the evaluation — not in image encoding, not in action transport, not in seeding, not in termination handling. A 500-episode average can hide compensating errors; ten independent per-task matches cannot.
What it took: the θ=π bug
The parity gate did not pass on the first try, and the failure is the most instructive artifact of this reproduction. Nine tasks matched; task 5 collapsed from 98% to 58%.
Root cause: OFT consumes the robot's orientation as axis-angle, converted
from the simulator's quaternion by the authors' own quat2axisangle.
Axis-angle is double-valued at θ=π — (axis, π) and
(−axis, π) describe the same rotation. Our adapter helpfully
canonicalized the quaternion before serving it. Mathematically identical;
numerically it flips which of the two representations the conversion lands on —
and the model was trained on the other branch.
One task happened to operate near θ=π. The model saw orientation values from a convention it had never seen in training, and only that task paid. We found it by diffing observations byte-by-byte between the two paths at the same seed. The fix: the adapter serves the native quaternion, and the client applies the authors' own conversion — the API's job is fidelity, not hygiene.
Two more conditions were required for parity, both now standing policy:
- Author-protocol seeding. Their harness seeds each environment once per task; episode variation then comes from an episode-order-dependent fixture sequence. Parity runs must replicate this quirk exactly.
- The client owns the model's preprocessing. OFT's tf.image resize 256→224 and its gripper normalize→binarize→invert chain live client-side, taken verbatim from the authors' code. Omitting the resize alone drives success to 0% — a reminder that "the model" includes its input pipeline.
The canonical row: 98.6%
The number on our leaderboard is 493/500 = 98.6% (Wilson 95% CI
97.1–99.3%) under libero-spatial@1, which deviates from the authors'
protocol in one documented way: every episode is seeded independently by a derived
per-episode seed, making runs bit-exact-reproducible and episodes independently
drawable. The seeding change moves the score 0.2 pp — within n=500 noise.
That delta is itself a finding. Before the quaternion bug was isolated, per-episode seeding appeared to cost 8 points — a plausible-sounding "seeding sensitivity" story that was actually an encoding artifact. Reproduction infrastructure has to be episode-exact before it can be trusted to attribute smaller effects like this one honestly.
How it fails, when it fails
All 7 failures are on video and hand-labeled (watch them on the result row). Five are grasp misses with a shared signature: the gripper closes on nothing, and the arm then completes the entire place trajectory empty-handed — the fingerprint of open-loop chunked execution with no grasp-failure feedback. The other two are placement misses, the bowl released at the plate's rim. OFT's failure profile is distinct from the other models on this suite: GR00T hesitates near the goal; Octo (on SimplerEnv) fails at grasp closure; OFT fails by not noticing it failed.
Evidence
- canonical bundle
cc9cfa3f43de49b0· root hashsha256:b41f633bb712956dc6b933308d8c08b07f260907121e17cca22ff1e304387ae6 - parity bundle (author-protocol mode)
1261a0e0698c49a3· both verify mechanically (per-file checksums + root hash) - protocol
libero-spatial@1: 10 tasks × 50 eps, max 220 steps, per-episode sha256-derived seeds, determinism classbit_exact, LIBERO pinned at8f1084e - failure labels:
pera.failure-taxonomy.v1sidecar, 7/7 episodes, videos retained for every failure - ground truth: authors'
openvla-oftharness, unmodified, same machine, same checkpoint — 494/500
Questions this report should answer but doesn't, or errors in it: that's what the evidence trail is for — every claim above is checkable against a bundle. Report № 2 (GR00T N1.6, cross-engine agreement at +0.2 pp) and № 3 (Octo-small on SimplerEnv, triple-anchored real-to-sim) will follow the same format.