← result row

Reproduction report № 3

Octo on SimplerEnv: reproducing the numbers nobody brags about

pera engine · July 2026 · protocol simpler-widowx@1 · Octo-small (27M) and Octo-base (93M)

Why reproduce a 27% score? Because reproduction infrastructure that only confirms flattering numbers is marketing. SimplerEnv exists to measure something uncomfortable — how policies trained on real robot data behave in a visually-matched simulator — and its published numbers for Octo on the WidowX bridge suite are low, scattered, and strange: the bigger model scores worse. If pera's evaluation stack is honest, it should reproduce exactly that: the low scores, the per-task scatter, and the scaling inversion. It does, on all three counts.

Octo-base, same triple: published 16.0 / same-machine 14.6 / pera 15.6. Per-task Fisher vs the same-machine anchor: all p ≥ 0.49 (small), all p ≥ 0.61 (base).

A triple anchor with a fourth leg: real robots

SimplerEnv's authors report both simulated and real-robot results for these exact four tasks, which lets this row do something rare — situate a simulation score against physical ground truth:

Two readings. First: 27% is not a broken harness — it's what this model does on these tasks on a physical robot too (25.6%). The suite is simply hard for this generation of policies; RT-1-X scores zero on three of the four tasks in both sim and reality. Second: a leaderboard where Octo's 27% sits beside LIBERO's 98s is telling the truth about benchmark difficulty being non-comparable across suites — which is why pera never aggregates across protocols.

The scaling inversion survives reproduction

Octo-base has 3.4× the parameters of Octo-small and scores roughly half as well here (15.6% vs 27.1% through pera). That inversion is in the authors' own tables (16.0 vs 29.5), in their harness on our machine (14.6 vs 25.0), and now in a third independent path. Whatever its cause — the authors' checkpoints, training recipes, or a real capability difference on out-of-distribution visuals — it is a stable property of the released artifacts, not an evaluation accident. Counterintuitive results that replicate three ways are the ones worth trusting.

What "failure" looks like at 27%

All 70 of octo-small's failures are labeled from video. The profile is dominated by grasp closure (43/70): the arm reliably reaches the right object, then nudges, tips, and pinches without ever securing it. The eggplant task inverts the pattern — the sink's walls funnel the approach so grasps succeed, and failures shift downstream to drops during transport (6 of its 9). A small tail never engages the object at all. This is a third distinct signature next to OFT's empty-handed places and GR00T's goal hesitancy — evidence that failure taxonomy, not just success rate, differentiates architectures.

Evidence