Reproduction report № 2
GR00T N1.6 on LIBERO: cross-engine agreement, a step-budget verdict, and the run we threw away
The situation this report handles: not every model comes with a runnable reference harness. NVIDIA reports 97.65% on LIBERO-Spatial for its official GR00T N1.6 fine-tune but ships no public LIBERO eval; the checkpoint everyone can download is a community fine-tune. The anchor strategy therefore differs from our other reports: the community checkpoint's own card (n=200/suite) plus an independent prior-art reproduction of the same checkpoint on a different evaluation engine. pera agrees with that independent engine to 0.2 points — the strongest form of agreement available when the original authors' harness doesn't exist.
Four suites, one checkpoint
| suite | pera | checkpoint card (n=200) | reading |
|---|---|---|---|
| object | 99.6% | 100.0% | agreement |
| spatial | 96.8% | 96.0% | agreement (+ cross-engine ±0.2) |
| goal | 95.2% | 98.0% | compatible (p = 0.13) |
| long | 87.4% | 97.5% | step-budget effect — see below |
The long-suite gap is a clock, not a skill deficit
A ten-point miss would normally end a reproduction claim. Here the bundle explains itself: all 63 failures are step-cap timeouts at the protocol's 520-step limit (the OpenVLA-lineage convention), zero task failures. Successful episodes crowd the cap — median 244 steps, 99th percentile 441 — and sampled failure videos show the model mid-progress when time expires: second moka pot in hand, approaching the stove. GR00T executes multi-stage tasks correctly and slowly; under a generous horizon its rate would approach the card's.
The cross-model control makes it a finding rather than an excuse: X-VLA, run under the same 520-step cap on the same machine, scores 92.4% — its fast chunked execution clears the tasks with room to spare (success p99 = 413). Same rules, different clocks. Execution speed is a scored trait of the policy under any finite-horizon protocol, and honest rows should say so rather than bury it: this one does, in its anchor note.
The run we threw away
Our first goal-suite run scored 84.4%. It was invalid, and it never touched the leaderboard: 52 of its 98 failure videos showed corrupted observations — black voids, displaced cameras — traced to two simulation sessions sharing one rendering process's EGL context. The model had been scored on garbage input. A clean, isolated rerun scored 95.2%; every published pera bundle since passes a per-episode frame integrity scan.
We publish this because it's the strongest argument in the report: evaluation infrastructure fails in ways that look like model failures. An 84.4% for GR00T-goal is a perfectly believable number. Only video evidence made it disprovable — which is why every pera score links to its episodes.
How it fails, when it genuinely fails
With infrastructure ruled out, GR00T's real failure profile is distinctive and consistent across suites: hesitancy at the goal. All eight of its wine-rack failures carry the bottle to the rack and hover without committing the insertion; its goal-suite losses are placement stalls and second-stage grasp timidity, not wrong objects or missed detections. Contrast OFT (grasps fail, but the place motion completes empty-handed) and Octo (grasp closure itself fails): three architectures, three signatures, all watchable on their rows.
Evidence
- spatial bundle
17f52e532b484b13(96.8%, 484/500, tagged) · object52d1c8a483414b5f(99.6%) · goal5e13a01e13f04166(95.2%, clean rerun, tagged) · longb9b1bd488574498d(87.4%, all-timeout analysis) - quarantined corrupted goal run
29c6fd4e5e5d40aa— retained as the incident record, never published - checkpoint: community GR00T-N1.6 LIBERO fine-tune (multi-suite; card n=200/suite); NVIDIA official-ft reference 97.65% spatial
- all bundles verify offline:
pera verify <bundle>