3.4 · Leakage-Free Splits, Evaluation, and Dataset Audit
Random sample splits often overstate robot-learning performance because adjacent frames, repeated objects, or the same site appear on both sides. Evaluation must test the generalization claim the system is expected to make.
- Design splits by episode, object, site, task, embodiment, or time.
- Connect offline metrics to closed-loop behavior.
- Run a practical data-and-pipeline audit before expensive training.
Split by the intended claim
Use episode-level splits at minimum. Hold out objects for object generalization, sites for environment generalization, tasks for composition, embodiments for transfer, and future time for drift. Store explicit split manifests and deduplicate trajectories.
From loss to behavior
Offline likelihood, MSE, endpoint error, smoothness, and calibration diagnose models. Closed-loop success, completion time, interventions, collisions, recovery, and constraint violations determine whether behavior is useful.
The smallest convincing baseline
Before fine-tuning a large VLA, overfit a tiny subset and train a compact state-to-action baseline. Failures at this stage often reveal timing, preprocessing, normalization, or controller defects that scale cannot repair.
A trustworthy dataset supports the exact generalization claim and can reproduce the deployed physical interface.
