Skip to content

Reproduce the paper from published results

The paper reports 123 results: π0.5, SmolVLA and X-VLA, each pretrained with one quality method, fine-tuned on the 24 held-out tasks and evaluated on 50 seeds per task. The per-seed rollout logs of every result are published. qvla paper tables recomputes from them every number the paper derives from its evaluation rollouts and checks them against the published outputs: the main results (each method relative to plain BC, per VLA), the data of the figures (the headline result, random selection against filtering at the same data budget, the methods, scaling with pretraining data size, conditioning), the success rates in the text and a results page.

You need a CPU and the reproduce extra (Install).

Recompute the tables

qvla paper tables --check

IDENTICAL  14 outputs match expected/

It reads the v1.0 results from the Hub and keeps the download in ./.qvla-cache (--cache DIR for another directory). Add --out DIR to keep the outputs in DIR: tables/table1.md, figures/*.csv, claims.json and results_page.html. How the check works.

Quality labels and the human study

The quality-label distributions and their rank correlation with human judgments come from the pretraining data's labels and the human study, not from the rollouts. qvla paper label-study recomputes them; the correlation needs numpy, which the method extra installs:

qvla data get corpus --include 'index/*' --include 'signals/v1/*/chunk/awr_weights.json' --dest corpus
qvla paper label-study --pretraining-data corpus --check

--out DIR keeps fig4a_sim_pe.csv and fig4.json. How the check works.

Next