LingBot-VLA 2.0

LingBot-VLA 2.0 is Robbyant’s vision-language-action model, the successor to LingBot-VLA 1.0. It pairs a Qwen3-VL-4B backbone with a sparse Mixture-of-Experts Qwen2 action expert (32 experts, top-4 routing) and generates action chunks with flow matching over a 55-D canonical state/action space shared across embodiments.

The LeRobot port is the lingbot_vla_v2 policy. It uses standard LeRobot datasets, processors, lerobot-train, lerobot-eval and lerobot-rollout, including upstream’s dual-query depth and video distillation.

For model details, see the upstream repository and the technical report.

Installation

pip install -e ".[training,lingbot_vla_v2]"

The Qwen3-VL tokenizer and image processor are loaded from Qwen/Qwen3-VL-4B-Instruct (--policy.tokenizer_path) when the processor is built. flash-attn is not needed; attention uses PyTorch SDPA.

Checkpoints

Load either one with --policy.path; the policy type is read from the checkpoint’s config.json.

Mapping a robot onto the canonical layout

The model sees a 55-D state and predicts 55-D actions. The layout is set by canonical_joints in the config:

--policy.state_slots and --policy.action_slots map spans of your dataset’s observation.state and action onto these slots, as JSON:

{
  "observation.state.arm.position": {
    "origin_keys": [{ "observation.state": { "start": 0, "end": 5 } }]
  }
}

• Slot names are the canonical names above, optionally prefixed with observation.state. or action.. Unknown names raise an error.
• Each slot reads one or more {start, end} spans from observation.state (state slots) or action (action slots). Spans are concatenated and packed from the start of the slot; the rest of the slot is zero-padded and excluded from the loss.
• With no slot mapping, the raw features are taken as the 55-D canonical layout, as in the base checkpoint.
• Relative actions are off by default. --policy.use_relative_actions=true subtracts the current state from the action, except for dims whose name contains an entry of --policy.relative_exclude_joints (default ["gripper"]).

The postprocessor inverts the mapping, so select_action returns actions in your robot’s raw action space. Fine-tuned checkpoints save the slot mappings and dataset statistics with their processors.

Cameras

The image step reads the cameras listed in the checkpoint’s input_features, in that order. For the base checkpoint they are camera_top, camera_wrist_left and camera_wrist_right. Map your dataset cameras onto them with --rename_map; views your robot does not have are masked out.

Fine-tuning

Example on maximellerbach/omx_multicubes, a single arm with 5 joints and a gripper and two cameras (top, wrist):

lerobot-train \
  --dataset.repo_id=maximellerbach/omx_multicubes \
  --policy.path=lerobot/lingbot_vla_v2_base \
  --policy.state_slots='{"observation.state.arm.position": {"origin_keys": [{"observation.state": {"start": 0, "end": 5}}]}, "observation.state.effector.position": {"origin_keys": [{"observation.state": {"start": 5, "end": 6}}]}}' \
  --policy.action_slots='{"action.arm.position": {"origin_keys": [{"action": {"start": 0, "end": 5}}]}, "action.effector.position": {"origin_keys": [{"action": {"start": 5, "end": 6}}]}}' \
  --rename_map='{"observation.images.top": "observation.images.camera_top", "observation.images.wrist": "observation.images.camera_wrist_left"}' \
  --policy.dtype=float32 \
  --accelerator.mixed_precision=bf16 \
  --policy.repo_id=your_repo_id \
  --batch_size=1 \
  --steps=30000 \
  --save_freq=5000 \
  --output_dir=outputs/train/lingbot_vla_v2_omx

Keep the weights in fp32 and compute in bf16 (--policy.dtype=float32 --accelerator.mixed_precision=bf16), as upstream does. With --policy.dtype=bfloat16 the optimizer updates bf16 weights directly, and most updates are too small to change them: on this dataset 66% of the weights stayed bit-identical to the base checkpoint, and the fine-tuned policy was much jerkier. Run inference in fp32 as well (the default of a checkpoint trained this way); bf16 inference makes the action chunks noticeably less smooth.

Normalization statistics come from the dataset. Multi-GPU training uses the standard torchrun launcher with the --parallelism.* and --accelerator.* flags. The fp32 weights, gradients and optimizer state take about 96 GB, so shard them with --parallelism.dp_shard=<num_gpus> when one GPU is too small.

Dual-query distillation

LingBot-VLA 2.0 appends learned query tokens (current and future depth/video) after the language tokens, and the action expert attends to them. Both checkpoints carry upstream’s recipe in align_params, so the query tokens are always used at inference. During training, small heads read those tokens and are distilled from frozen teachers (LingBot-Depth with MoGe-2, and DINO-Video) with upstream’s loss weights; the extra losses are logged as depth_loss, future_depth_loss, current_video_loss and future_video_loss.

The teachers are downloaded on the first training step (about 3 GB from robbyant/lingbot-vla-v2-6b and Ruicheng/moge-2-vitb-normal), take about 2 GB of GPU memory and add roughly 130 ms per sample to each step. Each sample also loads the frame chunk_size - 1 steps ahead, which the future targets need. Nothing extra runs at inference.

Key training parameters

To fit a smaller GPU, combine --policy.train_expert_only=true and --policy.gradient_checkpointing=true with a small --batch_size.

Evaluation

Evaluate the RoboTwin checkpoint in LeRobot’s RoboTwin environment (see RoboTwin for setup). The rename map aligns the env camera names with the checkpoint’s:

lerobot-eval \
  --policy.path=lerobot/lingbot_vla_v2_robotwin \
  --env.type=robotwin \
  --env.task=beat_block_hammer \
  --eval.batch_size=1 \
  --eval.n_episodes=100 \
  --rename_map='{"observation.images.head_camera": "observation.images.cam_high", "observation.images.left_camera": "observation.images.cam_left_wrist", "observation.images.right_camera": "observation.images.cam_right_wrist"}'

Average success over the 50 RoboTwin 2.0 tasks (100 episodes each), as reported in the upstream README for the released RoboTwin checkpoint (Muon post-training on clean and randomized data, FP32 inference with the upstream RoboTwin launcher). These are upstream numbers, not measured through this port.

Inference speed-ups

Four flags, all off by default, speed up select_action. They need CUDA and fixed input shapes, and no extra dependencies:

• --policy.use_cudagraph_denoise=true: capture the denoising loop as one CUDA graph.
• --policy.use_cudagraph_prefix=true: capture the vision-language prefix KV fill.
• --policy.use_cudagraph_prefix_full=true: also capture the vision tower in the prefix graph.
• --policy.compile_predict_velocity=true: torch.compile the per-step velocity prediction. The first call compiles for about a minute.

Time for one 50-step action chunk with 10 denoising steps, RoboTwin checkpoint, one RTX PRO 6000:

In bf16 the CUDA graphs give the same outputs as eager; torch.compile changes the numerics slightly.

Deployment

Run a fine-tuned checkpoint on a robot with lerobot-rollout, passing the same --rename_map as in training:

lerobot-rollout \
  --strategy.type=base \
  --policy.path=outputs/train/lingbot_vla_v2_omx/checkpoints/last/pretrained_model \
  --robot.type=<your_robot_type> \
  --robot.port=<port> \
  --task="pick up the cube" \
  --fps=30 \
  --device=cuda

Real-Time Chunking is available with --inference.type=rtc, as for the other flow-matching policies.

Citation

@article{lingbotvla2,
      title={From Foundation to Application: Improving VLA Models in Practice},
      author={Wei Wu and Fangjing Wang and Fan Lu and He Sun and Shi Liu and Yunnan Wang and Yibin Yan and Yong Wang and Shuailei Ma and Xinyang Wang and Yibin Liu and Shuai Yang and Tianxiang Zhou and Kejia Zhang and Lei Zhou and Cheng Su and Nan Xue and Bin Tan and Han Zhang and Youchao Zhang and Fei Liao and Xing Zhu and Yujun Shen and Kecheng Zheng},
      journal={arXiv preprint arXiv:2607.06403},
      year={2026}
}

License

The code and the converted checkpoints are Apache-2.0, as the upstream LingBot-VLA 2.0 and Qwen3-VL releases.

Distillation training downloads three frozen teachers under their own licenses (inference never loads them):

The DINO-Video weights are derived from Meta’s DINOv3, so training with distillation means using them under the DINOv3 License, which among other terms asks you to acknowledge DINOv3 in publications and restricts some end uses.