LeRobot v0.7.0: Large-scale training, remote inference

1. Split large models for training and reduce checkpoint overhead

The new training system supports FSDP2 and HSDP. It distributes weights, the values adjusted during training, and other model data across GPUs to train models that are hard to fit in a single GPU's memory. Separately, distributed checkpoints split training states for storage and reload them across multiple GPUs.

According to the LeRobot team's distributed training announcement, saving distributed checkpoints was 3x faster. This was measured while training the 6-billion-parameter FastWAM on a single server with 8 H200 GPUs. Parameters are numerical values a model learns. Under the same conditions, peak memory per GPU when resuming training fell from 143GB to 42GB. These figures imply a reduction of about 71%, easing saving and resuming overhead separately from distributing memory during training.

Table: Peak memory per GPU when resuming training

Conditions: LeRobot team report, one 8×H200 server, 6B FastWAM, before and after distributed checkpoints

Unit: GB

Trackio replaces Weights & Biases for experiment tracking. Local runs can be monitored without signing up or logging in, and console output, including commands and errors, is recorded. Uploading checkpoints to the tracking service as artifacts, bundles of files storing experiment results, is now disabled by default.

2. Read only the video segments needed

The new streaming reader fetches only the video byte ranges needed for training from the Hub or a storage bucket. The team's streaming announcement reports 2.1–2.5x the previous samples per second with four data-reading workers. Each full training pass visits every frame exactly once, and the order is reproduced when resuming.

Table: Change in streaming data loading throughput

Conditions: LeRobot team report, 4 DataLoader workers with the new streaming reader, measured in samples/second

Unit: Multiple of previous throughput

Datasets stored in LanceDB can be read without re-encoding the original MP4 files. Below is the blog's example for loading a dataset.

from lerobot.datasets import LeRobotDataset

dataset = LeRobotDataset("lance-format/pusht-lance")

Augmentation, which expands training data by transforming training images, can also run on the GPU. The team's GPU augmentation announcement reports training speedups of 2.8x on H100 and 3.3x on A100 for ACT, a robot action learning model.

The team says physical AI data in LeRobot format publicly available on the Hub totals nearly 300TB. This helps explain the growing scope for streaming where storing entire datasets on local disks is difficult.

3. Keep the robot on site and run the model on a server

A policy server lets cameras, motors, and recording run near the robot while the policy, a trained action model, runs on a remote GPU server. Sending observations, inputs describing the robot's state, and instructions returns the next action chunk from the server. Computing the next inference result while the robot performs actions already received overlaps computation and movement. New action chunks align with the remaining trajectory, dropping expired steps and blending overlapping targets for smooth transitions.

Interactive execution lets users change instructions or ask about what the robot sees while keeping the robot connected and policy weights loaded. It also supports giving a goal and letting the policy select subtasks. Questions and autopilot require a text-generating policy or an external planner. The external planner connects a vision-language model, which handles images and text, through an OpenAI-compatible API to answer questions and choose the next task steps.

Remote execution reduces model memory demands on the computer beside the robot. Developers must instead operate an inference server and network, adjusting execution to latency, the time it takes for a response to return. Transport uses Zenoh, which provides TLS encryption for secure connections and access control settings.

4. Unify simulator and motor connections on common foundations

A common runtime replaces the setup that installed a separate simulator in the same environment for each benchmark, a task used to compare performance. Simulators can run inside the LeRobot process or as servers in separate Docker images. Execution, recording, and teleoperation workflows can be tested without a physical robot.

Evaluation results show success counts alongside 95% Wilson confidence intervals, ranges indicating uncertainty in the success rate. This lets users assess uncertainty when interpreting success rates from a small number of trials.

Feetech and Dynamixel motors share a serial bus implementation, a connection method in which devices communicate in sequence. A rustypot-based implementation replaces vendor software development kits, while existing calibration files for setting motor reference values are retained. Motor operating modes are now set by sending commands directly to the bus.

5. Check installation and training resume paths before upgrading

The installation section of the release notes says Linux PyTorch and torchvision switch to CUDA 13 builds, requiring NVIDIA driver 580.65 or later. For older drivers, it outlines installing PyTorch from the CUDA 12 index first.

The training section of the release notes says the new distributed training system replaces the FSDP1 path, requiring migration of old settings to new options. FSDP checkpoints saved through v0.6.x can load weights but cannot resume the training state as it was. Accelerate YAML and environment variable settings are rejected by default; continuing to allow them requires a separate setting described in the release notes.

The inference and execution section of the release notes says the old asynchronous inference module was removed, requiring migration to remote inference. Code that calls policies directly must also handle the new return format, which can return both actions and prediction information.

LeRobot v0.7.0 expands the scope for handling larger models and datasets, but requires existing users to migrate their installation, training, and execution paths.