Isaac 0.5 by Perceptron

Introducing Isaac 0.5, our open foundation model for robot learning.

Isaac 0.5 brings multimodal video understanding, embodied reasoning, spatial grounding, task-progress estimation, and robot control into one 36-billion-parameter sparse model. It reads images, video, language instructions, robot state, and previous actions, then produces text, normalized coordinates, task-state outputs, or robot actions.

To our knowledge, Isaac 0.5 is the first open model operating at the frontier of multimodal video understanding, embodied reasoning, and robot control.

Read the paper · Download the weights · View the code

Using this checkpoint

This checkpoint is consumed through the Perceptron Isaac repository. Direct Transformers and stock LeRobot usage are not currently supported. It is compatible with Perceptron Isaac commit be6507b.

git clone https://github.com/perceptron-ai-inc/isaac.git
cd isaac
git checkout be6507b4aed7472f2029606c22684d4ebc9d73e6
git submodule update --init --recursive
git -C lerobot fetch origin main
git -C lerobot checkout e12389c1f8f591ad05dced4e284d4e92e48c5df4
cd lerobot
uv sync --locked --extra perceptron_isaac

The pinned repository lockfile defines the supported runtime versions.

Extending the frontier of open robot learning

Isaac 0.5 is trained on more than 35 robot systems, 100,000 hours of robot experience, one million hours of general video, and three trillion multimodal tokens. Video understanding, spatial grounding, task progress, future-percept prediction, and robot action are co-trained from the beginning on one shared backbone.

Teams can fine-tune Isaac as a robot policy or use its visual outputs inside a planner, controller, or data engine. This model repository provides checkpoint weights and portable runtime manifests. The companion Perceptron Isaac repository provides action-training and inference code, LeRobot integration, a reference policy server, evaluation tools, and reproduction guides.

What's new in Isaac 0.5

• A scaling law for video and robot data: Isaac measures how general video, egocentric video, UMI data, and teleoperation data trade off as training scales.
• Semantic world modeling: The model predicts compact future percepts that capture task-relevant changes such as object motion, contact, grasp state, spatial relationships, and task progress.
• Unified perception, reasoning, and control: One shared sparse backbone supports video understanding, pointing, tracking, task-state estimation, and robot action generation.
• Continuous and discrete action interfaces: Isaac supports continuous control through a dedicated Flow expert and diffusion transformer, plus discrete control through a 2,048-token FAST action vocabulary.
• Real-time closed-loop control: Isaac predicts the next action chunk while the current chunk is still executing, using the latest observation and previously issued commands.
• Open training and deployment stack: This model repository publishes checkpoint weights and portable manifests; the companion Perceptron Isaac repository provides training, inference, LeRobot integration, the reference policy server, and evaluation code.

A scaling law for video and robot experience

Isaac 0.5 establishes a scaling law for choosing the mix of broad video and expensive robot demonstrations.

We trained Isaac across a grid of general-video, egocentric-video, UMI, and teleoperation mixtures, then measured action-prediction loss on robot trajectories held out from training. The resulting contours map data mixtures that reach the same held-out action loss.

At a target action loss of 2.50:

• a model trained on 1,000 hours of general video requires about 5,900 hours of teleoperation;
• a model trained on one million hours of general video requires about 28 hours of teleoperation.

This is a 210× reduction in required teleoperation. The result gives teams a practical way to decide when another order of magnitude of video is valuable and when collecting more robot experience matters more.

General video contributes broad coverage of objects, activities, environments, and tasks. Egocentric video adds close-up, task-relevant views of hands, motion, and contact. UMI data connects visual knowledge to manipulation. Teleoperation grounds that knowledge in a specific embodiment and action space. Isaac trains on these sources together so each shapes the representation used for control.

Semantic world modeling

Isaac learns from video by predicting future percepts: compact representations of changes that matter for a task. These include which object moved, how objects are related, whether contact occurred, whether a grasp held, and which stage of a task has been reached.

Targets are derived from later frames, including general video without robot action labels. Given observations through time t, Isaac predicts the percept at a later time t + Δ:

$$ L_{\text{percept}} = \mathbb{E}\left[\ell\left(g_\theta(o_{\leq t}), z_{t+\Delta}\right)\right] $$

Here, o≤t is the observation history and z(t + Δ) is the future percept. This objective trains the same backbone Isaac uses to produce actions.

Model architecture

Isaac uses a 36B Qwen-family vision-language backbone with sparse experts. Text, vision, time, robot state, and action history enter one shared sequence.

The autoregressive interface produces text, normalized coordinates, task-state outputs, and discrete actions. Discrete control uses a separate vocabulary of 2,048 FAST action tokens. For continuous control, backbone states condition a dedicated Flow expert and 36-block diffusion transformer that generates an action chunk.

Each mixture-of-experts layer gives every token access to 256 learned experts and a null route. A token may use zero to eight routed experts, while a shared expert and residual path remain active. This lets Isaac vary routed compute across visual, language, state, and action tokens. The checkpoint contains 36B parameters in total.

Performance benchmarks

We evaluate the same Isaac checkpoints across multimodal video understanding, spatial grounding, held-out action prediction, and closed-loop robot control.

Up and to the left is better. Ringed Isaac 0.5 results sit on the frontier; hollow markers are dominated.

Where Isaac sits among open models

Building with Isaac in the open

Isaac 0.5 can be fine-tuned on new demonstrations, used as a visual backbone inside a larger embodied system, or deployed as an action policy through LeRobot or our reference server.

This model repository provides:

• checkpoint weights;
• continuous Flow and discrete FAST action configurations;
• text, pointing, tracking, and task-state output schemas;
• checkpoint, data, and model-I/O manifests;
• the technical report and model card.

The companion Perceptron Isaac repository provides:

• action training, fine-tuning, and inference code;
• LeRobot integration and a reference policy server;
• evaluation code, task definitions, and rollout manifests;
• reproduction and deployment guides.

Resources

• Weights: Hugging Face
• Code: GitHub
• Technical report: Read the paper

Open models are essential to robotics progress. The weights and portable manifests are published here; code, interfaces, evaluation tools, and deployment guides are maintained in the companion Perceptron Isaac repository.

For help deploying Isaac on your infrastructure, contact sales@perceptron.inc.