lingbot-va-mot
Converted, not fine-tuned, base checkpoint for WanMoTTransformer3DModel — the Mixture-of-Transformers (MoT) video-action architecture from joliachen/lingbot-va (model_mot.py). 10.0B total parameters. Produced from the shared-backbone lingbot-va-base (WanTransformer3DModel) checkpoint via convert_to_mot.py.
A fine-tuned derivative of this checkpoint (LIBERO-Goal post-train) is available at ZhuoranChen/lingbot-va-mot-posttrain-libero-goal-gradaccum43.
How the action-stream weights were initialized
The action stream is losslessly cloned from the corresponding video-stream weights at conversion time — no interpolation, no random initialization, no dimension change (both streams are the same width, see below):
| video-stream source | action-stream target |
|---|---|
blocks.N.attn1.* | blocks.N.action_attn1.* |
blocks.N.attn2.* | blocks.N.action_attn2.* |
blocks.N.norm2.* | blocks.N.action_norm2.* |
blocks.N.ffn.* | blocks.N.action_ffn.* |
blocks.N.scale_shift_table | blocks.N.action_scale_shift_table |
top-level scale_shift_table | action_scale_shift_table_final |
applied identically for all 30 transformer blocks. action_embedder, action_proj_out, and condition_embedder_action.* were already present in the base checkpoint and are kept as-is (not cloned). patch_embedding.* from the base checkpoint is dropped (unused by the MoT model). At this conversion step — before any post-training — the action stream is a byte-identical copy of the video stream; the two only diverge once fine-tuning starts.
Architecture: 3072-dim action stream, not 768 (deviation from the lingbot-va paper)
The lingbot-va paper describes the action stream operating through a narrower 768-dim bottleneck (raw 30-dim action → 768 → … → 768 → 30). This checkpoint's architecture has no such bottleneck: the action stream runs at the full video-stream width, d_v = 3072, throughout:
action_embedder:Linear(30 → 3072)(weight shape[3072, 30])action_proj_out:Linear(3072 → 30)(weight shape[30, 3072])condition_embedder_action.*(text/time embedders, time projection): all at 3072, shape-parallel to the video-streamcondition_embedder- every per-block action module (
action_attn1/2,action_norm2,action_ffn,action_scale_shift_table) is shape-identical to its video-stream counterpart at 3072
Net effect: the action stream holds ~5.0B of the model's 10.0B params — full width/capacity parity with the video stream, rather than the paper's narrower 768-dim bottleneck design.
Config
num_layers | 30 |
num_attention_heads / attention_head_dim | 24 / 128 (→ d_v = 3072) |
in_channels / out_channels (video, patchified) | 48 / 48 |
action_dim | 30 |
text_dim | 4096 |
ffn_dim | 14336 |
attn_mode | flex (train/eval), torch also supported for inference |
Repo contents
Full standalone layout, ready to load directly:
transformer/— this checkpoint's 10.0B-param MoT weightstext_encoder/—google/umt5-xxl(stock, unmodified)tokenizer/— matching T5 tokenizervae/— Wan2.1AutoencoderKLWan(stock, unmodified)