NVILA-8B HD High-Frame v2 (filter 0.15/0.15)

This private repository contains an inference-ready research snapshot of the NVILA-8B-HD-Video model trained with AutoGaze-selected visual tokens.

Snapshot

• Training run: nvila8b_hd_highframe_v2_framecap512_batch12k_resume1000_filter1515_nan_guardbf16_24gpu_20260726_205300
• Snapshot: checkpoint-latest, global step 1400
• Parameters: 8,087,084,656
• Weight format: BF16, four safetensors shards
• Tensor payload size: 16,174,169,312 bytes
• Four shard files including safetensors headers: 16,174,270,912 bytes
• Base vision tile: 392 x 392
• Training sequence limit: 16,384 tokens
• Configured text positions: 32,768

Only inference artifacts are uploaded. DeepSpeed optimizer states, RNG states, and rank-local training state are intentionally excluded.

Important Runtime Requirement

The checkpoint uses custom NVILA model code plus the AutoGaze sparse-video runtime from the accompanying AutoGaze codebase. Loading the HF weights alone does not reproduce video input construction. In particular, the runtime must:

1. construct 16-frame, stride-15 gaze windows;
2. preserve the overlapping context frame inside SigLIP and drop its duplicate tokens before the LLM;
3. encode each complete 392 tile through the dense SigLIP prefix;
4. gather the selected patch positions;
5. run the sparse temporal SigLIP tail;
6. apply TokenShuffle(9) and preserve frame/tile/patch order.

See Video Input and Benchmark Evaluation.

Runnable examples:

• Stage 1: prepare reusable gaze payload
• Stage 2: lmms-eval with or without semantic filter
• Interactive single-video gaze/filter/QA demo

Additional experiment records:

• High-frame v2 training input
• Evaluation scoreboard snapshot
• Temporal and HD benchmark protocol

Loading

The model weights and trusted model code can be loaded with Transformers:

import torch
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "yinanzhou1/nvila8b-hd-highframe-v2-filter1515",
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
).cuda().eval()

Use the repository's dynamic payload collator for real video inference. A plain processor call cannot reconstruct the selected-patch sequence.

Training Input Summary

• Each emitted frame starts on a new line.
• Text timestamps are emitted on the first frame and when the sampled time crosses a new one-second bucket.
• Exact time and global normalized patch position remain attached to every visual token.
• Scene isolation is disabled.
• Multiple-choice option positions are balanced during training.
• Semantic filtering is selected for a complete batch with probability 0.10.
• Training filter thresholds are 0.15 for both 224- and 112-level patches.
• Peak learning rate is 1e-5; the checkpoint starts a fresh optimizer from the recovery model rather than resuming a decayed scheduler.

Generation

Evaluation uses deterministic greedy decoding (do_sample=False). Multiple-choice tasks force-decode one option token. Open-ended tasks should reserve output context and use at least max_new_tokens=2048; do not consume the entire context with visual input and then truncate the answer.

Limitations

• This is an intermediate research checkpoint, not a final release.
• Evaluation results from other checkpoints must not be attributed to this step-1400 snapshot.
• Long-context inference beyond the configured training/context lengths is extrapolative and should be reported explicitly.
• Benchmark-specific frame and tile policies materially affect both accuracy and visual-token count.