Aero Realtime
Aero Realtime is a native proactive audio-video language model whose input and output share one aligned autoregressive stream. The released 4B checkpoint is available at kcz358/aero-realtime-4B.
Realtime Interaction
Turn-based models separate input prefill from output decoding. Micro-turn systems shorten that boundary but repeatedly restart interaction. Aero Realtime instead advances audio, video, silence, and lexical output on one causal clock.
Model Architecture
Each approximately 80 ms audio slot is fused with the preceding output token. The language-model head predicts either the next lexical token or the learned silence token, jointly modeling when and what to generate.
Examples
Offline video demo
Run the bundled 60-second video through the realtime engine. The script feeds
80-ms audio chunks and 1-fps video frames in timestamp order and writes the
per-interval outputs to results/offline_demo.jsonl.
python examples/offline/demo.py \
--model kcz358/aero-realtime-4B \
--deploy-config /path/to/vllm-omni/vllm_omni/deploy/aero_realtime.yaml
Pass --video /path/to/video.mp4 to use another video. Add
--tensor-parallel-size N when the model should span multiple GPUs.
Online WebSocket demo
Start the online server in the first terminal. It keeps one Aero generation session alive and accepts chunkwise audio/video input over WebSocket.
python examples/online/server.py \
--model kcz358/aero-realtime-4B \
--deploy-config /path/to/vllm-omni/vllm_omni/deploy/aero_realtime.yaml \
--port 8090
In a second terminal, stream the bundled video at its original wall-clock rate:
python examples/online/client.py --video assets/demo.mp4
Use --url ws://HOST:8090/v1/aero/realtime when the server runs remotely.
For B200 inference, add --mm-encoder-attn-backend TRITON_ATTN to the server
command.
Interactive 4B outputs, including seekable video, chunked audio waveforms, and per-slot silence/lexical predictions, are available on the project website. You can also try the model directly in this HF Space demo.
Training
Training uses the main branch of
LMMs-Engine.
The two-stage 4B configs and launcher are under scripts/train/.
export AERO_WORKSPACE=/path/to/aero-workspace
bash scripts/train/launch.sh stage1
export AERO_STAGE1_MODEL=/path/to/merged-stage1
bash scripts/train/launch.sh stage2
The workspace must contain the dataset and initialization paths referenced by the YAML files. The launcher only starts training; environment and data setup remain external.
Evaluation
Evaluation uses the main branch of
LMMs-Eval and the shard runner
under scripts/eval/.
export AERO_MODEL_PATH=kcz358/aero-realtime-4B
export AERO_OUTPUT_DIR=$PWD/results/ovobench
bash scripts/eval/launch.sh
See scripts/eval/README.md for sharding and hardware overrides.