Cardano — learning when to stop watching
Project page: ostadabbas.github.io/cardano.github.io · Code: github.com/ostadabbas/Cardano
Cardano decides, while a video is still playing, that watching more of it would not change its answer — and stops there. The decision is made by a 40-parameter linear head reading nine features of the model's own answer distribution. It reads no hidden state, runs at one global operating point, and is never tuned per dataset.
Video arrives one window at a time. The backbone keeps a KV cache, so each frame is encoded once and windows past the commit are never encoded at all. The head reads the answer distribution after each window and asks a single question: would the rest of the video change this answer?
Why a stopping rule is not a confidence threshold
Confidence asks how sure am I? Cardano asks would more evidence move me? Those come apart: the head is trained against the model's own final answer, so its target is agreement, not correctness. Concretely, it matches a tuned per-dataset confidence threshold while itself being untuned — which is the point, since the threshold is what you would otherwise have to fit for every new dataset.
Results
Accuracy and fraction of video watched, Qwen2.5-VL-7B, against a matched causal budget that watches the same number of leading windows:
| Benchmark | Cardano | Watched | vs. matched budget |
|---|---|---|---|
| STAR | 67.2 | 0.69 | +3.7 [+2.4, +5.1] |
| NExT-QA | 81.8 | 0.31 | +1.3 [+0.3, +2.3] |
| FunQA | 76.0 | 0.41 | +1.9 [+0.9, +3.0] |
| Video-MME | 65.1 | 0.59 | +2.5 [+0.6, +4.4] |
All intervals exclude zero and hold on 10 of 10 head-fit seeds. On LLaVA-OneVision-7B the same unrefit head gives +3.5 / +2.3 / +2.4 on the first three.
The saving is what the stopping head buys. Where accuracy also improves on an off-the-shelf backbone, that comes from fine-tuning the answerer.
The green points are the four budget settings. Cardano sits above the causal fixed-budget curve at every one, and on FunQA above the whole-video backbone while watching under half the video.
Where it stops early, and where it correctly refuses to
The mechanism is most visible on safety-critical footage. Here a pedestrian is already crossing in the first window; the head commits at the third of sixteen and answers correctly, so the answer exists while the clip is still playing rather than after it ends:
The converse matters as much. On egocentric planning — where the question asks what to do next, so the decisive evidence is the final frame — the same head watches 0.89 of the stream and declines to commit early on about two thirds of items. That is the correct behaviour, and it is not something a fixed budget can do.
What is in this repository
| path | contents |
|---|---|
heads/commit_head_final/ | the 10 deployed commit heads (one per fit seed), 40 parameters each |
heads/commit_head_frozen_fit/ | 10 heads fit on frozen-backbone trajectories, for frozen-model experiments |
results/ | every number in the paper's tables, with confidence intervals and per-seed counts |
adapters/qwen2.5-vl-7b/ | LoRA adapter for the 7B answerer (the headline backbone) |
adapters/qwen2.5-vl-3b/ | LoRA adapter for the 3B answerer |
adapters/llava-onevision-7b/ | LoRA adapter for the LLaVA-OneVision answerer |
assets/ | figures used above |
The commit heads are the contribution and they are tiny — 40 parameters, about 5 KB each.
Note on
gate.pt. Each adapter directory contains anadapter.ptand agate.pt. Thegate.ptis a commit head that was co-trained during answerer fine-tuning and is not the deployed head — its gradient never reached the backbone and it was discarded. The heads that produce every number on this page are the ones inheads/commit_head_final/. Load those.
Using a head
import torch
from commit_head import CommitHead # see the code release
head = CommitHead(d_h=0, d_mid=0) # linear on 9 features, 4 budget weights
head.load_state_dict(torch.load("heads/commit_head_final/e150_head_CA_star-nextqa-funqa_s0.pt"))
head.eval()
# per window: features are 9 chance-normalised statistics of the answer distribution
logits = head(features) # (W, 4), one column per budget weight kappa
commit = (torch.sigmoid(logits[:, 2]) > 0.5).nonzero()[0] # kappa = 0.05, the shipped setting
Geometry is sixteen windows of two frames spanning the clip, so watching everything is 32 frames. Clips shorter than about four seconds yield fewer windows.
How it was trained
Two stages, in this order. The two halves are not trained jointly.
- Answerer — LoRA fine-tune, with the answer supervised on a prefix plus three real future windows, and a consistency term pulling the prefix answer towards the prefix-plus-future answer. No gradient from the stopping decision reaches the backbone.
- Commit head — fit afterwards, offline, on the frozen answerer's recorded answer trajectories. Fit on STAR / NExT-QA / FunQA only, then applied unrefit to every backbone and to benchmarks it was never fit on.
Two details of the fit matter more than they look. The nine features are standardised with statistics computed once over the whole training corpus and then frozen, and the learning rate decays to zero. Without both, the fit does not converge and the operating point ends up depending on the order the training data happened to arrive in — across ten seeds that moved the watched fraction by ±0.07. With them it moves by ±0.003.
Authors
Bishoy Galoaa1, Sadid A. Hasan2, Sarah Ostadabbas1
1 Northeastern University 2 Microsoft
Augmented Cognition Lab, Northeastern University.