Nouns DAO Proposal Predictor (DistilBERT)
Fine-tuned DistilBERT that reads a Nouns DAO proposal's title + description and predicts whether it passed. Trained on gramajo/nouns-proposals (982 proposals, pass/fail labels).
⚠️ Read this before using the model
Do not use this model to predict real proposal outcomes. It is published as a negative result and as a reproducible artifact, not as a working predictor. See the numbers below and decide for yourself.
Results
Accuracy is meaningless without a baseline. The bar is majority-class accuracy — what you'd score by ignoring the proposal entirely and always guessing the more common outcome. A model only tells you something if it clears that bar.
| run | n train | n test | test pass rate | majority baseline | accuracy | lift | AUC | beats baseline? |
|---|---|---|---|---|---|---|---|---|
| A_stratified_random | 785 | 197 | 50.2% | 50.2% | 58.9% | +8.6% | 0.648 | yes |
| B_temporal_breakeven | 785 | 197 | 28.4% | 71.6% | 73.1% | +1.5% | 0.651 | yes |
| C_post_breakeven_only | 147 | 50 | 28.0% | 72.0% | 72.0% | +0.0% | 0.365 | NO |
What each run asks:
- A — stratified random split over all 982 proposals. Regime held constant. Does the text carry signal at all?
- B — temporal holdout at the BreakEven break. Train on props 1–785, test on 786–982. Does a model trained before the "BreakEven" voting bloc transfer to after it?
- C — post-break only, stratified inside the new regime. Within the new regime, is text predictive? (n is small — read with caution.)
The BreakEven regime change
Around proposal ~786, a bloc of large Nouns holders ("BreakEven") began voting down spend proposals to bring outflows in line with revenue. The pass rate shifts sharply across that boundary:
- Proposals 1–785: 55.9% pass
- Proposals 786–982: 28.4% pass
This is a structural break in the decision rule, and the new rule is largely not a function of the proposal text — it's a function of who is voting and what the treasury looks like. Run B is therefore a test of transfer across concept drift, not a test of whether text is informative. Run A is the cleaner test of the latter.
Training details
- Base model:
distilbert-base-uncased, max_length 512, batch size 16, lr 3e-5, up to 6–8 epochs - Early stopping on eval_loss (not F1 — F1 rewards a degenerate always-PASS predictor)
- No class weighting. An earlier version used balanced class weights derived from the pre-break training distribution; on the post-break test set this pushed the model further toward predicting PASS, making the drift look worse than it was.
- Seed: 42. Exact split indices are published in
splits.jsonin this repo.
Reproduce it
Everything needed to check these numbers is public:
- Dataset: https://huggingface.co/datasets/gramajo/nouns-proposals
- Exact split IDs:
splits.jsonin this repo - Raw metrics:
results.jsonin this repo - Training notebook:
nouns_predictor_experiments.ipynbin this repo
Limitations
- Text-only. No proposer identity, no proposer track record, no requested amount, no treasury balance at submission, no voting history — i.e. none of the variables most likely to actually determine a Nouns vote.
- 982 examples is small for a transformer. Run C in particular is trained on ~150 examples.
- Labels are binary pass/fail; abstentions, cancellations, and vetoes are not modeled here.
- Single seed. Results have not been averaged over multiple seeds and should be treated as indicative, not precise.