piedomains-text
Classifies a website into one of 44 content categories from its homepage text. The default
model in piedomains.
0.818 accuracy / 0.758 macro-F1 on 4,382 held-out documents. Temperature-scaled: T = 1.2483, ECE 0.0490 to 0.0168.
Splits are a pure function of the domain
sha256(domain) % 100 decides train/val/test. No seed, no shuffle, no ordering. Every
script computes the same answer for the same domain, so the text corpus and the screenshot
corpus cannot disagree about where a domain belongs, and anyone can recompute what this
model trained on without being told.
That replaced six leak detectors. They existed because two preparers each shuffled their own list, and three separate times a model here was scored on its own training data -- image-only read 0.768 where the honest figure was 0.429, and text read 0.9357 against a real 0.71. Each looked plausible for a while.
The label set is 44, and two of them are not topics
parked (F1 0.987) and unavailable
(F1 0.915, precision
1.000) mean the domain resolves but there is no
site: a for-sale placeholder, a server autoindex, a suspended account. They are categories
rather than errors because the page states them plainly, and because leaving them out put
dead pages inside topic classes.
Known defect: drugs absorbs retail
Extracting those two classes did not finish the job, and the shipped model still gets this
wrong. Only 28% of the 672 drugs training documents mention any drug or pharmacy term
(against 4% of shopping). The rest are expired pharmacy domains recycled into SEO spam,
plus parked and dead pages the extraction missed -- one advertises the Squadhelp domain
marketplace, another says "blog not found".
So drugs learned "product listings and prices" rather than "drugs", and it competes with
retail. Live, today:
| domain | prediction | |
|---|---|---|
walmart.com | drugs 0.56 | shopping only 0.14 |
zappos.com | drugs 0.40 | a shoe catalogue |
amazon.com | shopping 0.37 | drugs second at 0.20 |
rei.com | sports 0.20 | drugs second at 0.17 |
drugs is in the top two for four of five retailers tested. Treat a drugs label on a
commercial site as unreliable. The fix is a labelling pass, not more data.
It is not mutually exclusive, deliberately
Four questions share one vocabulary, and the error rate tracks the axis rather than the
class: status ~1%, topic ~15%, risk ~21%, and the labels describing what a site is
(shopping, news, forum) ~31%. shopping says what a site does and automobile says
what it is about; a car dealership is honestly both.
So the library returns a categories list -- every label above a threshold -- alongside the
argmax. That lifts the chance of reporting the right label from 81.8% to 87.3% at 1.30
labels per domain. Read that as recall. The evaluation gold is single-label, so nothing
here establishes whether a second label is correct.
Weakest classes, named
urlshortener 0.154,
library 0.333,
socialnet 0.372,
shopping 0.453. The first three sit at their ceiling in
the corpus -- more data is not available for them. shopping is different: it has tens of
thousands of unused documents and still fails, because it is competing against topics
rather than being one.