gliner_datause_tiered

Fine-tune of urchade/gliner_large-v2.1 for data-use mention extraction with a single DATA_MENTION class, trained on rafmacalaba/data-use-mentions-tiered — the tiered copy of rafmacalaba/data-use-mentions where Luna/classifier-judged T3 (non-mention) and junk spans are untagged hard negatives (text stays, span removed). The extractor owns the mention boundary only (T1 evidential ∪ T2 declaration vs T3/junk); specificity detail is recovered downstream by the multitask SFT model.

Labels

• DATA_MENTION — a real data mention that carries an analytic or declarative use (T1 evidential ∪ T2 declaration)

Training

• base model: urchade/gliner_large-v2.1
• dataset: rafmacalaba/data-use-mentions-tiered (gliner_tiered config)
• epochs: 5
• learning rate: 5e-06
• batch size: 16
• precision: bf16
• checkpoint selection: val span-F0.5 (post-hoc sweep of epoch checkpoints; eval_loss was explicitly not used)

Evaluation (tiered holdout)

Gold = T1∪T2 spans; a true-FP cluster matching a dropped T3/junk span counts as a T3 leak (lower is better). Label-agnostic Hungarian matching, jaccard >= 0.5 — identical to prior data-use-mentions evals.

thr	tp	fp	fn	precision	recall	f0.5	f1	t3_leak	t3_leak%
0.10	16812	9110	353	0.6486	0.9794	0.6956	0.7804	2829	31.1%
0.20	16664	6850	501	0.7087	0.9708	0.7491	0.8193	2335	34.1%
0.30	16505	5607	660	0.7464	0.9615	0.7814	0.8404	2002	35.7%
0.40	16275	4623	890	0.7788	0.9482	0.8076	0.8552	1741	37.7%
0.50	15934	3701	1231	0.8115	0.9283	0.8325	0.8660	1498	40.5%
0.60	15195	2735	1970	0.8475	0.8852	0.8548	0.8659	1203	44.0%
0.70	13793	1808	3372	0.8841	0.8036	0.8667	0.8419	876	48.4%

Best F0.5: 0.8667 (thr=0.7) Best F1: 0.8660 (thr=0.5)

Full per-doc predictions (raw scores, gold spans with tier decisions): holdout_predictions.jsonl on this repo.

Corpus breakdown (holdout, best F0.5)

corpus	examples	spans	thr	precision	recall	f0.5	f1
prwp	7758	12359	0.70	0.8897	0.7777	0.8648	0.8299
fcv	7574	9693	0.70	0.8744	0.8542	0.8703	0.8642

T3/junk expulsion analysis (post-hoc, per-example)

Per-example analysis of holdout_predictions.jsonl (scored at threshold 0.0, thresholding applied post-hoc) to verify that tiered training actually expels T3/junk from tagged mentions. Predictions were matched to gold spans by normalized surface form within each doc (stored predictions lack offsets), so counts approximate the overlap-based Hungarian matching above.

Score separation

group	n	median score	p25	p75
keep-gold hits (T1∪T2)	18,156	0.854	0.624	0.943
drop-gold leaks (T3/junk)	4,665	0.228	0.059	0.548
other FPs	2,565,971	0.000	0.000	0.003

• AUROC (keep vs drop scores): 0.869 — the model suppresses T3/junk into low scores rather than tagging it.
• Leak composition: 4,430 tier3_nonmention, 235 junk.

Expulsion vs recall trade-off

thr	keep recall	leaks expelled
0.40	92.0%	61.2%
0.50	88.1%	67.3%
0.60	82.4%	74.1%
0.70	74.3%	81.2%
0.90	32.1%	97.3%

A modest operating threshold (0.4–0.5) expels **60–67% of T3/junk leaks while retaining 88–92% of T1∪T2 recall**.

Interpretation

• Expulsion largely works via scoring: leak scores are heavily suppressed (median 0.23 vs 0.85 for true mentions), so any post-hoc threshold in the 0.4–0.5 range removes the majority of T3/junk without meaningful recall loss.
• Residual leaks are high-confidence T3 lookalikes — not fixable by thresholding; the leak% rising with threshold (31%→48% above) reflects the FP pool shrinking faster than the leak pool, not worsening extraction.
• Full numbers: leak_analysis.json on this repo; reproduction script: training/analyze_gliner_leaks.py in the source repo.