SecPaper-Selector-4B

Select the passages that support a security-research question. Given a question and named excerpts, SecPaper-Selector-4B returns the supporting passage IDs or abstains when the supplied material does not establish an answer. It is a Qwen3.5-4B adapter for evidence selection, with an explicit JSON interface.

Use it between retrieval and answer writing: shortlist the supplied excerpts, display those excerpts to a reader, and keep an unsupported question unanswered. On the frozen 156-case test it matches 149 reference passage sets (95.5%), compared with 66.0% for the strongest tested untuned comparator. This measures agreement with fallible synthetic passage labels, not scientific-answer correctness or independent human-verified entailment.

Application example

A source-visible research assistant. Retrieve several abstracts or document excerpts and give each one an ID. Ask which mechanism a paper uses to protect stored data. If the supplied excerpt says AES-GCM, the desired response selects that excerpt's ID; if the mechanism is absent, the desired response is {"evidence": [], "abstain": true}. Show the selected original text to the reader instead of turning the ID selection into an unsupported answer. The runnable example below is illustrative, not an additional benchmark result.

Use

Download this repository and install requirements.txt. CUDA with BF16 support is the measured setup; evaluation used an A100 80GB, batch 8. The base weights are downloaded separately. The PEFT adapter contains about 57MB of original learned tensors before QKV fusion; this repository includes the converted HF weights.

from evidence import EvidenceSelector

selector = EvidenceSelector(adapter="itsalissonsilva/SecPaper-Selector-4B")
result = selector.select(
    "Which mechanism protects the stored data?",
    {"paper-a/p0": "The system encrypts stored data with AES-GCM.",
     "paper-b/p0": "The controller reduces network latency."},
)
print(result)  # JSON-compatible evidence IDs and an abstain flag

The example is illustrative, not a separately measured evaluation. The helper validates IDs and JSON, rejects truncated outputs, and preserves the FP32 adapter values. It does not execute passage content. Inputs exceeding 16,000 tokens are rejected. An empty evidence list means abstention, not proof that no answer exists. Keep the selected passages visible to the user; do not present selections as verified citations automatically.

Measured evidence

The saved HF adapter achieved 149/156 (95.5%) exact reference passage sets on a held-out 2026 security-paper metadata cohort, including 100/107 answerable cases and 49/49 abstentions. This is agreement with fallible synthetic labels, not 95.5% scientific-answer accuracy or human-verified entailment.

Condition	Exact set accuracy	Answerable	Abstention
SecPaper-Selector-4B (saved HF weights)	95.5%	93.5%	100.0%
Untuned Qwen3.5-4B (HF)	48.7%	28.0%	93.9%
Untuned Qwen3.6-35B-A3B (Tinker)	66.0%	50.5%	100.0%
Untuned GPT-OSS-20B, low reasoning (Tinker)	61.5%	44.9%	98.0%
Trained TF-IDF logistic retrieval (CPU)	59.6%	44.9%	91.8%
Fixed lexical overlap	26.9%	39.3%	0.0%

All conditions receive the same supplied passages and task. Qwen conditions disable thinking and allow 160 output tokens; GPT-OSS uses low reasoning and up to 4,096 tokens (all 156 terminated cleanly). The adapter reproduces all 156 native Tinker outputs exactly in the measured HF configuration. All 208 saved FP32 adapter tensors were verified after loading.

The publication rule was fixed before training: at least 90% balanced accuracy, at least 85% in each category, and a positive lower 95% paired anchor-paper-group bootstrap difference against every eligible baseline (or noninferiority with at least 30% lower cost). This release qualifies through quality; it makes no cross-provider cost-saving claim. The trained CPU and original GPT-OSS baselines were added after seeing the native candidate result; neither changed the candidate, labels, nor threshold. See evaluation.json for all comparisons and benchmark/ for rows and predictions. The cohort is small, and shared context passages can induce dependence beyond the 103 anchor groups used in the bootstrap.

Training and reproducibility

• Base: Qwen/Qwen3.5-4B at 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a.
• Tinker attention LoRA, rank 16, alpha 32, two fixed epochs, learning rate 8e-5, batch 16, seed 2026110607. No test-based checkpoint selection.
• 551 training, 101 validation, 156 test tasks. All source papers, including distractors, are disjoint between splits: 392 / 79 / 117 papers. Question templates and task types recur across splits.
• Frozen rows SHA256: b3c542237dba50153d88d9358db3c05351bdfbd99fa6b7b87362b9751de5bde3.
• Original training weights SHA256: 532513c9c29daa15fe0040d31db93fb26a82ad92aa4626e6947ed915e0bb878a. Converted weights SHA256: 075107999fddb3b2a28f424beac0a329199aa7f6689ed93df3ff14dd674d0a2d.
• Split Q/K/V LoRA updates are fused by concatenating A and block-diagonalizing B; fused rank/alpha are tripled without changing scaling. No lossy compression. conversion.json contains the full mapping.
• Reproduction sources are in reproduce/. The source runners use data/evidence/ for the benchmark and runs/ for results; copy benchmark/rows.json and its manifest there before use. Tinker training requires its SDK and credentials; HF evaluation uses the converted adapter and saved baseline traces. No credentials are distributed.

Source texts are arXiv descriptive metadata and abstracts, not full papers. The source manifest records API queries and hashes. Metadata reuse follows arXiv API terms (CC0); original authors and paper URLs are retained in benchmark/paper_metadata.json. Teacher-generated task labels were filtered using Qwen3.5-397B-A17B; the student shares that model family, so correlated annotation errors remain possible. Base-model pretraining contamination is unverified. These convenience samples do not establish general literature-review competence, full-paper understanding, robustness to adversarial passages, or general citation correctness.

To recompute every reported baseline accuracy and the release comparisons from the included raw outputs, run python verify_benchmark.py . with NumPy installed. This is an offline audit, not new inference. It checks the frozen row hash, regrades all 156 cases, and recomputes the paired bootstrap comparisons.

This is a separate narrow selector. The earlier SecPaperAgent-20B full-answer study remains private and failed; these results do not promote that model or transfer to its broader task.