TrialJev (pointer-head eligibility scorer)

Research preview. Not for clinical decisions, trial enrollment or patient care. TrialJev is independent research that adopts the design of the open-weight MedJev and kev releases (Apache-2.0). It is not affiliated with or endorsed by TypeSafe AI, the developer of Jev.

This repository holds a LoRA adapter (rank 16) and a bilinear pointer head (dimension 256) that score the two answer options in one forward pass and return P(eligible), without generating text. Each sub-folder is one selected checkpoint (lowest development loss), trained on TREC Clinical Trials 2021 and evaluated on 6,975 eligible-or-excluded pairs from 50 unseen TREC 2022 patients (always predicting "eligible" scores 56.47%).

Checkpoints

Sub-folder	Configuration	Backbone (revision)	Step	Accuracy	Macro-F1	AUROC	Brier	ECE
qwen3.5-0.8b-fresh-seed0	0.8B, fresh initialization, seed 0	Qwen/Qwen3.5-0.8B-Base (revision not recorded at training time)	1000	54.27%	53.9%	0.601	0.251	0.088
qwen3.5-0.8b-fresh-seed1	0.8B, fresh initialization, seed 1	Qwen/Qwen3.5-0.8B-Base (revision not recorded at training time)	100	55.23%	47.8%	0.549	0.245	0.015
qwen3.5-0.8b-fresh-seed2	0.8B, fresh initialization, seed 2	Qwen/Qwen3.5-0.8B-Base (revision not recorded at training time)	500	53.32%	51.9%	0.596	0.266	0.128
qwen3.5-0.8b-medjev-init-seed0	0.8B, MedJev clinical initialization, seed 0	Qwen/Qwen3.5-0.8B-Base (revision not recorded at training time)	400	73.48%	73.0%	0.797	0.190	0.085
qwen3.5-0.8b-medjev-init-seed1	0.8B, MedJev clinical initialization, seed 1	Qwen/Qwen3.5-0.8B-Base (revision not recorded at training time)	500	68.22%	68.2%	0.787	0.207	0.034
qwen3.5-0.8b-medjev-init-seed2	0.8B, MedJev clinical initialization, seed 2	Qwen/Qwen3.5-0.8B-Base (revision not recorded at training time)	500	70.44%	70.1%	0.776	0.190	0.012
qwen3.5-9b-seed0	9B, seed 0	Qwen/Qwen3.5-9B-Base (revision 68c46c4b3498)	400	78.52%	78.4%	0.860	0.161	0.078
qwen3.5-9b-seed1	9B, seed 1	Qwen/Qwen3.5-9B-Base (revision 68c46c4b3498)	400	77.65%	77.3%	0.849	0.159	0.059
qwen3.5-9b-seed2	9B, seed 2	Qwen/Qwen3.5-9B-Base (revision 68c46c4b3498)	1000	78.65%	78.6%	0.875	0.153	0.051
qwen3.5-27b	Qwen3.5-27B, seed 0	Qwen/Qwen3.5-27B (revision fc05daec18b0)	400	80.97%	80.7%	0.885	0.145	0.089
qwen3.8-27b	Qwen3.8-27B, seed 0	Qwen/Qwen3.8-27B (revision 1d4bf0f2ff60)	400	81.84%	81.7%	0.888	0.145	0.098

All metrics are on the 6,975 eligible-or-excluded TREC 2022 test pairs and are the numbers reported in the manuscript for that exact checkpoint: accuracy and macro-F1 of the model's decision, AUROC from P(eligible), the binary Brier score and a 10-bin expected calibration error (lower is better for both). For the fine-tuned LLM the accuracy and macro-F1 are those of the written verdict (an unparsable answer counts as wrong) and the AUROC comes from the first-token two-way score; Brier and ECE are not reported because a written verdict carries no probability. Higher AUROC with a higher ECE, as for TrialJev against the A/B-logit control, means the scores rank pairs equally well but their scale is less calibrated: choose a cutoff on development data. (The TrialJev numbers use the canonical two-pass evaluator; 27B and Qwen3.8-27B are single seeds, and 95% intervals are about plus or minus 3 points because the 50 test patients are the independent units). The backbone weights are not included: the adapter loads on the upstream model at the listed revision. Seed-0 checkpoints of the 9B and 27B models did not record their backbone revision; the revision shown is the only snapshot in the training cache. Every sub-folder also holds training_config.json, training_metrics.json and the training log, and SHA256SUMS.txt lists the checksum of every file.

How it works

The patient note, the question with the trial text and the two natural-language answer options are serialized once. A frozen Qwen backbone with a LoRA adapter encodes the sequence; a bilinear pointer head compares the hidden state at a decision token with the hidden state at the end of each option, and a softmax over the two scores gives the probabilities. No text is generated. This follows the typed-decision design of the open-weight MedJev and kev releases.

Usage

from huggingface_hub import snapshot_download
local = snapshot_download("omareng/TrialJev", allow_patterns=["qwen3.8-27b/*"])      # one sub-folder, not the whole repository
path = f"{local}/qwen3.8-27b"

import torch
from trialmatch.checkpoint import load            # code repository: https://github.com/oibrahi1/TrialMatch_J
from trialmatch.records import materialize

tok, model = load(path, "cuda", dtype=torch.bfloat16, merge=False)     # the backbone is fetched from its own Hub repository
model.lm.config.use_cache = True
request = {"state": "<patient note>",
           "questions": {"eligible": {"type": "noul", "instructions": "<question and trial text>",
                                       "criteria": {"false": "<excluded option>", "true": "<eligible option>"},
                                       "label": True, "src": "x"}}}      # the label is ignored when scoring
enc = model.encode(tok, materialize(request), max_state=512, max_branch=4096)
with torch.no_grad():
    probs, _ = model.probs_and_prefix(enc)         # probs[0] = [P(excluded), P(eligible)]
print(float(probs[0][1]))

The exact prompt serialization used in the manuscript (question wording, option text, trial sections) is in docs/REPRODUCE.md and build_data.py of the code repository. To reproduce a reported number, score a split with python -m trialmatch.evaluate --run <path> --data <dir> --split <split> --dtype fp32.

Training

Setting	Value
Training data	TREC Clinical Trials 2021: 60 training patients, 9,302 eligible-or-excluded pairs (development: 15 patients, 2,233 pairs)
Adapter	LoRA rank 16, alpha 32, dropout 0.05 on the attention, MLP and Gated DeltaNet projections
Optimization	AdamW, learning rate 1e-4 (one-cycle, 5% warm-up), weight decay 0.01, gradient clipping 1.0, 3 epochs (1,746 steps), effective batch 16
Inputs	patient note capped at 512 tokens, state plus trial text capped at 4,096 tokens
Selection	lowest development loss (checked every 100 steps at 0.8B, every 200 at 9B and above); the test set was scored once
Precision	fp32 backbone at 0.8B, bf16 at 9B and above

Training data

TREC Clinical Trials 2021 patient cases and ClinicalTrials.gov records. The processed data are not redistributed because they derive from TREC topics and registry text; the code repository explains how to rebuild them and lists SHA-256 hashes. No real patient data were used.

Limitations and intended use

Research use only. One benchmark of 50 synthetic patients; pairs that assessors had already judged relevant (not retrieval); synthetic, short patient notes; no explanation or rationale is produced; the scores are not calibrated probabilities and a cutoff should be chosen on development data; one seed at 27B. head.pt is a PyTorch pickle holding the pointer-head tensors, the special-token embeddings and the training arguments; load it only from this repository. See the manuscript and docs/RESULTS.md in the code repository.

Citation

@misc{ibrahim2026trialjev,
  title  = {TrialJev: Fast Single-Pass Patient--Trial Eligibility Screening with Pointer-Head Language Models},
  author = {Ibrahim, Omar},
  year   = {2026},
  note   = {Preprint; arXiv identifier to be added at posting}
}

Please also cite the work this builds on: MedJev (https://github.com/JunMa11/MedJev) and kev (https://github.com/jaredpalmer/kev).

License

Apache-2.0 for the adapters and heads. The backbones are released by Alibaba Cloud under their own licenses; the TREC data belong to NIST/TREC and are not included.

Framework versions

Large models: torch 2.14.0+cu130, transformers 5.17.0, peft 0.21.0, accelerate 1.15.0, flash-linear-attention 0.5.2 (H100 NVL, bf16 backbone). 0.8B models: torch 2.14.0, transformers 5.17.0, peft 0.21.0, accelerate 1.15.0, flash-linear-attention 0.5.2 (RTX 6000 Ada, fp32 backbone)