arxiv:2610.21046

PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviews

Published on Oct 15

· Submitted by

Dingjie Song on Oct 19

Upvote

307

Authors:

Dingjie Song ,

Abstract

PremiseCheck improves peer review verification by decomposing comments into explicit premises checked by decision models, achieving high accuracy with lower cost and latency.

Generated by thinkingmachines/Inkling-Small

Peer review is under strain from the growth of submissions, and review comments written by humans or AI can misstate what a paper says. We introduce ReviewFact, a benchmark that labels such factual errors in review comments under one definition across NLP, machine learning, and the natural sciences, and that measures cost and latency alongside accuracy. A frontier LLM checks comments well but must read the whole paper for each check, while cheap decision models, which score whether a statement holds in a text, fall far short on whole comments that mix several implicit assertions. PremiseCheck is a framework that combines the two. An LLM rewrites each comment as explicit premises without reading the paper, and a decision model checks each premise against the paper. When each comment is checked on its own, PremiseCheck reaches accuracy comparable to GPT-5.5 across all three fields at about half its cost, and on NLP reviews at a quarter of its cost and less than half its latency.

View arXiv page View PDF Project page GitHub 4 Add to collection

Community

songdj

Paper author Paper submitter 2 days ago

TL;DR: Review comments, written by humans or AI, sometimes misstate what the paper says. PremiseCheck flags them cheaply: an LLM rewrites each comment as explicit premises without reading the paper, and a small decision model checks each premise against the paper. The weakest premise sets the score, and each flag names the premise the paper contradicts.

ReviewFact: 9,267 comments on 217 papers from NLP, machine learning and the natural sciences, labeled for factual errors under one definition (from the expert annotations of ReviewCritique, ReviewScore and PeerReview Bench), with cost and latency measured alongside accuracy.
Per comment: AUROC comparable to holistic GPT-5.5 in all three fields at about half the cost. On NLP reviews with a no-reasoning decomposer: 63.1 vs. 63.2 AUROC at ~25% of the cost and 45% of the latency.
Per review: fusing batched PremiseCheck and GPT-5.5 scores gives 71.5 AUROC at 28% of the reference cost.
Limitations: the task is hard (AUROC ≈ 63 on NLP for both systems), the ML and science subsets are small with wide intervals, and judgment errors are out of scope. It is meant to point people to comments worth checking, not to replace reviewers.

Code: https://github.com/OpenLAIR/PremiseCheck
Data: https://huggingface.co/datasets/songdj/ReviewFact
Project page: https://openlair.github.io/premisecheck/

Questions and counterexamples are welcome here.

librarian-bot

about 23 hours ago

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend

researchstudio-bot

about 22 hours ago

This is an automated message from the ResearchStudio team.

We created an interactive ResearchStudio Reel for this paper. It includes a visual poster, a video, and a blog, all available for download in editable formats.