When Expert Disagreement Hurts: Prestige and Selectivity in LLM Decision Revision
Abstract
A controlled two-pass audit varying only the revision context reveals that language models often fail to selectively correct errors under disagreement, with expert labels increasing reversals but not reliably improving accuracy, so revision should be assessed using paired conditional metrics per model and protocol.
When language models revise a decision after another agent disagrees, neither reversal counts nor final accuracy shows whether they are fixing mistakes or abandoning correct answers. We introduce a controlled two-pass audit that holds the question, the evidence, and the model's initial answer fixed and varies only the revision context: neutral reflection, anonymous disagreement, or disagreement attributed to a domain expert. For binary decisions the accounting is exact: revision raises accuracy only if it is selective enough, reversing wrong answers more than a/(1-a) times as often as correct ones, where a is initial accuracy (nine times at 90%). Across five models and 1,989 evidence-grounded decisions in medicine, science, and law, no model clears this bar under expert disagreement, and accuracy falls by up to 16.1 points. The expert label, a prestige cue, adds reversals for four of the five models, yet its effect on accuracy ranges from a 5.2-point loss to a 2.4-point gain, so reversal counts are a poor proxy for harm. With identical arguments from both sources, the label still raises reversal for three proprietary models and in a pooled analysis of four newer models. Models also respond to stated source reliability. Revision pipelines should therefore be evaluated with paired, conditional metrics, per model and protocol.
Community
An LLM answers a question correctly. Then another assistant, introduced as a domain expert, says the opposite and gives no reason. Across five models and 1,989 decisions in medicine, science and law, this lowers accuracy for every model, by up to 16.1 points.
For yes/no decisions the math is exact: revising helps only if a model flips wrong answers more than a/(1-a) times as often as right ones (9x at 90% accuracy). Under expert disagreement, none of the 5 main models clears that bar. Of 4 newer ones, only Claude-Opus-4.8 does.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Behind Aggregate Scores: What Systematically Varying Arena-Hard's LLM Judge Reveals (2026)
- When Should LLMs Trust Their Own Revisions? A Risk-Aware Study of Intrinsic Self-Correction (2026)
- Who Convinced Whom? Tracing How LLMs Change Their Minds in Multi-Agent Systems (2026)
- Same Scores, Different Decisions: Evaluating JEV and Language Models for Legal Document Understanding (2026)
- Authority Bias in Language Models: Source Deference and User Agreement Are Not Interchangeable (2026)
- Correction or Manipulation? Measuring Epistemic Vigilance in LLM Agents Under User Pressure (2026)
- JEV-as-a-Judge: Accept When Confident, Escalate When Unsure (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Models citing this paper 0
No model linking this paper
Datasets citing this paper 1
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper