Anthropic’s Claude Fable 5.2 correctly identified the author of 56% of 398 unsigned peer reviews and meta-reviews in a small-scale test, researchers reported in a November 19 preprint. In the study, six volunteers wrote reports presented to the model one at a time, without web access, tools, a list of possible authors, or other writing samples. Atrey Desai and colleagues analyzed reviews written between 2015 and 2026 for 26 double-blind conferences and workshops in machine learning and natural language processing. Although the researchers knew each report’s author, the reviewers’ names had never been publicly linked to the texts; in 19 cases, the review text was already available without attribution.

When evaluated individually, Claude named the writer on 68.4% of 266 standard reviews and 31.1% of 132 meta-reviews, which summarize submissions. Meta-reviews were shorter and originated from just two of the six contributors. The study did not isolate the effects of review length, type, or authorship. The reported 56% accuracy weights each report equally, meaning contributors who submitted more reviews carried greater influence. When reviewers were given equal weight, Claude’s average accuracy dropped to 49.1%. The findings demonstrate that unsigned reports can inadvertently reveal their authors, though estimating the frequency of this occurrence in confidential peer review would require a larger, more diverse sample.

The researchers also tested the model on 1,559 public reviews from 47 open-review journals. Claude correctly identified the reviewer in 21.4% of 1,530 cases it answered. A subset of 789 reviews, which an AI-text detector (Pangram 4.1) had labeled as human or mixed, yielded 39.3% accuracy—a rate the paper’s abstract rounds to 39%. Accuracy was significantly higher among reviews where authors cited their own work: Claude achieved 80.2% accuracy on 106 self-cited reviews, compared to 17.1% on the 1,453 reviews without self-citations. Even in this high-performing group, however, an presented name from Claude would require independent evidence to confirm the true author.

The team noted that the model sometimes made errors and plans to expand the analysis with a substantially larger set of anonymous peer reviews in a future update to the preprint.

Source link

Exit mobile version