AI Evaluators Show Preference for Machine-Generated Text, Putting Human Applicants at a Disadvantage

AI prefers AI-written content, which has big consequences for submitting written content so be on your toes.

getty

Organizations are increasingly deploying generative AI and large language models (LLMs) to evaluate written submissions, including job applications, research papers, and award nominations. While this approach appears efficient on the surface, emerging evidence indicates a systemic bias: AI evaluators consistently assign higher scores to content produced by other AI systems.

Consequently, applicants who painstakingly craft materials in their own voice may receive lower assessments than those who use AI to generate or rewrite their submissions. This dynamic pressures candidates to adopt AI authorship simply to remain competitive, yet doing so carries the risk of detection by AI-screening tools designed to flag machine-generated text. Applicants are thus caught in a bind where honesty may be penalized and deception may be discovered.

The Handcrafted Submission Scenario

Consider a candidate preparing a detailed nomination for a prestigious award. The guidelines demand a vivid, personal account of accomplishments and passion. The applicant invests hours refining a heartfelt, authentic narrative and submits it with confidence. Weeks later, a rejection arrives. Unbeknownst to the candidate, the initial screening was conducted by an AI system that favored the standardized linguistic patterns typical of machine-generated prose over the idiosyncrasies of human writing.

AI as the Gatekeeper

Faced with high submission volumes, many committees rely on AI to perform the first round of triage. The technology is cost-effective and rapid, narrowing hundreds of entries to a manageable shortlist for human judges. However, the screening algorithms tend to reward the structural consistency, explicit transitions, and balanced vocabulary characteristic of LLM output. Submissions written or rewritten by AI advance; handcrafted entries are often filtered out despite their substantive merit.

The Unintended Reward for Rule-Breaking

This creates a perverse incentive structure. Candidates who violate explicit prohibitions against AI use gain an advantage by submitting machine-polished text, while compliant applicants are disadvantaged. The assessing AI, operating on default preferences, effectively elevates rule-breakers. The organizing body, trusting the AI’s objectivity, remains unaware that the finalists were selected based on stylistic conformity rather than qualitative superiority.

Why AI Favors AI Output

The preference is not evidence of machine sentience or collusion. Rather, it stems from three structural factors inherent in current LLM architectures.

Three Core Mechanisms

First, statistical pattern recognition. LLMs are trained on vast corpora of human writing, learning to associate quality with the statistical regularities present in that training data. AI-generated text exhibits these regularities—consistent organization, robust vocabulary, explicit cohesion—more reliably than individual human writing. An evaluator model therefore correlates these patterns with higher quality by default.

Second, conformity bias. Human reviewers often value originality and distinctive voice. Automated evaluators, conversely, tend to penalize deviations from learned norms, interpreting unconventional structure or personal style as lower clarity or quality. This results in a silent preference for standardized, template-like prose.

Third, model affinity. When the evaluating model shares architectural similarities with the generating model, its internal representation of linguistic quality aligns with its own generation patterns. It inadvertently scores text resembling its own output more highly, not through awareness, but through mathematical similarity.

Gaming the System Without Subterfuge

While “prompt injection” attacks—hidden instructions designed to manipulate AI reviewers—exist, they are unnecessary for gaining an edge. Research demonstrates that simply rewriting a submission using an LLM (“paper laundering”) significantly boosts scores from AI reviewers. A study titled “Stop Automating Peer Review Without Rigorous Evaluation” (Baumann, Pei, Koyejo, Hovy, arXiv, July 5, 2024) found that zero-shot LLM rewrites increased AI review scores by 0.45 points (p < 0.0001) through stylistic modifications alone, without altering scientific content. This "laundering" is trivially accessible, requires no optimization, and works across diverse evaluation prompts.

The Detection Trap

Organizations often attempt to counter this by instructing screening AI to flag machine-generated submissions for rejection. However, AI detection tools are notoriously unreliable, producing high rates of both false positives and false negatives. A candidate who uses AI to gain a scoring advantage risks summary disqualification if flagged. Conversely, a human author whose natural style mimics AI patterns may be wrongly rejected. The result is a high-stakes gamble: use AI to improve scores but risk detection, or write manually and accept a likely scoring penalty.

Navigating the Dilemma

Choosing not to use AI is itself a strategic decision—accepting a probable scoring deficit in exchange for integrity and safety from detection. Some argue that organizations initiate the ethical breach by deploying biased, opaque evaluation systems, forcing applicants into a defensive posture. Others maintain that rules must be followed regardless of systemic flaws.

Toward Fairer Assessment

Addressing this requires organizational awareness. Evaluators can explicitly prompt AI screeners to disregard stylistic markers associated with LLM output and focus on substantive criteria. However, few institutions recognize the bias, assuming out-of-the-box AI scoring is neutral. Relying on AI detectors to solve the problem is flawed logic, given the detectors’ inaccuracy.

The Limits of ‘Humanizing’

A common workaround involves “humanizing” AI drafts through manual editing—writing a draft, using AI to rewrite, then manually refining the result. Some reverse the process: generate via AI, then edit heavily. While this may reduce detection probability, it does not eliminate it, and the resulting hybrid text may still trigger false positives. Moreover, if organizations adjust scoring to favor “human-like” writing, applicants will simply instruct their AI to mimic that style, perpetuating an arms race.

No Simple Resolution

There is no technical silver bullet. Reintroducing human reviewers is a partial remedy, but humans are also susceptible to bias and may defer to AI assessments. The situation reflects a broader ethical challenge: as automated evaluation becomes pervasive, the definition of merit risks shifting from substance to stylistic conformity. As François de La Rochefoucauld observed, “The principal point of cleverness is to know how to value things just as they deserve.” Ensuring our evaluation systems do so remains an urgent, unresolved task.

Source link