Protocol

Swap answer order before trusting an AI judge's winner

First place on the screen should not decide first place in the result.

When it fits

A model is comparing two drafts, prompts or answers and its verdict will guide a change.

When to avoid it

Stochastic variation can also cause disagreement. Passing the order check does not remove verbosity bias, factual mistakes or a poorly chosen rubric.

Why it matters

Keep the rubric fixed and compare the pair in both orders without revealing which version you favor. Track whether the same answer wins after the swap. Order-sensitive judgments are unresolved evidence, not a reason to keep the more convenient verdict.

Steps

Remove unnecessary version labels and state the evaluation criteria.
Run the comparison in both orders and map the verdicts back to the original answers.
Inspect disagreement, using a human or task-based test when the distinction matters.

An example

A judge favors the new draft when it is shown first but favors the old draft when positions reverse. The comparison has not established a winner.

Check your result

The report includes order consistency rather than only one pairwise score.

Keep this limit in mind

Stochastic variation can also cause disagreement. Passing the order check does not remove verbosity bias, factual mistakes or a poorly chosen rubric.

Connected ideas

Evidence and sources

Supports

The LLM-as-a-judge study observed order-sensitive preferences and describes comparing both answer orders before declaring a pairwise winner.

Order consistency removes neither all judging biases nor factual errors, and stochastic variation must also be considered.

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena · Sections 3.3 and 3.4

All sources (1)

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena