Protocol
Swap answer order before trusting an AI judge's winner
First place on the screen should not decide first place in the result.
Before you use it
- Use when
- A model is comparing two drafts, prompts or answers and its verdict will guide a change.
Keep this limit in mind
- Stochastic variation can also cause disagreement. Passing the order check does not remove verbosity bias, factual mistakes or a poorly chosen rubric.
An idea to explore.
Try this idea.
Inspect the example, try the idea in your situation, then check your next move.
Why it matters
Keep the rubric fixed and compare the pair in both orders without revealing which version you favor. Track whether the same answer wins after the swap. Order-sensitive judgments are unresolved evidence, not a reason to keep the more convenient verdict.
Steps
- Remove unnecessary version labels and state the evaluation criteria.
- Run the comparison in both orders and map the verdicts back to the original answers.
- Inspect disagreement, using a human or task-based test when the distinction matters.
An example
A judge favors the new draft when it is shown first but favors the old draft when positions reverse. The comparison has not established a winner.
Check your result
The report includes order consistency rather than only one pairwise score.
Save and practise this idea
My learning
Read. Recall. Repeat.
Save this idea and return to it on any device.
Keep exploring
Connected ideas
You are readingSwap answer order before trusting an AI judge's winner
Evidence and sources
The LLM-as-a-judge study observed order-sensitive preferences and describes comparing both answer orders before declaring a pairwise winner.
Order consistency removes neither all judging biases nor factual errors, and stochastic variation must also be considered.
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena · Sections 3.3 and 3.4
All sources (1)
Edition and review
Version 0.1.0
No review details are supplied for this edition.