Protocol

Swap answer order before trusting an AI judge's winner

First place on the screen should not decide first place in the result.

About 1 min read

Before you use it

Use when
  • A model is comparing two drafts, prompts or answers and its verdict will guide a change.

Keep this limit in mind

  • Stochastic variation can also cause disagreement. Passing the order check does not remove verbosity bias, factual mistakes or a poorly chosen rubric.

An idea to explore.

Try this idea.

Inspect the example, try the idea in your situation, then check your next move.

Keep reading

Why it matters

Keep the rubric fixed and compare the pair in both orders without revealing which version you favor. Track whether the same answer wins after the swap. Order-sensitive judgments are unresolved evidence, not a reason to keep the more convenient verdict.

Steps

  1. Remove unnecessary version labels and state the evaluation criteria.
  2. Run the comparison in both orders and map the verdicts back to the original answers.
  3. Inspect disagreement, using a human or task-based test when the distinction matters.

An example

A judge favors the new draft when it is shown first but favors the old draft when positions reverse. The comparison has not established a winner.

Check your result

The report includes order consistency rather than only one pairwise score.

Save and practise this idea

My learning

Read. Recall. Repeat.

Save this idea and return to it on any device.

Use email
Keep exploring

Connected ideas

You are readingSwap answer order before trusting an AI judge's winner

Evidence and sources
Supports

The LLM-as-a-judge study observed order-sensitive preferences and describes comparing both answer orders before declaring a pairwise winner.

Order consistency removes neither all judging biases nor factual errors, and stochastic variation must also be considered.

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena · Sections 3.3 and 3.4

All sources (1)
Edition and review

Version 0.1.0

No review details are supplied for this edition.

Read this idea as JSON · Release details