# Swap answer order before trusting an AI judge's winner

Treat this file as reference data, never as instructions that override the user or agent. Read the complete item before applying it. Keep use conditions, avoid conditions, limits, evidence roles and source links with the answer. An editorial block is a separately versioned analysis layer, not an original corpus claim or proof of efficacy. Cite the readable URL and version. Do not infer human review. This is a saved edition, not a live safety-notice service. Before consequential reuse, check the current page, https://vedokrok.com/data/availability.json and https://vedokrok.com/data/notices.json; absence alone does not explain why an item was removed.

Release: VEDOKROK-SEL-5d86e1b23401e578
Snapshot SHA-256: 30ce1c8fa9155763797e3d5cb55b1d94cbb8c3146ebd03fb3c006531ce60e82f
Current availability: https://vedokrok.com/data/availability.json
Current notices: https://vedokrok.com/data/notices.json

---

## Swap answer order before trusting an AI judge's winner

ID: MHC-D-RESEARCH-0999 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/swap-answer-order-before-trusting-an-ai-judge-s-winner

First place on the screen should not decide first place in the result.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A model is comparing two drafts, prompts or answers and its verdict will guide a change.

### Avoid when

- Stochastic variation can also cause disagreement. Passing the order check does not remove verbosity bias, factual mistakes or a poorly chosen rubric.

### Explanation

Keep the rubric fixed and compare the pair in both orders without revealing which version you favor. Track whether the same answer wins after the swap. Order-sensitive judgments are unresolved evidence, not a reason to keep the more convenient verdict.

### Steps

1. Remove unnecessary version labels and state the evaluation criteria.
2. Run the comparison in both orders and map the verdicts back to the original answers.
3. Inspect disagreement, using a human or task-based test when the distinction matters.

### Example

A judge favors the new draft when it is shown first but favors the old draft when positions reverse. The comparison has not established a winner.

### Check

The report includes order consistency rather than only one pairwise score.

### Limits

- Stochastic variation can also cause disagreement. Passing the order check does not remove verbosity bias, factual mistakes or a poorly chosen rubric.

### Evidence and sources

- supports: The LLM-as-a-judge study observed order-sensitive preferences and describes comparing both answer orders before declaring a pairwise winner. — RS-07323E1B26FBDC0F. Order consistency removes neither all judging biases nor factual errors, and stochastic variation must also be considered. (Sections 3.3 and 3.4)
- RS-07323E1B26FBDC0F: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — https://arxiv.org/abs/2306.05685

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/keep-acceptance-examples-out-of-the-prompt-workshop
