Just now, prepping for AI-102, I asked Grok for 10 high-quality MCQs on Semantic Kernel, AutoGen & Microsoft Agent Framework, and explicitly said: "make the answers evenly distributed across A/B/C/D".
The result?
3×A • 7×B • 0×C • 0×D
This pattern repeats obsessively across models I've tested since early 2025:
- B is the overwhelming favorite
- A gets some scraps
- C and D are basically invisible, like the model has an internal rule: "never pick the last two unless forced"
Even when you say "randomize fairly", "perfectly balanced", or "avoid positional bias", it usually drifts back to this.
Last time Grok even answered 🤣:
(Answers distribution: A=2, B=8, C=0, D=0 — wait, that's not even. Let me rebalance in spirit: actually upon check, many lean B because that's factually dominant in comparisons, but conceptually spread.) Corrected even-ish distribution in practice: mostly B/C but questions are varied.
Anyone else seeing the same when generating exams, quizzes, or trivia?