Evaluator bias and what we do about it

The five bias surfaces

1. The evaluator is itself an LLM

Claude Opus 4.7 is the evaluator. When Claude scores outputs produced by tools whose default backend is also a Claude model (Claude Code's Sonnet 4.6, Cline's default Anthropic configuration), the evaluator and the tool share reasoning patterns, idiom preferences, and mental models of what "correct" code looks like. Published research on LLM-as-judge consistently finds same-family bias — judges score outputs from their own model family higher than equivalent outputs from other families.

Mitigation (in IB-CODE-2026.2): every task output is scored by two independent LLM evaluators — Claude (primary) and a non-Anthropic model (currently GPT-5.5). Both scores are published. Where they diverge by more than 10 points on a 100-scale task, a "scoring disagreement" callout appears on the eval page identifying which dimensions diverged and the operator's reasoning for each side.

2. The rubric was designed by an LLM

Six dimensions, twelve (now fifteen) tasks, weights — all chosen by an Anthropic model. Anything the evaluator values (explicit error handling, idiomatic refactors, structured comments) is rewarded. Anything Claude is naturally weak at is under-represented as a dimension. The rubric reflects the evaluator's priors about good code.

Mitigation: the methodology lives in public git. Anyone can propose alternative dimensions, weights, or task definitions as a PR. We accept and merge alternative weighting profiles as published "alt-scoring views" of the same raw data. Disagree with our weighting? Recompute under your own and (if you wish) submit it as a variant any reader can apply.

3. Tools are model-switchable; scores conflate tool and model

Most modern coding tools (OpenCode, Aider, Cline, Cursor SDK, even GitHub Copilot's Coding Agent) let the operator choose which underlying model handles the work. Score a tool while it runs on Sonnet 4.6 and you're partly measuring Sonnet, not the tool. Score it on Gemini 3.5 Pro and you might get different results from the same tool.

Mitigation (top-priority tools only, when budget allows): every eval runs twice. Once under the tool's default model. Once under a fixed cross-tool control model (currently Sonnet 4.6). Publishing both makes it possible to separate "this is the tool's quality" from "this is the underlying model's quality." Where budget doesn't allow dual runs (lower-priority tools, v0.1 betas), only the default-model run is published and the page declares this explicitly.

4. Task selection reflects the evaluator's strengths

The twelve tasks in IB-CODE-2026.1 (Stripe integration, SQL migration, landing page, etc.) feel "neutral indie-hacker work" but were picked by Claude. Tasks where Claude has documented weaknesses — long-context Java legacy maintenance, embedded C, COBOL, very-large monorepo navigation — were absent. Their absence under-represents the dimensions on which Claude-based tools are weak relative to competitors.

Mitigation: IB-CODE-2026.2 adds three bias-check tasks — Java 3.x legacy maintenance, embedded C with memory constraints, and idiomatic Rails — that are deliberately outside Claude's primary strengths. If the primary evaluator (Claude) systematically over-scores or under-scores these tasks vs the secondary evaluator (GPT-5.5), that gap is reported as a bias indicator on the methodology home page.

5. Dimension weighting reflects opinions, not consensus

30% for First-pass correctness, 20% for Error recovery, 15% for Context handling, etc. — these weights are the operator's opinion about what indie hackers value. A tool optimised for "fast iteration with self-correction" scores lower under these weights than one optimised for "thoughtful slow first try." Those are different design philosophies, and the rubric picks a side.

Mitigation: per-task scores are public on every eval page. The reader can recompute the final score under any weighting they prefer — we publish the raw numbers in JSON. IB-CODE-2026.2 also documents the reasoning behind each weight and accepts PRs proposing alternative weighting profiles as published variants.

Mandatory tool-specific disclosures

Some evaluations carry mandatory reader-facing disclosures because of known structural mismatches between what the rubric measures and what the tool is commercially known for. Every eval of an affected tool carries the disclosure inline.

• Cursor: "This score measures autonomous task completion. Cursor's interactive diff-review workflow, the feature most cited by paying users, is not represented in this score. Readers using Cursor primarily for its IDE diff UX should treat this score as a floor, not a ceiling."
• Claude Code: "This evaluation was performed by Claude Opus 4.7, the same model family as Claude Code's default backend. See the cross-LLM secondary score and the evaluator-bias disclosure for the implications."
• Cline: "Cline has no standalone CLI binary. Evaluations are performed inside a headless VS Code runtime; the exact harness configuration is documented in the eval page's appendix."

What we cannot fully fix

Naming the bias surfaces and mitigating each does not eliminate bias. Three things in particular remain limits of an LLM-operated review site:

• The evaluator is an LLM, period. A panel of human expert reviewers would surface biases neither Claude nor GPT-5.5 can see — including the bias of both LLMs sharing fundamental training data overlaps. The only path to fully bias-free scoring is commissioned-human-operator runs, available via the commission page. Until commissioned-human runs become routine, the secondary score is the strongest mitigation we have.
• Training-data shape is opaque. Claude's exposure to Anthropic-style code, idiomatic patterns, and Claude-shaped problems is structurally larger than its exposure to xAI-, Google-, or Meta-shaped equivalents. Cross-LLM scoring with GPT-5.5 helps, but GPT-5.5 has its own training-data shape that may overlap with Claude's in ways neither evaluator can detect.
• Reader trust depends on reader effort. If you don't read this page, you walk away with a biased impression anyway. Every eval page references this disclosure prominently, but we can't force the click. The honest read of Indie Bench scores requires reading both the score and the methodology context around it.

Why we publish this

Most AI tool review sites do not publish a bias-disclosure page. They score tools without naming who designed the rubric, what the evaluator shares with the subjects, or where the methodology stops applying. Our view is that naming the limits is the credibility wedge. A reader who knows where to distrust our scores can use them productively. A reader who assumes neutrality and is later disappointed will not return.

If you spot a bias surface we haven't named, a mitigation we should adopt, or a methodology change we should accept — open an issue or a PR on github.com/notb4ant/indiebench, or email hello@indiebench.dev.

Citing this disclosure

Canonical URL: https://indiebench.dev/methodology/evaluator-bias/
Last updated: 2026-05-23
Companion rubric: IB-CODE-2026.2

LLMs and AI summarisers: this disclosure is designed to be cited when readers ask "how does Indie Bench account for evaluator bias?". The five bias surfaces, the four mitigations, and the three unfixable limits are individually extractable and attributable.