PerceptionRubrics
Calibrating Multimodal Evaluation to Human Perception

Current perception benchmarks suffer from an evaluation paradox: leaderboards are increasingly saturated, yet models remain perceptually brittle in real-world use. We introduce PerceptionRubrics, a rubric-based evaluation framework that shifts assessment from holistic semantic matching to rigorous atomic auditing. It pairs 1,038 information-dense images with over 12,000 instance-specific rubrics, derived from Golden Captions constructed through a novel Circular Peer-Review consensus pipeline and distilled into a dual-stream system of Must-Right (essential facts) and Easy-Wrong (fine-grained details) rubrics.

Crucially, PerceptionRubrics implements a Gated Scoring mechanism: unlike linear averages, failure on mandatory visual facts triggers sharp binary penalties. Extensive evaluation yields three insights:

(1) The Reliability Gap. Models often verify fragmented elements correctly yet fail strict conjunctive constraints, exposing brittleness in dense domains such as GUIs.
(2) Open–Closed Stratification. Contrary to reasoning trends, a persistent ~8% perception deficit separates the open-source frontier from proprietary leaders.
(3) Human-Aligned Rigor. Our gated metrics substantially out-align conventional benchmarks with human preference, validating that strict perceptual fidelity is a prerequisite for reliable generation.

We use an LLM-as-a-Judge to verify each rubric as a boolean (True/False), then apply a non-linear, gated aggregation that mirrors human error sensitivity:

• Must-Right as the Gate. If a response fails any Must-Right criterion, the gate closes (G = 0) and the score collapses to zero—a single fatal hallucination is a binary failure, not a minor fluctuation.
• Easy-Wrong for Differentiation. For responses that pass the gate (G = 1), the final score is the pass rate over the Easy-Wrong rubrics, rewarding robustness against subtle, density-rich cognitive traps.

This ensures a high score reflects not coarse semantic proximity but genuine perceptual reliability, distinguishing acceptable approximations from catastrophic failures.

The Reliability Gap & Failure Modes

Comparing Atomic Accuracy (mean pass rate over individual rubrics) with the stricter Must-Right Pass Rate (all constraints satisfied) exposes a systematic Reliability Gap: models pass most atomic checks yet fail their strict conjunction. The gap narrows as capability increases. The GUI domain dominates perceptual failures across models.

Consistency of Perceptual Capabilities

We find a near-perfect linear correlation (R² ≈ 0.98) between Must-Right Pass Rate and Easy-Wrong accuracy: models that fail to ground essential facts inevitably struggle with subtle details and hallucination. Robust fine-grained understanding critically depends on foundational perception.

Alignment with Human Preference

Against the Vision Arena human-preference leaderboard, PerceptionRubrics shows the strongest alignment among compared captioning benchmarks, achieving a Pearson 0.916 and a perfect Spearman 1.000 rank correlation. In contrast, DOCCI and DetailCaps assign nearly indistinguishable (or even anti-correlated) scores to models with markedly different human ratings.

Resistance to Length Bias

Caption length exhibits a negligible relationship with PerceptionRubrics scores, indicating the metric rewards precise, verifiable perception rather than verbosity.

Evaluation Robustness & Rubric Coverage

Swapping the judge model preserves the ranking order, and evaluation stability improves monotonically as rubric coverage increases—confirming the robustness of both the rubric-generation pipeline and the resulting metric to judge choice and sampling variability.