Reliability and Methodology

Reliability describes whether the available evidence is fit for interpreting a particular metric or conclusion. It is not a guarantee that an AI response is objectively correct.

What reliability may consider

Eligible sample size
Prompt-system coverage
Response depth
AI-system coverage
Prompt-family coverage
Invalid-response rate
Entity-resolution quality
Extraction quality
Metric reconciliation
Historical compatibility
Verification state for sensitive claims

Public states include High, Medium, Low, and Suppressed. The UI displays reasons for a reliability state, not only a color or badge.

Run Quality versus Metric Reliability

Run Quality describes the fitness of the overall report run.
Metric Reliability describes the fitness of one metric or conclusion.

A report may pass Run Quality while a specific metric remains Low or Suppressed because its eligible population is limited.

A validated or published report is an immutable snapshot. Corrections create a visible revision or superseding report rather than silently rewriting the original result.

Consensus

Consensus describes how consistently eligible responses support a defined observation under the displayed scope. It must identify what is being compared, the eligible population, supporting observations, and included AI systems. Consensus is not objective truth.

Methodology versions

Each report should preserve Prompt Set, Prompt Universe, AI-system policy, competitor snapshot, metric-definition, extraction, validation, and aggregation versions. These support reproducibility without exposing proprietary weights.