SpyderBot Documentation

Evidence quality, coverage and consistency

Evidence quality tells you how well a result is supported by the declared measurement scope. It does not certify that every AI answer is true. Collection completeness, repeated-answer consistency, cross-system agreement and reportability describe different properties and must remain separate.

For balanced multi-prompt or multi-system headlines, SpyderBot does not present a statistical confidence interval unless a separately identified, appropriate estimation method is actually provided. A High evidence label is not “95% confidence” and is not a probability that a brand claim is correct.

On this page

Evidence Level · Metric Availability and Reportability · Collection Coverage · Executions, Attempts and Answer Artifacts · Run Completion, Success and Metric Readiness · Response Sampling Depth · Repeat Outcome Agreement · Modal Outcome Share · Cross-System Dominant Outcome Agreement · System Variation and Dispersion · Run Scope and Prompt-System Coverage · Valid Response Rate and Execution Coverage · Prompt Set Health Diagnostics · Median Response Length and Citation Count · Extraction Coverage · Entity Identity and Resolution Status

Evidence Level

Unit: High / Medium / Low, with reasons

A summary of the support available for the declared measurement result.

How it is assessed

The assessment considers applicable scope, completed planned observations, independent evidence, repeated measurement and comparability. The result exposes the scope, support counts and reasons affecting its level. Evidence level is separate from whether a particular metric is reportable.

How to interpret it

A paid plan or large attempt count does not automatically guarantee High evidence. The label is not a proprietary numeric confidence score or a statistical probability.

What to do next

Read the evidence reasons and missing scope before treating a headline as decision-ready.

Related concepts: Metric Availability and Reportability ↗ · Collection Coverage ↗ · Repeat Outcome Agreement ↗

Measurement scope and aggregation ↗ · Evidence and availability ↗

Metric Availability and Reportability

Unit: Status and reason

Whether a metric has an applicable and sufficiently supported value for its declared scope.

How it is assessed

A metric has a numeric result only when its own applicability and evidence requirements are met. Measured zero, not applicable, missing or unavailable, insufficient evidence and suppressed describe different conditions. Known inapplicable groups may be excluded from a conditional calculation; unresolved missing groups may not. The accompanying reason identifies the evidence condition that limits the result.

How to interpret it

High evidence in one area cannot override a failed requirement in another. A reduced-scope result has its own label and support, rather than replacing the original headline silently.

What to do next

Resolve the stated gap or inspect an explicitly scoped diagnostic result.

Related concepts: Collection Coverage ↗ · Baseline and History Comparability ↗

Measurement scope and aggregation ↗ · Evidence and availability ↗

Collection Coverage

Unit: Counts and percent of planned applicable observations

How much of the planned measurement scope has usable observed evidence.

How it is calculated

Completed usable observations are compared with planned applicable observations at the same measurement unit. The planned denominator stays fixed. Missing prompts, systems, surfaces or repeats are evidence gaps; collecting extra answers in another cell does not repair them. Metric-specific observability is assessed separately from operational completion.

How to interpret it

Coverage is not brand visibility. Completed negative answers are valid evidence; failed collection is not negative performance.

What to do next

Identify missing measurement cells before comparing headlines.

Measurement scope and aggregation ↗ · Evidence and availability ↗

Executions, Attempts and Answer Artifacts

Unit: Separate operational counts

The planned work completed, collection attempts made and distinct answer artifacts obtained.

How it is calculated

A logical execution is a scheduled measurement unit. An attempt is a try to collect it, so one logical execution can have several retry attempts. Answer Artifacts counts distinct accepted physical outputs. Reusing an output does not create fresh evidence. These three counts answer different operational questions.

How to interpret it

“100 attempts” does not mean 100 usable answers or 100 independent observations. Never use attempt totals as the denominator of a performance metric.

What to do next

Use these fields to understand collection progress and the actual evidence depth.

Measurement scope and aggregation ↗ · Evidence and availability ↗

Run Completion, Success and Metric Readiness

Unit: Separate operational statuses or rates

Whether planned work has reached a terminal state, succeeded in collecting evidence, and produced reportable metrics.

How it is calculated

Completion measures terminal planned work, including terminal failures. Success measures the declared successful collection outcomes. Metric Readiness checks whether the accepted evidence meets each metric’s conditions. Always show the denominator and state being reported.

How to interpret it

A run can be finished while a metric remains unavailable. Progress bars must not imply that all results are usable.

What to do next

Open the collection status or the metric-specific missing-evidence reason.

Measurement scope and aggregation ↗ · Evidence and availability ↗

Response Sampling Depth

Unit: Planned and observed repeats per exact configuration

The number of repeated observations collected for the same measurement configuration.

How it is calculated

Sampling depth counts separately collected accepted repetitions within the same prompt, system, surface, locale and declared time configuration. Reused artifacts are not fresh observations. Planned and observed depth can differ. Additional repetitions add support without increasing the strategic headline weight of that prompt or system.

How to interpret it

Depth means repetition count, not response word length. Repeats capture observed variation but do not make a probabilistic system deterministic.

What to do next

Use comparable depth when reading observed prompt reach and consistency.

Related concepts: Repeat Outcome Agreement ↗ · Observed Prompt Reach ↗

Measurement scope and aggregation ↗ · Evidence and availability ↗

Repeat Outcome Agreement

Unit: Percent of comparable repeated-answer pairs

How often repeated answers in the same configuration agree on the named categorical outcome.

How it is calculated

Compare every unordered pair of eligible repeated outcomes in the exact same cell. Divide matching pairs by all comparable pairs. A no-primary-choice outcome is a real category, not missing evidence. With fewer than two outcomes, pair agreement is unavailable. The outcome being compared must be named.

How to interpret it

Two matching repeats show observed agreement at shallow depth, not statistical certainty. Recommendation agreement and sentiment agreement are different diagnostics.

What to do next

Inspect the disagreeing answers to understand what varies.

Illustrative example

Outcomes A, A, B create three pairs: one A–A match and two mismatches, so agreement is 1/3 = 33.3%.

Measurement scope and aggregation ↗ · Evidence and availability ↗

Modal Outcome Share

Unit: Percent of repeated outcomes

The proportion of repeated outcomes belonging to the most frequent observed category.

How it is calculated

Within the same cell, divide the count of the most frequent outcome by all observed eligible outcomes. Preserve ties and the no-choice category. This denominator is individual outcomes, not pairs.

How to interpret it

Modal share differs from pair agreement and must not share its tooltip or label.

What to do next

Read it with the complete distribution.

Illustrative example

For A, A, B, modal share is 2/3 = 66.7%, while pair agreement is 33.3%.

Measurement scope and aggregation ↗ · Evidence and availability ↗

Cross-System Dominant Outcome Agreement

Unit: Percent of comparable measured systems

How many comparable systems share the most common uniquely dominant outcome.

How it is calculated

Determine each fully measured system’s unique dominant categorical outcome on the same scope. Count the systems supporting the most common unique outcome and divide by all comparable fully measured systems. A tied system stays in the denominator but casts no unique vote. One measured system is not enough for cross-system agreement.

How to interpret it

Agreement does not prove correctness. Systems can agree while varying in strength or citations.

What to do next

Inspect per-system distributions and ties before summarising a consensus.

Measurement scope and aggregation ↗ · Evidence and availability ↗

System Variation and Dispersion

Unit: Percentage points

How much the same metric varies across comparable AI systems.

How it is calculated

Range is the highest minus lowest system rate. Deviation is a system’s difference from the named baseline. When dispersion is shown, it is the weighted population standard deviation of system rates under the declared profile. Each uses the same entity, task and measurement scope.

How to interpret it

Dispersion is not standard error, a confidence interval or the probability of a future answer. Do not mix it with categorical outcome agreement.

What to do next

Focus investigation on the systems that differ most, then inspect evidence.

Measurement scope and aggregation ↗ · Evidence and availability ↗

Run Scope and Prompt-System Coverage

Unit: Counts, range and matrix-cell percent

The actual questions, system configurations and measurement cells represented by a run.

How it is calculated

Prompt Set counts distinct confirmed Core Prompts; wording variants are counted separately. AI Systems counts distinct actual independent configurations, with selected and observed totals separate. Prompts per System is a min–max range when coverage differs. Prompt-System Coverage is the number of planned base cells with at least one complete protocol-matched answer divided by planned base cells. Agents Used counts actual distinct runner identities only when that identity is available; execution counts must not be relabelled as agents.

How to interpret it

Operational cell coverage does not mean every metric has complete extraction or meets reportability requirements. Variants, retries and proxy names do not manufacture additional strategic units.

What to do next

Inspect the missing cells and selected-versus-observed scope before interpreting a headline.

Related concepts: Collection Coverage ↗ · Response Sampling Depth ↗

Measurement scope and aggregation ↗ · Evidence and availability ↗

Valid Response Rate and Execution Coverage

Unit: Separate operational counts and rates

Delivered answer integrity and completion of the planned logical sample manifest.

How it is calculated

Valid Responses counts integrity-complete, protocol-matched delivered answers; a complete refusal can qualify. Valid Response Rate divides these complete delivered answers by terminal attempted logical samples, including successful and failed logical work; retries remain one logical sample. Pending work makes the diagnostic provisional. Execution Coverage and Executions Completed / Planned use complete matched delivered logical samples divided by all planned logical samples. Pending and cancellation states remain visible.

How to interpret it

A valid delivered answer can still have an unresolved metric extraction. Terminal run completion is a different measure and can include failures. Neither rate is a brand outcome.

What to do next

Check delivery, extraction and metric readiness separately.

Related concepts: Run Completion, Success and Metric Readiness ↗ · Executions, Attempts and Answer Artifacts ↗

Measurement scope and aggregation ↗ · Evidence and availability ↗

Prompt Set Health Diagnostics

Unit: Counts, proportions and states

Operational and planning context for the confirmed prompt panel.

How it is calculated

Count confirmed Core Prompts distinctly. Stable/adaptive selection roles come from frozen selection metadata; direct input is a separate category when present. Intent composition uses primary intent; question mode and funnel stage are different dimensions. System Participation is systems with at least one complete matched response divided by selected planned systems. Failed-Chart Count includes distinct required results with failed computation or collection, not N/A, paywall or low-evidence states. Snapshot Freshness shows the pinned as-of time, age and revision; Quality Gate shows publication validation state and reasons.

How to interpret it

These diagnostics do not form a composite brand-health or confidence score. Rejected planning candidates are not missing observations and do not enter metric denominators.

What to do next

Use the named diagnostic to resolve the relevant scope or collection issue.

Related concepts: Valid Response Rate and Execution Coverage ↗ · Median Response Length and Citation Count ↗ · Run Scope and Prompt-System Coverage ↗

Measurement scope and aggregation ↗ · Evidence and availability ↗

Median Response Length and Citation Count

Unit: Median words or validated citations per complete response

The typical text length and citation count of complete observed answers.

How it is calculated

Compute the median word count and, separately, the median validated citation count among complete responses using the declared counting method. A legacy Median Response Depth label refers to these text/content diagnostics; it must not be used for repetition depth.

How to interpret it

Longer answers and more citations are not automatically more reliable. This is different from Response Sampling Depth, which counts repeated observations.

What to do next

Use these diagnostics to understand answer format, not as a confidence score.

Related concepts: Response Sampling Depth ↗

Measurement scope and aggregation ↗ · Evidence and availability ↗

Extraction Coverage

Unit: Percent at the declared extraction unit

How much submitted or delivered evidence has completed the named extraction step.

How it is calculated

Operational Extraction Completion divides completed extraction work by submitted extraction work at the same unit. Metric-specific extraction coverage instead measures whether the assertions needed by that metric are resolved on its eligible observations. Completing one stage does not imply complete extraction for every metric.

How to interpret it

Completing one extraction stage does not prove that every metric is observable. No positive assertion row is not enough to establish an observed negative.

What to do next

Read the metric-specific missing or unresolved evidence reason.

Measurement scope and aggregation ↗ · Evidence and availability ↗

Entity Identity and Resolution Status

Unit: Identity mapping and evidence status

Which organisation, product, person or domain an observed reference has been associated with for the selected result.

How it is calculated

References are interpreted in context and linked to the declared entity mapping for the result. Ambiguous or unresolved identities remain unresolved; a similar name alone does not establish that two references describe the same entity. Company, product, retailer, founder and current leader are distinct identities or roles. Corrections belong to a new result revision rather than silently changing a historical publication.

How to interpret it

This status explains attribution of observed evidence. It is not the GEO Audit assessment of how clearly a website describes an entity, and it does not disclose the internal recognition algorithm.

What to do next

Inspect the original permitted evidence and entity context when a mention appears to refer to the wrong subject.

Measurement scope and aggregation ↗ · Evidence and availability ↗