Search Sign in

SpyderBot Documentation

Understanding Report Scope

Every Report View displays its analytical scope above the tabs. Scope is the boundary for interpreting every metric and finding.

Scope strip

The scope may include Active LLM Agents versus planned LLM Agents, Prompt Set size, Stable and Adaptive prompts, AI systems, completed versus planned executions, valid responses, prompt-system coverage, Run Quality, market, language, and collection period.

Active LLM Agents: 99/100
Prompt Set: 30 prompts · 19 Stable · 11 Adaptive
AI Systems: 7
Executions: 495/500
Valid Responses: 489/495
Prompt-system Coverage: 210/210 cells
Run Quality: Pass

How to interpret the scope

Active LLM Agents

99/100 means that 99 distinct planned operational agents produced at least one terminal execution outcome—valid or invalid—under the run policy.

An agent may run several prompts across several AI systems. An agent does not necessarily run every prompt on every AI system. Agent count is operational metadata and must not be used as the denominator for analytical metrics.

Prompt Set

Prompt Set is the number of unique canonical prompts included in the report. Repeated executions do not increase the Prompt Set count.

Executions

An execution is one attempt to run one canonical prompt on one eligible AI system. Executions are not unique prompts.

Valid Responses

Valid responses passed validation and are eligible for analytical metrics. Invalid responses remain available in Evidence where permissions allow, but they are excluded from metrics.

assigned executions = completed executions + not-completed executions
completed executions = valid responses + invalid responses

If the counts do not reconcile, the report enters Needs Attention and cannot be validated for publication.

Prompt-system Coverage

Complete coverage means every eligible prompt–AI-system pair contains at least one valid response. Equal response depth per cell is not required.

Run Quality

Run Quality summarizes whether execution completion, response validity, prompt-system coverage, Prompt Set integrity, reconciliation, and required report metadata are sufficient for publication. It is not a statement that every AI response is objectively correct.