SpyderBot Documentation

Recommendations and ranking

Mentions, candidate inclusion, active recommendations and primary recommendations describe different outcomes. Their conditions are assessed in context, not from list formatting or isolated keywords. A task’s eligibility is established before its answer is observed.

The rates below use the balanced aggregation method ↗. Different cards can have different eligible prompt populations; their percentages should not be compared as if they all used the same denominator.

On this page

Organic Recommendation Rate · Primary Recommendation Rate · Candidate Inclusion Rate · Branded Endorsement Rate · Comparative Preference Rate · Alternative Recommendation Rate · Ordered Top-1 Rate · Ordered Top-3 Rate · Average Rank When Ranked · Ranked-Target Coverage · Ranking Instruction Compliance · Observed Leader

Organic Recommendation Rate

Unit: Percent

How often AI actively advises considering, choosing or using the target in eligible open recommendation tasks.

How it is calculated

Within every eligible measurement group, count active recommendations among all complete, usable answers to recommendation-eligible questions. Include relevant explicit conditional advice. A complete neutral list, refusal or answer without a recommendation remains an observed opportunity with no positive recommendation. Combine group rates using balanced aggregation.

How to interpret it

The denominator is not restricted to answers that already recommend something. A mention, quoted praise or negative recommendation is not automatically an active recommendation.

What to do next

Review the needs and conditions for which the target is or is not recommended.

Illustrative example

“Options include A and B” can be candidate inclusion without active advice. “Consider A if you need offline access” can be an active recommendation when that condition fits the request.

Related concepts: Candidate Inclusion Rate ↗ · Primary Recommendation Rate ↗

Measurement scope and aggregation ↗ · Evidence and availability ↗

Primary Recommendation Rate

Unit: Percent

How often the target is explicitly preferred or placed in an explicitly preferred top tier.

How it is calculated

Use the same eligible open recommendation panel as Organic Recommendation Rate, but count only an explicit primary choice or preferred top tier. Apply balanced aggregation.

How to interpret it

An unordered shortlist does not establish a primary choice. Several entities may share an explicitly preferred top tier.

What to do next

Inspect whether the target is considered but rarely preferred.

Measurement scope and aggregation ↗ · Evidence and availability ↗

Candidate Inclusion Rate

Unit: Percent

How often the target is presented as an admissible option in an eligible open recommendation task.

How it is calculated

Count inclusion as an admissible candidate among all usable answers in the eligible recommendation panel, then aggregate with the same balanced method. Neutral option lists can qualify; a mere reference or explicit rejection does not automatically qualify.

How to interpret it

Inclusion is broader than active recommendation, but narrower than any substantive mention on the same answer pool. Headlines on different prompt populations need not follow this numerical ordering.

What to do next

Compare inclusion and active recommendation on the same scope to find consideration without endorsement.

Measurement scope and aggregation ↗ · Evidence and availability ↗

Branded Endorsement Rate

Unit: Percent

How often AI affirmatively endorses a choice or claim that explicitly introduces the target.

How it is calculated

Use aided questions that actually ask for endorsement or choice validation. Count an affirmative endorsement of that requested choice in context, then balance within that cohort.

How to interpret it

A general trust or factual question is not automatically a purchase endorsement opportunity. This rate does not enter organic recommendation.

What to do next

Examine qualifications and cautions alongside affirmative endorsements.

Measurement scope and aggregation ↗ · Evidence and availability ↗

Comparative Preference Rate

Unit: Percent

How often the target is explicitly preferred in a named comparison.

How it is calculated

Use the declared named-candidate comparison panel. Count a supported explicit preference for the target, including an explicitly tied preference where applicable, across all usable eligible answers. Apply balanced aggregation.

How to interpret it

The specified candidate set constrains the result. This does not measure open-market discovery.

What to do next

Keep comparison criteria and candidates stable when evaluating changes.

Measurement scope and aggregation ↗ · Evidence and availability ↗

Alternative Recommendation Rate

Unit: Percent

How often AI actively recommends the target as an alternative to the specified competitor.

How it is calculated

Use the competitor-led alternative questions for the stated anchor. Count active target recommendations among all usable eligible answers, then balance within that anchor-specific scope.

How to interpret it

It is narrower than Competitive Capture, which measures substantive presence. Changing the anchor changes the question.

What to do next

Review the use cases where the target is a credible alternative.

Related concepts: Competitive Capture ↗

Measurement scope and aggregation ↗ · Evidence and availability ↗

Ordered Top-1 Rate

Unit: Percent

How often the target receives a validated first rank when the question requests an ordered comparison.

How it is calculated

Use all usable answers to eligible ordered-ranking tasks. Count a validated target rank of 1; target absence, lack of required ordering or no qualifying rank contributes no Top-1 outcome. Apply balanced aggregation.

How to interpret it

Formatting position alone is not a semantic rank. Explicit rank-one ties may allow several entities to qualify; this is not a share that must sum to 100%.

What to do next

Check ordering compliance and the common ranking criteria.

Measurement scope and aggregation ↗ · Evidence and availability ↗

Ordered Top-3 Rate

Unit: Percent

How often the target receives a validated rank of 1, 2 or 3 in eligible ordered-ranking tasks.

How it is calculated

Calculate validated Top-3 outcomes across all usable answers to ranking-eligible questions, including observed noncompliance and target absence as non-positive outcomes. Use balanced aggregation.

How to interpret it

The denominator includes answers where the target has no rank. A naturally numbered answer to a task that did not request ranking is diagnostic evidence, not automatically an eligible ranking observation.

What to do next

Read Top-3 with ranked-target coverage and ordering compliance.

Measurement scope and aggregation ↗ · Evidence and availability ↗

Average Rank When Ranked

Unit: Rank

The target’s average validated rank among applicable answers where it was ranked.

How it is calculated

Average known target ranks within applicable measurement groups, then use conditional balanced aggregation. Groups known to contain no target rank are not applicable; unresolved extraction is missing. No target ranks means no average rank.

How to interpret it

A rank of 1 based on rare appearances can coexist with low coverage. Missing ranks are not replaced by zero or an invented last position. Lower is better only within comparable ranking criteria.

What to do next

Always inspect ranked-target coverage and supporting counts.

Related concepts: Ranked-Target Coverage ↗

Measurement scope and aggregation ↗ · Evidence and availability ↗

Ranked-Target Coverage

Unit: Percent

How often a usable answer to an eligible ranking task assigns the target a validated rank.

How it is calculated

Calculate the balanced share of all usable eligible ranking answers containing a valid target rank.

How to interpret it

This is the companion needed to understand Average Rank. It does not say whether the rank is high.

What to do next

Investigate low coverage before interpreting a good conditional average.

Measurement scope and aggregation ↗ · Evidence and availability ↗

Ranking Instruction Compliance

Unit: Percent

How often the answer provides the requested meaningful ordering on the stated comparison basis.

How it is calculated

Calculate the balanced share of usable eligible ranking answers that comply with the requested semantic ordering.

How to interpret it

A numbered list is not sufficient by itself. Low compliance limits what the ranking outcomes can tell you.

What to do next

Review task clarity and observed answers; keep any redesigned prompt separate in history.

Measurement scope and aggregation ↗ · Evidence and availability ↗

Observed Leader

Unit: Entity and observed lead state

The entity with the highest observed result on the explicitly named metric and shared comparison scope.

How it is calculated

Entities are compared on one task-aligned metric and a common observation pool. The highest observed value determines the leader; exact ties remain ties. The leader state distinguishes no positive outcome, a single candidate, narrow leadership and insufficient evidence. The runner-up margin is the difference on that same metric when a comparable runner-up exists.

How to interpret it

This replaces the old blended Winner Score. An observed leader is not a statistically proven superior brand. Pure factual or trust evaluation has no default cross-entity winner.

What to do next

Check the basis, candidate roster, margin and evidence before acting.

Illustrative example

An enumeration task can have an observed mention leader; it must not be labelled the recommendation winner.

Measurement scope and aggregation ↗ · Evidence and availability ↗