SpyderBot Documentation

AI Traffic: observed activity, referrals and outcomes

AI Traffic separates observed AI bot activity from human visits referred by AI services. Bot activity can occur without a human visit, and a human referral can occur without a bot event in your available data. Neither establishes that an AI model trained on a page.

Every result identifies its source, date window, timezone, filters and collection grain. A server request, collector event, session, page, conversion event and order are different units. Only call activity “requests” when the source actually provides request-level evidence.

Use one declared primary session source for a result. GA4 and the SpyderBot browser script must not be added together as if they were independent audiences. Google Tag Manager is a deployment mechanism, not an additional traffic source. Browser-only instrumentation does not establish complete bot collection, bot identity, HTTP status or server access. These observations require appropriate server, edge or CDN evidence.

Estimated traffic in Quick Overview is separate from observed connected traffic. Keep attribution methods such as native referrer, UTM, GA4 classification, supported joins and modelled attribution distinguishable. Missing referrers and unobserved activity can lead to undercounting.

On this page

Observed AI Bot Activity · AI Bot Identities and Operators · AI-Active Pages · Observed Page Coverage · Critical Page Coverage · Known HTTP Status Coverage · Successful Access Rate · Successful Access Page Coverage · HTTP Errors and Observed Blocks · AI Referral Sessions · AI Referral Share · AI Referral Users and Landing Pages · AI Referral Engagement · Scroll and Short-Session Diagnostics · AI Referral Conversions · AI-Attributed Orders and Revenue · Attribution Coverage · Traffic Data Freshness and Collection Health

Observed AI Bot Activity

Unit: Requests or events, explicitly labelled

Qualifying activity attributed to supported AI-related bots in the connected collection source.

How it is calculated

Accepted qualifying observations are counted after source-appropriate deduplication. Request-level sources produce request counts; event-level sources produce event counts. The source, time window, supported identity basis and collection coverage define what the count includes.

How to interpret it

This is observed activity, not a complete count of all AI crawlers. Identifying an operator does not prove the purpose of a particular request or training use.

What to do next

Inspect the source, bot identity and access outcomes before changing infrastructure.

Measurement scope and aggregation ↗ · Evidence and availability ↗

AI Bot Identities and Operators

Unit: Counts or share of qualified activity

The distinct supported bot identities or operators observed, and their share of the selected activity.

How it is calculated

Deduplicate identities and operators at the explicitly named level. Activity share is each category’s qualifying observations divided by qualifying observations in the same source and scope. Preserve unknown identity rather than inventing a named bot.

How to interpret it

An operator can run several bots. Bot identity and inferred use are not interchangeable, and an unknown identity is not automatically non-AI.

What to do next

Open identity evidence where available and use supported classifications.

Measurement scope and aggregation ↗ · Evidence and availability ↗

AI-Active Pages

Unit: Distinct eligible canonical pages

Eligible canonical pages with at least one qualifying observed AI activity event in the selected period.

How it is calculated

Normalize pages using the declared inventory and canonical mapping, then count distinct eligible pages with qualifying activity. Multiple events on the same page do not increase the page count.

How to interpret it

No observed activity is not proof that a page is invisible or blocked. The result is limited by connected collection coverage.

What to do next

Compare active pages with the eligible page inventory.

Related concepts: Observed Page Coverage ↗

Measurement scope and aggregation ↗ · Evidence and availability ↗

Observed Page Coverage

Unit: Percent of eligible inventory pages

How much of the declared eligible page inventory has qualifying observed AI activity.

How it is calculated

Observed Page Coverage = distinct AI-active eligible canonical pages / all eligible canonical pages in the declared inventory. The inventory date and eligibility scope define the denominator. Without a valid inventory denominator, coverage is unavailable.

How to interpret it

This is observed collection coverage across pages, not indexing, ranking or guaranteed discoverability.

What to do next

Review unobserved important pages and whether collection is complete.

Illustrative example

40 active pages out of an eligible inventory of 200 give 20% observed page coverage.

Measurement scope and aggregation ↗ · Evidence and availability ↗

Critical Page Coverage

Unit: Percent of eligible critical pages

The share of the declared critical-page set with qualifying observed AI activity.

How it is calculated

Distinct active eligible critical pages are divided by all eligible critical pages in the declared set. The selected critical set and observation period define this result. An empty critical-page set makes the measure not applicable rather than 0%.

How to interpret it

The result depends on the declared critical set; it is not directly comparable to overall page coverage without that context.

What to do next

Review key product, pricing, support and trust pages according to your own declared inventory.

Measurement scope and aggregation ↗ · Evidence and availability ↗

Known HTTP Status Coverage

Unit: Percent of qualifying activity with a known status

How much qualifying AI activity has an observable HTTP status outcome.

How it is calculated

Qualifying activity with a known HTTP status is divided by all qualifying activity in the same source, unit and scope. Observations without status remain in the coverage denominator but cannot be classified as known successes, errors or blocks.

How to interpret it

Missing status is not a success, error or block. Browser data alone must not fabricate HTTP evidence.

What to do next

Check status coverage before interpreting access rates.

Measurement scope and aggregation ↗ · Evidence and availability ↗

Successful Access Rate

Unit: Percent of qualifying known-status activity

How often qualifying activity with a known HTTP status receives a 2xx response.

How it is calculated

Qualifying observations with a known 2xx status are divided by qualifying observations with a known HTTP status, at the same measurement unit. Request-level rates require request-level evidence. Known Status Coverage indicates how much of the activity supports this calculation.

How to interpret it

A 2xx response does not prove that the full intended content was rendered, understood, cited or used for training.

What to do next

Inspect representative responses and content delivery when access appears successful but outcomes remain poor.

Related concepts: Successful Access Page Coverage ↗ · Known HTTP Status Coverage ↗

Measurement scope and aggregation ↗ · Evidence and availability ↗

Successful Access Page Coverage

Unit: Percent of pages with known-status activity

How many observed eligible pages with known-status evidence have at least one qualifying 2xx access.

How it is calculated

Divide distinct eligible pages with at least one observed qualifying 2xx outcome by distinct eligible pages with qualifying known-status activity in the same scope. A page with both successful and failed requests counts once in this page-level measure.

How to interpret it

This page-level coverage differs from request-level Successful Access Rate. It can be high even when a page also experiences repeated failures.

What to do next

Inspect failure rates alongside page coverage.

Measurement scope and aggregation ↗ · Evidence and availability ↗

HTTP Errors and Observed Blocks

Unit: Same-grain rates and distinct page counts

Observed failed status outcomes or supported blocking evidence within the connected activity source.

How it is calculated

For an HTTP error rate, use the named error-status outcomes divided by qualifying known-status observations. For blocking, use the explicitly supported block-evidence population and show its denominator. Count blocked pages distinctly when a page count is shown. Block and HTTP-error categories can overlap; unknown status is not a confirmed block.

How to interpret it

Do not sum overlapping error and block rates into a pie or infer blocking from a lack of traffic.

What to do next

Review the status and block evidence, then verify the relevant server or edge configuration.

Measurement scope and aggregation ↗ · Evidence and availability ↗

AI Referral Sessions

Unit: Distinct sessions from the declared primary source

Human visit sessions attributed to supported AI referral sources by the selected measurement method.

How it is calculated

Count deduplicated qualifying sessions in the selected primary analytics source and window. Preserve source sessionisation, attribution method and identity rules. Do not add overlapping GA4 and browser-script sessions. Bot activity is not a human referral session.

How to interpret it

Unattributed direct visits may include AI-influenced visits, but they are not silently relabelled as measured AI referrals.

What to do next

Inspect AI source, landing page and attribution coverage.

Related concepts: Attribution Coverage ↗ · AI Referrals — Estimated ↗

Measurement scope and aggregation ↗ · Evidence and availability ↗

AI Referral Share

Unit: Percent of eligible measured sessions

The proportion of eligible measured sessions attributed to AI referrals.

How it is calculated

Qualifying AI referral sessions are divided by all eligible measured sessions in the same primary source, window and filter scope. The separately labelled AI Provider Share instead divides one provider’s qualifying referral sessions by all qualifying AI referral sessions.

How to interpret it

This is not a share of all website visitors outside your connected measurement scope. Provider-share and total-traffic-share percentages are different.

What to do next

Check total-traffic coverage and attribution limitations before comparison.

Measurement scope and aggregation ↗ · Evidence and availability ↗

AI Referral Users and Landing Pages

Unit: Distinct measured users or pages

The measured users or landing pages associated with qualifying AI referral sessions.

How it is calculated

Use the primary source’s supported user identity for distinct users; use normalized landing-page identity for pages. Do not infer person-level identity across unjoined devices or sources. Landing-page rows use the same qualifying session scope.

How to interpret it

Users are not sessions. Privacy settings and incomplete identity affect user counts.

What to do next

Use page-level patterns to investigate the content users actually reach.

Measurement scope and aggregation ↗ · Evidence and availability ↗

AI Referral Engagement

Unit: Source-defined sessions, percent or time

Engagement observed among eligible AI referral sessions using the selected source’s supported signals.

How it is calculated

Engaged Sessions follows the named source’s engagement definition. Engagement Rate divides qualifying engaged sessions by signal-eligible AI referral sessions. Time metrics use the stated measurable session or user denominator. The source definition and signal coverage govern interpretation; definitions from different analytics sources are not assumed identical.

How to interpret it

An unmeasured engagement signal is missing, not zero. Short-session diagnostics are not automatically GA4 bounce rate.

What to do next

Compare like-for-like measurement sources and inspect landing pages.

Measurement scope and aggregation ↗ · Evidence and availability ↗

Scroll and Short-Session Diagnostics

Unit: Percent of signal-eligible measured sessions or pages

Behavioural diagnostics based on supported browser or analytics signals.

How it is calculated

The configured scroll or duration rule determines which measured outcomes qualify. The rate divides those outcomes by the sessions or pages for which that signal was measurable, using the stated unit. Missing signals do not count as negative outcomes. The rule and measurement scope explain differences between results.

How to interpret it

These are engagement diagnostics, not proof of user satisfaction or content quality. A page-level scroll denominator must not be labelled as a session denominator.

What to do next

Review measurement configuration before interpreting low engagement.

Measurement scope and aggregation ↗ · Evidence and availability ↗

AI Referral Conversions

Unit: Distinct converting sessions, events or percent

Configured goal outcomes observed in qualifying AI referral sessions.

How it is calculated

Goal Sessions counts each eligible session once if it meets the selected goal. Goal Events counts supported goal occurrences and can exceed the number of sessions. Session Conversion Rate = qualifying converting sessions / eligible AI referral sessions in the same source and attribution scope. Goal configuration and effective dates define comparability.

How to interpret it

A change in goal setup can break comparability. Attribution associates outcomes with measured sessions; it does not prove AI caused incremental sales.

What to do next

Review the configured goal and attribution basis.

Illustrative example

5 converting sessions among 100 eligible AI referral sessions give 5% session conversion, even if those sessions fire 8 goal events.

Measurement scope and aggregation ↗ · Evidence and availability ↗

AI-Attributed Orders and Revenue

Unit: Orders and currency

Orders and revenue associated with qualifying AI referral sessions under the selected source’s attribution basis.

How it is calculated

Use supported order identifiers and source-reported revenue in the same attribution window. State currency and the source’s treatment of refunds, discounts and taxes. Average Order Value = attributable revenue / attributable orders on the same accounting basis. With no attributable orders, average order value is not applicable.

How to interpret it

Do not invent revenue from visits or multiply by an assumed conversion rate. Different attribution windows can produce different totals.

What to do next

Reconcile against the connected commerce source before using the result for budgeting.

Measurement scope and aggregation ↗ · Evidence and availability ↗

Attribution Coverage

Unit: Percent of eligible observations with a supported attribution method

How much of the eligible measured population has the attribution evidence required by the displayed result.

How it is calculated

Eligible observations with supported attribution evidence are divided by the eligible population at the same unit: sessions, conversions or orders, as identified by the result. Native referrer, UTM, analytics-classified, joined and modelled attribution remain distinguishable methods.

How to interpret it

Coverage describes attribution availability, not attribution accuracy or causal incrementality.

What to do next

Investigate missing attribution and compare only compatible methods.

Measurement scope and aggregation ↗ · Evidence and availability ↗

Traffic Data Freshness and Collection Health

Unit: Timestamp, lag and status

How current and complete the connected traffic observations appear.

How it is calculated

The source’s latest available observation and the time it was ingested or synchronised are different timestamps. Collection health describes received evidence against the configured collection scope, including known delays. A lack of recent observations can indicate delayed collection rather than zero activity.

How to interpret it

A successful integration connection does not prove complete historical coverage.

What to do next

Check collection health before responding to abrupt traffic drops.

Measurement scope and aggregation ↗ · Evidence and availability ↗