SpyderBot Documentation
AI Traffic: observed activity, referrals and outcomes
AI Traffic separates observed AI bot activity from human visits referred by AI services. Bot activity can occur without a human visit, and a human referral can occur without a bot event in your available data. Neither establishes that an AI model trained on a page.
Every result identifies its source, date window, timezone, filters and collection grain. A server request, collector event, session, page, conversion event and order are different units. Only call activity “requests” when the source actually provides request-level evidence.
Use one declared primary session source for a result. GA4 and the SpyderBot browser script must not be added together as if they were independent audiences. Google Tag Manager is a deployment mechanism, not an additional traffic source. Browser-only instrumentation does not establish complete bot collection, bot identity, HTTP status or server access. These observations require appropriate server, edge or CDN evidence.
Estimated traffic in Quick Overview is separate from observed connected traffic. Keep attribution methods such as native referrer, UTM, GA4 classification, supported joins and modelled attribution distinguishable. Missing referrers and unobserved activity can lead to undercounting.
On this page
Observed AI Bot Activity · AI Bot Identities and Operators · AI-Active Pages · Observed Page Coverage · Critical Page Coverage · Known HTTP Status Coverage · Successful Access Rate · Successful Access Page Coverage · HTTP Errors and Observed Blocks · AI Referral Sessions · AI Referral Share · AI Referral Users and Landing Pages · AI Referral Engagement · Scroll and Short-Session Diagnostics · AI Referral Conversions · AI-Attributed Orders and Revenue · Attribution Coverage · Traffic Data Freshness and Collection Health
Observed AI Bot Activity
Unit: Requests or events, explicitly labelled
Qualifying activity attributed to supported AI-related bots in the connected collection source.
How it is calculated
Accepted qualifying observations are counted after source-appropriate deduplication. Request-level sources produce request counts; event-level sources produce event counts. The source, time window, supported identity basis and collection coverage define what the count includes.
How to interpret it
This is observed activity, not a complete count of all AI crawlers. Identifying an operator does not prove the purpose of a particular request or training use.
What to do next
Inspect the source, bot identity and access outcomes before changing infrastructure.
Measurement scope and aggregation ↗ · Evidence and availability ↗
AI Bot Identities and Operators
Unit: Counts or share of qualified activity
The distinct supported bot identities or operators observed, and their share of the selected activity.
How it is calculated
Deduplicate identities and operators at the explicitly named level. Activity share is each category’s qualifying observations divided by qualifying observations in the same source and scope. Preserve unknown identity rather than inventing a named bot.
How to interpret it
An operator can run several bots. Bot identity and inferred use are not interchangeable, and an unknown identity is not automatically non-AI.
What to do next
Open identity evidence where available and use supported classifications.
Measurement scope and aggregation ↗ · Evidence and availability ↗
AI-Active Pages
Unit: Distinct eligible canonical pages
Eligible canonical pages with at least one qualifying observed AI activity event in the selected period.
How it is calculated
Normalize pages using the declared inventory and canonical mapping, then count distinct eligible pages with qualifying activity. Multiple events on the same page do not increase the page count.
How to interpret it
No observed activity is not proof that a page is invisible or blocked. The result is limited by connected collection coverage.
What to do next
Compare active pages with the eligible page inventory.
Related concepts: Observed Page Coverage ↗
Measurement scope and aggregation ↗ · Evidence and availability ↗
Observed Page Coverage
Unit: Percent of eligible inventory pages
How much of the declared eligible page inventory has qualifying observed AI activity.
How it is calculated
Observed Page Coverage = distinct AI-active eligible canonical pages / all eligible canonical pages in the declared inventory. The inventory date and eligibility scope define the denominator. Without a valid inventory denominator, coverage is unavailable.
How to interpret it
This is observed collection coverage across pages, not indexing, ranking or guaranteed discoverability.
What to do next
Review unobserved important pages and whether collection is complete.
Illustrative example
40 active pages out of an eligible inventory of 200 give 20% observed page coverage.
Measurement scope and aggregation ↗ · Evidence and availability ↗
Critical Page Coverage
Unit: Percent of eligible critical pages
The share of the declared critical-page set with qualifying observed AI activity.
How it is calculated
Distinct active eligible critical pages are divided by all eligible critical pages in the declared set. The selected critical set and observation period define this result. An empty critical-page set makes the measure not applicable rather than 0%.
How to interpret it
The result depends on the declared critical set; it is not directly comparable to overall page coverage without that context.
What to do next
Review key product, pricing, support and trust pages according to your own declared inventory.
Measurement scope and aggregation ↗ · Evidence and availability ↗
Known HTTP Status Coverage
Unit: Percent of qualifying activity with a known status
How much qualifying AI activity has an observable HTTP status outcome.
How it is calculated
Qualifying activity with a known HTTP status is divided by all qualifying activity in the same source, unit and scope. Observations without status remain in the coverage denominator but cannot be classified as known successes, errors or blocks.
How to interpret it
Missing status is not a success, error or block. Browser data alone must not fabricate HTTP evidence.
What to do next
Check status coverage before interpreting access rates.
Measurement scope and aggregation ↗ · Evidence and availability ↗
Successful Access Rate
Unit: Percent of qualifying known-status activity
How often qualifying activity with a known HTTP status receives a 2xx response.
How it is calculated
Qualifying observations with a known 2xx status are divided by qualifying observations with a known HTTP status, at the same measurement unit. Request-level rates require request-level evidence. Known Status Coverage indicates how much of the activity supports this calculation.
How to interpret it
A 2xx response does not prove that the full intended content was rendered, understood, cited or used for training.
What to do next
Inspect representative responses and content delivery when access appears successful but outcomes remain poor.
Related concepts: Successful Access Page Coverage ↗ · Known HTTP Status Coverage ↗
Measurement scope and aggregation ↗ · Evidence and availability ↗
Successful Access Page Coverage
Unit: Percent of pages with known-status activity
How many observed eligible pages with known-status evidence have at least one qualifying 2xx access.
How it is calculated
Divide distinct eligible pages with at least one observed qualifying 2xx outcome by distinct eligible pages with qualifying known-status activity in the same scope. A page with both successful and failed requests counts once in this page-level measure.
How to interpret it
This page-level coverage differs from request-level Successful Access Rate. It can be high even when a page also experiences repeated failures.
What to do next
Inspect failure rates alongside page coverage.
Measurement scope and aggregation ↗ · Evidence and availability ↗
HTTP Errors and Observed Blocks
Unit: Same-grain rates and distinct page counts
Observed failed status outcomes or supported blocking evidence within the connected activity source.
How it is calculated
For an HTTP error rate, use the named error-status outcomes divided by qualifying known-status observations. For blocking, use the explicitly supported block-evidence population and show its denominator. Count blocked pages distinctly when a page count is shown. Block and HTTP-error categories can overlap; unknown status is not a confirmed block.
How to interpret it
Do not sum overlapping error and block rates into a pie or infer blocking from a lack of traffic.
What to do next
Review the status and block evidence, then verify the relevant server or edge configuration.
Measurement scope and aggregation ↗ · Evidence and availability ↗
AI Referral Sessions
Unit: Distinct sessions from the declared primary source
Human visit sessions attributed to supported AI referral sources by the selected measurement method.
How it is calculated
Count deduplicated qualifying sessions in the selected primary analytics source and window. Preserve source sessionisation, attribution method and identity rules. Do not add overlapping GA4 and browser-script sessions. Bot activity is not a human referral session.
How to interpret it
Unattributed direct visits may include AI-influenced visits, but they are not silently relabelled as measured AI referrals.
What to do next
Inspect AI source, landing page and attribution coverage.
Related concepts: Attribution Coverage ↗ · AI Referrals — Estimated ↗
Measurement scope and aggregation ↗ · Evidence and availability ↗
AI Referral Share
Unit: Percent of eligible measured sessions
The proportion of eligible measured sessions attributed to AI referrals.
How it is calculated
Qualifying AI referral sessions are divided by all eligible measured sessions in the same primary source, window and filter scope. The separately labelled AI Provider Share instead divides one provider’s qualifying referral sessions by all qualifying AI referral sessions.
How to interpret it
This is not a share of all website visitors outside your connected measurement scope. Provider-share and total-traffic-share percentages are different.
What to do next
Check total-traffic coverage and attribution limitations before comparison.
Measurement scope and aggregation ↗ · Evidence and availability ↗
AI Referral Users and Landing Pages
Unit: Distinct measured users or pages
The measured users or landing pages associated with qualifying AI referral sessions.
How it is calculated
Use the primary source’s supported user identity for distinct users; use normalized landing-page identity for pages. Do not infer person-level identity across unjoined devices or sources. Landing-page rows use the same qualifying session scope.
How to interpret it
Users are not sessions. Privacy settings and incomplete identity affect user counts.
What to do next
Use page-level patterns to investigate the content users actually reach.
Measurement scope and aggregation ↗ · Evidence and availability ↗
AI Referral Engagement
Unit: Source-defined sessions, percent or time
Engagement observed among eligible AI referral sessions using the selected source’s supported signals.
How it is calculated
Engaged Sessions follows the named source’s engagement definition. Engagement Rate divides qualifying engaged sessions by signal-eligible AI referral sessions. Time metrics use the stated measurable session or user denominator. The source definition and signal coverage govern interpretation; definitions from different analytics sources are not assumed identical.
How to interpret it
An unmeasured engagement signal is missing, not zero. Short-session diagnostics are not automatically GA4 bounce rate.
What to do next
Compare like-for-like measurement sources and inspect landing pages.
Measurement scope and aggregation ↗ · Evidence and availability ↗
Scroll and Short-Session Diagnostics
Unit: Percent of signal-eligible measured sessions or pages
Behavioural diagnostics based on supported browser or analytics signals.
How it is calculated
The configured scroll or duration rule determines which measured outcomes qualify. The rate divides those outcomes by the sessions or pages for which that signal was measurable, using the stated unit. Missing signals do not count as negative outcomes. The rule and measurement scope explain differences between results.
How to interpret it
These are engagement diagnostics, not proof of user satisfaction or content quality. A page-level scroll denominator must not be labelled as a session denominator.
What to do next
Review measurement configuration before interpreting low engagement.
Measurement scope and aggregation ↗ · Evidence and availability ↗
AI Referral Conversions
Unit: Distinct converting sessions, events or percent
Configured goal outcomes observed in qualifying AI referral sessions.
How it is calculated
Goal Sessions counts each eligible session once if it meets the selected goal. Goal Events counts supported goal occurrences and can exceed the number of sessions. Session Conversion Rate = qualifying converting sessions / eligible AI referral sessions in the same source and attribution scope. Goal configuration and effective dates define comparability.
How to interpret it
A change in goal setup can break comparability. Attribution associates outcomes with measured sessions; it does not prove AI caused incremental sales.
What to do next
Review the configured goal and attribution basis.
Illustrative example
5 converting sessions among 100 eligible AI referral sessions give 5% session conversion, even if those sessions fire 8 goal events.
Measurement scope and aggregation ↗ · Evidence and availability ↗
AI-Attributed Orders and Revenue
Unit: Orders and currency
Orders and revenue associated with qualifying AI referral sessions under the selected source’s attribution basis.
How it is calculated
Use supported order identifiers and source-reported revenue in the same attribution window. State currency and the source’s treatment of refunds, discounts and taxes. Average Order Value = attributable revenue / attributable orders on the same accounting basis. With no attributable orders, average order value is not applicable.
How to interpret it
Do not invent revenue from visits or multiply by an assumed conversion rate. Different attribution windows can produce different totals.
What to do next
Reconcile against the connected commerce source before using the result for budgeting.
Measurement scope and aggregation ↗ · Evidence and availability ↗
Attribution Coverage
Unit: Percent of eligible observations with a supported attribution method
How much of the eligible measured population has the attribution evidence required by the displayed result.
How it is calculated
Eligible observations with supported attribution evidence are divided by the eligible population at the same unit: sessions, conversions or orders, as identified by the result. Native referrer, UTM, analytics-classified, joined and modelled attribution remain distinguishable methods.
How to interpret it
Coverage describes attribution availability, not attribution accuracy or causal incrementality.
What to do next
Investigate missing attribution and compare only compatible methods.
Measurement scope and aggregation ↗ · Evidence and availability ↗
Traffic Data Freshness and Collection Health
Unit: Timestamp, lag and status
How current and complete the connected traffic observations appear.
How it is calculated
The source’s latest available observation and the time it was ingested or synchronised are different timestamps. Collection health describes received evidence against the configured collection scope, including known delays. A lack of recent observations can indicate delayed collection rather than zero activity.
How to interpret it
A successful integration connection does not prove complete historical coverage.
What to do next
Check collection health before responding to abrupt traffic drops.
Measurement scope and aggregation ↗ · Evidence and availability ↗