SpyderBot Documentation
Fix issues affecting important pages
A commercially important page has an access warning, unusual bot activity, or little observed activity. Should you change the page, run an audit, or fix the measurement first?
Use this playbook to move from an AI Traffic finding to a specific repair and an appropriate verification plan. You will avoid treating every traffic pattern as a technical failure and distinguish a successful technical fix from a business outcome.
Your outcome: a page-level issue with evidence, an owner, a justified action, and clear verification criteria-or a documented reason that no page change is needed.
Start from the symptom on your page
| Your situation | First useful move | Expected next output |
|---|---|---|
| "An important page has repeated access failures." | Check source health, then inspect the finding and affected URL. | A technical diagnosis and a scoped repair task. |
| "Our product page has no observed bot activity." | Check data coverage, then investigate discovery and access separately. | A supported issue or a monitoring decision-not an assumed block. |
| "AI visitors arrive but do not take the next step." | Check human referral and goal measurement. | A landing-page or measurement task; use technical audits only when evidence points to a technical issue. |
Marketing or growth can choose the page, explain its business role, and prepare the evidence. A site owner or developer should handle server, rendering, redirect, or access-policy changes. SpyderBot supplies the findings and checks available in your setup; the website change is an implementation task for your team.
Before you begin
Choose a page that matters to a real business objective: a product page, an integration page, a key comparison, or another important customer destination.
Your available evidence depends on your measurement setup. Browser-based tracking can support human referral and engagement analysis. Request-level bot activity, HTTP responses, and access decisions require an appropriate server or edge data source.
If you only have browser tracking, you can still assess human referral outcomes. You cannot use it to establish that a bot received a particular HTTP status.
1. Check Measurement health first
Open AI Traffic and review Measurement health before interpreting a drop or absence of activity.
Check whether the relevant source is connected, current, and capable of answering your question. A stopped feed can look like a traffic decline. A source without request-level visibility cannot establish a bot-access failure.
| What you find | Next step |
|---|---|
| Tracking has stopped or become delayed | Investigate measurement before changing the page. |
| Human referral attribution is unreliable | Review measurement setup and attribution. |
| Request-level evidence is unavailable | Obtain suitable evidence before diagnosing HTTP or policy failures. |
| Measurement is healthy for the question | Continue to the page finding. |
Keep machine requests separate from human referral sessions. Do not add two human analytics sources together as though they contain different visitors.
2. Choose the issue that matters to the business
Review What Needs Attention and the Critical Pages summary. Start with a finding on a page that supports your current business objective.
Confirm that the page's business priority is appropriate. A Critical Page is a business choice you can review; it is not proof that the page has a technical defect.
Read the finding's impact and evidence together:
- Severity describes potential consequence.
- Confidence describes the support for the finding.
- Priority helps determine the order of work.
A severe finding with weak evidence calls for investigation. A well-supported access failure on an important product page may justify immediate technical work. A low-value page with an uncertain signal may only need monitoring.
Use SpyderBot's recommendation as the starting point, then account for the page's actual business role. Do not create another scoring formula.
For example, when a supported failure affects a product page used in an active campaign, investigate it before an uncertain signal on an old blog post. If no finding is available for the page you care about, open its Page Intelligence and check coverage. You can still investigate the page, but absence of a finding is not a clean bill of health when data is missing.
3. Inspect the page-level evidence
Open Page Intelligence for the selected URL. Confirm the canonical page and any relevant URL variants, language, or market.
Review the evidence needed for this finding: bot activity and access, human referrals, source coverage, related findings, and existing audit or verification results. You do not need to review every panel.
Write a short diagnostic statement:
"Request-level evidence shows repeated access failures for the selected bot on our UK product page during the observed period."
Or, when evidence is limited:
"No activity was observed for this page in the available data. We have not established whether the page is blocked, undiscovered, or simply not requested."
Those statements lead to different work. No observed activity is not proof of a block, and a successful request is not proof that an AI system understood or cited the page.
4. Route the issue to the right investigation
Use the finding to select the relevant audit or operational task.
| Evidence or symptom | Where to investigate | Question to answer |
|---|---|---|
| Repeated HTTP failures, redirects, canonical conflicts, or policy-related access issues | URL Optimization; access checks in LLM Readability when relevant | Can the intended requester reach the correct page? |
| No observed activity on an important page | Discovery checks in LLM Readability; URL Optimization if discovery exposes URL issues | Is the page discoverable, and is there evidence of an access problem? |
| Important content is absent from extracted or rendered output | LLM Readability | Is the information available in a form the tested retrieval process can access? |
| Duplicate parameters or incorrect locale routing | URL Optimization | Are requests reaching the intended canonical and market page? |
| Conflicting entity facts or structured information | Entity Understanding | Do page content and structured facts describe the entity consistently? |
| Human referrals arrive but engagement or conversion is weak | Content or conversion review | Does the landing page satisfy the visitor's intent and make the next step clear? |
| Attribution is broken or tracking has stopped | Measurement Setup | Can the data be trusted? |
| Request identity or an access rule is uncertain | AI Access Policies and the relevant server/edge evidence | Is the requester identified appropriately, and what rule affected it? |
Do not run every audit by default. Start with the one that can test the suspected problem, then follow evidence to another investigation if necessary.
For access-policy issues, distinguish guidance from enforcement. A robots.txt directive and an llms.txt file are not equivalent to a WAF or CDN access rule. Review the specific policy and affected requester; do not disable security controls broadly to resolve a single page finding.
5. Turn the diagnosis into an owned action
Create an action where the workflow supports it, or record the task in your team's issue tracker. Include:
- The canonical page and affected variants.
- The observed issue and supporting evidence.
- The specific change to make.
- The person responsible.
- The technical acceptance check.
- Any later bot or business outcome you need to observe.
A useful task is specific:
"Review the rule causing verified requests to the product page to fail. Apply the narrow correction approved by the site owner, then retest the affected URL and relevant variants."
"Make the site AI-friendly" is too broad to implement or verify.
If the audit does not support the suspected issue, do not manufacture a repair. Record the result and either investigate the remaining uncertainty or monitor for recurrence.
A repair brief you can hand to a developer
Illustrative example; replace every detail with observed evidence.
Page: our UK product page and the affected URL variant.
Business role: primary destination for the current product campaign.
Observed issue: repeated failed requests in the selected period; attach the finding, timestamps, request statuses, and identity evidence.
Investigation: determine which rule or routing behavior caused those failures.
Acceptance: the corrected route returns the intended page under the relevant test conditions, with essential product content present; unrelated access controls remain as intended.
Follow-up: check a subsequent real request by the relevant bot. Track referral outcomes separately if they are part of the goal.
If the underlying cause is not established, assign the investigation first. Do not write "remove the WAF rule" as the repair just because the symptom is an access failure.
6. Verify the change at the right level
After implementation, report the fix where supported and run the appropriate verification. Reporting a fix is not the same as verifying it.
Choose checks that match your objective:
| Verification level | What it can establish | What it does not establish |
|---|---|---|
| Configuration or content review | The intended change was applied. | That the affected retrieval path now works. |
| SpyderBot probe or relevant audit rerun | The tested technical issue is resolved under the test conditions. | That an external bot has revisited. |
| Actual bot revisit | A real subsequent request occurred and its observed outcome. | That the content was understood, cited, or recommended. |
| Human referral or business outcome | The measured visitor outcome, using configured measurement and goals. | That the repair alone caused the outcome. |
| Regression monitoring | Whether the observed issue recurs. | That future failures are impossible. |
If the technical check passes but no real revisit has occurred, record "technically verified; awaiting real revisit" in your working record or the equivalent available product state. Do not keep making speculative changes while waiting for a bot you do not control.
If the objective was technical access, a successful technical check can complete the repair task while real-world observation remains separate. If the objective included referral or business improvement, keep that outcome open until suitable data is available. Conversion evaluation requires configured goals; it cannot be reconstructed from bot requests.
If verification does not resolve the issue
| What happens next | What to do |
|---|---|
| The same technical check still fails | Return the evidence to the implementer; check deployment, affected variants, and the diagnosed cause before marking the repair complete. |
| A probe passes, but a real request still fails | Compare requester, URL variant, time, and access path. Investigate the mismatch; keep the real-world issue open. |
| A probe passes, but no real revisit is observed | Set a review date, retain the pending state, and check coverage again at review. Do not assume an external bot will revisit on demand. |
| Access succeeds, but no referral uplift is observed | Keep technical verification intact. Investigate referral measurement and the separate visibility or visitor-intent question. |
| The issue returns | Reopen the task with new evidence and inspect changes since the last successful check. |
Example: repair an access issue without overstating the result
Illustrative example. A product page is marked as critical. AI Traffic highlights repeated access failures, and Measurement health confirms that request-level data is current.
The team opens Page Intelligence, checks the affected URL and requester evidence, and investigates the relevant access rule. The site owner applies a targeted correction. A subsequent technical check passes.
At that point, the team records:
| Check | Result |
|---|---|
| Intended rule correction applied | Confirmed by the site owner. |
| Technical access check | Passed for the tested page and relevant variants. |
| Actual external bot revisit | Not yet observed; continue monitoring. |
| AI citation or recommendation | Not established by these access checks. |
| Human referral or conversion improvement | Not yet evaluated. |
Later, a successful request by the relevant bot confirms a real revisit. The team can update that verification item without claiming an increase in AI recommendations or revenue.
Record the repair and verification
| Page and business role | Finding and evidence | Action and owner | Technical acceptance check | Later outcome check | Current state |
|---|---|---|---|---|---|
| Fill in | Fill in | Fill in | Fill in | Use only if relevant | Fill in |
You are done when the suspected issue has been supported or ruled out, any justified repair has an owner, and the appropriate verification result is recorded. If measurement is inadequate, completing the measurement repair is the correct next outcome. If no page defect is supported, monitoring with a clear review trigger is preferable to an unnecessary change.
Continue with: Find AI referral pages worth improving or expanding when the remaining question concerns human visitors; or Find where competitors are preferred over your brand when you need to investigate AI recommendation behavior separately.