SpyderBot Documentation
SpyderBot Optimizer Bot
Public documentation for SpyderBot Optimizer, the verified crawler operated by SpyderBot for GEO (Generative Engine Optimization) analysis.
This page describes what the bot does, how to identify it, and how site owners can allow or block it.
Overview
SpyderBot Optimizer fetches HTML pages only when a SpyderBot customer runs a GEO Optimizer analysis on a URL they control or have asked us to review. It is not a general search engine crawler and does not bulk-crawl the open web.
The bot collects on-page signals (title, meta description, headings, schema markup, link structure, and related GEO metrics) to power optimizer recommendations inside the SpyderBot product.
Bot identification
| Bot name | SpyderBot Optimizer |
|---|---|
| Operator | SpyderBot (spyderbot.net) |
| User-Agent | SpyderBot-Optimizer/1.0 (+https://docs.spyderbot.net/bot) |
| Documentation | https://docs.spyderbot.net/docs/integrations/spyderbot-optimizer-bot |
| Request method | GET (HTTPS) |
| Typical Accept header | text/html, application/xhtml+xml |
Web Bot Auth (cryptographic verification)
Outbound requests are signed using Web Bot Auth (HTTP Message Signatures, Ed25519).
- Key directory: https://docs.spyderbot.net/.well-known/http-message-signatures-directory
- Signature-Agent: points to the directory URL above
- Signature-Input tag:
web-bot-auth
Verifiers (including Cloudflare) can validate the Signature and Signature-Input headers against the public keys published in the directory.
What this bot does
- Fetches a single URL per optimizer job (no recursive site crawl)
- Reads publicly accessible HTML over HTTPS
- Extracts GEO-relevant page signals for analysis dashboards
- Honors
robots.txtand page-levelnoindex/ robots meta where applicable
What this bot does not do
- Does not log in, submit forms, or bypass authentication walls
- Does not execute JavaScript rendering (HTML response only)
- Does not crawl entire sites without a customer-initiated optimizer task
- Does not collect personal data from page visitors
Allow or block
To allow SpyderBot Optimizer, ensure your robots.txt does not disallow our User-Agent. To block it, add:
User-agent: SpyderBot-Optimizer
Disallow: /Blocking the bot may prevent GEO Optimizer analysis for URLs on your site.
Contact
Questions about this crawler or unexpected traffic:
- Email: [email protected]
- Product: LLM Tracking & GEO Optimizer
Next step
=> Learn how SpyderBot tracks LLM crawlers on your site in LLM Tracking