SpyderBot Documentation
Log import (Server / CDN)
Import nginx, Apache, or Cloudflare Logpush access logs into AI Traffic to measure AI crawler activity — including historical crawls before live Worker or script tracking started.
What it measures
- AI crawler hits (for example GPTBot, ClaudeBot, and similar known bot user-agents)
- HTTP status codes and requested pages/paths from server or CDN access logs
- Historical crawl windows when those days are present in the uploaded or Logpush payload
Imported rows appear alongside other bot visit evidence used by AI & Bot Activity views.
What it does not measure
- Human AI referrals or chat-referred humans (access logs do not contain reliable browser sessions)
- Engagement, scroll, or conversion outcomes that require client-side tracking
- Proof of citation, recommendation, or training use — request evidence only
Supported formats
- nginx combined
- Apache combined / common
- Cloudflare Logpush JSON (NDJSON / JSONL)
- Gzip-compressed files (
.gz) are accepted
Format is auto-detected from the file contents. Typical extensions: .log, .txt, .gz, .json, .ndjson, .jsonl.
Upload in AI Traffic
- Open AI Traffic and select your website.
- Go to Setup → Log import.
- Choose an access-log file and upload. The job runs asynchronously; the panel polls progress until completed or failed.
- Review counters (lines read, bot hits imported, skipped rows, parse errors) and the recent-imports list.
If Log import is not available for your account, contact support.
Cloudflare Logpush (HTTP destination)
Use Cloudflare Logpush with an HTTP destination — SpyderBot does not create or manage Logpush jobs via the Cloudflare account API for you.
- In Cloudflare, create or edit a Logpush job with an HTTP destination.
- Destination URL:
https://spyderbot.net/api/logpush - Auth header name:
X-Tracking-Key. Header value: the AI Traffic project tracking key for that website (exact format:X-Tracking-Key: <project tracking key>). Do not use a legacy website key. - Include request fields needed for bot identification (at least URI, user-agent, status, timestamp, and host when available). Gzip content encoding is accepted.
Logpush availability: Cloudflare documents Logpush as available on Free, Pro, Business, and Enterprise plans (dataset availability can still depend on products on the account) — see Cloudflare Logpush docs. If your dashboard does not offer an HTTP destination, check your Cloudflare plan.
The same destination is shown under Setup → Log import in the product UI.
Overlap and dedupe
- Default overlap mode is historical only: only days before live Worker/script tracking started are imported, so live ingest and log import do not double-count the same days.
- Duplicate lines across uploads or Logpush batches are skipped.
- Rows outside the configured retention window are skipped.
Retention
Imported bot visit rows follow the same AI Traffic bot-log retention as live ingest.