SpyderBot Documentation

Log import (Server / CDN)

Import nginx, Apache, or Cloudflare Logpush access logs into AI Traffic to measure AI crawler activity — including historical crawls before live Worker or script tracking started.


What it measures

  • AI crawler hits (for example GPTBot, ClaudeBot, and similar known bot user-agents)
  • HTTP status codes and requested pages/paths from server or CDN access logs
  • Historical crawl windows when those days are present in the uploaded or Logpush payload

Imported rows appear alongside other bot visit evidence used by AI & Bot Activity views.


What it does not measure

  • Human AI referrals or chat-referred humans (access logs do not contain reliable browser sessions)
  • Engagement, scroll, or conversion outcomes that require client-side tracking
  • Proof of citation, recommendation, or training use — request evidence only

Supported formats

  • nginx combined
  • Apache combined / common
  • Cloudflare Logpush JSON (NDJSON / JSONL)
  • Gzip-compressed files (.gz) are accepted

Format is auto-detected from the file contents. Typical extensions: .log, .txt, .gz, .json, .ndjson, .jsonl.


Upload in AI Traffic

  1. Open AI Traffic and select your website.
  2. Go to Setup → Log import.
  3. Choose an access-log file and upload. The job runs asynchronously; the panel polls progress until completed or failed.
  4. Review counters (lines read, bot hits imported, skipped rows, parse errors) and the recent-imports list.

If Log import is not available for your account, contact support.


Cloudflare Logpush (HTTP destination)

Use Cloudflare Logpush with an HTTP destination — SpyderBot does not create or manage Logpush jobs via the Cloudflare account API for you.

  1. In Cloudflare, create or edit a Logpush job with an HTTP destination.
  2. Destination URL: https://spyderbot.net/api/logpush
  3. Auth header name: X-Tracking-Key. Header value: the AI Traffic project tracking key for that website (exact format: X-Tracking-Key: <project tracking key>). Do not use a legacy website key.
  4. Include request fields needed for bot identification (at least URI, user-agent, status, timestamp, and host when available). Gzip content encoding is accepted.

Logpush availability: Cloudflare documents Logpush as available on Free, Pro, Business, and Enterprise plans (dataset availability can still depend on products on the account) — see Cloudflare Logpush docs. If your dashboard does not offer an HTTP destination, check your Cloudflare plan.

The same destination is shown under Setup → Log import in the product UI.


Overlap and dedupe

  • Default overlap mode is historical only: only days before live Worker/script tracking started are imported, so live ingest and log import do not double-count the same days.
  • Duplicate lines across uploads or Logpush batches are skipped.
  • Rows outside the configured retention window are skipped.

Retention

Imported bot visit rows follow the same AI Traffic bot-log retention as live ingest.