SpyderBot Documentation
AI Access Policies
Access control for AI systems and bots involves four distinct layers. Do not treat guidance documents as enforcement.
| Layer | Meaning |
|---|---|
| Declared crawl directive | robots.txt expresses requested crawler behavior |
| Content guidance / discovery | llms.txt may provide content guidance; it is not reliable enforcement |
| Enforced policy | Cloudflare, CDN, WAF, server, or supported plugin rule |
| Observed outcome | Captured allow, block, or HTTP response evidence |
- Do not present
llms.txtas an access-control enforcement mechanism. - Do not document a policy toggle unless the connected integration can enforce it.
- With no enforcement source, show guidance plus a connection path.