Dark Visitors
Directory and robots.txt rules for AI crawlers
Tracks known AI crawlers and agents, generates robots.txt rules to allow or block them and shows analytics on agent traffic. It keeps a public directory listing each crawler and its owner.
More in llms.txt & schema
Featured
llms.txt
llms.txt & schema
Proposed standard for LLM-friendly site summaries
JSON-LD
llms.txt & schema
W3C format for linked data in JSON
llms-txt-hub
llms.txt & schema
Directory of sites that publish llms.txt
Mintlify
llms.txt & schema
Docs platform that generates llms.txt files
Rich Results Test
llms.txt & schema
Google's tool for testing structured data
Schema App
llms.txt & schema
Platform for managing schema markup at scale