AI Crawler Observatory: Who Reads This Site
Every other page on this site reports what someone else published. This one reports what we measured ourselves — a Cloudflare Worker in front of the origin records every AI crawler, answer-engine fetch, and machine-endpoint read, and those logs become the datasets below.
The questions this section exists to answer: does anything actually read llms.txt, does publishing machine-readable JSON get it consumed, how much traffic do AI answers send back, and how often does a request wearing a crawler's name turn out not to be that crawler.
- AI Agent Payments 2026: What a Crawl Toll Really Earns — Aug 18, 2026
Cloudflare gave AI agents wallets. We priced our own measured crawler traffic against it: 5,227 crawls in 20 days, and what a per-request toll would actually pay. - AI Crawler Traffic 2026: 747 Crawls per Human Visit — Aug 13, 2026
Twenty days of first-party Cloudflare logs: AI crawlers made 5,227 requests to this site and sent back 7 human visitors. Full dataset, method, and limits.