Buying a Site?
Crawler Revenue Diligence
AI crawlers already pull real traffic from most content sites. Whether that traffic is being monetized, or just given away, is a signal worth knowing before you buy one, and standard diligence doesn't check it. Two endpoints, live now, priced per call with no account.
Screening · live
Needs nothing but a domain, and no cooperation from the seller. Reads per-crawler robots.txt rules for 18 named AI crawlers, then checks whether access is already being sold through TollBit, RSL, x402, or a 402 returned to a declared crawler. Returns one of four verdicts: unmonetized crawler magnet, already monetized, blocked but not monetized, or low appeal. Cheap enough to run across a whole marketplace category as a first pass.
Verified · live
For a site you're already in diligence on, once you have real server logs from the seller. Breaks crawler traffic down by crawler and operator, and uses reverse DNS on source IPs to flag addresses that claim an AI crawler User-Agent but don't belong to that operator, so spoofed traffic isn't counted as demand. Reads up to 20,000 log lines per call.
What it won't tell you
It won't invent a revenue number. There is no published industry rate for crawler access: TollBit and Cloudflare prices are set by each publisher individually and aren't disclosed. Supply your own per-crawl rate and volume and you get a band back with the arithmetic shown; supply nothing and you get no dollar figure at all. A confident number here would be a made-up one.
Log verification is reverse DNS, not cryptographic. Signature-based crawler verification only works on a live request, because ordinary access logs don't record the signature headers. Source IPs are resolved and checked against the operator the User-Agent claims, which is the standard method, and results distinguish verified, spoofed, and simply unchecked.
robots.txt is a request, not enforcement. A crawler that ignores it won't show as blocked and won't appear in screening results at all.
Timing worth knowing
From 15 September 2026 Cloudflare blocks mixed-use AI crawlers by default on ad-hosting pages for new customers, new sites, and all free-plan customers. Cloudflare has also replaced per-fetch Pay Per Crawl with Pay Per Use, which pays when content appears in an AI answer rather than per fetch. Both change what crawler access on a given domain is worth. Every screening response carries this note.
Both endpoints are paid per call in USDC over x402: call once with no payment to get the exact price, pay, and retry. Full parameters and output shapes are in /docs, and the price table is on /pricing.