Most publishers have lived with a false choice: block AI crawlers and lose distribution, or allow free access and watch downstream products monetize their work. Pay Per Crawl introduces a third path by turning authenticated crawler requests into billable events anchored on HTTP 402. Using cryptographic headers and a Cloudflare-managed settlement flow, it allows domain owners to set a flat per-request price, selectively exempt partners, and still block bad actors. This moves content monetization from bespoke deals into programmable infrastructure, which matters as AI shifts from search referrals to agentic retrieval and synthesis. The bet: standardized access plus receipts will create a liquid content market that compensates creators without wrecking openness.

Technically, Pay Per Crawl relies on Web Bot Auth proposals: crawlers register a key directory, sign requests with Ed25519 HTTP Message Signatures, and present payment intent via headers. If a URL is priced, the edge returns HTTP 402 with a price, or fulfills when the crawler declares willingness to pay (exact or max price), logging the charged amount. Crucially, identity-bound signatures help prevent user-agent spoofing and enable differentiated policies post-WAF and bot management. For publishers, this creates auditability and levered negotiating power; for AI developers, it introduces predictable procurement and compliance logs. The shift from robots.txt etiquette to verifiable receipts is what makes large-scale, multi-party access governable.

Economically, per-request pricing unlocks segmentation: charge agents and training bots, exempt traditional search, and zero-rate promotional paths. Early pricing will be discovery-led (reactive 402) until buyers learn content value; expect convergence toward intent-led max-price headers as budgets and SLAs mature. Risks remain: fragmentation across CDNs, bypass via screenshotting or cached mirrors, and adversarial routing through unpriced syndication. Yet if enough high-signal domains participate, agents will prefer authenticated routes for freshness, coverage, and reliability. For operators, the minimally viable pilot is scoped pricing on high-value paths, strict crawler allowlists, and instrumentation that ties charges to downstream outcomes like leads, conversions, or agent task success.