Crawler identity checks
A user agent is a claim, not an identity: anything on the internet can sendGPTBot/1.3. DevTune checks the address a matched crawler request came from against the networks its operator publishes, then discards the address. Edge middleware sends that address; the Cloudflare log pull uses it when Cloudflare’s analytics include it.
Each request gets one of three verdicts:
- Verified — the address, or a valid Web Bot Auth signature, matches the claimed operator.
- Failed — the address contradicts the claimed operator. AI Traffic shows these as identity mismatches.
- Unknown — the check could not be made. DevTune records why: the sensor supplied no address, DevTune has no check for that operator, or the check could not complete. Requests recorded before DevTune kept that reason appear as before verification.
Supported operators
Mistral’s published feeds cover
MistralAI-User and MistralAI-Index, not its training crawler. ByteDance has no identity check and stays unknown. A platform name inferred from a user agent or referral is not proof of identity.
Coding agents such as Claude Code and Gemini CLI fetch from their user’s own network. DevTune recognises their user agents but keeps their identity unknown unless an independent operator check succeeds; an address outside hosted crawler ranges is not a mismatch for them. OpenCode retries, Qwen Code, Crush and Aider also stay unknown. Their fetches count as answer fetches, not training crawls.
Signed requests
Some agents sign their requests with Web Bot Auth. When edge middleware forwards a signature that validates against the operator’s published key, the request is verified by its signature instead of its address. The AI Traffic API and MCP report how many verified requests were verified by signature. That count is null when the method was not recorded for some verified requests in the window, including requests from before DevTune recorded it.Trusted proxies
The middleware reads the sending address only from a source your deployment declares as trustworthy, because a forgeable address proves nothing. On Vercel it reads the platform’s own forwarded header. On Express and Node it uses the socket peer. It readsX-Forwarded-For only when you select CloudFront or set a numeric offset, because a visitor can add entries before the ones your proxies append.
An edge runtime with no socket and no declared proxy sends no address, so its traffic stays unknown. With
'none', traffic is stored as unverified.
CloudFront
For CloudFront connecting directly to the app that runs the middleware, add the option to your existing setup:X-Forwarded-For, and this setting reads the rightmost entry. See AWS’s custom-origin documentation.
Use it only when requests reach the app through your own CloudFront distribution. If an ALB or another proxy between CloudFront and the app appends addresses, use a numeric offset instead: 0 selects the rightmost entry, 1 skips one appended address, and so on. Check whether each proxy appends, preserves or replaces the header before you choose an offset.
This applies to middleware at the origin, not to CloudFront Functions or Lambda@Edge. After you redeploy, new requests with a usable address can be checked; earlier requests stay unknown.
Delivery
Edge middleware batches matched requests and sends them without delaying your responses. By default a batch is sent when 10 events are queued or after 250 ms, with up to 100 events per request. Next.js hands delayed sends toevent.waitUntil(). If a short-lived runtime can’t reliably finish delayed work, set batchSize: 1 and flushIntervalMs: 0 to send each event immediately.
Delivery is best effort, and lost batches are not resent later:
- A rate-limited response (429) is retried with backoff, three times per batch by default with each wait capped at five seconds. If throttling lasts longer, the batch is dropped.
- Any other failure, such as a network error or a 5xx, drops the batch immediately.
- With a linked Vercel project, a batch DevTune can’t check the Vercel token for (503) is resent with
ingestKeyif one is set, and otherwise retried like a 429. - Drops are silent unless you pass a
logger, and its warnings are sampled at 1% by default.
Response statuses
404 Demand needs each request’s response status. Express and Node record the completed status. A Next.js proxy can’t see the final route status after it passes a request on, so the middleware leaves the status unknown instead of guessing. To capture it:- When your proxy returns a response itself, use
createDevTuneAiTraffic()and pass the knownresponse.status. - Where you own a Fetch-compatible route handler, wrap it in
withDevTuneAiTrafficRoute()to capture its exact status. - List known missing-page paths with
notFoundPathPatterns.
Package versions
AI Traffic pages show an upgrade notice, with the install command, when your edge middleware runs an older release than the latest, or a release too old to report its version. The notice clears within about an hour of upgrading. Dismissing it hides it on AI Traffic pages until the next release; Traffic settings keeps showing it, with Show on every AI Traffic page to bring it back. The API and MCP traffic summary return the same information.
From 0.5.0 the middleware refreshes DevTune’s list of AI bots in the background about once an hour; until a refresh completes, a newly added bot may be missed, and a failed refresh keeps the last working list. Browser request data is not sent. Changes to address handling, adapters or the event format can still need a package upgrade. The Cloudflare log pull needs no package.