Before you start
- A paid plan that includes AI Traffic, and project management permission. See Set up AI Traffic.
- An app you can deploy with request middleware, such as Next.js, Express or Node.
Generate the ingest key
- In Settings › Integrations, open Edge middleware under Crawler sensors.
- Select Generate ingest key.
- Copy the Server-side ingest key into a server or edge environment variable as
DEVTUNE_AI_TRAFFIC_INGEST_KEY. Never expose it in browser code.
Install the package
DEVTUNE_AI_TRAFFIC_ORIGIN to the application’s canonical public origin, such as https://www.example.com. Pinning the origin prevents client-supplied forwarded headers from changing the hostname recorded in DevTune. If one process serves multiple tracked hostnames, omit the fixed origin only when a trusted proxy overwrites X-Forwarded-Host and X-Forwarded-Proto, or create a separate adapter instance with its own origin for each hostname.
Next.js
For Next.js 16, addproxy.ts. For older Next.js projects, use the same body in middleware.ts and export middleware() instead.
Express
next() immediately and records the actual response status after Express emits finish.
Node HTTP
response.statusCode without blocking or changing the response.
Vercel OIDC without an ingest key
On Vercel, edge middleware can send traffic withoutDEVTUNE_AI_TRAFFIC_INGEST_KEY. Link the Vercel project in DevTune, and each deployment authenticates with the short-lived OIDC token Vercel issues to it.
You need:
- A site deployed on Vercel. Other hosts keep using the ingest key.
@devtune/ai-traffic0.11.0 or later.
Link the Vercel project
In Settings › Integrations, open Edge middleware under Crawler sensors. Under Vercel projects:- Enter the Vercel team ID. In Vercel, it is in the team’s Settings › General, under Team ID, and starts with
team_. - Enter the Vercel project ID. It is in the project’s Settings › General, under Project ID, and starts with
prj_. - Select Add Vercel project.
Configure the middleware
The same screen shows the DevTune project ID. Copy it into an environment variable asDEVTUNE_AI_TRAFFIC_PROJECT_ID in every linked Vercel project. The project ID is not a secret.
Pass it as projectId and leave out ingestKey:
projectId option when they run on Vercel.
Switch from an ingest key
You can keepingestKey set while you switch. When both are set, the middleware sends with the Vercel token and uses the key whenever the token can’t be used: Vercel issued no token, Vercel couldn’t exchange it, DevTune rejected it, or DevTune couldn’t check it.
ingestKey and DEVTUNE_AI_TRAFFIC_INGEST_KEY from that project and redeploy. The ingest key is shared by every site that sends traffic to this DevTune project, so retire it only after every one of them has stopped using it: each Vercel project has its own link showing Last seen, and no other host still sends with the key. Then retire the key in Settings: select Rotate key, then either Revoke previous key right away or leave the previous key to stop working after 24 hours. If Rotate key is unavailable, a previous key is still active, so first select Revoke previous key once no deployment uses it.
Preview deployments
DevTune accepts Vercel-token batches from production deployments only. To accept a linked project’s previews too, turn on Preview deployments on its link. Preview traffic then mixes into the same AI Traffic data as production. Turn it off to go back to production only. Deployments in a custom Vercel environment are never accepted with the Vercel token, even with Preview deployments on. This setting covers only the Vercel token. A preview deployment that also hasingestKey set falls back to the key, and DevTune accepts the key from any deployment. To keep previews out while you still use the key, set DEVTUNE_AI_TRAFFIC_INGEST_KEY only for Vercel’s Production environment. Do this only when DEVTUNE_AI_TRAFFIC_PROJECT_ID is set in every environment: a deployment with neither fails at startup.
Link status
A new link shows Not seen yet until DevTune accepts its first batch. After that, it shows the Vercel team and project names and when the link was last seen. If a link stays Not seen yet after a deploy that received crawler traffic:- Check both IDs on the link against Vercel. A mistyped ID never matches, and DevTune records nothing from that deployment. Remove the link and add it again with the right IDs.
- Check that
projectIdin the middleware matches the DevTune project ID shown in Settings. - Check that Snippet and middleware collection is switched on in Settings › Integrations. While it is off, DevTune rejects every batch.
- Check which Vercel environment received the traffic. Previews count only with Preview deployments on, and custom environments never count.
Remove a link
Select Remove on a link to stop accepting that Vercel project’s batches. If the middleware still has an ingest key configured, it falls back to the key. Otherwise its traffic stops reaching DevTune.Verifying crawler identity
A user agent is a claim, not an identity. The middleware sends the address each matched crawler request came from, so DevTune can check it against the operator’s published networks and then discard it. Requests that can’t be checked are kept with their identity unknown. See crawler identity checks for the operators DevTune can check and what each verdict means. On Vercel, and on Express or Node with no proxy in front, nothing extra is needed. If a proxy sits in front of your app, settrustedProxy so the middleware can find the crawler’s own address:
Keep
@devtune/ai-traffic up to date. AI Traffic pages show an upgrade notice when your middleware is behind the latest release; see package versions for what each release adds.
Delivery and statuses
Delivery is best effort. A batch that fails, or stays rate-limited past its retries, is dropped and not resent, and drops are silent unless you pass alogger. To send events immediately on short-lived runtimes, or to tune retries and queue size, see delivery.
Express and Node record the completed response status. A Next.js proxy can’t see the final route status after pass-through, so those requests are status unknown and don’t reach 404 Demand unless you capture the status.
Check the middleware is working
-
Deploy, then request a page the middleware runs on with a crawler user agent. Replace the address with your own:
- In Settings › Integrations, reload the page. Until DevTune accepts its first event, the Edge middleware row reads Installation has not been confirmed. Within a few minutes it reads Active, with Last ingest and a time, and lists your hostname. A linked Vercel project also changes from Not seen yet to Last seen.
- Don’t expect the test request in the verified totals. It didn’t come from OpenAI’s network, so it isn’t counted as a verified crawl.
If something goes wrong
If your app fails at startup, the middleware couldn’t read its settings:
On Vercel, variables are set per environment, so a variable set only for Production leaves Preview deployments without it.
If the row doesn’t change:
- Check that
DEVTUNE_AI_TRAFFIC_INGEST_KEY, orDEVTUNE_AI_TRAFFIC_PROJECT_IDon Vercel, is set in the environment you deployed. - Check that Snippet and middleware collection is switched on in Settings › Integrations. While it is off, DevTune rejects every batch.
- Pass a
loggerto see failed or dropped sends. See delivery. - On Vercel without an ingest key, work through Link status.
Next steps
- Advanced sensor settings - identity checks, proxies, delivery tuning and package versions
- Set up AI Traffic - connect website analytics, Search Console and the tracking snippet
- Reading AI Traffic - what each view counts