Skip to main content
Use the middleware package for application servers and edge runtimes. It forwards only matched AI bot requests, so normal browser traffic is never sent to DevTune.

Before you start

  • A paid plan that includes AI Traffic, and project management permission. See Set up AI Traffic.
  • An app you can deploy with request middleware, such as Next.js, Express or Node.

Generate the ingest key

  1. In Settings › Integrations, open Edge middleware under Crawler sensors.
  2. Select Generate ingest key.
  3. Copy the Server-side ingest key into a server or edge environment variable as DEVTUNE_AI_TRAFFIC_INGEST_KEY. Never expose it in browser code.
On Vercel, you can link the Vercel project instead.

Install the package

Keep the ingest key in a server-side environment variable. Choose the adapter for the runtime that receives your application requests; every adapter performs the same package-owned crawler filtering before sending matched traffic. For a single-origin Express or Node server, also set DEVTUNE_AI_TRAFFIC_ORIGIN to the application’s canonical public origin, such as https://www.example.com. Pinning the origin prevents client-supplied forwarded headers from changing the hostname recorded in DevTune. If one process serves multiple tracked hostnames, omit the fixed origin only when a trusted proxy overwrites X-Forwarded-Host and X-Forwarded-Proto, or create a separate adapter instance with its own origin for each hostname.

Next.js

For Next.js 16, add proxy.ts. For older Next.js projects, use the same body in middleware.ts and export middleware() instead.

Express

The middleware calls next() immediately and records the actual response status after Express emits finish.

Node HTTP

The hook records the final response.statusCode without blocking or changing the response.

Vercel OIDC without an ingest key

On Vercel, edge middleware can send traffic without DEVTUNE_AI_TRAFFIC_INGEST_KEY. Link the Vercel project in DevTune, and each deployment authenticates with the short-lived OIDC token Vercel issues to it. You need:
  • A site deployed on Vercel. Other hosts keep using the ingest key.
  • @devtune/ai-traffic 0.11.0 or later.
In Settings › Integrations, open Edge middleware under Crawler sensors. Under Vercel projects:
  1. Enter the Vercel team ID. In Vercel, it is in the team’s Settings › General, under Team ID, and starts with team_.
  2. Enter the Vercel project ID. It is in the project’s Settings › General, under Project ID, and starts with prj_.
  3. Select Add Vercel project.
The link works as soon as you add it. Anyone with project management permission can add or remove links. To send traffic from several Vercel projects, such as a docs site and a marketing site, add a link for each one.

Configure the middleware

The same screen shows the DevTune project ID. Copy it into an environment variable as DEVTUNE_AI_TRAFFIC_PROJECT_ID in every linked Vercel project. The project ID is not a secret. Pass it as projectId and leave out ingestKey:
The Express and Node HTTP adapters take the same projectId option when they run on Vercel.

Switch from an ingest key

You can keep ingestKey set while you switch. When both are set, the middleware sends with the Vercel token and uses the key whenever the token can’t be used: Vercel issued no token, Vercel couldn’t exchange it, DevTune rejected it, or DevTune couldn’t check it.
Once the link shows Last seen with a time, remove ingestKey and DEVTUNE_AI_TRAFFIC_INGEST_KEY from that project and redeploy. The ingest key is shared by every site that sends traffic to this DevTune project, so retire it only after every one of them has stopped using it: each Vercel project has its own link showing Last seen, and no other host still sends with the key. Then retire the key in Settings: select Rotate key, then either Revoke previous key right away or leave the previous key to stop working after 24 hours. If Rotate key is unavailable, a previous key is still active, so first select Revoke previous key once no deployment uses it.

Preview deployments

DevTune accepts Vercel-token batches from production deployments only. To accept a linked project’s previews too, turn on Preview deployments on its link. Preview traffic then mixes into the same AI Traffic data as production. Turn it off to go back to production only. Deployments in a custom Vercel environment are never accepted with the Vercel token, even with Preview deployments on. This setting covers only the Vercel token. A preview deployment that also has ingestKey set falls back to the key, and DevTune accepts the key from any deployment. To keep previews out while you still use the key, set DEVTUNE_AI_TRAFFIC_INGEST_KEY only for Vercel’s Production environment. Do this only when DEVTUNE_AI_TRAFFIC_PROJECT_ID is set in every environment: a deployment with neither fails at startup. A new link shows Not seen yet until DevTune accepts its first batch. After that, it shows the Vercel team and project names and when the link was last seen. If a link stays Not seen yet after a deploy that received crawler traffic:
  1. Check both IDs on the link against Vercel. A mistyped ID never matches, and DevTune records nothing from that deployment. Remove the link and add it again with the right IDs.
  2. Check that projectId in the middleware matches the DevTune project ID shown in Settings.
  3. Check that Snippet and middleware collection is switched on in Settings › Integrations. While it is off, DevTune rejects every batch.
  4. Check which Vercel environment received the traffic. Previews count only with Preview deployments on, and custom environments never count.
Select Remove on a link to stop accepting that Vercel project’s batches. If the middleware still has an ingest key configured, it falls back to the key. Otherwise its traffic stops reaching DevTune.

Verifying crawler identity

A user agent is a claim, not an identity. The middleware sends the address each matched crawler request came from, so DevTune can check it against the operator’s published networks and then discard it. Requests that can’t be checked are kept with their identity unknown. See crawler identity checks for the operators DevTune can check and what each verdict means. On Vercel, and on Express or Node with no proxy in front, nothing extra is needed. If a proxy sits in front of your app, set trustedProxy so the middleware can find the crawler’s own address: Keep @devtune/ai-traffic up to date. AI Traffic pages show an upgrade notice when your middleware is behind the latest release; see package versions for what each release adds.

Delivery and statuses

Delivery is best effort. A batch that fails, or stays rate-limited past its retries, is dropped and not resent, and drops are silent unless you pass a logger. To send events immediately on short-lived runtimes, or to tune retries and queue size, see delivery. Express and Node record the completed response status. A Next.js proxy can’t see the final route status after pass-through, so those requests are status unknown and don’t reach 404 Demand unless you capture the status.

Check the middleware is working

  1. Deploy, then request a page the middleware runs on with a crawler user agent. Replace the address with your own:
  2. In Settings › Integrations, reload the page. Until DevTune accepts its first event, the Edge middleware row reads Installation has not been confirmed. Within a few minutes it reads Active, with Last ingest and a time, and lists your hostname. A linked Vercel project also changes from Not seen yet to Last seen.
  3. Don’t expect the test request in the verified totals. It didn’t come from OpenAI’s network, so it isn’t counted as a verified crawl.

If something goes wrong

If your app fails at startup, the middleware couldn’t read its settings: On Vercel, variables are set per environment, so a variable set only for Production leaves Preview deployments without it. If the row doesn’t change:
  • Check that DEVTUNE_AI_TRAFFIC_INGEST_KEY, or DEVTUNE_AI_TRAFFIC_PROJECT_ID on Vercel, is set in the environment you deployed.
  • Check that Snippet and middleware collection is switched on in Settings › Integrations. While it is off, DevTune rejects every batch.
  • Pass a logger to see failed or dropped sends. See delivery.
  • On Vercel without an ingest key, work through Link status.

Next steps