> ## Documentation Index > Fetch the complete documentation index at: https://docs.devtune.ai/llms.txt > Use this file to discover all available pages before exploring further. # AI Traffic Setup > Choose and install the right DevTune AI Traffic sensor for snippets, Cloudflare-proxied hostnames, and edge middleware deployments. Setting up AI Traffic means choosing the sensor that matches each hostname. This page covers the JavaScript snippet for AI referral traffic, server-side crawler sensors for Cloudflare and edge middleware deployments, and optional first-party proxy routing when your deployment or network setup requires it. ## Where Setup Lives AI Traffic setup now lives inside **Project Settings > AI Traffic**. To generate the snippet: 1. Open your project 2. Go to **Project Settings** 3. Open the **AI Traffic** tab 4. Click **Generate Tracking Snippet** DevTune generates a unique snippet key for your project. The snippet code appears on the settings page, ready to copy. Cloudflare and edge middleware crawler sensors are configured in the same **AI Traffic** tab when those controls are available for your account. ## Install the Snippet Copy the snippet and add it to your website's layout or template file so it loads on every page. The snippet should be placed before the closing `` tag. ### GitBook If your documentation is hosted on **GitBook**, you can use the official DevTune integration instead of manually installing the snippet: 1. Go to your GitBook space settings 2. Navigate to **Integrations** 3. Find and install the **DevTune** integration from the marketplace 4. Enter your project snippet key when prompted The integration will automatically add the tracking snippet to all your GitBook pages. Before enabling it, make sure your consent flow and any region-aware handling are configured so tracking only starts after the required opt-in. [View the integration on GitBook](https://www.gitbook.com/integrations/devtune). ### Next.js Add the snippet to your root layout file (`app/layout.tsx` or `pages/_document.tsx`): ```jsx theme={null} import Script from 'next/script'; export default function RootLayout({ children }) { return ( {children} ``` ## Enable or Disable Tracking On the Project Settings **AI Traffic** tab, use the toggle switch to enable or disable tracking at any time. When disabled, incoming events are rejected and no new data is recorded. ## Rate Limits Each tracking snippet is rate-limited to **1,000 requests per minute**. This is enough for most sites. If your site exceeds this limit, excess requests are silently dropped. ## Crawler Sensor Capabilities AI crawlers and answer fetchers often do not run JavaScript, so DevTune also supports server-side crawler sensors. Use one server-side crawler sensor per hostname to avoid double-counting. | Capability | Cloudflare pull | Edge middleware | | ------------ | ------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | Best for | Hostnames already proxied by Cloudflare, including docs or static hosts that cannot run app middleware. | Vercel, Next.js, or self-hosted apps where request middleware can run. | | Setup | Zone ID, readonly API token, and an orange-cloud proxied hostname. | Server-side ingest key plus `@devtune/ai-traffic` in your proxy or middleware. | | Data flow | DevTune pulls hourly Cloudflare analytics buckets. | Your edge code pushes matched AI bot requests asynchronously. | | Freshness | Hourly sync after Cloudflare analytics are available. | Near real-time push into DevTune's ingest queue. | | Granularity | Hourly aggregate buckets by page, user agent, bot class, and status. | One event per matched request with URL, path, user agent, bot class, and optional status. | | History | Can backfill the recent Cloudflare analytics window available to the zone. | Starts collecting only after middleware is installed and deployed. | | Status codes | Real edge status classes for proxied traffic, including 404s. | Known for redirects and denials returned by middleware. Normal page responses are status unknown unless you add route wrappers or known 404 path patterns. | | Coverage | Only traffic that passes through the orange-cloud hostname. | Only routes reached by your app middleware. | | Avoid | Using Cloudflare and middleware on the same hostname. | Double-proxying an app through Cloudflare just to collect traffic. | Recommended split: * Use **Cloudflare pull** for docs, marketing, or static hostnames that already sit behind Cloudflare. * Use **edge middleware** for Vercel, Next.js, and app hostnames where you do not want a Cloudflare-to-app double proxy. * Keep a hostname on one crawler sensor at a time. If you move a hostname from Cloudflare pull to middleware, remove or pause the Cloudflare hostname connection. ## Edge Middleware Setup Install `@devtune/ai-traffic`, keep your ingest key in a server-side environment variable, and call the package only for likely machine user agents. Keep the prefilter broad enough for future crawler and fetcher names; the package still performs the authoritative registry match before ingesting an event, but the cheap prefilter keeps normal browser traffic out of the middleware tracking path. Set `batchSize: 1`, `flushIntervalMs: 0`, and `waitUntilRegistryRefresh: false` in middleware so background ingest work is scheduled immediately, and hourly registry refreshes do not extend middleware `waitUntil` duration. ```typescript theme={null} // proxy.ts (Next.js 16; use middleware.ts/middleware() on older Next.js) import { createDevTuneAiTraffic } from '@devtune/ai-traffic'; import { NextResponse, type NextFetchEvent, type NextRequest, } from 'next/server'; const aiTraffic = createDevTuneAiTraffic({ ingestKey: process.env.DEVTUNE_AI_TRAFFIC_INGEST_KEY!, batchSize: 1, defaultStatus: null, flushIntervalMs: 0, waitUntilRegistryRefresh: false, }); const aiTrafficMachineUserAgentHints = [ 'bot', 'crawler', 'spider', 'fetch', 'agent', 'gpt', 'claude', 'anthropic', 'perplexity', 'openai', 'notebooklm', 'google-extended', 'bytespider', 'meta-external', '-ai', 'ai-', ]; function isLikelyMachineTraffic(request: NextRequest) { const userAgent = request.headers.get('user-agent')?.toLowerCase(); return aiTrafficMachineUserAgentHints.some((hint) => userAgent?.includes(hint), ); } export function proxy(request: NextRequest, event: NextFetchEvent) { const response = NextResponse.next(); if (isLikelyMachineTraffic(request)) { aiTraffic.trackRequest(request, event, null); } return response; } ``` When your middleware returns a redirect, denial, or custom response, pass `response.status` instead of `null`. For normal pass-through page requests, keep `null` because the middleware cannot observe the final page status. ## Proxy Setup You can proxy the tracking endpoint through your own domain when you need first-party routing for your deployment setup. Teams should still honor consent requirements and local privacy rules before collecting traffic data. ### How It Works Instead of the snippet sending data to `https://devtune.ai/api/v1/llm-traffic/collect`, you configure a reverse proxy, so requests go to a path on your own domain. ### Vercel / Next.js Add a rewrite to `next.config.ts`: ```javascript theme={null} async rewrites() { return [ { source: '/dt/:path*', destination: 'https://devtune.ai/api/v1/llm-traffic/:path*', }, ]; } ``` ### Netlify Add to `netlify.toml`: ```toml theme={null} [[redirects]] from = "/dt/*" to = "https://devtune.ai/api/v1/llm-traffic/:splat" status = 200 force = true ``` ### Cloudflare Create a redirect rule or Worker that proxies `/dt/*` to `https://devtune.ai/api/v1/llm-traffic/*`. ### nginx ```nginx theme={null} location /dt/ { proxy_pass https://devtune.ai/api/v1/llm-traffic/; proxy_set_header Host devtune.ai; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Host $host; } ``` After setting up the proxy, update the beacon URL in your snippet to use the first-party path. ## Multi-Site Tracking You can use the same tracking snippet across multiple websites or subdomains. The dashboard automatically detects distinct domains and provides a domain filter so you can view traffic for each site individually or in aggregate. ## Verifying Installation After installing the snippet: 1. Visit your website in a browser 2. Open the DevTune AI Traffic dashboard 3. You should see traffic appearing within a few minutes If no data appears, check: * The snippet is present in your page source * The snippet key matches the one shown in Project Settings * Ad blockers are not blocking the request * Tracking is enabled on the AI Traffic tab ## Next Steps * **[AI Traffic Dashboard](/ai-traffic/dashboard)** - Explore your traffic data * **[AI Traffic Overview](/ai-traffic/overview)** - Learn what gets tracked and why