Technical AEO Audit

Documentation Index

Fetch the complete documentation index at: https://docs.devtune.ai/llms.txt

Use this file to discover all available pages before exploring further.

The technical AEO audit shows whether AI crawlers and answer engines can discover, fetch, extract, and understand your primary brand domains. It appears in Owned Content above Content Explorer as the AEO health report. Use the audit to find technical blockers before you spend time rewriting content. If AI systems cannot crawl a page, extract meaningful text, or discover your canonical URLs, they are less likely to cite or represent that content accurately.

​

What It Answers

The audit helps answer:

• Can AI search crawlers access the public site?
• Did DevTune capture enough owned content to assess the domain?
• Are important pages extractable as meaningful text or markdown?
• Do pages expose useful titles and basic metadata?
• Does the sitemap help crawlers discover URLs and freshness signals?
• Is optional llms.txt guidance present?

​

Where To Find It

1. Open your project
2. Go to Owned Content
3. Review the AEO health section above Content Explorer

Each audited domain appears as a compact row with its status, score, checked date, and finding count. Expand a domain to inspect the checks that produced the score.

​

How It Runs

DevTune runs the audit from two low-cost sources: Owned crawl checks reuse existing crawl data rather than starting a new paid crawl. Live probes are small HTTP requests against public resources such as /robots.txt, /sitemap.xml, /llms.txt, and the homepage. Audits refresh after completed domain crawls when the prior result is stale. Eligible primary domains are also scheduled for a monthly refresh.

​

Checks

​

Crawl Coverage

Checks whether active crawled pages are available for the domain and whether the crawl includes the homepage. If no pages are available yet, run or wait for the owned-content crawl before treating the result as a website remediation task.

​

Content Extractability

Looks for pages that extracted as nearly empty or unusually thin. This can indicate JavaScript-only content, bot protection, consent gates, redirects, or templates that do not expose meaningful crawler-readable text.

​

Markdown Extraction

Checks whether the crawl returned markdown as the primary content format. Markdown is easier to review and generally indicates that the page structure was captured cleanly.

​

Metadata

Checks for missing page titles and descriptions across active crawled pages. Titles and descriptions help crawlers and answer systems identify page purpose and brand context.

​

Content Freshness

Uses sitemap lastmod coverage as a freshness signal. Accurate lastmod values help crawlers decide which pages deserve recrawling. Missing or partial lastmod coverage is a warning, not a crawler block.

​

AI Crawler Access

Checks robots rules and homepage responses for AI search and retrieval crawlers, including OAI-SearchBot, ChatGPT-User, Claude-SearchBot, and PerplexityBot. Training crawler restrictions are reported separately because they may be intentional.

​

Sitemap

Checks whether a valid XML sitemap exists at the standard location or is declared in robots.txt. The audit validates sitemap-like content instead of treating every successful /sitemap.xml response as valid.

​

Homepage HTML

Checks whether the homepage returns enough raw HTML text and whether it includes noindex directives in robots meta tags or X-Robots-Tag headers.

​

llms.txt

Checks whether /llms.txt exists. This is optional and not a blocker, but it can provide concise guidance and canonical links for AI agents.

​

Reading The Report

When you expand a domain, the left side lists checks with their score and status. Selecting a check opens three tabs: The Copy full report button exports the audit evidence for review. The Copy prompt for all findings button appears only when there are actionable website fixes that can be safely handed to an implementation agent.

​

Multi-Domain Projects

DevTune audits each primary domain separately. For example, your main marketing site and docs site can have separate AEO health rows if both are configured as primary domains for the project. The top-level status is intentionally conservative: it highlights the weakest domain so a broken docs site or blocked homepage does not get hidden behind a stronger domain.

​

Public Brand Pages

Public brand report pages may show a lightweight AEO readiness scorecard. That public scorecard shares broad crawlability signals only. Exact findings, affected URLs, remediation details, and audit history remain inside the signed-in DevTune app.

​

Acting On Findings

Use the remediation guidance to separate website fixes from DevTune data freshness:

• No crawled pages available means the owned-content crawl has not produced active pages yet. Run or wait for the crawl first.
• Crawler access blocked usually points to robots rules, CDN/WAF policy, bot protection, or non-OK responses to AI crawler user agents.
• Thin or empty extracted pages usually points to client-only rendering, gated content, redirects, extraction-unfriendly markup, or pages that genuinely need more indexable explanation.
• Missing sitemap or low lastmod coverage usually points to sitemap generation, canonical host, or robots.txt discovery.
• Missing metadata usually points to page templates that do not generate unique titles or descriptions.

After making website changes, run a fresh domain crawl or wait for the next crawl and audit refresh. The report should update once DevTune has new crawl data or live probe results.

​

Next Steps

• Browsing Content - Review the crawled pages behind the audit
• Version History - Confirm content updates were captured after a crawl
• AI Traffic - See which AI bots and AI referrals reach your site