Skip to main content

Losses

A scenario is a loss when a competing tool or Built its own was clearly picked more often than your tool: the 90% interval on the difference between their pick rates excludes zero. A scenario where the difference is not clear is level, not a loss. Only a competing tool or Built its own can make a loss; No provider chosen never does. Built its own runs that DevTune recorded only as a fallback guess are left out of the decision. For example, 1 pick out of 12 against a competitor’s 9 is a loss. 1 out of 4 against 3 is level: four runs cannot tell the two apart. Tests requested before this rule keep the rule they were decided by: a loss when a competing tool or Built its own was a top pick, ties included, and your tool was not picked in a majority of completed runs. Their losses and Actions are not recalculated.

Actions

Open Actions to find Agent Picks Actions in the shared queue. Use the Source filter (AI Search only, Agent Picks only or Both) to choose the work shown. Both includes the sources you can access. AI Search tracking requires its own plan. Completed tests create two kinds of Action, unless the test is directional, such as a Quick test or a scenario re-run:
  • A loss Action for a losing test app, request, agent and model combination.
  • A reason Action when the same recorded reason appears across several losing scenarios, such as a failed page fetch. It lists every affected scenario and its evidence.
Each setup has its own Actions. A test’s losses create Actions for its setup only, and updating one setup’s Actions never changes another’s. When a project has several setups, each Agent Picks Action in the queue shows a small label with its setup’s category or job. Removing a setup removes its suggested Actions that nobody has accepted; accepted Actions stay. Open the Action to review the original pick count, completed and planned runs, likely range and recorded evidence. A failed fetch is an observation, not proof it caused the loss. Copied public tests can supply baseline rates while their original run evidence is unavailable; the Action says when evidence is missing.

Draft

Use the Action’s brief and draft controls to prepare a fix. Review the proposed content and approve it before publishing. If a page is locked, the suggestion names it and calls for supplementary content around it. When the brief is ready, select Copy as a coding-agent prompt to hand the fix to your own coding agent. The prompt contains the brief, what agents picked instead and how often, the target files or pages, the acceptance check and every locked page the fix steers around. Review the agent’s change before you ship it; copying the prompt publishes nothing. You can dismiss an Action and restore it later. If a newer Action already tracks the loss, restoring the older one is refused and the newer Action stays in the queue.

Shipping and re-testing

  1. Publish the fix and select Mark live on the Action.
  2. Open the live Action and select Estimate a re-test under Re-test this fix.
  3. Review the estimate, then select Confirm and reserve credits.
A complete original measurement is required. If a recorded page on your domain returned 404, that page must load before the re-test can start. If the original test app is no longer supported, run a fresh test first. The re-test uses the Action’s original requests, test apps, agents and models. Changing your saved setup does not widen it. Only one paid test, including a re-test, can run in a setup at a time. A re-test runs in the setup of its Action. Re-tests run in batches of eight, up to 40 runs total. They stop early when the change from the baseline is clear at 90%. A baseline scenario that stopped after four agreeing runs is complete, but its range is wider than at eight runs, so a re-test from it can take one more batch to separate. For a reason Action, runs rotate across its recorded scenarios and stop early only when every scenario’s change is clear. Unstarted runs stopped at that point are reported separately from failures, and unused credits return through the normal settlement.

Results

Use the Source filter in Results to find Agent Picks results. Each comparison keeps its before-and-after counts, likely ranges and coverage. A re-test compares its pick rate with the baseline’s by the 90% interval on their difference, and shows 90% Wilson ranges. Re-tests started before this method keep the one they started with, and their results are never recalculated: they compare 95% ranges, show 95% ranges, and stop early when those ranges separate. Each result’s API method says which it used, v2 or v1, and Results notes Method changed when a list shows both. Shipping alone does not establish improvement. A later re-test sets the current result back to Still measuring until it finishes. These are observed pick results, not citation or traffic results, and they do not prove causation.