Losses
A scenario is a loss when a competing tool or Built its own was clearly picked more often than your tool: the 90% interval on the difference between their pick rates excludes zero. A scenario where the difference is not clear is level, not a loss. Only a competing tool or Built its own can make a loss; No provider chosen never does. Built its own runs that DevTune recorded only as a fallback guess are left out of the decision. For example, 1 pick out of 12 against a competitor’s 9 is a loss. 1 out of 4 against 3 is level: four runs cannot tell the two apart. Tests requested before this rule keep the rule they were decided by: a loss when a competing tool or Built its own was a top pick, ties included, and your tool was not picked in a majority of completed runs. Their losses and Actions are not recalculated.Actions
Open Actions to find Agent Picks Actions in the shared queue. Use the Source filter (AI Search only, Agent Picks only or Both) to choose the work shown. Both includes the sources you can access. AI Search tracking requires its own plan. Completed tests create two kinds of Action, unless the test is directional, such as a Quick test or a scenario re-run:- A loss Action for a losing test app, request, agent and model combination.
- A reason Action when the same recorded reason appears across several losing scenarios, such as a failed page fetch. It lists every affected scenario and its evidence.
Draft
Use the Action’s brief and draft controls to prepare a fix. Review the proposed content and approve it before publishing. If a page is locked, the suggestion names it and calls for supplementary content around it. When the brief is ready, select Copy as a coding-agent prompt to hand the fix to your own coding agent. The prompt contains the brief, what agents picked instead and how often, the target files or pages, the acceptance check and every locked page the fix steers around. Review the agent’s change before you ship it; copying the prompt publishes nothing. You can dismiss an Action and restore it later. If a newer Action already tracks the loss, restoring the older one is refused and the newer Action stays in the queue.Shipping and re-testing
- Publish the fix and select Mark live on the Action.
- Open the live Action and select Estimate a re-test under Re-test this fix.
- Review the estimate, then select Confirm and reserve credits.
Results
Use the Source filter in Results to find Agent Picks results. Each comparison keeps its before-and-after counts, likely ranges and coverage. A re-test compares its pick rate with the baseline’s by the 90% interval on their difference, and shows 90% Wilson ranges. Re-tests started before this method keep the one they started with, and their results are never recalculated: they compare 95% ranges, show 95% ranges, and stop early when those ranges separate. Each result’s APImethod says which it used, v2 or v1, and Results notes Method changed when a list shows both.
Shipping alone does not establish improvement. A later re-test sets the current result back to Still measuring until it finishes. These are observed pick results, not citation or traffic results, and they do not prove causation.