What a test measures
Every request ends: “Make any decisions yourself. If a service needs an account or keys, assume I’ll create them and leave placeholders.” Results show what an agent picks when it works for a team that will create the accounts and keys a service needs. Tests judged beforeagent-picks-classifier/18 ended each request with “Make any decisions yourself; I won’t be able to answer questions.” They show what an agent picked when it could not get an account. Trends mark the change Judging changed and don’t compare tests across it.
These rules decide what counts as a pick:
- A product the agent runs on its own servers counts as that product. The vendor’s own server image in Docker Compose is a pick of that product, not Built its own: for example self-hosted Supabase, Appwrite, PocketBase, Sentry, GlitchTip, Logto or Lago, or a Jenkins, GitLab or TeamCity server. Error tracking also reads Kamal accessories. A server image of a tool the category doesn’t measure makes no pick. Rails gems that keep or send errors themselves, such as Solid Errors and Exception Notification, are named picks too.
- A Sentry SDK counts as the server it sends to, when DevTune can tell. GlitchTip, Bugsink and Better Stack accept Sentry’s SDK. The run counts as GlitchTip or Bugsink when the change runs that server, and as GlitchTip or Better Stack when the DSN is on their hosted service. Any other DSN, or a placeholder, counts as Sentry. Two servers in one change are Not classified.
- A tool a project already has counts when the agent keeps using it. Some projects start with a tool in their files: Next.js with Supabase, Next.js with GitHub Actions, and Rails, whose starter includes a GitHub Actions workflow, Kamal and Solid Queue. Results count these projects like any other. Deleting the tool’s config, or leaving it unable to run (a workflow with every job commented out), stops counting it. Improve existing requests run only in projects that start with one of the category’s tools.
Classifier accuracy
A classifier reads each run’s code change and final answer and records its outcome. These figures show how often each classifier version was right on a labelled test set: recorded runs and agent answers where a person wrote down the right outcome. They are not measured on your runs or on production runs. The set is small and covers only the categories and jobs labelled so far, so read each percentage with its count. For figures from published test runs, see Precision by category. Tests of a covered category, including the public leaderboards, useagent-picks-classifier/18. Tests that describe a job use agent-picks-classifier/18+job/7. Earlier versions’ figures stay here for tests they judged. Run detail names the version that judged the run. REST and MCP return its figures in pick.classifier_accuracy; it is null for a version with no labelled figures, and for job tests that read named choices with a model-based judge, which this set does not measure.
- Right when reported is how often the outcome was correct when the classifier reported it.
- Named choices found is how often the classifier found a tool the agent named as its choice.
- Not classified, where one answer was right is how often the classifier reported Not classified although one outcome was correct. If the classifier avoided hard cases by not answering, this figure would rise.
agent-picks-classifier/12
agent-picks-classifier/12+job/7
agent-picks-classifier/13
agent-picks-classifier/13+job/7
agent-picks-classifier/14
agent-picks-classifier/14+job/7
agent-picks-classifier/15
agent-picks-classifier/15+job/7
agent-picks-classifier/16
agent-picks-classifier/16+job/7
agent-picks-classifier/17
agent-picks-classifier/17+job/7
agent-picks-classifier/18
agent-picks-classifier/18+job/7
Precision by category
These figures come from a labelled sample of DevTune’s own category tests: 40 completed runs per category from its latest public test, spread across outcomes and tools, and all 62 runs of the first CI/CD test. A person read each run’s code change, dependency files and transcript and wrote down the right outcome. They are not measured on your runs. A run nobody could decide is left out and counted separately. Each cell is how often an outcome was right when the classifier reported it, with the count. An outcome reported fewer than 10 times in a category shows only its count, because that is too few to state a rate. Not reported means the classifier never reported that outcome in the category’s sample. REST and MCP return the figures for a run’s category and judging version inpick.category_precision.