> ## Documentation Index > Fetch the complete documentation index at: https://docs.devtune.ai/llms.txt > Use this file to discover all available pages before exploring further. # Measurement methodology > Learn what an Agent Picks test measures, how it classifies outcomes, and how accurate and precise the classifier is in each category. Use these figures to assess the classifier that recorded a test's outcomes. For pick rates, install rates and run coverage, see [Reading results](https://docs.devtune.ai/agent-picks/reading-results.md). ## What a test measures Every request ends: "Make any decisions yourself. If a service needs an account or keys, assume I'll create them and leave placeholders." Results show what an agent picks when it works for a team that will create the accounts and keys a service needs. Tests judged before `agent-picks-classifier/18` ended each request with "Make any decisions yourself; I won't be able to answer questions." They show what an agent picked when it could not get an account. Trends mark the change **Judging changed** and don't compare tests across it. These rules decide what counts as a pick: * **A product the agent runs on its own servers counts as that product.** The vendor's own server image in Docker Compose is a pick of that product, not **Built its own**: for example self-hosted Supabase, Appwrite, PocketBase, Sentry, GlitchTip, Logto or Lago, or a Jenkins, GitLab or TeamCity server. Error tracking also reads Kamal accessories. A server image of a tool the category doesn't measure makes no pick. Rails gems that keep or send errors themselves, such as Solid Errors and Exception Notification, are named picks too. * **A Sentry SDK counts as the server it sends to, when DevTune can tell.** GlitchTip, Bugsink and Better Stack accept Sentry's SDK. The run counts as GlitchTip or Bugsink when the change runs that server, and as GlitchTip or Better Stack when the DSN is on their hosted service. Any other DSN, or a placeholder, counts as Sentry. Two servers in one change are **Not classified**. * **A tool a project already has counts when the agent keeps using it.** Some projects start with a tool in their files: Next.js with Supabase, Next.js with GitHub Actions, and Rails, whose starter includes a GitHub Actions workflow, Kamal and Solid Queue. Results count these projects like any other. Deleting the tool's config, or leaving it unable to run (a workflow with every job commented out), stops counting it. **Improve existing** requests run only in projects that start with one of the category's tools. ## Classifier accuracy A classifier reads each run's code change and final answer and records its outcome. These figures show how often each classifier version was right on a labelled test set: recorded runs and agent answers where a person wrote down the right outcome. They are not measured on your runs or on production runs. The set is small and covers only the categories and jobs labelled so far, so read each percentage with its count. For figures from published test runs, see [Precision by category](#precision-by-category). Tests of a covered category, including the public leaderboards, use `agent-picks-classifier/18`. Tests that describe a job use `agent-picks-classifier/18+job/7`. Earlier versions' figures stay here for tests they judged. Run detail names the version that judged the run. REST and MCP return its figures in `pick.classifier_accuracy`; it is `null` for a version with no labelled figures, and for job tests that read named choices with a model-based judge, which this set does not measure. * **Right when reported** is how often the outcome was correct when the classifier reported it. * **Named choices found** is how often the classifier found a tool the agent named as its choice. * **Not classified, where one answer was right** is how often the classifier reported **Not classified** although one outcome was correct. If the classifier avoided hard cases by not answering, this figure would rise. Labelled cases also include answers that name two tools, where **Not classified** is the correct outcome. An outcome the classifier never reported on the set has no row. ### `agent-picks-classifier/12` | Measure | Value | | - | - | | Labelled cases | 4 | | Installed: right when reported | 100.0% (2 of 2) | | Built its own: right when reported | 100.0% (2 of 2) | | Not classified, where one answer was right | 0.0% (0 of 4) | ### `agent-picks-classifier/12+job/7` | Measure | Value | | - | - | | Labelled cases | 328 | | Installed: right when reported | 96.7% (29 of 30) | | Named as its choice: right when reported | 97.6% (82 of 84) | | No tool found: right when reported | 100.0% (9 of 9) | | Named choices found | 91.1% (82 of 90) | | Not classified, where one answer was right | 10.4% (14 of 134) | ### `agent-picks-classifier/13` | Measure | Value | | - | - | | Labelled cases | 26 | | Installed: right when reported | 100.0% (9 of 9) | | Built its own: right when reported | 100.0% (9 of 9) | | No tool found: right when reported | 100.0% (8 of 8) | | Not classified, where one answer was right | 0.0% (0 of 26) | ### `agent-picks-classifier/13+job/7` | Measure | Value | | - | - | | Labelled cases | 328 | | Installed: right when reported | 96.7% (29 of 30) | | Named as its choice: right when reported | 97.6% (82 of 84) | | No tool found: right when reported | 100.0% (9 of 9) | | Named choices found | 91.1% (82 of 90) | | Not classified, where one answer was right | 10.4% (14 of 134) | ### `agent-picks-classifier/14` | Measure | Value | | - | - | | Labelled cases | 54 | | Installed: right when reported | 100.0% (9 of 9) | | Built its own: right when reported | 100.0% (27 of 27) | | No tool found: right when reported | 100.0% (18 of 18) | | Not classified, where one answer was right | 0.0% (0 of 54) | ### `agent-picks-classifier/14+job/7` | Measure | Value | | - | - | | Labelled cases | 328 | | Installed: right when reported | 96.7% (29 of 30) | | Named as its choice: right when reported | 97.6% (82 of 84) | | No tool found: right when reported | 100.0% (9 of 9) | | Named choices found | 91.1% (82 of 90) | | Not classified, where one answer was right | 10.4% (14 of 134) | ### `agent-picks-classifier/15` | Measure | Value | | - | - | | Labelled cases | 112 | | Installed: right when reported | 100.0% (51 of 51) | | Built its own: right when reported | 100.0% (35 of 35) | | No tool found: right when reported | 100.0% (25 of 25) | | Not classified, where one answer was right | 0.0% (0 of 111) | ### `agent-picks-classifier/15+job/7` | Measure | Value | | - | - | | Labelled cases | 328 | | Installed: right when reported | 96.7% (29 of 30) | | Named as its choice: right when reported | 97.6% (82 of 84) | | No tool found: right when reported | 100.0% (9 of 9) | | Named choices found | 91.1% (82 of 90) | | Not classified, where one answer was right | 10.4% (14 of 134) | ### `agent-picks-classifier/16` | Measure | Value | | - | - | | Labelled cases | 112 | | Installed: right when reported | 100.0% (51 of 51) | | Built its own: right when reported | 100.0% (35 of 35) | | No tool found: right when reported | 100.0% (25 of 25) | | Not classified, where one answer was right | 0.0% (0 of 111) | ### `agent-picks-classifier/16+job/7` | Measure | Value | | - | - | | Labelled cases | 328 | | Installed: right when reported | 96.7% (29 of 30) | | Named as its choice: right when reported | 97.6% (82 of 84) | | No tool found: right when reported | 100.0% (9 of 9) | | Named choices found | 91.1% (82 of 90) | | Not classified, where one answer was right | 10.4% (14 of 134) | ### `agent-picks-classifier/17` | Measure | Value | | - | - | | Labelled cases | 112 | | Installed: right when reported | 100.0% (51 of 51) | | Built its own: right when reported | 100.0% (35 of 35) | | No tool found: right when reported | 100.0% (25 of 25) | | Not classified, where one answer was right | 0.0% (0 of 111) | ### `agent-picks-classifier/17+job/7` | Measure | Value | | - | - | | Labelled cases | 328 | | Installed: right when reported | 96.7% (29 of 30) | | Named as its choice: right when reported | 97.6% (82 of 84) | | No tool found: right when reported | 100.0% (9 of 9) | | Named choices found | 91.1% (82 of 90) | | Not classified, where one answer was right | 10.4% (14 of 134) | ### `agent-picks-classifier/18` | Measure | Value | | - | - | | Labelled cases | 145 | | Installed: right when reported | 100.0% (81 of 81) | | Built its own: right when reported | 100.0% (32 of 32) | | No tool found: right when reported | 100.0% (29 of 29) | | Not classified, where one answer was right | 0.0% (0 of 142) | ### `agent-picks-classifier/18+job/7` | Measure | Value | | - | - | | Labelled cases | 328 | | Installed: right when reported | 96.7% (29 of 30) | | Named as its choice: right when reported | 97.6% (82 of 84) | | No tool found: right when reported | 100.0% (9 of 9) | | Named choices found | 91.1% (82 of 90) | | Not classified, where one answer was right | 10.4% (14 of 134) | ## Precision by category These figures come from a labelled sample of DevTune's own category tests: 40 completed runs per category from its latest public test, spread across outcomes and tools, and all 62 runs of the first CI/CD test. A person read each run's code change, dependency files and transcript and wrote down the right outcome. They are not measured on your runs. A run nobody could decide is left out and counted separately. Each cell is how often an outcome was right when the classifier reported it, with the count. An outcome reported fewer than 10 times in a category shows only its count, because that is too few to state a rate. **Not reported** means the classifier never reported that outcome in the category's sample. REST and MCP return the figures for a run's category and judging version in `pick.category_precision`. ### Published test runs, `agent-picks-classifier/15` | Category | Labelled runs | Installed | Built its own | No provider chosen | No tool found | | - | - | - | - | - | - | | Authentication | 40 | 88.2% (30 of 34) | 5 of 5, too few for a rate | Not reported | 0 of 1, too few for a rate | | CI/CD | 62 | 100.0% (61 of 61) | Not reported | Not reported | 0 of 1, too few for a rate | | Databases & BaaS | 40 | 100.0% (16 of 16) | 8 of 8, too few for a rate | 8 of 8, too few for a rate | 0 of 8, too few for a rate | | Error tracking | 40 | 100.0% (20 of 20) | 100.0% (20 of 20) | Not reported | Not reported | | Payments | 40 | 100.0% (39 of 39) | Not reported | Not reported | 0 of 1, too few for a rate | | Transactional email | 40 | 95.0% (19 of 20) | 60.0% (6 of 10) | 100.0% (10 of 10) | Not reported | ### Published test runs, `agent-picks-classifier/16` | Category | Labelled runs | Installed | Built its own | No provider chosen | No tool found | | - | - | - | - | - | - | | Authentication | 40 | 100.0% (35 of 35) | 5 of 5, too few for a rate | Not reported | Not reported | | CI/CD | 62 | 100.0% (62 of 62) | Not reported | Not reported | Not reported | | Databases & BaaS | 40 | 100.0% (16 of 16) | 100.0% (16 of 16) | 8 of 8, too few for a rate | Not reported | | Error tracking | 40 | 100.0% (20 of 20) | 100.0% (20 of 20) | Not reported | Not reported | | Payments | 40 | 100.0% (40 of 40) | Not reported | Not reported | Not reported | | Transactional email | 40 | 100.0% (19 of 19) | 54.5% (6 of 11) | 100.0% (10 of 10) | Not reported | ### Published test runs, `agent-picks-classifier/17` | Category | Labelled runs | Installed | Built its own | No provider chosen | No tool found | | - | - | - | - | - | - | | Authentication | 40 | 100.0% (35 of 35) | 5 of 5, too few for a rate | Not reported | Not reported | | CI/CD | 62 | 100.0% (62 of 62) | Not reported | Not reported | Not reported | | Databases & BaaS | 40 | 100.0% (16 of 16) | 100.0% (16 of 16) | 8 of 8, too few for a rate | Not reported | | Error tracking | 40 | 100.0% (20 of 20) | 100.0% (20 of 20) | Not reported | Not reported | | Payments | 40 | 100.0% (40 of 40) | Not reported | Not reported | Not reported | | Transactional email | 40 | 100.0% (19 of 19) | 60.0% (6 of 10) | 100.0% (11 of 11) | Not reported | ### Published test runs, `agent-picks-classifier/18` | Category | Labelled runs | Installed | Built its own | No provider chosen | No tool found | | - | - | - | - | - | - | | Authentication | 40 | 100.0% (35 of 35) | 5 of 5, too few for a rate | Not reported | Not reported | | CI/CD | 62 | 100.0% (62 of 62) | Not reported | Not reported | Not reported | | Databases & BaaS | 40 | 100.0% (16 of 16) | 100.0% (16 of 16) | 8 of 8, too few for a rate | Not reported | | Error tracking | 40 | 100.0% (20 of 20) | 100.0% (20 of 20) | Not reported | Not reported | | Payments | 40 | 100.0% (40 of 40) | Not reported | Not reported | Not reported | | Transactional email | 40 | 100.0% (19 of 19) | 60.0% (6 of 10) | 100.0% (11 of 11) | Not reported | This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.