Skip to main content
For a new project, choose Agent Picks during setup and enter your product name and domain. For an existing project, open Agent Picks and select Set up Agent Picks. To review or change a saved setup, open the project’s Settings › Agent Picks. Only members who can manage the project can save changes.

Several setups

A project can have several setups: one for each category, and as many custom jobs as you need. Each setup has its own tool, competitors, test apps, agents and schedule, and its own tests, results and Actions. For example, a company that sells both CI and container builds can test each category in one project. Settings › Agent Picks lists the project’s setups with their tool, competitor count and schedule.
  • Add a setup opens the setup steps: starting point, competitors, then test apps and agents. A category that already has a setup stays in the list, disabled, with Edit that setup. Start from scratch is always available.
  • Edit opens that setup’s form and test schedule. Saving changes that setup only.
  • Remove asks you to confirm. Removing a setup stops its schedule and removes its suggested Actions that nobody has accepted. Accepted Actions stay, and its tests and results stay readable from their links and test history. Removing the only setup leaves the project with no Agent Picks setup until you add one.
When a project has several setups, the Agent Picks tabs show one at a time. Use the Setup picker at the right of the tabs to switch. Agent Picks opens on the setup you last viewed in this browser, or on the first one added. With one setup the picker doesn’t appear.

Covered category

  1. Choose Use a category as your starting point.
  2. Select the category and your tool. If your tool is absent, choose Your tool isn’t listed. Results can show what agents pick, but cannot measure your tool until it is recognized in the test.
  3. Select up to six competitors.
  4. Choose the test apps agents start in, then the agents and models.
  5. Select Save setup.
A category supplies its tools and requests. Requests ask agents to add an integration, recommend a tool, or use a hosted service. Categories without a hosted option use the first two requests. Your setup is your project’s own copy of DevTune’s setup for the category: later changes DevTune makes to the category don’t change your setup. To change a new copy before its first test, open Review requests and recognition on the last step. Untouched, the setup saves the category as it is. You can edit it later in Settings › Agent Picks either way. Some categories also have an improve request, which asks agents to improve a tool the test app already has, such as making a slow build faster. It runs only in test apps that already have one of the category’s tools set up. If your project has no finished test, Agent Picks can show the category’s results. The page labels them Category, with the date and run count. They are not a test of your project.

Your own job

Choose Start from scratch to define a job instead of using a category. If your product has no covered category, setup opens the job editor.
  1. Describe the work in one sentence, such as “Add AI traffic analytics to this app.” This becomes the build request.
  2. Review the drafted recommendation request and, when relevant, the hosted-service request. Edit, reset or leave them out before saving. Use plain requests without links, code or environment variable names.
  3. Confirm your tool and the competitors to compare.
  4. Review how DevTune recognizes each tool: its packages, environment names, API hosts, code patterns and domains. Use Draft again or correct the fields yourself if a draft is missing or wrong.
  5. Review Tools that run inside another. Confirm the suggestions that are right, or add one when your tool runs inside, or replaces part of, another tool in the job, such as faster runners inside GitHub Actions or a plugin inside a framework. A run that uses both then counts as the tool that runs inside. Remove any that are wrong.
  6. Review How we decide Built its own. Leave fields empty when they do not apply. Ask a teammate familiar with your integration to check package names and code patterns if needed.
  7. Choose test apps and models, then save.
When a tool’s own site says it runs inside another tool in the job, its draft suggests that relationship and shows the sentence and page it came from. Select Confirm to use it, or Dismiss to drop it; a suggestion changes nothing until you confirm it. A dismissed suggestion, or a relationship you remove, stays dropped after you save. Drafting a tool again can suggest it again. Without a relationship, a run that uses both tools is Not classified, and its run detail names both tools with a link to this setting. A relationship applies to tests run after you save. If you change the job sentence, redraft the requests or select Confirm requests before saving. A saved job compares with its own earlier tests and has no category results.
If the app says runs for a custom job are unavailable, your setup stays saved. You can run its starter test from Agent Picks when the app offers it.

Edit requests, the ending and recognition

Every setup, a category copy or your own job, can be edited in Settings › Agent Picks on every plan. Select the setup, change the fields below, then select Save setup. Tests you run after saving use the new setup; earlier tests keep what they ran.
  • Requests. Edit any request’s text. Select Leave this request out to stop asking it; its text stays, read-only, and Ask this request again brings it back as it was. A setup asks at least one request, so the last one can’t be left out.
  • Improve request. A copy of a category with an improve request keeps it. You can reword it or leave it out, but you can’t add one to a setup that doesn’t have one.
  • Request ending. The text added after every request. Leave it empty to send each request as written.
  • Recognition. How DevTune recognizes each tool in a run: its packages for each registry, config file names, import patterns and domains. Expand a tool to edit them. How we decide Built its own holds the setup’s building blocks.
Saving checks the setup, and says beside the field what to fix when it can’t save:
  • A request or the ending can’t name a tool you’re testing, by its name, an alias or its domain: “Requests can’t name a tool you’re testing. Describe the job instead.” Describe the job; a scenario that needs a tool belongs in the test app the agent starts in.
  • A tool’s recognition can’t match the files a selected test app begins with, unless that test app already has the tool. Every run there would count as a pick.
  • A building-block package can’t also be one of a tool’s packages.
  • Every pattern must be one DevTune can run.
If someone else saves the same setup while you’re editing it, your save stops and asks you to reload, so your changes never overwrite theirs unseen. Reload, then make your changes again. If you run fast track again on a project whose copy you’ve edited, it leaves your setup as it is and says so.

Test apps

A test app is the code an agent starts in. Choose test apps that match where buyers use your product. Options include an empty folder and the official starters for Next.js, TanStack Start, SvelteKit, Express, FastAPI and Rails. Some test apps vary the starting state, such as Next.js with an existing tool. The starter test uses one plain framework starter, or the empty folder. During onboarding, mark the test app to use for the starter test. Paid tests use the saved test app selections.

Hosting and requirements

Some test apps add context to a starter: the host’s own deploy setup committed in the test app, or a sentence agents are told before every request, such as where customer data must stay. They’re listed unticked under Hosting and requirements, below the other test apps. Each row says what the test app starts with or what agents are told, any request it doesn’t ask, and how many runs it adds to a Full test. Tick one to include it in your tests; untick it to take it out. A new setup leaves them unticked, so the default test doesn’t change. The section appears only when there’s at least one such test app. When a test app like this is revised, it’s replaced by a new one. A setup that already has the earlier test app keeps it, and its tests and re-tests keep running there; a new setup can only choose the replacement.

Agents and models

Paid tests offer these models: Pi and OpenCode need Agent Picks Pro or Enterprise. On Plus they show as locked, with a link to Billing for members who can change the plan. If you move from Pro to Plus, your saved Pi and OpenCode choices stay saved, but paid tests leave them out. The quote and the Agents section say which agents were left out. Upgrade to Pro to test them again, or remove them from the Agents section. The defaults are Claude Sonnet 5.5 and GPT-6.1 Sol. A project can test up to seven agent and model pairs. Selecting two models for one agent tests both separately and increases the run count. These choices do not change the starter test’s fixed models. Save changes before requesting a paid test. If the app warns that saved models are unavailable, review the current choices and save again.