Several setups
A project can have several setups: one for each category, and as many custom jobs as you need. Each setup has its own tool, competitors, test apps, agents and schedule, and its own tests, results and Actions. For example, a company that sells both CI and container builds can test each category in one project. Settings › Agent Picks lists the project’s setups with their tool, competitor count and schedule.- Add a setup opens the setup steps: starting point, competitors, then test apps and agents. A category that already has a setup stays in the list, disabled, with Edit that setup. Start from scratch is always available.
- Edit opens that setup’s form and test schedule. Saving changes that setup only.
- Remove asks you to confirm. Removing a setup stops its schedule and removes its suggested Actions that nobody has accepted. Accepted Actions stay, and its tests and results stay readable from their links and test history. Removing the only setup leaves the project with no Agent Picks setup until you add one.
Covered category
- Choose Use a category as your starting point.
- Select the category and your tool. If your tool is absent, choose Your tool isn’t listed. Results can show what agents pick, but cannot measure your tool until it is recognized in the test.
- Select up to six competitors.
- Choose the test apps agents start in, then the agents and models.
- Select Save setup.
Your own job
Choose Start from scratch to define a job instead of using a category. If your product has no covered category, setup opens the job editor.- Describe the work in one sentence, such as “Add AI traffic analytics to this app.” This becomes the build request.
- Review the drafted recommendation request and, when relevant, the hosted-service request. Edit, reset or leave them out before saving. Use plain requests without links, code or environment variable names.
- Confirm your tool and the competitors to compare.
- Review how DevTune recognizes each tool: its packages, environment names, API hosts, code patterns and domains. Use Draft again or correct the fields yourself if a draft is missing or wrong.
- Review Tools that run inside another. Confirm the suggestions that are right, or add one when your tool runs inside, or replaces part of, another tool in the job, such as faster runners inside GitHub Actions or a plugin inside a framework. A run that uses both then counts as the tool that runs inside. Remove any that are wrong.
- Review How we decide Built its own. Leave fields empty when they do not apply. Ask a teammate familiar with your integration to check package names and code patterns if needed.
- Choose test apps and models, then save.
If the app says runs for a custom job are unavailable, your setup stays saved.
You can run its starter test from Agent Picks when the app offers it.
Edit requests, the ending and recognition
Every setup, a category copy or your own job, can be edited in Settings › Agent Picks on every plan. Select the setup, change the fields below, then select Save setup. Tests you run after saving use the new setup; earlier tests keep what they ran.- Requests. Edit any request’s text. Select Leave this request out to stop asking it; its text stays, read-only, and Ask this request again brings it back as it was. A setup asks at least one request, so the last one can’t be left out.
- Improve request. A copy of a category with an improve request keeps it. You can reword it or leave it out, but you can’t add one to a setup that doesn’t have one.
- Request ending. The text added after every request. Leave it empty to send each request as written.
- Recognition. How DevTune recognizes each tool in a run: its packages for each registry, config file names, import patterns and domains. Expand a tool to edit them. How we decide Built its own holds the setup’s building blocks.
- A request or the ending can’t name a tool you’re testing, by its name, an alias or its domain: “Requests can’t name a tool you’re testing. Describe the job instead.” Describe the job; a scenario that needs a tool belongs in the test app the agent starts in.
- A tool’s recognition can’t match the files a selected test app begins with, unless that test app already has the tool. Every run there would count as a pick.
- A building-block package can’t also be one of a tool’s packages.
- Every pattern must be one DevTune can run.
Test apps
A test app is the code an agent starts in. Choose test apps that match where buyers use your product. Options include an empty folder and the official starters for Next.js, TanStack Start, SvelteKit, Express, FastAPI and Rails. Some test apps vary the starting state, such as Next.js with an existing tool. The starter test uses one plain framework starter, or the empty folder. During onboarding, mark the test app to use for the starter test. Paid tests use the saved test app selections.Hosting and requirements
Some test apps add context to a starter: the host’s own deploy setup committed in the test app, or a sentence agents are told before every request, such as where customer data must stay. They’re listed unticked under Hosting and requirements, below the other test apps. Each row says what the test app starts with or what agents are told, any request it doesn’t ask, and how many runs it adds to a Full test. Tick one to include it in your tests; untick it to take it out. A new setup leaves them unticked, so the default test doesn’t change. The section appears only when there’s at least one such test app. When a test app like this is revised, it’s replaced by a new one. A setup that already has the earlier test app keeps it, and its tests and re-tests keep running there; a new setup can only choose the replacement.Agents and models
Paid tests offer these models:
Pi and OpenCode need Agent Picks Pro or Enterprise. On Plus they show as locked, with a link to Billing for members who can change the plan.
If you move from Pro to Plus, your saved Pi and OpenCode choices stay saved, but paid tests leave them out. The quote and the Agents section say which agents were left out. Upgrade to Pro to test them again, or remove them from the Agents section.
The defaults are Claude Sonnet 5.5 and GPT-6.1 Sol. A project can test up to seven agent and model pairs. Selecting two models for one agent tests both separately and increases the run count. These choices do not change the starter test’s fixed models.
Save changes before requesting a paid test. If the app warns that saved models are unavailable, review the current choices and save again.