agentpicks run tests a coding agent installed on your machine. It gives the agent a category’s request, such as adding error tracking, in a starter app or your own repository inside a fresh Docker container, and writes what the agent did to a bundle on your disk: its transcript, its code changes and the requests it made through the container’s proxy. It then uploads the bundle to your DevTune project, which analyses the run.
A run uses your agent’s own sign-in or API key, so its model calls are charged to that account. It spends no DevTune credits. To keep bundles on your machine only, pass --no-upload.
Requirements
- Docker, running. Each run gets a fresh container with 2 CPUs and 4 GB of memory. It needs 20 GB free on Docker’s disk and may write up to 15 GB there.
- Node.js 20 or later.
- A DevTune project API key with the Agent Picks Run scope, on an Agent Picks Cloud, Local Runner or Enterprise plan, to upload. See Sign in to DevTune.
--no-uploadneeds neither. - A supported coding agent, installed and signed in on this machine:
Cursor and Muse Code can’t be tested on your machine:
agentpicks refuses them by name. Test Muse Code from Agent Picks in DevTune.
The container runs on your machine’s own architecture, x86_64 or arm64, including Apple Silicon. Rails isn’t available on arm64: on an arm64 machine, --stack rails and --stack rails-own-servers-eu are refused before anything is built. Choose another stack.
Sign in to DevTune
- Create a key under API Keys in DevTune for the project the runs belong to, with the Agent Picks Run scope. See API authentication.
- Run
agentpicks loginand paste the key at the prompt, which doesn’t echo it, or pipe it in.
login asks DevTune which project the key belongs to, and refuses a key without the Agent Picks Run scope. To save a key without checking it, pass --project <project id>.
The key and project are saved to agentpicks/credentials.json under $XDG_CONFIG_HOME, or else under %APPDATA% on Windows and ~/.config elsewhere. Only your user can read the file. agentpicks never prints the key.
The member who created the key needs DevTune management permission on the account and edit rights on the project. A key stops uploading when its creator leaves or loses either.
In CI
Set these instead of runninglogin. They override a saved login.
DEVTUNE_API_BASE_URL must be an HTTPS address ending in /api/v2.
Run a test
run is refused before anything starts. Pass --no-upload to run without one.
Before the first run, agentpicks prints the agent, its version and model, the test, what it passes and copies from your machine, the DevTune project it uploads to, and what each run needs from Docker. The first run builds the sandbox image; later runs reuse it.
Runs go one after another. Each run asks the category’s add request. Claude Code is stopped 60 seconds after agentpicks sees it install or wire in a tool of the category, and is limited to 100 turns. Every agent has 15 minutes to finish.
npx @devtune/agentpicks --version prints the version, and --help prints the usage.
An unknown category, stack or agent is refused with the values you can use. agentpicks exits 0 when every run wrote a bundle and, unless you passed --no-upload, every bundle uploaded. It exits 1 otherwise.
Stacks
Your repository
--repo <path> tests the git repository at that path at one commit: HEAD, or the one --ref names. Each run copies that commit’s committed files into its container. Uncommitted changes and untracked files aren’t included, so commit what you want tested. Your checkout is only read.
Dependencies install frozen from the lockfiles at the repository’s root. A package.json, pyproject.toml or Gemfile without its lockfile is refused before a container starts.
To upload a --repo run, the repository needs a remote with a network address: origin, or else its first remote. DevTune groups a repository’s runs by that remote. Without one, run is refused before anything starts: add an origin remote, or pass --no-upload.
Choosing the agent and model
agentpicks finds the agent on your PATH, reads its version, and installs that same version in the container. It reads the model from the agent’s own config: ANTHROPIC_MODEL or settings.json for Claude Code, config.toml for Codex, opencode.json for OpenCode, and settings.json for Pi. --model overrides it. Without either, Claude Code and Codex use their default model.
OpenCode and Pi need a provider and a model, such as --model anthropic/claude-sonnet-4-5, and call Anthropic or OpenAI directly.
Credentials
Your sign-in or API key reaches the agent in the container.agentpicks prints what it passes and copies before the first run.
A copied sign-in leaves out its refresh token, so the container can’t renew it or sign this machine out. It is deleted with the container.
The access token has to last the run. A sign-in that expires within 40 minutes is refused: run the agent on this machine so it renews, then try again.
Claude Code on macOS
Claude Code on macOS keeps its sign-in in the Keychain, whichagentpicks never reads, so the run is refused. Create a token instead:
ANTHROPIC_API_KEY works too.
Codex with a keyring
Codex set to keep its sign-in in the system keyring (cli_auth_credentials_store = "keyring" in config.toml) is refused the same way. Set OPENAI_API_KEY, or set cli_auth_credentials_store = "file" and run codex login.
The bundle
Each completed run writes one directory,<out>/<run id>/. A run that fails, or that you stop with Ctrl-C, writes nothing, and agentpicks removes its container.
Each artifact is written only when the run produced it:
The container’s public IP address is read from
https://api.ipify.org before setup. When it can’t be read, it is null in the manifest.
agentpicks prints each file’s size and the number of redactions when a run finishes. Bundle files are readable only by you.
Redaction
Artifacts are redacted before they are written, and each masked value reads[redacted]. Redaction masks the credentials the run used, credentials in URLs, private keys, common API key and token formats, and secret values the agent wrote into the app, such as a generated AUTH_SECRET in .env.local, wherever they then appear. It can miss a secret. To read a bundle before it leaves your machine, run with --no-upload, check the bundle, then send it with agentpicks upload.
A run that wrote more than 200 secrets writes no bundle, and no further runs start.
Upload to DevTune
Once a run’s bundle is written,run prints the bundle’s directory, each file it sends with its size, and the number of redactions in each artifact. It then uploads the bundle at once: manifest.json to DevTune, and each artifact straight to storage. redactions.json stays on your machine.
After the upload, run prints the run id, the state of its analysis, and the upload’s API address.
If DevTune refuses the upload, run prints DevTune’s message and the command that sends the bundle later, and starts no further runs. The bundle stays on disk either way. See Refusals.
Upload a bundle later
upload takes one bundle directory, or the --out directory of run to upload every bundle in it. It prints each bundle’s files, sizes and redactions before sending it.
Uploading a bundle again is safe. An upload that stopped partway sends only the artifacts DevTune hasn’t stored, and one that completed is left as it is. A bundle whose files changed after agentpicks wrote it is refused before anything is sent; the other bundles in the directory still upload.
What DevTune does with an upload
DevTune analyses an uploaded run with the same pipeline as the tests it runs itself: it records the agent’s pick, any tool the agent named as its choice without installing it, and the trail of what the agent searched for, fetched and installed. The upload’sanalysis reads pending until then, analysed once the run has its outcome, or failed with the reason when it can’t be analysed. A failed analysis isn’t retried. Uploaded runs don’t create Actions.
Runs that share an agent and its version, a model, a starter and its lock digest or a repository, a category and a request are one series. A series holds one test a UTC month. A self-run’s results compare only with runs in its own series: never with tests DevTune runs, and never with the public board. Read them over the API and MCP.
Plans and the monthly limit
Uploading needs an Agent Picks Cloud, Local Runner or Enterprise plan. Plus and Pro can’t upload; use--no-upload.
On Local Runner, each project can upload 500 runs for analysis a month. Cloud and Enterprise have no limit. The count resets at 00:00 UTC on the first of each month. Deleting an uploaded run doesn’t give it back.
A run counts when its upload completes, in the UTC month it completes, not the month the run started, and whether or not its analysis then succeeds. An upload that never completes doesn’t count, and uploading a completed bundle again doesn’t count twice.
Refusals
run and upload print DevTune’s message. Over the API, the response’s error names the kind of refusal, and most also carry a reason.
Clean up
agentpicks made, including a container an interrupted run left behind. Bundles stay where they were written. Building the image for a new version of agentpicks removes older versions’ images.