> ## Documentation Index > Fetch the complete documentation index at: https://docs.devtune.ai/llms.txt > Use this file to discover all available pages before exploring further. # Run tests on your machine > Test the coding agents installed on your machine with the agentpicks CLI, upload each run to your DevTune project, and read the bundle each run writes. `agentpicks run` tests a coding agent installed on your machine. It gives the agent a category's request, such as adding error tracking, in a starter app or your own repository inside a fresh Docker container, and writes what the agent did to a bundle on your disk: its transcript, its code changes and the requests it made through the container's proxy. It then uploads the bundle to your DevTune project, which analyses the run. A run uses your agent's own sign-in or API key, so its model calls are charged to that account. It spends no DevTune credits. To keep bundles on your machine only, pass `--no-upload`. ## Requirements * **Docker**, running. Each run gets a fresh container with 2 CPUs and 4 GB of memory. It needs 20 GB free on Docker's disk and may write up to 15 GB there. * **Node.js 20** or later. * **A DevTune project API key** with the **Agent Picks Run** scope, on an Agent Picks Cloud, Local Runner or Enterprise plan, to upload. See [Sign in to DevTune](#sign-in-to-devtune). `--no-upload` needs neither. * **A supported coding agent**, installed and signed in on this machine: | Agent | `--agent` | | - | - | | Claude Code | `claude-code` | | Codex | `codex` | | OpenCode | `opencode` | | Pi | `pi` | Cursor and Muse Code can't be tested on your machine: `agentpicks` refuses them by name. Test Muse Code from [Agent Picks](https://docs.devtune.ai/agent-picks/overview.md) in DevTune. The container runs on your machine's own architecture, x86\_64 or arm64, including Apple Silicon. Rails isn't available on arm64: on an arm64 machine, `--stack rails` and `--stack rails-own-servers-eu` are refused before anything is built. Choose another stack. ## Sign in to DevTune ```bash theme={null} npx @devtune/agentpicks login ``` 1. Create a key under **API Keys** in DevTune for the project the runs belong to, with the **Agent Picks Run** scope. See [API authentication](https://docs.devtune.ai/api-reference/authentication.md#creating-an-api-key). 2. Run `agentpicks login` and paste the key at the prompt, which doesn't echo it, or pipe it in. `login` asks DevTune which project the key belongs to, and refuses a key without the Agent Picks Run scope. To save a key without checking it, pass `--project `. The key and project are saved to `agentpicks/credentials.json` under `$XDG_CONFIG_HOME`, or else under `%APPDATA%` on Windows and `~/.config` elsewhere. Only your user can read the file. `agentpicks` never prints the key. The member who created the key needs DevTune management permission on the account and edit rights on the project. A key stops uploading when its creator leaves or loses either. ### In CI Set these instead of running `login`. They override a saved login. | Variable | Value | | - | - | | `DEVTUNE_API_KEY` | The project API key | | `DEVTUNE_PROJECT_ID` | The project's id | | `DEVTUNE_API_BASE_URL` | The API address. Defaults to `https://devtune.ai/api/v2`. | `DEVTUNE_API_BASE_URL` must be an HTTPS address ending in `/api/v2`. ## Run a test ```bash theme={null} npx @devtune/agentpicks run --category error-tracking --stack nextjs --runs 3 ``` Without a saved login or the CI variables, `run` is refused before anything starts. Pass `--no-upload` to run without one. Before the first run, `agentpicks` prints the agent, its version and model, the test, what it passes and copies from your machine, the DevTune project it uploads to, and what each run needs from Docker. The first run builds the sandbox image; later runs reuse it. Runs go one after another. Each run asks the category's `add` request. Claude Code is stopped 60 seconds after `agentpicks` sees it install or wire in a tool of the category, and is limited to 100 turns. Every agent has 15 minutes to finish. | Flag | Value | | - | - | | `--category` | Required. `authentication`, `payments`, `transactional-email`, `databases`, `error-tracking` or `ci`. | | `--stack` | The starter app the agent works in; see [Stacks](#stacks). Required unless you pass `--repo`. | | `--repo` | Test the git repository at this path instead of a starter; see [Your repository](#your-repository). | | `--ref` | With `--repo`, the commit, tag or branch to test. Defaults to `HEAD`. | | `--runs` | How many runs. A whole number, 1 or more. Defaults to `1`. | | `--agent` | `claude-code`, `codex`, `opencode` or `pi`. Required when more than one is installed. | | `--model` | The model to ask for. Defaults to the model your agent's own config sets. OpenCode and Pi take `anthropic/` or `openai/`. | | `--out` | Where bundles are written. Defaults to `./agentpicks-runs`. | | `--no-upload` | Keep bundles on this machine only. Send them later with [`agentpicks upload`](#upload-a-bundle-later). | `npx @devtune/agentpicks --version` prints the version, and `--help` prints the usage. An unknown category, stack or agent is refused with the values you can use. `agentpicks` exits `0` when every run wrote a bundle and, unless you passed `--no-upload`, every bundle uploaded. It exits `1` otherwise. ### Stacks | `--stack` | Starter app | Categories | | - | - | - | | `empty` | An empty folder | All | | `nextjs` | Next.js | All | | `nextjs-noinstr` | Next.js without agent instructions | All | | `nextjs-supabase` | Next.js with Supabase | All | | `nextjs-github-actions` | Next.js with GitHub Actions | `ci` | | `tanstack` | TanStack Start | All | | `sveltekit` | SvelteKit | All | | `express` | Express | All | | `fastapi` | FastAPI | All | | `rails` | Rails, x86\_64 only | All | | `fastapi-aws` | FastAPI on AWS | `error-tracking` | | `rails-own-servers-eu` | Rails 8 on your own servers, EU data, x86\_64 only | `error-tracking` | ### Your repository `--repo ` tests the git repository at that path at one commit: `HEAD`, or the one `--ref` names. Each run copies that commit's committed files into its container. Uncommitted changes and untracked files aren't included, so commit what you want tested. Your checkout is only read. Dependencies install frozen from the lockfiles at the repository's root. A `package.json`, `pyproject.toml` or `Gemfile` without its lockfile is refused before a container starts. To upload a `--repo` run, the repository needs a remote with a network address: `origin`, or else its first remote. DevTune groups a repository's runs by that remote. Without one, `run` is refused before anything starts: add an `origin` remote, or pass `--no-upload`. ### Choosing the agent and model `agentpicks` finds the agent on your `PATH`, reads its version, and installs that same version in the container. It reads the model from the agent's own config: `ANTHROPIC_MODEL` or `settings.json` for Claude Code, `config.toml` for Codex, `opencode.json` for OpenCode, and `settings.json` for Pi. `--model` overrides it. Without either, Claude Code and Codex use their default model. OpenCode and Pi need a provider and a model, such as `--model anthropic/claude-sonnet-4-5`, and call Anthropic or OpenAI directly. ## Credentials Your sign-in or API key reaches the agent in the container. `agentpicks` prints what it passes and copies before the first run. | Agent | Passed from your environment, for the run only | Copied into the container | | - | - | - | | Claude Code | `ANTHROPIC_AUTH_TOKEN`, `ANTHROPIC_API_KEY` or `CLAUDE_CODE_OAUTH_TOKEN`, with `ANTHROPIC_BASE_URL`. Also read from the `env` block of Claude Code's `settings.json`. | Otherwise, the sign-in in `~/.claude/.credentials.json` | | Codex | `OPENAI_API_KEY` or `CODEX_API_KEY` | Otherwise, the sign-in in `~/.codex/auth.json` | | OpenCode, Pi | `ANTHROPIC_API_KEY` or `OPENAI_API_KEY`, for the model's provider | Nothing | A copied sign-in leaves out its refresh token, so the container can't renew it or sign this machine out. It is deleted with the container. The access token has to last the run. A sign-in that expires within 40 minutes is refused: run the agent on this machine so it renews, then try again. ### Claude Code on macOS Claude Code on macOS keeps its sign-in in the Keychain, which `agentpicks` never reads, so the run is refused. Create a token instead: ```bash theme={null} claude setup-token export CLAUDE_CODE_OAUTH_TOKEN= ``` An `ANTHROPIC_API_KEY` works too. ### Codex with a keyring Codex set to keep its sign-in in the system keyring (`cli_auth_credentials_store = "keyring"` in `config.toml`) is refused the same way. Set `OPENAI_API_KEY`, or set `cli_auth_credentials_store = "file"` and run `codex login`. ## The bundle Each completed run writes one directory, `//`. A run that fails, or that you stop with Ctrl-C, writes nothing, and `agentpicks` removes its container. | Path | Contents | | - | - | | `manifest.json` | The run's context: the agent's kind, version and model, your OS and architecture, the image digest, the container's public IP address, the stack, the category, the exact prompt, start and end times, and each artifact's size and SHA-256 | | `redactions.json` | How many redactions were made, in total and per artifact | | `artifacts/` | What the run captured, already redacted | Each artifact is written only when the run produced it: | Artifact | Contents | | - | - | | `transcript.jsonl` | The agent's transcript | | `agent.err` | The agent's error output | | `meta.json` | The agent's exit code and how long it ran | | `earlystop.json` | Whether the agent was stopped after a pick | | `diff.patch` | The agent's code changes, without `package-lock.json`, `pnpm-lock.yaml`, `yarn.lock` and `bun.lock` changes, cut at 400 KB | | `diff-truncated.txt` | The full size of the diff, when `diff.patch` was cut | | `diffstat.txt` | The files the agent changed, with lines added and removed, lockfiles included | | `source-files.json` | The JavaScript, TypeScript, Svelte, Vue and Astro files the agent changed, with the local files they import | | `ignored-env.txt` | The paths, not the contents, of git-ignored `.env` files the agent created or changed | | `package-json.txt` | The app's `package.json` files after the run | | `manifests.txt` | `Gemfile`, `requirements.txt`, `requirements-dev.txt` and `pyproject.toml` after the run | | `baseline-manifests.txt` | The same manifests and `package.json` files as the starter app had them before the run | | `lockfiles.txt` | The names of the lockfiles and Python and Go dependency files at the app's root after the run | | `deps.txt` | The runtimes, agent CLIs and app dependencies installed at setup, before the agent started | | `requests.jsonl` | One line per finished request the agent's processes sent through the container's proxy: method, host, path, status or error, response size and user agent, without other headers or bodies | | `requests-started.jsonl` | One line per request through the proxy as it started, including requests that never finished | | `llm-usage.jsonl` | Token usage of each model response from the Anthropic and OpenAI APIs | | `mitm.log` | The proxy's own log | | `setup.json` | How long setup took | | `env.json` | The agent instruction files the starter app shipped, and whether they were removed | | `starter-extra.txt` | Extra starter files for some stacks | The container's public IP address is read from `https://api.ipify.org` before setup. When it can't be read, it is `null` in the manifest. `agentpicks` prints each file's size and the number of redactions when a run finishes. Bundle files are readable only by you. ### Redaction Artifacts are redacted before they are written, and each masked value reads `[redacted]`. Redaction masks the credentials the run used, credentials in URLs, private keys, common API key and token formats, and secret values the agent wrote into the app, such as a generated `AUTH_SECRET` in `.env.local`, wherever they then appear. It can miss a secret. To read a bundle before it leaves your machine, run with `--no-upload`, check the bundle, then send it with `agentpicks upload`. A run that wrote more than 200 secrets writes no bundle, and no further runs start. ## Upload to DevTune Once a run's bundle is written, `run` prints the bundle's directory, each file it sends with its size, and the number of redactions in each artifact. It then uploads the bundle at once: `manifest.json` to DevTune, and each artifact straight to storage. `redactions.json` stays on your machine. After the upload, `run` prints the run id, the state of its analysis, and the upload's API address. If DevTune refuses the upload, `run` prints DevTune's message and the command that sends the bundle later, and starts no further runs. The bundle stays on disk either way. See [Refusals](#refusals). ### Upload a bundle later ```bash theme={null} npx @devtune/agentpicks upload ./agentpicks-runs npx @devtune/agentpicks upload ./agentpicks-runs/ ``` `upload` takes one bundle directory, or the `--out` directory of `run` to upload every bundle in it. It prints each bundle's files, sizes and redactions before sending it. Uploading a bundle again is safe. An upload that stopped partway sends only the artifacts DevTune hasn't stored, and one that completed is left as it is. A bundle whose files changed after `agentpicks` wrote it is refused before anything is sent; the other bundles in the directory still upload. ### What DevTune does with an upload DevTune analyses an uploaded run with the same pipeline as the tests it runs itself: it records the agent's pick, any tool the agent named as its choice without installing it, and the trail of what the agent searched for, fetched and installed. The upload's `analysis` reads `pending` until then, `analysed` once the run has its outcome, or `failed` with the reason when it can't be analysed. A failed analysis isn't retried. Uploaded runs don't create Actions. Runs that share an agent and its version, a model, a starter and its lock digest or a repository, a category and a request are one **series**. A series holds one test a UTC month. A self-run's results compare only with runs in its own series: never with tests DevTune runs, and never with the public board. Read them over the [API and MCP](https://docs.devtune.ai/agent-picks/api-and-mcp.md#self-runs). ### Plans and the monthly limit Uploading needs an Agent Picks Cloud, Local Runner or Enterprise plan. Plus and Pro can't upload; use `--no-upload`. On Local Runner, each project can upload 500 runs for analysis a month. Cloud and Enterprise have no limit. The count resets at 00:00 UTC on the first of each month. Deleting an uploaded run doesn't give it back. A run counts when its upload completes, in the UTC month it completes, not the month the run started, and whether or not its analysis then succeeds. An upload that never completes doesn't count, and uploading a completed bundle again doesn't count twice. ### Refusals `run` and `upload` print DevTune's message. Over the API, the response's `error` names the kind of refusal, and most also carry a `reason`. | Refusal | What to do | | - | - | | Uploading needs a DevTune project API key | Run `agentpicks login`, set `DEVTUNE_API_KEY` and `DEVTUNE_PROJECT_ID`, or pass `--no-upload`. | | The key isn't a DevTune API key | Copy the key again from **API Keys**. A key starts with `dtk_live_`. | | The key needs the Agent Picks Run scope | Edit the key under **API Keys** and select **Agent Picks Run**, or create a new key with it. | | DevTune doesn't accept the key (`401`) | The key was revoked or expired. Create a new key and run `agentpicks login` again. | | The repository has no remote (`repository_remote_missing`) | Add an `origin` remote with a network address and run the test again. A bundle keeps the remote it was written with, so adding one doesn't fix an existing bundle. To run without uploading, pass `--no-upload` to `run`. | | `plan_excludes_uploads` | Uploading needs an Agent Picks Cloud, Local Runner or Enterprise plan. | | `analysis_cap_reached` | The project has used its 500 uploads this month. Upload again after the time DevTune names, the start of the next UTC month, or move to Agent Picks Cloud. | | `forbidden`, with no `reason` | The key's creator can no longer manage the project. Use a key created by a member with DevTune management permission and edit rights on the project. | | `paused` | DevTune support has paused the account. Uploads wait until it is resumed; keep the bundles and upload them then. | | The bundle changed after `agentpicks` wrote it, or isn't a bundle `agentpicks` wrote (`invalid_manifest`) | Run the test again. A bundle can't be uploaded once its files change. | | `upload_expired`, `upload_deleted`, `artifact_size_mismatch` | The upload wasn't completed within 24 hours, was deleted, or stored a file at the wrong size, and its files are being removed. Run `agentpicks upload` again at least 2 hours 15 minutes after the last attempt; if it is still refused, try again 15 minutes later. | | An artifact is larger than DevTune takes (`artifact_too_large`) | The run can't be uploaded. Run the test again. | | `artifact_missing` | A file didn't reach storage. Run `agentpicks upload` again; it sends the files DevTune hasn't stored, then completes the upload. | | `not_found` | The project has no upload with that run id, or it was deleted. Run `agentpicks upload` again. | | `unavailable` | The account lost Agent Picks, or the project is being deleted, during the upload. Check the account's Agent Picks plan, then run `agentpicks upload` again. | | `series_busy` | Another upload changed the series at the same time. Run `agentpicks upload` again. | | `run_exists` | Another project already has a run with this run id. Upload the bundle to the project it was first uploaded to. | | `unknown_category`, `unknown_request`, `unknown_stack` | DevTune doesn't offer the bundle's category, request or stack. Run the test with another. | | Couldn't reach DevTune | Check the connection, then run `agentpicks upload` with the bundle's directory. | ## Clean up ```bash theme={null} npx @devtune/agentpicks cleanup ``` Removes the containers and images `agentpicks` made, including a container an interrupted run left behind. Bundles stay where they were written. Building the image for a new version of `agentpicks` removes older versions' images. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.