Skip to main content
The devtunehq/agentpicks action runs a setup from your repository in a GitHub Actions workflow, with the coding agent you install on the runner, and uploads each run to your DevTune project. Add it to a scheduled workflow to test the agents on a schedule, with no machine of your own. A run uses the agent’s own API key, so its model calls are charged to that account. It spends no DevTune credits.

Requirements

Create the API key

The key needs these scopes:
  1. Create a key under API Keys in DevTune for the project the runs belong to, with those scopes. See API authentication.
  2. Find the project’s id: it is data.project.id in the key’s connection, and data.scopes lists the scopes the key holds now.
  3. In the repository on GitHub, open Settings › Secrets and variables › Actions. Add the key as the secret DEVTUNE_API_KEY, and the project id as the variable DEVTUNE_PROJECT_ID. The project id isn’t secret.
The member who created the key needs the project Admin role. A key stops uploading when its creator leaves the team or stops being an Admin.

Commit a setup file

A setup file says what to test: the job, your tool and its competitors, the stacks or your repository, the agents and the number of runs. Write one on your machine with agentpicks init. Then link it to a DevTune setup with agentpicks push from your own machine, after agentpicks login, so its runs are listed on that setup. The sign-in needs Visibility and Actions Write; the CI key doesn’t need Actions Write. See Write a setup and Link a setup to DevTune.
For a new file with repo, push saves the stacks it detects in your repository, and the file keeps repo, so the action tests your repository. If push detects no supported stack, replace repo with stacks in the file, push it, and set repo: . on the action to test your repository. Keep a linked file in sync with DevTune. A run is refused before Docker starts when the file has changes not pushed, or when its DevTune setup changed since the file last saw it. Run agentpicks push or agentpicks pull on your own machine, signed in with agentpicks login, then commit the file.

Add the workflow

Save this as .github/workflows/agentpicks.yml. It runs agentpicks/error-tracking.yaml with Claude Code every Monday at 06:00 UTC, and whenever you start it from the Actions tab:
  • Pin a release tag, such as v0.4.10. A tag runs the same version of the agentpicks CLI every time.
  • Install and authenticate each agent the setup lists, or pick one with agent. Install it at a pinned version before the action; agentpicks installs the same version in its container. A run whose agent isn’t installed is refused.
  • Check out the repository first. The setup file is read from the checkout, and a setup with repo: . tests the checked-out commit. Only committed files are tested.

Agent credentials

Add the agent’s key as a repository secret and pass it to the action through env:, under one of these names: To use a Claude subscription, run claude setup-token on your machine and store the token it prints as CLAUDE_CODE_OAUTH_TOKEN. Other subscription sign-ins are saved on the machine where you log in, so a fresh runner has none to use.

Inputs

Set setup or category, not both. The action refuses both, or neither. A name in setup always runs the DevTune setup of that name. If the checkout’s agentpicks folder has a file of that name, the action refuses and names the file’s path to set instead. Set devtune-api-key and devtune-project-id together. If one is set and the other is empty, such as from a misnamed secret, the action refuses to upload. With both empty, the action uses DEVTUNE_API_KEY, DEVTUNE_PROJECT_ID and DEVTUNE_API_BASE_URL from the job’s environment. The action never prints the key. The action’s bundle-dir output is the absolute path of out, with one bundle per completed run. With no-upload: true, keep the bundles with actions/upload-artifact.

Where results appear

Each run uploads as soon as its bundle is written. The step’s log prints the run id and the state of its analysis. DevTune analyses the run, and the runs of a linked setup are listed on that setup in Agent Picks history. See Where uploaded runs are listed. The step fails when a run fails or DevTune refuses an upload, and the log prints DevTune’s message. See Refusals.

Other CI systems

Call the CLI directly. Install Docker, Node.js 20 or later and the agent, set DEVTUNE_API_KEY, DEVTUNE_PROJECT_ID and the agent’s key from the CI system’s secrets, and run the setup from the checkout at a pinned version:
See Run tests on your machine for every run option.