devtunehq/agentpicks action runs a setup from your repository in a GitHub Actions workflow, with the coding agent you install on the runner, and uploads each run to your DevTune project. Add it to a scheduled workflow to test the agents on a schedule, with no machine of your own.
A run uses the agent’s own API key, so its model calls are charged to that account. It spends no DevTune credits.
Requirements
- A setup file committed to the repository. See Commit a setup file.
- A Linux runner with Docker and Node.js 20 or later.
ubuntu-latesthas both. - A project API key with the scopes below, on a plan that can upload. See Plans and the monthly limit.
- An API key or token for the agent you test. See Agent credentials.
Create the API key
The key needs these scopes:- Create a key under API Keys in DevTune for the project the runs belong to, with those scopes. See API authentication.
- Find the project’s id: it is
data.project.idin the key’s connection, anddata.scopeslists the scopes the key holds now. - In the repository on GitHub, open Settings › Secrets and variables › Actions. Add the key as the secret
DEVTUNE_API_KEY, and the project id as the variableDEVTUNE_PROJECT_ID. The project id isn’t secret.
Commit a setup file
A setup file says what to test: the job, your tool and its competitors, the stacks or your repository, the agents and the number of runs. Write one on your machine withagentpicks init. Then link it to a DevTune setup with agentpicks push from your own machine, after agentpicks login, so its runs are listed on that setup. The sign-in needs Visibility and Actions Write; the CI key doesn’t need Actions Write. See Write a setup and Link a setup to DevTune.
repo, push saves the stacks it detects in your repository, and the file keeps repo, so the action tests your repository. If push detects no supported stack, replace repo with stacks in the file, push it, and set repo: . on the action to test your repository.
Keep a linked file in sync with DevTune. A run is refused before Docker starts when the file has changes not pushed, or when its DevTune setup changed since the file last saw it. Run agentpicks push or agentpicks pull on your own machine, signed in with agentpicks login, then commit the file.
Add the workflow
Save this as.github/workflows/agentpicks.yml. It runs agentpicks/error-tracking.yaml with Claude Code every Monday at 06:00 UTC, and whenever you start it from the Actions tab:
- Pin a release tag, such as
v0.4.10. A tag runs the same version of theagentpicksCLI every time. - Install and authenticate each agent the setup lists, or pick one with
agent. Install it at a pinned version before the action;agentpicksinstalls the same version in its container. A run whose agent isn’t installed is refused. - Check out the repository first. The setup file is read from the checkout, and a setup with
repo: .tests the checked-out commit. Only committed files are tested.
Agent credentials
Add the agent’s key as a repository secret and pass it to the action throughenv:, under one of these names:
To use a Claude subscription, run
claude setup-token on your machine and store the token it prints as CLAUDE_CODE_OAUTH_TOKEN. Other subscription sign-ins are saved on the machine where you log in, so a fresh runner has none to use.
Inputs
Setsetup or category, not both. The action refuses both, or neither.
A name in
setup always runs the DevTune setup of that name. If the checkout’s agentpicks folder has a file of that name, the action refuses and names the file’s path to set instead.
Set devtune-api-key and devtune-project-id together. If one is set and the other is empty, such as from a misnamed secret, the action refuses to upload. With both empty, the action uses DEVTUNE_API_KEY, DEVTUNE_PROJECT_ID and DEVTUNE_API_BASE_URL from the job’s environment. The action never prints the key.
The action’s bundle-dir output is the absolute path of out, with one bundle per completed run. With no-upload: true, keep the bundles with actions/upload-artifact.
Where results appear
Each run uploads as soon as its bundle is written. The step’s log prints the run id and the state of its analysis. DevTune analyses the run, and the runs of a linked setup are listed on that setup in Agent Picks history. See Where uploaded runs are listed. The step fails when a run fails or DevTune refuses an upload, and the log prints DevTune’s message. See Refusals.Other CI systems
Call the CLI directly. Install Docker, Node.js 20 or later and the agent, setDEVTUNE_API_KEY, DEVTUNE_PROJECT_ID and the agent’s key from the CI system’s secrets, and run the setup from the checkout at a pinned version:
run option.