> ## Documentation Index
> Fetch the complete documentation index at: https://docs.devtune.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Run tests in GitHub Actions

> Run an Agent Picks setup on a schedule in GitHub Actions and upload each run to your DevTune project.

The `devtunehq/agentpicks` action runs a setup from your repository in a GitHub Actions workflow, with the coding agent you install on the runner, and uploads each run to your DevTune project. Add it to a scheduled workflow to test the agents on a schedule, with no machine of your own.

A run uses the agent's own API key, so its model calls are charged to that account. It spends no DevTune credits.

## Requirements

* **A setup file** committed to the repository. See [Commit a setup file](#commit-a-setup-file).
* **A Linux runner** with Docker and Node.js 20 or later. `ubuntu-latest` has both.
* **A project API key** with the scopes below, on a plan that can upload. See [Plans and the monthly limit](https://docs.devtune.ai/agent-picks/run-on-your-machine.md#plans-and-the-monthly-limit).
* **An API key or token for the agent** you test. See [Agent credentials](#agent-credentials).

## Create the API key

The key needs these scopes:

| Scope | Needed to |
| - | - |
| **Agent Picks Run** | Upload each run |
| **Visibility** | Upload the runs of a linked setup file (one with an `id`), run a DevTune setup by its name, or run a `category` |
| **Actions Write** | Run a `category` the project has no copy of yet: the run creates the copy first |

1. Create a key under **API Keys** in DevTune for the project the runs belong to, with those scopes. See [API authentication](https://docs.devtune.ai/api-reference/authentication.md#creating-an-api-key).
2. Find the project's id: it is `data.project.id` in the key's connection, and `data.scopes` lists the scopes the key holds now.
   ```bash theme={null}
   curl -s -H "Authorization: Bearer <your API key>" https://devtune.ai/api/v2/connection
   ```
3. In the repository on GitHub, open **Settings › Secrets and variables › Actions**. Add the key as the secret `DEVTUNE_API_KEY`, and the project id as the variable `DEVTUNE_PROJECT_ID`. The project id isn't secret.

The member who created the key needs the project Admin role. A key stops uploading when its creator leaves the team or stops being an Admin.

## Commit a setup file

A setup file says what to test: the job, your tool and its competitors, the stacks or your repository, the agents and the number of runs. Write one on your machine with `agentpicks init`. Then link it to a DevTune setup with `agentpicks push` from your own machine, after `agentpicks login`, so its runs are listed on that setup. The sign-in needs Visibility and Actions Write; the CI key doesn't need Actions Write. See [Write a setup](https://docs.devtune.ai/agent-picks/run-on-your-machine.md#write-a-setup) and [Link a setup to DevTune](https://docs.devtune.ai/agent-picks/run-on-your-machine.md#link-a-setup-to-devtune).

```bash theme={null}
npx @devtune/agentpicks init
npx @devtune/agentpicks login
npx @devtune/agentpicks push error-tracking
git add agentpicks/error-tracking.yaml
git commit -m "Add the Agent Picks error-tracking setup"
```

For a new file with `repo`, `push` saves the stacks it detects in your repository, and the file keeps `repo`, so the action tests your repository. If `push` detects no supported stack, replace `repo` with `stacks` in the file, push it, and set `repo: .` on the action to test your repository.

Keep a linked file in sync with DevTune. A run is refused before Docker starts when the file has changes not pushed, or when its DevTune setup changed since the file last saw it. Run `agentpicks push` or `agentpicks pull` on your own machine, signed in with `agentpicks login`, then commit the file.

## Add the workflow

Save this as `.github/workflows/agentpicks.yml`. It runs `agentpicks/error-tracking.yaml` with Claude Code every Monday at 06:00 UTC, and whenever you start it from the **Actions** tab:

```yaml theme={null}
name: Agent Picks
on:
  schedule:
    - cron: '0 6 * * 1'
  workflow_dispatch:

permissions:
  contents: read

jobs:
  agentpicks:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - run: npm install -g @anthropic-ai/claude-code@2.1.323
      - uses: devtunehq/agentpicks@v0.4.10
        with:
          setup: agentpicks/error-tracking.yaml
          agent: claude-code
          devtune-api-key: ${{ secrets.DEVTUNE_API_KEY }}
          devtune-project-id: ${{ vars.DEVTUNE_PROJECT_ID }}
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
```

* **Pin a release tag**, such as `v0.4.10`. A tag runs the same version of the `agentpicks` CLI every time.
* **Install and authenticate each agent the setup lists, or pick one with `agent`.** Install it at a pinned version before the action; `agentpicks` installs the same version in its container. A run whose agent isn't installed is refused.
* **Check out the repository first.** The setup file is read from the checkout, and a setup with `repo: .` tests the checked-out commit. Only committed files are tested.

## Agent credentials

Add the agent's key as a repository secret and pass it to the action through `env:`, under one of these names:

| Agent | `--agent` | Variable |
| - | - | - |
| Claude Code | `claude-code` | `ANTHROPIC_API_KEY`, `ANTHROPIC_AUTH_TOKEN` or `CLAUDE_CODE_OAUTH_TOKEN` |
| Codex | `codex` | `OPENAI_API_KEY` or `CODEX_API_KEY` |
| OpenCode, Pi | `opencode`, `pi` | `ANTHROPIC_API_KEY` with an `anthropic/` model, or `OPENAI_API_KEY` with an `openai/` model |

To use a Claude subscription, run `claude setup-token` on your machine and store the token it prints as `CLAUDE_CODE_OAUTH_TOKEN`. Other subscription sign-ins are saved on the machine where you log in, so a fresh runner has none to use.

## Inputs

Set `setup` or `category`, not both. The action refuses both, or neither.

| Input | Value |
| - | - |
| `setup` | A setup file's path in the checkout, such as `agentpicks/error-tracking.yaml`, or the name of a DevTune setup with no file, as `agentpicks list` shows it. A job, or a copy that asks an add request of its own, runs only from a committed file. |
| `category` | A built-in category to test with no setup file, with `stack` or `repo`. Uploading, it runs the project's copy of the category, and refuses a copy that asks its own add request: commit that copy as a setup file and set `setup`. |
| `stack` | A stack to test in, in place of the setup's stacks or repo. Set `stack` or `repo`, not both. |
| `repo` | A path in the checkout to test, such as `.`, in place of the setup's stacks or repo. |
| `runs` | Runs per agent and stack, in place of the setup's `runs`. |
| `agent` | The agent to test, in place of the setup's agents: `claude-code`, `codex`, `opencode` or `pi`. |
| `model` | The model to ask for. OpenCode and Pi need one, such as `anthropic/claude-sonnet-4-5`. |
| `out` | Where bundles are written. Defaults to `./agentpicks-runs`. |
| `no-upload` | `true` keeps the bundles on the runner and uploads nothing. Defaults to `false`. |
| `devtune-api-key` | The project API key, from a secret, with the Agent Picks Run scope; Visibility too for a linked setup file, a DevTune setup's name or a category, and Actions Write for a category the project has no copy of yet. |
| `devtune-project-id` | The project's id. |
| `devtune-api-base-url` | The API address, ending in `/api/v2`. Defaults to `https://devtune.ai/api/v2`. |

A name in `setup` always runs the DevTune setup of that name. If the checkout's `agentpicks` folder has a file of that name, the action refuses and names the file's path to set instead.

Set `devtune-api-key` and `devtune-project-id` together. If one is set and the other is empty, such as from a misnamed secret, the action refuses to upload. With both empty, the action uses `DEVTUNE_API_KEY`, `DEVTUNE_PROJECT_ID` and `DEVTUNE_API_BASE_URL` from the job's environment. The action never prints the key.

The action's `bundle-dir` output is the absolute path of `out`, with one bundle per completed run. With `no-upload: true`, keep the bundles with `actions/upload-artifact`.

## Where results appear

Each run uploads as soon as its bundle is written. The step's log prints the run id and the state of its analysis. DevTune analyses the run, and the runs of a linked setup are listed on that setup in Agent Picks history. See [Where uploaded runs are listed](https://docs.devtune.ai/agent-picks/run-on-your-machine.md#where-uploaded-runs-are-listed).

The step fails when a run fails or DevTune refuses an upload, and the log prints DevTune's message. See [Refusals](https://docs.devtune.ai/agent-picks/run-on-your-machine.md#refusals).

## Other CI systems

Call the CLI directly. Install Docker, Node.js 20 or later and the agent, set `DEVTUNE_API_KEY`, `DEVTUNE_PROJECT_ID` and the agent's key from the CI system's secrets, and run the setup from the checkout at a pinned version:

```bash theme={null}
npm install -g @anthropic-ai/claude-code@2.1.323
npx --yes @devtune/agentpicks@0.4.10 run error-tracking --agent claude-code
```

See [Run tests on your machine](https://docs.devtune.ai/agent-picks/run-on-your-machine.md) for every `run` option.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.