Skip to content

Testing

Philosophy

Developer tests the deterministic orchestration, not the agent's reasoning.

The agent is non-deterministic. It might triage the same issue differently on two runs. Testing that would be flaky and expensive. What is deterministic: the code that decides which stage runs next, the tool permissions that control what the agent can do, the git operations that commit and push code, the config validation that catches bad YAML.

That's what the tests cover. When triage returns action_direct, the pipeline must plan and implement; when the quality tools still fail after the last attempt, it must open a draft PR. Both are provably correct or not. The agent's judgment is out of scope.

Running tests

uv run pytest

This runs the full suite with coverage. The minimum coverage threshold is enforced by pytest-cov (see pyproject.toml for the current value). Coverage reports show missing lines so you know where gaps are.

Run a specific test file:

uv run pytest tests/test_resolve.py

Run a specific test:

uv run pytest tests/test_pipelines.py -k resolve

What's tested

Pipelines end to end -- tests/test_pipelines.py runs each pipeline through its service with a scripted agent, FakeGitHub and real git. There is one test per triage decision, the quality loop (passing at once, fixed by the agent, still failing into a draft PR), address with and without action items, and review. Each checks which stages ran, which sessions they continued, what they were shown, and what reached git and GitHub.

Stage runner -- a stage that returns no session stops the pipeline, and each stage's metrics are recorded under its name.

Config validation -- malformed YAML gets specific error messages naming the exact field that's wrong.

Git operations -- workspaces, commit, diff and push, against real repositories in temp directories. A fixture points git's https://github.com/ URLs at local bare repos, so clones, fetches and pushes run for real, including a push refused because someone else pushed first.

GitHub client -- GitHubClient against a stand-in GitHub REST API that serves responses recorded from the playground repo (tests/fixtures/github). Covers reading lists past the first page, both shapes of issue label, tokens from the installation that can reach the repo, and a new token for every push. Pipeline tests use FakeGitHub instead, which records every write.

Template rendering -- every stage template renders from the context its command actually builds. A contract test runs each CLI command down every pipeline branch with a scripted agent, and templates fail on any undefined variable.

CLI parsing -- URL formats parse correctly. Missing args produce usage errors.

API endpoints -- every endpoint against fake services, with tasks run synchronously: bearer-token auth, repo resolution (by name and owner/repo), error responses, one run per target at a time, and what a task's result and log hold.

Claude provider -- the SDK options each tool profile produces, the guard hook, the sandbox settings, and, from the SDK's real message types, each stage's cost, model usage and rate-limit warnings.

Run records -- a run leaves its transcript, metrics and result in runs/, or the error it stopped on.

Writing tests

  • Test public behaviour. Importing private names from other modules is a lint error (ruff PLC2701).
  • Don't test library behaviour, such as Pydantic validation, Click option parsing or dataclass defaults, and don't test states the code can't reach.
  • Fake at two seams: the agent (pass ScriptedProvider from tests/fakes.py as a service's agent=) and the GitHub client (FakeGitHub). Git tests use real repositories rather than mocking subprocess; the github fixture stands in for github.com.
  • Developer logs through logging, so check its output with caplog. Use capsys only for what a command prints itself, and CliRunner only to test Click commands.
  • The coverage floor in pyproject.toml is a floor, not a target.

Live tests

tests/live/ runs real agent sessions against a clone of the playground repo. They check the agent boundary against the real Claude CLI: - a read-only stage can't create files; - Bash can't write outside the worktree, push, or see GitHub secrets; - the repo's own tests can run inside the sandbox.

They are skipped unless you opt in, because they need network access, a Claude login and the sandbox prerequisites (see developer doctor):

DEVELOPER_LIVE=1 uv run pytest tests/live -m live --no-cov

Run them after any upgrade of claude-agent-sdk or the Claude CLI. DEVELOPER_LIVE_MODEL picks the model (default haiku).

What's not tested

Agent output quality. The tests use a scripted agent that returns canned responses. They verify that the pipeline does the right thing given a particular response, not that the agent produces good responses.

This is intentional. The pipeline is the safety net. If it works correctly, the worst an agent can do is produce bad code that a human reviews in a PR. It can't push or file issues itself, and Developer only pushes to a new branch or to the PR it was asked to address, never over someone else's commits.