Skip to content

Audit pipeline

Audit commands scan a repo and file GitHub issues for problems they find. There are three audit types: docs, refactor, and tests. Each is a single-stage pipeline.

developer audit docs assistant
developer audit refactor assistant --limit 5
developer audit tests assistant --dry-run

How audits work

All three follow the same pattern:

  1. Clone the repo on first use, otherwise fetch
  2. Create a detached worktree of the default branch and run setup in it
  3. Run a Claude agent with the audit's tools (read-only, or read-and-run for audit tests) and output_model=AuditReport
  4. Validate the structured output (a list of AuditFinding objects)
  5. Create GitHub issues for each finding

The agent never creates issues directly. It returns structured data, Python makes the API calls.

With --dry-run, findings print to stdout instead of becoming issues. The --limit flag caps the number of findings (default is 3).

Audit types

docs

Checks documentation for drift from the actual codebase. The agent reads through docs and source code, looking for places where documentation describes behavior that no longer matches the code. Outdated API references, wrong file paths, missing new features, that kind of thing.

refactor

Runs the repo's configured quality tools first, then gives the agent the tool output as context. The agent uses that signal (plus its own codebase exploration) to identify refactoring opportunities. Dead code, duplicated logic, overly complex functions.

This is the only audit type that uses quality tool output as input to the agent.

tests

Evaluates test coverage against risk. The agent looks at what code does (is it reversible? does it touch external systems? does it handle money?) and whether the test coverage matches that risk level. Low-risk utility functions with no tests? Probably fine. Payment processing with no tests? That's a finding.

If the repo has a testing_philosophy in its config, the agent uses that to calibrate expectations.

Adding a new audit type

  1. Create a template at src/developer/templates/audit/<name>.md.j2
  2. Add an AuditSpec to AUDITS in src/developer/services/audit.py: the template, the tool profile, a one-line help text, and a function that gathers the template's extra variables from the worktree. The developer audit <name> command and the API's /audit/<name> endpoint come from that entry.

The shared run_audit() service handles the clone, the worktree, the agent run, output validation and issue creation.