Skip to content

Audit pipeline

Audit commands scan a repo and file GitHub issues for problems they find. There are three audit types: docs, refactor, and tests. Each is a single-stage pipeline.

developer audit docs assistant
developer audit refactor assistant --limit 5
developer audit tests assistant --dry-run

How audits work

All three follow the same pattern:

  1. Clone/pull the repo and run setup
  2. Create a detached worktree from origin/main
  3. Run a Claude agent with read-only tools and output_model=AuditReport
  4. Validate the structured output (a list of AuditFinding objects)
  5. Create GitHub issues for each finding

The agent never creates issues directly. It returns structured data, Python makes the API calls.

With --dry-run, findings print to stdout instead of becoming issues. The --limit flag caps the number of findings (default is 3).

Audit types

docs

Checks documentation for drift from the actual codebase. The agent reads through docs and source code, looking for places where documentation describes behavior that no longer matches the code. Outdated API references, wrong file paths, missing new features, that kind of thing.

refactor

Runs the repo's configured quality tools first, then gives the agent the tool output as context. The agent uses that signal (plus its own codebase exploration) to identify refactoring opportunities. Dead code, duplicated logic, overly complex functions.

This is the only audit type that uses quality tool output as input to the agent.

tests

Evaluates test coverage against risk. The agent looks at what code does (is it reversible? does it touch external systems? does it handle money?) and whether the test coverage matches that risk level. Low-risk utility functions with no tests? Probably fine. Payment processing with no tests? That's a finding.

If the repo has a testing_philosophy in its config, the agent uses that to calibrate expectations.

Adding a new audit type

Two files to touch:

  1. Create a template at src/developer/templates/audit/<name>.md.j2
  2. Add a Click subcommand in src/developer/commands/audit.py that calls run_audit() with the template name

The shared run_audit() function handles cloning, worktree creation, agent invocation, output validation, and issue creation.