Repeatable instructions
YAML stores prompt IDs, instructions, repository location, routing, verification commands, and limits in a reviewable file.
Engineering case study · Controlled agent execution
AgentBatchRunner
Local engineering infrastructure for repeatable coding-agent work: explicit YAML tasks, verified attempts, persistent state, and review decisions before a folder pipeline moves forward.
Source-reviewed at commit 4b82cba (opens in a new tab). Implementation evidence, not a production benchmark.
02 / The problem
A prompt history is not an execution record.
Repeatedly copying instructions into an agent leaves the operator to track which task ran, which provider handled it, what changed, and whether a failed check was ever repaired. Interruptions turn that informal process into a recovery problem.
AgentBatchRunner makes that work explicit. A batch runs prompts sequentially; the shared core records each attempt and executes the configured verification commands. Folder pipelines add ordering, dependency checks, review artifacts, and next-file decisions.
Failures with context
A retry includes the original task and the failed command, exit code, output, and timeout details instead of a context-free “try again.”
An inspectable handoff
JSON state and Markdown reports preserve the task, provider, attempts, and result for the next operator decision.
Batch execution and artifact boundaries (opens in a new tab)
03 / Architecture
One execution core. Two operator interfaces.
Console CLI / Windows WPF UI → shared .NET 8 Core
- Load & route
PromptFileLoader,EffectiveAgentPolicy, and executable preflight resolve valid work and providers. - Checkpoint
BatchRunnerasksGitCheckpointManagerto record the starting Git state and create a branch reference. - Invoke an adapter
IAgentAdapterdispatches Claude Code, Codex CLI, or the built-in dry-run adapter. - Verify the attempt
VerificationRunnerexecutes configured shell commands and returns their results to the retry loop.
State & evidence throughout
RunStateStore persists configuration, routing, task and attempt JSON. ReportGenerator produces the run summary and final Markdown report. Events feed the GUI.
Folder-level orchestration
PipelineFolderRunner wraps batch execution. PipelineReviewRunner produces review evidence; NextPipelineFileSelector applies eligibility and advancement rules.
04 / Execution lifecycle
Task success and pipeline approval are different states.
Inside a prompt batch
YAML validation and executable preflight happen before task checkpoints. Labels below use the real RunStatus values; invocation and verification are operations within Running.
PendingTask announced in the configured sequence.
RunningCheckpoint → agent invocation → configured verification.
SucceededAgent succeeds and all configured verification commands pass.
UnverifiedSuccessAgent succeeds with no verification commands. Explicitly weaker than verified success.
Failed → retryA failed attempt feeds diagnostics back into the next invocation while the normal attempt budget remains.
NeedsHumanReviewNormal attempts are exhausted. The failed attempt history and final diff remain available.
Stop paths: RateLimited without an eligible fallback, ToolchainFailure, and explicit Blocked, NeedsHumanDecision, PrerequisiteMissing, or Canceled outcomes are handled separately from ordinary verification retries.
NeedsHumanReview does not itself halt the remaining prompts in a plain batch. Folder orchestration evaluates the batch outcome before advancing to another file.Inside a reviewed folder pipeline
The happy path uses PipelineFileStatus, a separate model from individual task status.
PendingDiscovered in the queue.
EligibleDependencies, gates, and agent availability allow selection.
RunningThe selected YAML batch executes.
ExecutionSucceededBatch has Succeeded or UnverifiedSuccess outcomes.
ReviewingA separate agent invocation produces review artifacts.
ApprovedParsed review approves the file; next-file rules still apply.
CompletedWithoutReviewIf review is not required, execution follows this branch and the next-file decision requires confirmation.
ApprovedWithWarningsDoes not auto-advance by default. The core exposes a separate policy option for allowing warnings.
ReviewFailed / BlockedInvalid review evidence or blocking findings prevent normal advancement. Other explicit stop verdicts retain their own status.
Approved review is machine-produced evidence. It is not proof that a person reviewed the code, nor is ExecutionSucceeded proof that verification commands existed.05 / YAML task model
Make the contract visible before execution.
Sanitized example · illustrative repository
project: Sample.Repository
repoPath: C:/work/sample-repository
defaultAgent: codex
defaultMaxRetries: 2
autoSwitchOnRateLimit: false
prompts:
- id: P001
title: Add a focused regression test
prompt: >
Add a test for the documented edge case.
Preserve existing public behavior.
verify:
- dotnet build
- dotnet test
maxRetries: 2
agentTimeoutSeconds: 1800
verifyTimeoutSeconds: 900prompt- The instruction sent to the selected agent. Scope and behavior constraints are part of the task definition.
repoPath- The shared working directory for the batch, agent process, Git operations, and verification. The example path is a placeholder, not a private repository location.
verify- Ordered shell commands supplied by the operator. Build and test are examples; the runner does not infer a project's checks.
defaultAgent/agent- A task can override the YAML default. An explicit run-level override takes precedence over both.
maxRetries- Despite the name, this implementation uses the value as the maximum number of normal attempts, including the first. Here, that means two attempts, not three.
- Checkpoints and folder metadata
- Task checkpoints are runner behavior, not a YAML checkpoint toggle. Optional
pipelinemetadata defines dependency, review, gate, and next-file rules.
06 / Verification model
The agent’s explanation is not the acceptance test.
Commands produce the result
The runner executes the configured commands in order, captures stdout, stderr, exit code, duration, and timeout information, and stops at the first failed command.
Failure informs the next attempt
A failed agent invocation or verification command becomes retry feedback. Output in that feedback is capped at the last 24,000 characters.
Missing checks stay visible
An empty verification list becomes UnverifiedSuccess. It is not silently relabeled Succeeded, but plain-batch resume still treats it as a completed task.
Verification and review answer different questions. Commands establish only the checks they actually perform. A folder review adds a parsed JSON verdict and a separate Markdown artifact; neither guarantees correctness or replaces code review.
07 / Checkpoint / recovery
Keep the evidence. Make recovery deliberate.
Git checkpoint, not automatic rollback
Before a task, the runner creates an agentbatchrunner/... branch at the current HEAD without checking it out. It records Git status, saves an initial unstaged diff when dirty, and later captures a final unstaged diff. It does not auto-commit, reset, or restore the worktree.
Resume from stored run state
The CLI loads the normalized config and run summary, restores routing, skips Succeeded and UnverifiedSuccess tasks, and appends attempts for remaining work. Existing checkpoint references and attempt folders are retained.
Batch evidence
.agentbatchrunner/runs/<run-id>/
run-config.normalized.json
run-routing.json
run-summary.json
final-report.md
tasks/<task-id>/
checkpoint.txt
status.json
git-diff-before.patch
git-diff-after.patch
attempts/attempt-N/Folder evidence
.agentbatchrunner/pipelines/<run-id>/
pipeline-state.json
pipeline-summary.json
pipeline-report.md
queue.json
execution-diffs/
generated-reviews/
review-runs/
reviews/Representative paths from the source. These are an artifact map, not captured run results.
A checkpoint is not a complete backup. A branch references a commit; plain git diff does not capture staged changes or untracked file contents. Partial agent edits remain in place, including after a provider switch. An interrupted execution with no run summary requires inspection rather than a silent replay.
08 / Provider routing
Resolve the provider once, then record every change.
- 1. Run override
Explicit CLI
--agentor GUI global override. - 2. Prompt agent
The individual task's
agentvalue. - 3. YAML default
The batch's
defaultAgent. - 4. Reject missing routing
No implicit provider when all three are absent.
Three implemented adapters
Claude Code and Codex are invoked as local CLIs. The dry-run adapter records the prompt without calling either provider; configured verification commands and runner Git/state operations still run.
Switches are explicit policy
Automatic rate-limit fallback is off by default. Configured fallbacks must pass executable preflight. Manual pending switches apply at prompt boundaries and do not interrupt the current process. Cross-provider switches start a fresh session and retain the current worktree.
Reports distinguish configured, base, effective, and attempt agents. Rate-limit attempts do not consume the normal attempt budget; provider switches have their own limit. Switching providers can change data handling, cost, and output behavior.
09 / Human-in-the-loop
The operator chooses the level of autonomy.
Folder pipeline controlOne selected file Default Opt-in
Successful execution does not authorize the next phase.
The folder runner separates execution results, review verdicts, dependency satisfaction, gate decisions, and next-file selection. A person chooses the mode, reviews the evidence, and decides how to resolve blockers.
Manual
Run the selected eligible file and stop at the boundary.
Confirm Each
Execute and review one file, then pause for approval to run the next recommendation.
Auto Advance
Requires an allowed review verdict, canAutoAdvance, an eligible next target, and policy checks. Automatic transitions are capped at 20 by default.
- Skipping is not approval.
SkippedByUserdoes not satisfy a dependency or gate. - Manual completion is explicit. Dependency satisfaction is opt-in; gate approval requires a separate explicit override with evidence and audit information.
- Ambiguity stops automation. Unresolved choices, missing prerequisites, and blocking review verdicts require operator attention.
- Review remains accountable.
Approvedcan come from the review agent. Auto Advance does not require a human click at every boundary, and a plain batch has no per-task approval gate.
Eligibility and advancement policy (opens in a new tab)
Confirmation, manual-action, and gate tests (opens in a new tab)
10 / Interfaces
CLI precision and a Windows operator console.
Validate, execute, resume, report
dotnet run --project src/AgentBatchRunner -- validate prompts.yaml
dotnet run --project src/AgentBatchRunner -- run prompts.yaml
dotnet run --project src/AgentBatchRunner -- resume --run-id <run-id>
dotnet run --project src/AgentBatchRunner -- report --run-id <run-id>The folder command adds planning, running, status, resume, reporting, and manual queue actions. CLI validation checks configuration; execution performs agent preflight.
Inspect and control the run
The Batch File tab exposes YAML selection, validation, routing, Run/Cancel, live logs, task diagnostics, and report links. The Folder Pipeline tab exposes the queue, execution/review agents, dependencies, gates, next-file approval, pause/stop, and manual status actions.
GUI validation includes executable preflight. The global routing selector is disabled during execution; pending-agent switches are handled separately.
Interface descriptions are verified from code. No screenshots, video, or recorded run logs were found in the inspected repository; no simulated screenshot or live-demo claim is presented.
11 / Failure scenarios
Different failures need different recovery paths.
Agent or verification timeout
The process runner attempts process-tree termination, records exit code 124, and feeds the failed attempt into normal retry handling. Detached children may require manual cleanup.
Verification fails repeatedly
Each failed check is saved with its output. Exhausting normal attempts produces NeedsHumanReview. It does not trigger an automatic Git reset.
Provider reports a rate limit
Stop as RateLimited, or use an available, preflighted fallback when explicitly enabled and within the switch limit. Old attempts remain; the replacement provider starts fresh.
Missing or incompatible executable
Preflight blocks invocation before checkpoints. A detected runtime toolchain failure is non-retryable and untouched prompts are marked Skipped.
Invalid review or review mutation
Malformed/mismatched review JSON or a detected product Git-state change becomes ReviewFailed. Changes are preserved for inspection, not reset.
Interrupted run or Git checkpoint error
Existing artifacts support diagnosis and boundary-based resume when a summary exists. A missing summary or failed branch creation requires operator inspection; there is no automatic conflict-resolution or rollback engine.
Process timeout and cancellation handling (opens in a new tab)
12 / Security / safety
Controls with explicit boundaries.
Trusted local commands
Verification uses PowerShell on Windows and /bin/sh elsewhere. repoPath is a working directory, not a filesystem sandbox. Review YAML and commands before execution; the runner is not an untrusted-code execution service.
Provider permissions are separate
The published defaults use Claude acceptEdits and Codex workspace-write. Folder reviews request Claude plan or Codex read-only and compare product Git state before/after. These controls rely on external CLI behavior and do not prove OS-level isolation.
Local evidence can be sensitive
Pattern-based redaction exists for selected secret-like strings. Prompts, configuration, logs, state, and reports are local files; the patterns do not guarantee complete secret removal. Provider authentication remains with the installed CLIs.
Git mutation is bounded in the runner
The checkpoint service creates branch references and captures status/diffs. It does not auto-commit, reset, delete product files, or force-push. Invoked agents and verification commands can still modify the working tree.
The CLI refuses administrator/root execution. This is a verified CLI check, not a claim that every entry point or child process is universally sandboxed. Review mutation detection compares Git status/diff, not a complete filesystem snapshot.
Redaction implementation and limits (opens in a new tab)
Read-only review options and Git-state guard (opens in a new tab)
13 / Trade-offs
More control means more state to operate.
What the architecture gains
Sequential execution, explicit routing, repeatable command checks, and durable evidence make a multi-step agent workflow easier to inspect and continue. Shared Core logic gives CLI and GUI a common execution model.
What it adds
Operators must maintain task definitions, verification commands, provider installations, routing policy, and stored run artifacts. Review adds another agent invocation and more decisions; shell checks are only as useful as their coverage.
Where it fits
Repeated repository maintenance, modernization steps, or an ordered engineering pipeline where a task needs a known starting point, observable checks, and a reviewable handoff.
Where it is excessive
A small exploratory edit or a single interactive question may not justify YAML, persistent run history, and a separate review workflow. Provider-independent orchestration also cannot remove provider-specific behavior.
These are architectural trade-offs inferred from the implementation, not measured productivity or cost claims.
14 / Limitations
A local tool, with operator-owned acceptance.
- Maturity: the README describes an MVP. This case study demonstrates implemented code paths and committed tests, not production adoption, uptime, or independently measured reliability.
- Verification is configured, not guaranteed: missing commands yield
UnverifiedSuccess, which is skipped by plain-batch resume and accepted as execution success by the folder layer. Require meaningful checks and review for consequential work. - Recovery is not rollback: branch references and unstaged diffs are incomplete backups. Resume depends on saved state; partial edits remain. A plain batch can continue after a task exhausts its normal attempt budget.
- Session continuity is conditional: Claude needs a captured session ID. Codex prefers an exact ID, but the published adapter falls back to
resume --lastwhen none is available; concurrent sessions can make that ambiguous. - Platform and execution scope: WPF is Windows-only. Generated folder-review verification commands target Windows PowerShell. The runner is sequential, not a distributed worker service or a multi-user approval system.
- Safety is bounded: shell commands are trusted input, redaction is pattern-based, and provider permission modes plus Git-state checks are not complete isolation. Detached child processes can outlive cancellation.
- Future work, not current claims: stronger content-level recovery snapshots, stricter review isolation, and deterministic session handling across every provider/version would need separate implementation and validation.
15 / Evidence
Trace the claims back to the implementation.
Project and task definitions
The public implementation at commit 4b82cba, with configuration and usage examples.
Runner, checks, checkpoints
Inspect the normal attempt budget, verification results, saved state, and actual Git commands.
Pipeline decisions
Execution modes, review outcomes, dependency eligibility, and explicit manual actions.
Behavior contracts
Test source covers retries, no-verification outcomes, resume, routing, confirmation boundaries, manual gates, and rejected review mutations. No fresh AgentBatchRunner test-pass claim is made here.
Evidence reviewed on 22 September 2026 against the public commit above and the matching local source. Unpublished local terminal-classification changes are excluded. No customer deployment, quantified productivity gain, live provider acceptance, or media evidence is claimed.