Skip to content

Verification

The Architect proves its own work by running it, before registering anything. A workflow that looks plausible and a workflow that works are different things, and only one of them is checkable.

Two stages.

1. Derive acceptance criteria

One LLM call turns the user's intent plus the staged graph's declared inputs into an AcceptancePlan:

  • 2–6 explicit success criteria — what "this worked" means for this intent.
  • Concrete test inputs — real values, derived from the declared inputs.
  • A Fixture, when the workflow reads a file or a directory.

Fixtures

A workflow that reads a file cannot be tested against a sentence.

The run happens in a throwaway sandbox directory that the fixture script populates first. A source path with nothing behind it is a hard failure that names the fixture it needs — not a placeholder string handed to read_file so it can fail as "no such file".

That distinction is the whole point: the second produces a diagnosis about a missing file, which is true and useless. The first produces a diagnosis about a missing fixture, which is the actual problem.

2. Run it and score it

verify_workflow runs the staged package on those inputs — in a worker thread, because the runner is synchronous — and then splits on what happened:

Outcome How it is scored
The run failed A deterministic diagnosis from the node errors. No judge call — criteria cannot pass on a crashed run, so paying a model to say so is waste.
The run was clean An LLM judge scores it per criterion, and produces a diagnosis plus design suggestions for anything failing.

The judge fails closed

A criterion the judge does not rule on counts as failed. An unscored criterion is not a pass; treating it as one would make a judge that returned nothing look like a perfect result.

What the agent does with the report

The report is rendered for the agent's test_workflow tool, and the agent applies fixes with its normal graph-editing tools — update_node, add_node, rewiring depends_on.

This is deliberate: it is design revision, not field patching. A verification failure is evidence the design is wrong, and the repair path is the same one that built it.

See The Agent for the "fix the smallest thing first" rule that governs how it responds.

Verification is fingerprinted

An unchanged design is not re-run. The verification result is fingerprinted against the design it describes, so a build that loops back through validation without changing the graph does not pay for another full run and another judge call.

Change the graph and the fingerprint no longer matches, so the next check is real.

Gating

Whether verification blocks registration is the verify setting on ArchitectAgent — "required" by default.

ArchitectAgent(provider, verify="required")   # must pass before it registers
ArchitectAgent(provider, verify="encouraged") # prompted, not gated
ArchitectAgent(provider, verify="off")        # skipped

"required" is the default because "encouraged" is prompt-only, and a weaker model simply skipped the step: two bug-report transcripts went validate → ok → register with no test_workflow call at all.

The loop is bounded

After three judged failures (max_verification_attempts) the workflow registers anyway, carrying the loudest caveat in the codebase — the same trade already made for a test rig that cannot be set up.

It needs a bound because the model writes the criteria it is then judged against, and the judge fails closed. A more capable model writes a stricter bar: gpt-5.1 derived "no information not present in the source" for a summariser, which no reading of one output can certify and no prompt can promise. With an unbounded loop its only exits were grind forever or give up — and it gave up, turning a two-node summarise-and-title request into WorkflowInfeasible while smaller models built it.

Structural gates are unaffected: an invalid graph stays unregisterable however many attempts were spent. Only the judge can be overruled, and never silently.

Criteria that demand a guarantee or the absence of something unstated are dropped before the run for the same reason — they are aspirations, not tests. Narrowly: "exactly three sentences", "no more than 200 words" and "does not include the raw table" all survive.

A complete design is never "blocked"

declare_blocked is for a capability nothing can provide — a missing integration, a credential nobody supplied, an unsafe or self-contradictory request. It is not for a workflow that merely fails its own verification, and the tool refuses a build whose design is complete, valid and fully grounded, naming the alternative instead.

Registration tracks the design

A fix applied after register_workflow — which is exactly what a warned design review asks for — is re-saved to the registry when the build ends. register() snapshots to disk, so without this the last fix of a build was the one that never landed. A re-save runs the same gates; if the late edit broke something, the earlier good copy stays and the caller is told.

The API

from neurosurfer.architect.agent.verify import (
    AcceptanceCriterion, AcceptancePlan, Fixture,
    VerificationReport, derive_acceptance, verify_workflow,
)

Next