Agentic Fitness Functions: When the Evaluator Has Judgement

An API change came through review a while back. Every automated check was green: the contract tests passed, the schema diff was clean, consumers were untouched. A human reviewer read it anyway and pointed out that the domain decomposition had shifted. Responsibility that belonged in one bounded context had quietly moved into another, and the contract was intact because the contract has no vocabulary for where a boundary sits.

It was caught in review and it was cheap to fix. That is the honest ending, and it is also the uncomfortable part. It was cheap because a person happened to be looking that day, at that diff, with enough context loaded to notice. That is not a control. That is luck with a good reputation. Deterministic fitness functions are excellent at the questions you can express as a predicate, and I have written before about catching architectural drift before it reaches production. "Did the boundary between these two contexts move, and does that still match the decision we wrote down?" is not one of them.

There is now a name and a shape for the check that does ask that question. An agentic fitness function uses a calibrated AI agent as the evaluator, expresses its criteria as a rubric instead of an assertion, and returns a structured verdict rather than a pass or a fail. This post is about building one, and about the single design rule that makes it safe to have: it advises, it never blocks.

The Passport Gate and the Customs Officer

At the border there are two checks and they are not the same kind of check. The automated gate reads the chip, matches the face, and either opens or does not. It is fast, binary, and it blocks. It has no opinion about you. Then there is the customs officer standing past the gate, who pulls a traveller aside because the passport is entirely valid and the story does not add up. The officer cannot express the reason as a predicate. That is precisely why the officer is standing there.

What the officer produces is a written note: a risk level, what they observed, what they inferred. They cannot revoke a passport. They flag the traveller for a supervisor and someone with authority decides. And before that officer works a lane alone, they shadow experienced staff and have their calls reviewed, because an officer who stops every nervous tourist is as useless as one who waves through the person they should have stopped.

Your deterministic fitness functions are the gate. Your agentic fitness function is the customs officer, and its verdict is the incident note rather than the refusal of entry.

The analogy breaks in two places worth naming. A real officer has legal authority to detain, and reaches a bounded decision in one sitting. An agentic evaluator has neither. It can only escalate, and its verdict is a continuous score that re-runs against every subsequent change rather than a question settled once.

What Changes When the Evaluator Has Judgement

The split worth holding onto is this: deterministic checks block, agentic checks advise and escalate. An agentic fitness function is an architecture governance check whose evaluator is a calibrated AI agent, whose criteria are expressed as an analytic rubric, and whose output is a structured verdict carrying evidence, a confidence level, and a rationale.

Three things follow from that.

Verdicts are probabilistic, so treat them that way

A verdict arrives as a fitness score with a confidence level attached, and neither number is a threshold. A middling score at middling confidence does not become a red build at any cutoff you care to pick. It is a number that says look here, and the value of it collapses the moment you dress it up as a gate.

The rubric is not a rulebook for the agent

AGENTS.md, donated to the Linux Foundation's Agentic AI Foundation in December 2025 and now used by more than 60,000 open-source projects, tells an agent how to work in your repository. A rubric is the opposite direction of travel. It is the standard the change gets held against, read by an evaluator that had no hand in producing the change.

This is not a security boundary

A judgement-based check can be argued with, worked around, and wrong. Anything you actually need to be true, a secret out of source control, an unauthorised network egress, belongs in the deterministic layer or in a sandbox. The customs officer is not the reason the door is locked.

Where Agentic Checks Go Wrong

The failure modes are structural, and the design has to assume them rather than hope they do not appear. An uncalibrated evaluator hallucinates violations, flagging an innocent refactor as a boundary breach with a confident-sounding rationale. The same evaluator, tuned the other way, misses a real drift and says so at high confidence. Both produce a number that looks like signal.

This is why the advise-only rule is not timidity. A reviewer that is wrong often enough to notice and blocks the pipeline is a broken build system. The same reviewer, posting a comment a human reads, is a useful colleague with an occasional bad day.

Building an Agentic Fitness Function on Your Own Machine

Everything below runs on a clean workstation with the .NET SDK and a GitHub repository. Layer 1 is NetArchTest, which blocks. Layer 2 is the rubric evaluator, which does not.

Step 1: Create the solution with boundaries worth checking

mkdir agentic-ff && cd agentic-ff
dotnet new sln -n Shop
dotnet new classlib -n Shop.Orders -o src/Shop.Orders
dotnet new classlib -n Shop.Billing -o src/Shop.Billing
dotnet new xunit -n Shop.ArchTests -o tests/Shop.ArchTests
dotnet sln add src/Shop.Orders src/Shop.Billing tests/Shop.ArchTests
dotnet add tests/Shop.ArchTests reference src/Shop.Orders src/Shop.Billing
dotnet add tests/Shop.ArchTests package NetArchTest.Rules

You should see Build succeeded and a Shop.sln listing three projects. Two bounded contexts, and somewhere to assert about them.

The evaluator needs real git history to diff against, so initialise the repository now, and keep the API key and the generated verdict out of it. Create .gitignore:

.env
verdict.json
bin/
obj/

Then make the first commit:

git init && git add -A && git commit -m "Shop solution with boundary tests"

git log --oneline shows one commit, and git status is clean.

Step 2: Add the deterministic gate

Replace tests/Shop.ArchTests/UnitTest1.cs with a real boundary assertion. This is the passport gate: fast, binary, and it fails the build.

using NetArchTest.Rules;
using Xunit;

namespace Shop.ArchTests;

public class BoundaryTests
{
    [Fact]
    public void Orders_Should_Not_Depend_On_Billing()
    {
        var result = Types.InAssembly(typeof(Orders.Class1).Assembly)
            .That().ResideInNamespace("Shop.Orders")
            .ShouldNot().HaveDependencyOn("Shop.Billing")
            .GetResult();

        Assert.True(result.IsSuccessful,
            $"Boundary violated by: {string.Join(", ", result.FailingTypeNames ?? new List<string>())}");
    }
}

dotnet test prints Passed! - Failed: 0, Passed: 1, Skipped: 0, Total: 1. To watch it go red, make Shop.Orders actually use the other context. Add the project reference:

dotnet add src/Shop.Orders reference src/Shop.Billing

Then create src/Shop.Orders/Violation.cs:

namespace Shop.Orders;

public class Violation
{
    public void Use() => System.Console.WriteLine(new Shop.Billing.Class1());
}

dotnet test now fails with Boundary violated by: Shop.Orders.Violation. Delete Violation.cs and drop the reference again before moving on:

dotnet remove src/Shop.Orders reference src/Shop.Billing/Shop.Billing.csproj

That is the whole vocabulary of layer 1, and it is why layer 2 exists.

Step 3: Write the rubric next to the ADRs

The rubric belongs with the architecture decision records, because it is the executable half of the decision record. It should version with them, review with them, and go stale with them.

That means there has to be a decision record to sit next to. Create docs/adr/007-split-orders-and-billing.md:

# Decision Record: Split Orders and Billing into Separate Contexts

## Status
Accepted

## Context
Order capture and invoicing changed together for two years because they
shared a model. Tax rules changed on a regulator's timetable, order flow
changed on the product team's, and every tax change required regression
testing the checkout path.

## Decision
Orders owns the lifecycle of an order: lines, quantities, fulfilment
state. Billing owns money: VAT rates, invoice totals, payment terms.
Orders never computes a tax amount. It publishes an order, and Billing
prices it.

## Consequences
The contexts version independently and a tax change no longer touches
checkout. The cost is an integration boundary where there used to be a
method call, and a VAT rate table with exactly one home.

## Alternatives Considered
A shared kernel holding the money types, with both contexts depending on
it. Rejected because the rate table is the thing that changes, and a
shared kernel would have put the change back in both contexts at once.

## Related
Superseded ADR-0004, the single-model order design.

The rubric's criteria are only as good as the decision they point at. "Do the assumptions in this ADR still hold" is unanswerable against an ADR that says the split exists but never says what belongs on which side.

Now create docs/adr/rubrics/boundary-intent.yml:

rubric: boundary-intent
version: 1
applies_to: ["src/Shop.Orders/**", "src/Shop.Billing/**"]
adrs: ["docs/adr/007-split-orders-and-billing.md"]
criteria:
  - id: bounded-context-intent
    question: >
      Does this change move responsibility across the Orders/Billing
      boundary described in ADR-007, even where no compile-time
      dependency was added?
  - id: adr-assumptions
    question: >
      Do the assumptions recorded in the referenced ADRs still hold
      after this change, or has one been silently invalidated?
  - id: consumer-shaped-api
    question: >
      Is the public API still shaped around what consumers need, or
      has it started to expose internal domain structure?
  - id: ubiquitous-language
    question: >
      Does naming in this diff match the ubiquitous language of the
      context it lives in, or is vocabulary drifting between contexts?
output:
  score: "0.0-1.0 overall fitness"
  confidence: "0.0-1.0 evaluator confidence"
  evidence: "file:line references from the diff"
  rationale: "two sentences per triggered criterion"

Step 4: Hand the diff and the rubric to an evaluator

The evaluator is a script that assembles a prompt and demands structured JSON back. Create .github/scripts/evaluate.py:

import json, os, subprocess, sys, urllib.request

base = os.environ["BASE_SHA"]
result = subprocess.run(["git", "diff", base, "HEAD"], capture_output=True, text=True)
if result.returncode != 0:
    sys.exit(f"git diff against {base} failed: {result.stderr.strip()}")

diff = result.stdout[:60000]
if not diff.strip():
    sys.exit(f"no changes between {base} and HEAD, nothing to evaluate")

rubric = open("docs/adr/rubrics/boundary-intent.yml").read()

prompt = (
    "You are an architecture reviewer. Evaluate the diff against the rubric.\n"
    "Return ONLY JSON: {\"score\": float, \"confidence\": float, "
    "\"findings\": [{\"criterion\": str, \"evidence\": str, \"rationale\": str}]}\n"
    "Advisory only. Do not recommend blocking.\n\n"
    f"RUBRIC:\n{rubric}\n\nDIFF:\n{diff}"
)

key = os.environ.get("ANTHROPIC_API_KEY")
if not key:
    sys.exit("ANTHROPIC_API_KEY is not set. Locally it comes from .env; "
             "in Actions it comes from the repository secret.")

req = urllib.request.Request(
    "https://api.anthropic.com/v1/messages",
    data=json.dumps({
        "model": "claude-opus-5",
        "max_tokens": 2000,
        "messages": [{"role": "user", "content": prompt}],
    }).encode(),
    headers={
        "x-api-key": key,
        "anthropic-version": "2023-06-01",
        "content-type": "application/json",
    },
)
try:
    body = json.loads(urllib.request.urlopen(req).read())
except urllib.error.HTTPError as exc:
    detail = exc.read().decode(errors="replace")[:300]
    sys.exit(f"Anthropic API returned {exc.code}: {detail}")

# The model thinks before it answers, so content[0] is a thinking block.
# Select the text block by type rather than by position.
text = next(b["text"] for b in body["content"] if b["type"] == "text")

# Models wrap JSON in a markdown fence even when told not to.
fence = "`" * 3
text = text.strip().removeprefix(fence + "json").removeprefix(fence).removesuffix(fence)

verdict = json.loads(text.strip())
json.dump(verdict, open("verdict.json", "w"), indent=2)
print(f"score {verdict['score']} at confidence {verdict['confidence']}, "
      f"{len(verdict['findings'])} finding(s)")

Create a .env, which step 1 already gitignored:

# .env
ANTHROPIC_API_KEY=sk-ant-api03-REPLACE-ME
BASE_SHA=HEAD~1

HEAD~1 compares against your previous commit, which is what you want locally. In the workflow it becomes the pull request's base commit, and origin/main only resolves once a remote exists.

The evaluator diffs two commits, so commit the tooling before you use it. Otherwise HEAD~1 has nothing to reach and git says the revision is unknown:

git add -A && git commit -m "Add boundary test, rubric, ADR and evaluator"

Now give it something worth judging. The interesting case is a change that layer 1 waves through, so put billing logic inside Shop.Orders without any reference to Shop.Billing. Create src/Shop.Orders/Order.cs:

namespace Shop.Orders;

public class Order
{
    public Guid Id { get; init; }
    public IReadOnlyList<OrderLine> Lines { get; init; } = [];

    public decimal CalculateVatAmount(string countryCode) =>
        Lines.Sum(l => l.UnitPrice * l.Quantity) * VatRateFor(countryCode);

    private static decimal VatRateFor(string countryCode) => countryCode switch
    {
        "NL" => 0.21m,
        "DE" => 0.19m,
        _ => 0.20m,
    };
}

public record OrderLine(string Sku, int Quantity, decimal UnitPrice);

dotnet test still passes, because nothing here imports the other context. Commit it and run the evaluator:

git add -A && git commit -m "Add VAT calculation to Order"
set -a && . ./.env && set +a && python .github/scripts/evaluate.py && cat verdict.json

It prints one summary line, then writes verdict.json with a score, a confidence, and a findings array. The score comes back low with findings against the boundary and the ADR, while the deterministic gate stays green, which is the whole point. The call takes a few seconds, so it is not instant.

Step 5: Post the verdict, never fail the build

The verdict has to become a comment. Keep that in its own file rather than inlining it in the workflow, because a heredoc inside a YAML block scalar carries the block's indentation and breaks in ways that are tedious to debug. Create .github/scripts/comment.py:

import json

v = json.load(open("verdict.json"))
print(f"**Architecture fitness: {v['score']} (confidence {v['confidence']})**\n")
for f in v["findings"]:
    print(f"- `{f['criterion']}` at {f['evidence']}: {f['rationale']}")

The runner has no .env, so the key has to come from a repository secret. Add it in your repository on GitHub:

1. Open Settings, then Secrets and variables, then Actions.
2. Stay on the Secrets tab and choose New repository secret.
3. Name it ANTHROPIC_API_KEY.
4. Paste the key into Secret and save.

It then appears under Repository secrets, with the value masked. GitHub never shows a secret again after you save it, so replacing a lost key means updating it rather than reading it back.

The workflow then runs the gate first and the evaluator only after it passes. Create .github/workflows/fitness.yml:

name: fitness
on: pull_request

permissions:
  contents: read
  pull-requests: write

jobs:
  deterministic:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-dotnet@v6
        with:
          dotnet-version: '9.0.x'
      - run: dotnet test   # this one blocks

  agentic:
    needs: deterministic
    runs-on: ubuntu-latest
    continue-on-error: true   # advisory, never red
    steps:
      - uses: actions/checkout@v7
        with:
          fetch-depth: 0
      - env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          BASE_SHA: ${{ github.event.pull_request.base.sha }}
        run: python .github/scripts/evaluate.py
      - env:
          GH_TOKEN: ${{ github.token }}
        run: |
          python .github/scripts/comment.py > comment.md
          gh pr comment ${{ github.event.pull_request.number }} --body-file comment.md

Open a pull request and you get one comment with a score, a confidence, and file-level evidence. The check mark stays green regardless. That is the design.

Step 6: Calibrate before you trust it

The usual advice is to calibrate the judge on a few dozen prior changes before trusting it. I would put it differently: size the calibration set to the risk and impact of the judgement being made. A rubric criterion about naming drift is low risk and low impact, and ten replays will tell you whether it is useful. A criterion that flags bounded-context violations in a payments domain is neither, and fifty is the floor.

Replay merged pull requests and compare the verdict to what actually happened:

for sha in $(git log --merges -n 30 --format=%H); do
  BASE_SHA=$sha~1 python .github/scripts/evaluate.py
  echo "$sha $(python -c "import json;v=json.load(open('verdict.json'));print(v['score'],v['confidence'])")"
done | tee calibration.tsv

Read calibration.tsv against your own memory of those merges. If low scores cluster on changes nobody objected to, the rubric is too suspicious. If the drift you remember scores high, it is asleep. Tune the criteria text, bump version in the rubric, replay.

Which Layer Does Which Job

	Deterministic (the gate)	Agentic (the officer)
Expressed as	An assertion	An analytic rubric
Output	Pass or fail	Score, confidence, evidence, rationale
On violation	Blocks the build	Posts a comment, escalates
Runs	Always, first	Only after layer 1 passes
Catches	Dependency, schema, policy, cost	Boundary intent, stale ADRs, language drift
Wrong answer costs	A broken pipeline	A human reading a paragraph
Preparation needed	Write the rule	Calibrate on prior changes

On build versus buy: start in-house. The rubric encodes decisions specific to your domain, and writing it yourself is how you find out what you actually believe. Expect to migrate to a product once the thing works, because maintaining an evaluator is not the job you were hired for.

The Rubric Is Half the Decision Record

The gate blocks, the officer advises. Nothing agentic fails a build, ever. The moment it does, you have handed a probabilistic evaluator authority it cannot carry.

Calibrate to consequence. A few dozen replays is a starting point rather than a constant. Low-risk criteria need less, payments boundaries need more.

Ship the rubric with the ADR. An architecture decision record says what you decided. The rubric says how you would know if it stopped being true. They should live in the same folder, review together, and version together.

Assume it will be wrong. Design so that a hallucinated violation costs a paragraph of someone's attention and nothing else.

That API change got caught because a person was paying attention on the right afternoon. The rubric is an attempt to write that attention down, so it runs on every diff instead of the lucky ones. It will never be as good as the reviewer who noticed. It does not have to be. It only has to notice more often than nobody.

Which architecture decisions in your repository would fail today without anything turning red? And if one did, who would find out, and how long would it take?