Saumil Srivastava
Engineering Leader
I am building a production-style inference PoC across Kubernetes, NVIDIA Triton, execution backends, and GPUs. I am also using a coding agent to implement parts of it.
That creates a second engineering problem:
How do I let an agent make meaningful progress without allowing it to wander, repeat the same mistakes, weaken tests, or quietly spend GPU money?
Running an agent in a loop is easy:
while not done:
call_agent()The useful work is everything around that loop.
The harness decides what the agent may work on, how much it may spend, what counts as success, what happens after failure, and how another engineer resumes the work later.
This post covers the harness I am using for the inference project. Not the framework. The control system around it.
This is a living engineering note. The control model is implemented, and the first real slice has run: a ResNet18 model exported to ONNX, validated against its PyTorch reference, and served through a Triton CPU container end-to-end. Three blocks below contain evidence from that first runtime slice. The fourth contains control-plane evidence: the GPU budget gate remained closed, so there is deliberately no paid benchmark yet.
Each run claims one finite unit of work.
Project truth lives outside the conversation.
The worker cannot approve its own change.
Deterministic failures are not repeated unchanged.
Time, retries, commands, and GPU cost are bounded.
Useful defects become regressions or preflight checks.
A long-running coding task spans multiple context windows, commands, failures, and partial implementations.
The next session should not have to reconstruct what was attempted, what actually worked, which failure was already investigated, or whether the repository is still healthy.
The problem is not only model capability. The problem is missing execution structure.
I separate four responsibilities.
Architecture, decomposition, acceptance, experiments, risk, and interpretation.
State, commands, timeouts, retry policy, limits, error history, and checkpoints.
Bounded implementation, tests, routine refactoring, and documentation updates.
Outcome checks, regressions, policy enforcement, evidence, and final pass/fail.
The important boundary: The agent that produced the change does not decide whether the change is correct.
The agent never receives: Build the complete Triton inference platform.
It receives something closer to:
{
"id": "M1-06",
"title": "Add Triton model-readiness verification",
"status": "pending",
"acceptance": [
"Triton server reports live",
"Expected model reports ready",
"Missing model produces a useful diagnostic",
"Verification terminates within 90 seconds",
"Existing harness regression tests pass"
]
}A small work item gives us a reviewable patch, a finite verification surface, a clear stopping point, a clean rollback boundary, and a useful handoff artifact.
while work_remains():
item = claim_one_work_item()
preflight(item)
run_worker(item)
result = run_verifier(item)
if result.passed:
create_green_checkpoint(item)
else:
route_failure(item, result)The agent can propose completion. Only the verifier can move the item to passed.
Conversation history is useful context. It should not be the project database.
harness/
├── feature_list.json
├── state.json
├── progress.jsonl
├── error_ledger.jsonl
├── known_errors.yaml
├── budget.json
├── decisions.md
└── prompts/
├── initializer.md
├── worker.md
└── verifier.md| Artifact | What it answers |
|---|---|
| feature_list.json | What remains, priority, dependencies, and acceptance. |
| state.json | What is active, where it started, and the last green commit. |
| progress.jsonl | What happened, in append-only order. |
| decisions.md | Why a design was chosen and alternatives rejected. |
| error_ledger.jsonl | What failed, under which conditions, and how it was resolved. |
| budget.json | How much time, retry capacity, compute, and money remain. |
{
"active_item": "M1-06",
"state": "verifying",
"starting_commit": "fbd3da9",
"last_green_commit": "fbd3da9",
"session_id": "20260720T153000Z-fbd3da9"
}An agent saying "the model loaded successfully" is not evidence.
Useful, but insufficient.
Useful, but insufficient.
Better, but still not the outcome.
Required alongside metrics and regressions.
def verify_model_ready(base_url: str, model: str) -> None:
assert get(f"{base_url}/v2/health/live").status_code == 200
assert get(f"{base_url}/v2/health/ready").status_code == 200
assert get(
f"{base_url}/v2/models/{model}/ready"
).status_code == 200For autoscaling, "Kubernetes created another pod" is not success.
I use the word verifier for three different layers of authority. They should not be collapsed into a single LLM call.
Tests, schemas, probes, numerical comparisons, process cleanup, budgets, metrics, and policy checks. If this layer fails, the work does not pass.
A fresh-context reviewer inspects intent, omissions, overengineering, test integrity, and operational risks. It cannot override deterministic failure.
Paid GPU work, destructive tests, architecture changes, benchmark fairness, and public claims remain human decisions.
The same underlying model can be useful as the secondary reviewer when it receives a fresh context, a fixed rubric, the original acceptance contract, the exact diff, and test evidence. It is still a critic—not proof.
M1 showed why verification needs layers. Deterministic checks found two issues; human review showed that one was a verifier false positive. A model review was unnecessary.
Not every failure deserves another attempt.
| Failure | Harness action |
|---|---|
| Temporary network timeout | Retry with bounded backoff. |
| API throttling | Retry more slowly with jitter. |
| Preempted process | Resume from checkpoint. |
| Test assertion failure | Do not retry unchanged. |
| Invalid Triton configuration | Block until configuration changes. |
| Authentication failure | Stop and request intervention. |
| GPU out of memory | Replan workload or memory strategy. |
| Budget exhausted | Stop. |
Critical rule: An identical deterministic failure cannot run again unless a relevant input changed.
work item
+ normalized command
+ exit code
+ normalized error
+ configuration hash
+ environment hash
+ Git commitBlocked: identical deterministic failure already observed.
Required change:
Update the model configuration or exported tensor contract.The repeat guard originally existed only in a test. I moved it into the harness and probed it with a deterministic verifier failure: the first attempt ran, while unchanged retries were blocked.
A functionally correct harness that can leave an A100 running overnight is not correct.
worst_case_cost = timeout_hours * gpu_hourly_price
if worst_case_cost > remaining_budget:
raise BudgetExceeded(
f"Maximum run cost ${worst_case_cost:.2f} "
f"exceeds remaining budget ${remaining_budget:.2f}"
)GPU: NVIDIA A6000
Price: US$1.09/hour
Command timeout: 45 minutes
Worst-case cost: US$0.82
Remaining budget: US$0.55
Decision: rejected before executionThis is control-plane evidence, not benchmark evidence. The first GPU request was denied because both GPU-minute and cost budgets were zero.
What has actually happened so far — three failures encountered in this repository, each turned into a permanent control:
| Encountered so far | Permanent improvement |
|---|---|
| Incorrect ONNX argument contract | Static / type verification. |
| Cleanup false positive | Outcome-based cleanup check. |
| Repeat guard existed only in test code | Shipped circuit-breaker implementation. |
Upcoming failure classes I expect the GPU and Kubernetes milestones to surface — listed as design targets, not as things that have happened yet:
The error ledger preserves the incident. The regression prevents its return. The preflight prevents another engineer from encountering it in the first place.
The harness changes the quality of team handoffs.
"I tried a few things. Check the branch and the logs."
Work item, current state, last green commit, failure class, evidence path, budget consumed, and next decision.
That shape is not hypothetical. The handoff below is a real one from this project — durable enough that the next session resumes from state, not from chat history.
People stop collaborating through memory and chat history. They collaborate through explicit intent, durable state, bounded changes, reproducible evidence, and green checkpoints.
The repository now contains enough durable state for a fresh session to resume without the original conversation. The remaining test is whether the recipient selects the correct next action from these artifacts alone.
handoff @ 27c32c2
state M0 green; M1 CPU path passed
contract verification/contracts/M1-model-correctness.yaml
evidence results/golden-path/<stamp>/
decisions harness/decisions.md
resume make golden-path ARCH=resnet18
next freeze tolerance, then run ARCH=resnet50
to prove fresh recipient resumes without chat historyYou do not need a complex multi-agent framework to begin.
Bounded work items and acceptance criteria.
An append-only record of what happened.
Timeouts, process cleanup, and structured output.
Outcome checks outside the worker's control.
A way to stop unchanged deterministic retries.
Explicit approval for paid or risky operations.
In a greenfield repository, the harness defines the initial operating system. In a mature repository, it should first discover the one already present: tickets encode intent, CI encodes verification, CODEOWNERS encodes authority, release approvals encode risk, and postmortems encode failure memory.
The first step is not replacing those controls. It is making them explicit and letting the agent operate through them.
The harness does not decide whether Triton is the right serving layer, which experiment matters, whether a benchmark is fair, whether p99 latency is acceptable, or whether the result generalizes to production.
| Owner | Responsibility |
|---|---|
| Human | Architecture, hypotheses, acceptance, and interpretation. |
| Agent | Bounded implementation. |
| Deterministic tools | Tests, measurements, and limits. |
| Harness | State, orchestration, evidence, and recovery. |
I do not need the agent to be infallible. I need the development system around it to make failures visible, bounded, and recoverable.
The interesting shift is not that agents can write more code. It is that we now need to engineer the system in which they write code.
For this inference PoC, the coding agent is replaceable. A different model can claim the next work item tomorrow.
The durable assets are the specification, acceptance criteria, decisions, checkpoints, error memory, and experiment evidence.
Three blocks report evidence from the ONNX → Triton runtime slice. The budget block reports an enforced non-event: the system refused paid GPU execution under a zero budget.
A loop keeps the agent running. A harness makes the work trustworthy.
That is the engineering.
Get weekly insights on AI implementation, performance measurement, and technical case studies.
Get weekly insights on AI implementation and technical case studies.