A small validation change in a customer-onboarding application looked like routine work for an AI coding agent. The requirement was clear. The acceptance tests were known. The agent read the relevant files, proposed the change and updated the tests.
The first attempt failed. The agent read more files, explored another path and checked additional dependencies. The final code passed.
The change was accepted. The consumption was much higher than for a similar change.
Our first design treated token usage mainly as a reporting issue. We expected teams to review model spend after the work, much like cloud usage. The complete delivery flow showed why that was too late.
The real problem was not the token count. It was whether the consumption could be connected to an authorised feature, the appropriate model class and an accepted engineering result.
This field note describes the internal architecture experiment that changed our view. It explains what we moved into the delivery path, what we measure against a baseline and what we are still refining.
“Our current view: token cost is part of the engineering control system - not only a finance report.”
At a glance
- Token consumption is useful only when it is tied to an authorised task and accepted output.
- Context selection, retries and restarts can consume as much attention as code generation itself.
- Different engineering tasks need different model capacity, limits and approval paths.
- The useful metric is not tokens per engineer. It is cost per accepted engineering outcome.
What we tried
We built a controlled path for a coding agent to work on bounded application changes. The path combined code generation, architecture checks, testing and traceability.
For each run, we captured the model used, the context supplied and the consumption created. The work used a controlled repository and simulated delivery conditions. It was an internal architecture experiment, not a client production deployment.
We began with a simple operating model: allow the agent to work, record usage and review unusual spend later.
Four observations changed that design.
What changed our view
1. Usage without purpose was only an activity measure
Our first dashboard could show model usage by team and period. It could not show why the usage happened.
A rise in tokens could mean productive adoption. It could also mean an unnecessarily large context window, repeated failures or exploration that never contributed to the accepted change.
We attached every run to an authorised feature, a stated purpose and an expected output. Consumption then became attributable to a piece of work rather than to a team in the abstract.
2. The cost often started before generation
We initially focused on the code the model generated. The experiment showed that the input context could be just as important.
Supplying too many files made every later step heavier. Failed attempts also carried test output, alternative code and prior reasoning into the next cycle.
We narrowed the starting context according to the task. The agent could request more information, but each expansion was visible. That made unnecessary context growth easier to identify.
3. One model and one ceiling were too blunt
A unit-test change and a cross-system identity change do not need the same reasoning capacity or risk control.
A single model and token cap treated unlike work as though it were the same.
We moved to task tiers. Bounded work could use a smaller, faster model. Ambiguous or cross-system work could move to a stronger one. Higher-risk changes still required human approval.
Each tier had an expected consumption range and an exception path.
We are still comparing alternative models by task type. The comparison includes accepted output, review effort, delivery speed, failures and total cost. Token price alone does not identify the more economical model.
4. Cost became useful only beside evidence and a baseline
The final code file did not prove value. We also needed passed tests, architecture compliance, review effort, rework and acceptance status.
We grouped completed changes into comparable categories such as defect fixes, minor enhancements, refactoring and cross-system changes. Recent work from the same codebase gave us a baseline for engineering effort, time to acceptance, review effort and defects.
This changed the metric. Tokens per engineer were easy to report. Cost per accepted engineering outcome was more useful.
A model may save 90 minutes of coding but add two hours of review and correction. The code appeared faster. The accepted change did not.
The assumptions that changed
Our initial assumptions changed in four practical ways.
1. Monthly usage would explain cost
It did not identify the feature or activity that caused the consumption.
We now attach each run to an authorised feature and expected result.
2. Generation would be the main cost driver
The experiment showed that broad context, retries and restarts could also create substantial consumption.
We now limit the initial context and make each expansion visible.
3. One model and one cap would simplify control
Different tasks required different capabilities, risk controls and consumption ranges.
We now tier models by activity and provide rerouting or approval paths when a run crosses its expected boundary.
4. More AI activity would imply more output
High usage did not prove speed, quality or completion.
We now compare the full cost with the accepted output and an established delivery baseline.
The control pattern we currently use
The pattern has four parts: define, bound, observe and verify.
The important change is where the control sits. It has moved from a central cost dashboard into the engineering flow where consumption and risk are created.
1. Define the work: Every run starts with an approved feature, a clear purpose and a required result.
This connects consumption to authorised work before the agent begins.
2. Bound the run: The task receives a default model tier, an initial context limit and an expected consumption range.
The boundary can change, but expansion should be deliberate and visible.
3. Observe the activity: The delivery path records context growth, retries, restarts and cost while the work is in progress.
This allows the workflow to intervene before an unproductive loop becomes an expensive completed run.
4. Verify the result: Consumption is assessed alongside tests, architecture controls, review effort, acceptance status and the pre-AI baseline.
The question is not merely what the run consumed. It is what accepted engineering result the consumption produced.
What technology leaders should test next
1. Establish one baseline before scaling: Choose one class of software change and measure recent comparable work.
Use engineering effort, time to acceptance, review effort, rework and defects. Without a baseline, growth in model usage is difficult to interpret.
2. Tier work by activity and risk: Separate bounded work, complex reasoning and higher-risk changes.
Give each category a default model, expected consumption range and escalation path.The tiers should reflect both task complexity and the consequences of an incorrect change.
3. Measure the accepted result, not the generated code: Compare the complete cost with accepted output, time to acceptance, review effort and quality.
“Faster code generation is useful only when it improves the delivery outcome.”
Where our thinking has landed - for now
In this experiment, token consumption stopped looking like a finance metric. It became part of the engineering control system.
Some difficult changes will justify more model work. The reason, limit and result should remain visible.
We are still refining task tiers, expected consumption ranges and model-comparison thresholds. We are also testing when a run should stop, request approval, change its model or alter its method.
Do not reward the tokens consumed. Control for the engineering result produced.
Are coding-agent costs rising faster than accepted delivery?
Bring us two or three recurring classes of software change where AI coding agents are being used or evaluated.
Ampersand can help determine where model tiers, context limits, runtime controls and acceptance measures should sit in the delivery flow.
Practical output: A bounded coding-agent control model linked to accepted engineering outcomes and a measurable baseline.
