7 min read

No coding agent rediscovered the incident rule, and that is what a team memory should hold

A closer look at the distributed round of the BAB tests: which lessons coding agents never rediscover, where local notes are enough, and what a shared memory costs a team per fix.

In one of my test repos, a geocoding service sends Retry-After-Ms in milliseconds, wants retries to stop after three attempts, and uses a 503 to say the quota is gone, so a 503 must never be retried. Each of those rules runs against what a sensible engineer, or a sensible model, would assume. Once one agent had learned them, no other agent in my tests worked them out again without being told.

That case is the most useful thing I found when I went back into the distributed round of my shared memory tests. The first post explains Big Agentic Brain (BAB), the open-source shared memory for coding agents that I am involved with, and gives the headline of that round in a few lines: a teammate fixes a related bug far more often with a shared memory than with only their own agent's notes. This post looks underneath that headline at three questions it skips. Which kinds of lessons are worth sharing, where are local notes enough, and what does the shared layer cost a team once the cost of writing everything down is included?

One of three lessons was never rediscovered without memory

Each simulated team had three agents in separate sandboxes with no shared files, and each team had to learn three lessons spread across two repos. The first was a vendor API quirk: Paylane wants amounts as minor-unit strings using ISO 4217 exponents, rounded half-up, with a lowercase currency code. The second was an internal convention: Gatekeeper tokens need a fixed issuer, an audience of the form svc:<name>, scopes as one space-separated string and 120 s of clock skew. The third was the geo-api incident described above.

Every agent first fixed a bug in a repo where its rule was documented. Memory was then frozen, and a related bug in a second repo went either back to the same agent or to a teammate who had worked on a different lesson. The second repo never states the rule, although it holds a log of 64 to 70 recorded requests from which a patient agent could work it out.

No model did. Every geo-api attempt without the relevant memory failed, for Sonnet 5.5, Haiku 5.5 and Opus 5.5 alike, whether the agent had no memory or only its own notes about another lesson. With the shared memory, every teammate fixed it.

Teammate fixesNo memoryLocal onlyShared
Paylane (vendor API quirk)
Sonnet 5.54/44/44/4
Haiku 5.51/83/87/8
Opus 5.52/22/22/2
Gatekeeper (internal auth convention)
Sonnet 5.54/44/44/4
Haiku 5.51/80/88/8
Opus 5.50/22/22/2
geo-api (incident that comes back)
Sonnet 5.50/40/44/4
Haiku 5.50/80/88/8
Opus 5.50/20/22/2
Related bugs handed to a teammate, by use case: fixes verified by hidden tests out of attempts. Without the relevant memory, no model fixed geo-api.

For Sonnet, this one lesson explains the whole teammate gap. With local notes only it fixed 8 of 12 teammate bugs, and all four misses were geo-api.

For rules a model can guess, memory mostly changed the price

Paylane and Gatekeeper behaved differently. Sonnet and Opus mostly solved them cold, because the rules sit close to what general knowledge suggests, and there memory moved the cost of a fix by 10 to 20% rather than deciding whether the fix landed.

Haiku, the smallest model, is the exception that matters for budgets. It failed most cold attempts on every lesson, so for Haiku even the guessable rules turned from failures into fixes once a teammate could read them. On Paylane, Haiku teammates went from 1 of 8 fixes with no memory to 7 of 8 with the shared layer.

The practical reading is that the value of a shared lesson depends on two things: how far the rule departs from reasonable defaults, and how capable the model is that has to reuse it.

Local notes are the right home for lessons the same agent will meet again

When the related bug went back to the agent that had already solved the first one, its own notes were enough. Sonnet's fix rate on those repeat bugs went from 58% with no memory to 100% with local notes, and its cost per fix fell from $0.0692 to $0.0427. Haiku went from 21% to 96%, and its median time to a verified fix dropped from the 900 s cap, meaning most attempts never got there, to 17.5 s.

A note on one machine did nothing for anyone else, though. I had registered in advance a check that an agent's local memory would not help a teammate, and it held for all three models: a teammate's session cost about the same with local notes as with no memory, at a ratio of 1.109 for Sonnet and 1.008 for Opus.

No memory
Local notes only
Local + shared BAB
Sonnet 5.5 (4 teams, 12 sessions per bar)
First solve (answer documented)
None
$0.0159
Local
$0.0219
Shared
$0.0319
Same agent, related bug
None
$0.0692
Local
$0.0427
Shared
$0.0398
Teammate, related bug
None
$0.0598
Local
$0.0663
Shared
$0.0372
Haiku 5.5 (8 teams, 24 sessions per bar)
First solve (answer documented)
None
$0.00126
Local
$0.00160
Shared
$0.00212
Same agent, related bug
None
$0.0176
Local
$0.00322
Shared
$0.00406
Teammate, related bug
None
$0.0450
Local
$0.0315
Shared
$0.00387
Opus 5.5 (2 teams, exploratory, 6 sessions per bar)
First solve (answer documented)
None
$0.0501
Local
$0.0520
Shared
$0.0744
Same agent, related bug
None
$0.1528
Local
$0.0843
Shared
$0.1039
Teammate, related bug
None
$0.3149
Local
$0.1587
Shared
$0.1067
Cost per verified fix (mean cost per session divided by fix rate), by phase. Bars are scaled per model. With the shared BAB, a teammate's fix costs about the same as a repeat fix by the same agent; with local notes only, the teammate stays expensive.

The team bill favors sharing once the writing cost is counted

Per-session numbers hide the agents that do the writing, so I also priced every team as a whole. The figures below divide all nine sessions per team, including the first solvers and their extra effort to record what they learned, by the number of fixes the team delivered. Sonnet ran with four teams, Haiku with eight and Opus with two.

ModelNo memoryLocalShared
Sonnet 5.5$0.0427$0.0408$0.0363
Haiku 5.5$0.00672$0.00414$0.00333
Opus 5.5$0.1285$0.0908$0.0950

For Haiku, the shared layer roughly halved the team's cost per delivered fix compared with no memory, and the full ordering of shared, then local, then none was significant across eight teams. For Sonnet the shared layer came out about 15% cheaper per fix than no memory, and the difference between shared and local was significant even with only four teams. The Opus row is exploratory, with two teams.

The shared teams also delivered more. Sonnet teams with the shared layer fixed 36 of 36 bugs against 27 of 36 with no memory, and Haiku teams fixed 70 of 72 against 31 of 72.

Checking the shared layer costs extra when the answer is already local

The shared layer is not free, and Opus shows it. At team level its local-only setup edged out the shared one, $0.0908 against $0.0950 per fix, although with two teams nothing in that comparison is significant.

The cause shows up in the repeat bugs. When an agent already had the answer in its own notes, also querying the shared memory cost Haiku about 26% and Opus about 23% more per session than local memory alone, while for Sonnet the two stayed within the margin I had set in advance. Seeding has a price too: the first solvers cost 1.4 to 2 times as much with the shared layer as with no memory, because they also wrote what they learned into it.

Where I would keep each kind of lesson

These results suggest a simple placement rule for a team that runs coding agents.

  • Incidents and rules that contradict defaults belong in the shared memory first, because in this test no agent rediscovered them from the evidence in the repo.
  • Guessable vendor and internal rules are worth sharing mainly when smaller models do the work, since stronger models solved them cold and memory only trimmed the cost.
  • Lessons only the same agent will need again fit well in local notes, which were cheap and effective for that job.
  • The shared tool surface should stay small, so that lookups on work the agent already understands add as little overhead as possible.

One thing I have not tested is an agent that checks its local notes first and queries the shared memory only when they come up empty, which could remove most of the overhead above.

Limits of this round

The three lessons are synthetic and were written for the study, and each agent had only one earlier session to learn from. Sonnet ran with four teams after a machine restart cost me the original eight-team run, and Opus ran with two teams as an exploratory arm, so its intervals are wide. The agent loop was my own minimal MCP loop rather than a commercial coding agent, which might load its local notes more cheaply, and the separate machines were isolated sandboxes on one host.

The hypotheses and analysis code were frozen before any paid run, because I have a stake in BAB. The round covered 378 graded sessions and cost $17.41 in total. BAB itself is open source at github.com/enhesa/big-agentic-brain, and the earlier rounds are documented in the full report (PDF).