Unlocking Swarm Intelligence with Big Agentic Brain and Claude Code

Agents fix far more real bugs when they share knowledge. On tasks that require a missing fact from another repo or a teammate, Sonnet went from 12% success without memory to 79–100% with a shared team memory and a small nudge to actually use it.

Here's one of the bugs from our benchmark. A service sends order webhooks to Shipfast, a carrier aggregator. Since this month, every request comes back 401 "signature verification failed". The unit tests pass. The code looks reasonable:

body = json.dumps(event).encode("utf-8")
signature = hmac.new(secret.encode("utf-8"), body, hashlib.sha256).hexdigest()
headers = {
  "Content-Type": "application/json",
  "X-Shipfast-Signature": f"sha256={signature}",
}

The root cause isn't in this repository. Shipfast switched off the legacy X-Shipfast-Signature header on 1 September. It now expects Shipfast-Signature: t=<unix seconds>,v1=<hmac>, where the HMAC is computed over "<timestamp>." + body, not over the body alone. The only copy of that spec lives in a different repo: the returns service, where a teammate's agent already hit the same 401 and fixed it.

An agent working on the webhook repo has no way to infer this. It can read every file and rerun the tests all day, and it will still never see the new header format. That's team knowledge: vendor quirks known from a support reply, a runbook convention, a partner's ID check digit. It lives in someone's head or in someone else's repo. AI coding agents start every session from zero, so whatever one agent learned is gone when its session ends.

Why I tested this properly

I'm involved with Big Agentic Brain (BAB), an open-source, self-hosted shared memory for a team's agents. Agents search it before they start, record what they learned when they finish, and react to each other’s entries. It's exposed over MCP, so any MCP-capable agent can use it.

Because I have a stake in it, I didn't want to write a post that just says memory makes agents better. So we measured it: agents fixing bugs with and without shared memory, graded by hidden tests. The second round's tasks and hypotheses were frozen before any of its runs.

What we found

Setup

• Models: Claude Sonnet 5.5 (primary), Haiku 5.5, and Opus 5.5 on the team-knowledge tasks.
• Tasks: 12 easy bug tasks plus 8 team-knowledge tasks.
• Scale: 1,104 graded runs, all on 19 October 2026, for about $29 in API spend.
• Memory: recorded by a "colleague" agent that had solved the source tasks, not written by hand.

Round 1: bugs the code already explains

These were classic traps:

• bool("false") being True in env-config parsing;
• Python's hash() changing across restarts and breaking a disk cache;
• JavaScript parsing "2026-10-19" as UTC midnight;
• a regex $ matching before a trailing newline;
• an Excel byte-order mark breaking a CSV import.

Every condition solved every task: 108/108 Sonnet runs and 180/180 Haiku runs. Memory had nothing to add. It did add cost: 1.50× per run on Sonnet. Most of that comes from BAB's tool definitions and instructions, which add 8,598 tokens to every request. Agents also rarely used the memory: only 22% of Sonnet runs searched it.

Round 2: bugs that need team knowledge

Here the results reversed:

Each cell is 24 runs (8 tasks × 3 trials). For Sonnet, default BAB vs no memory was +0.67 (95% CI +0.33 to +0.92, p = 0.031). Without memory, Sonnet solved only the one family where the prompt’s example IDs were enough to reverse-engineer a mod-11 check character. On the Shipfast family it went 0/3 without memory and 3/3 with it.

A caveat on how to read this table: these tasks were built so the missing fact (a vendor spec, a team convention) exists only in a colleague's memory. So the gap mostly shows that the memory channel works when the knowledge is there, not how often that situation comes up in real work. With 8 tasks per condition, the differences between the BAB variants are also within noise.

Opus, the largest model, didn't make memory unnecessary: without it, Opus also fixed only 12%. With default BAB it reached 88%, the best default result of the three models, and outdated memories pulled it back to baseline just like the others.

Hazards. With only outdated memories, success fell back to the no-memory baseline. Agents read the stale entry in 88% of runs but flagged it as possibly outdated in only 4%. When the current fact was also present, success matched default BAB (79%).

Retrieval at scale. In an offline test with 118 queries, recall@5 dropped from 0.93 with 20 distractor memories to 0.81 with 1,000, mostly on paraphrased queries. And BAB's own search advice, filtering by kind: standard, decision, practice, returned nothing at all (recall 0). Agents record their memories as kind learning, which that filter excludes.

What we learned about building it

Keep the tool surface small. Tool schemas are sent with every request, used or not. A lean setup with only bab_search, bab_learn and bab_react and short descriptions costs 648 tokens instead of 8,598. On team-knowledge tasks it kept most of the benefit at 1.18× baseline cost, against 1.51× for the full toolset.

Don't wait for the agent to search. On easy tasks agents skip the memory because the problem looks solvable. On hard ones they sometimes search once with a weak query and stop. A one-line nudge, or retrieving the top hits before the first turn, removes that failure mode. Auto-retrieval was also cheaper than the default (1.34× vs 1.51× baseline).

Prompt for structured learnings. Sonnet recorded learnings only when nudged at the end of a task. Storing raw finish summaries instead of structured bab_learn entries worked less well: 71% vs 79% on Sonnet.

Treat staleness as a first-class problem. A shared memory amplifies whatever it holds. Visible age, verify-before-acting prompts and down-weighting old entries matter as much as ranking.

Test your own advice. We only found the kind-filter problem because we measured it.

All of this has turned into concrete changes for BAB:

• a lean default toolset;
• first-prompt auto-retrieval;
• a search-first hint;
• corrected search instructions.

The limits

This is a first study. The benchmark tasks were written by an AI agent for this study, before any runs. The samples are small: 8 team-knowledge families with 3 trials each, so some comparisons are suggestive rather than conclusive. We used our own minimal MCP agent loop, not Claude Code, so the nudge and auto-retrieval variants simulate what a plugin hook would do. One planned arm, Sonnet on the easy-task variants, was only partly run and isn't used. The full report has the methodology, every condition, the statistics and the deviations.

Try it, or check our work

If your team runs coding agents and keeps re-solving problems someone already solved, this may be worth an afternoon:

• BAB is open source and runs as one container on one SQLite file: Big Agentic Brain on GitHub. If it's useful to you, a star helps others find it.
• The eval suite is being contributed to the repo (pull request #26). It includes both task suites, every condition above and the offline retrieval test, so you can rerun it or point it at your own tasks and agents.
• The full report (PDF) covers everything this post leaves out.

If your results look different from ours, I'd like to hear about it.