Two red teams and a spike: designing agent memory that survives the agent
The thesis is simple: agents are ephemeral; the knowledge they produce should be persistent. When an agent observes a system failure, learns a configuration preference, or makes an architectural decision, that knowledge should survive the agent’s termination and be available to the next agent that needs it. The planet persists; the agents are forcing functions that populate it.
The architecture for making that work — a unified memory design where one Postgres database handles deterministic, semantic, and graph retrieval — seems straightforward enough that you might build it directly. We didn’t. This post is about what happened when we tested the design before implementing it, and why the testing changed the architecture.
The initial proposal
The first proposal was a two-store routing model. One store for deterministic facts (structured claims with exact-match queries), another for semantic similarity (vector embeddings of documents). A router in front decides which store to query based on the query type. This mirrors a common pattern in RAG systems: structured data in a database, unstructured data in a vector store, a router in the middle.
The proposal had a design doc, a query interface, and a working implementation of the deterministic side. It had 140 passing tests. It looked ready to extend with semantic search.
Then we ran the first red team.
Red team one: the implementation bugs
The first adversarial pass examined the existing implementation. It found four critical bugs, six high-severity issues, and three strategic concerns. The criticals were implementation defects — a prompt/schema mismatch that would have caused eval failures, a severity ordering bug that sorted alphabetically instead of by severity level, an unbounded query that could OOM the server, and an exposed password in a config file.
These were fixable bugs, not design flaws. But the strategic concerns were more interesting: the routing logic lived in the wrong place, the natural-language adapter was naive, and the contract was a proposal, not a ratified standard. The red team suggested the architecture itself needed examination before more implementation work.
So we ran a second red team — this time against the design, not the implementation.
Red team two: the design flaws
The second pass was ruthless. It found four critical design flaws, six high-severity issues, and three strategic concerns. The criticals were the kind that would have made the system silently wrong rather than visibly broken:
The embedding text format would have produced poor search results. The original design proposed embedding pipe-delimited structured identifiers: endpoint.registry-api | endpoint.reachability.failing | infrastructure_anomalies | {url:...}. This looks reasonable. It isn’t. The embedding model (nomic-embed-text-v1.5) was trained on natural language. Pipe-delimited identifiers produce vectors that cluster by field value — all infrastructure_anomalies claims cluster together — rather than by semantic meaning. A natural-language query like “endpoint having connection problems” lands in a different region of the embedding space and matches poorly. Semantic search would have returned results, but they would have been the wrong results. The operator would have concluded that pgvector doesn’t work, when the actual problem was the text format.
The composite scoring formula was dimensionally incoherent. The design proposed merging results from three modalities with a weighted formula: max(deterministic * 1.0, semantic * 0.8, graph * 0.6). But deterministic score is claim confidence, semantic score is cosine similarity, and graph score is inverse traversal depth. These measure different things. Multiplying by weights and taking the max produces a number that’s sortable but not meaningful. A low-confidence exact match would rank below a high-similarity semantic match, even though the exact match is more authoritative. The formula was replaced with a simple priority order: deterministic first, semantic second, graph third.
The design claimed routing “disappears” — it doesn’t. The original framing said consolidating three modalities into one database eliminates the routing problem. It doesn’t. The routing moves inside the database. Instead of deciding “which external service do I call?”, you decide “which SQL query do I run?” The simplification is operational (no network hops, no service health checks), not conceptual. The honest framing — “routing moves inside the database” — replaced the overclaim.
The Cosmos integration plan undersold Cosmos’s value. The original design proposed reducing Cosmos (a physics-based knowledge graph with Boltzmann emission and thermal focus) to a display layer that queries the unified store. But Cosmos’s physics-based retrieval is a genuinely novel retrieval signal, not just a visualization. The revised design preserves Cosmos as a specialized retrieval engine with its own data store, fed by Planets via a sync path.
The humanization spike
The embedding text format was the highest-risk component. If the humanization — the conversion of structured claims to natural-language sentences — produced poor embeddings, the entire semantic search modality would be useless. Rather than build the infrastructure and find out, we spiked.
The spike was simple: write humanization functions for the current predicate vocabulary, generate 100 representative claims across all domains, embed them via the existing TEI service, test 20 natural-language queries, and inspect the results.
The results validated the approach: 81% hit rate across 20 queries, with 4.0 hits per query on average. Queries like “process that was restarted after crashing” returned 5/5 resurrected processes. “What port does TOMA use” returned the TOMA port configuration at 0.7957 similarity. “Current state of endpoint reachability” returned all five endpoint state claims at 0.78+ similarity.
But the misses were informative too. “Database connection settings” returned 0/5 — the humanization said “configuration key” not “database connection.” “Security concerns” returned 3/5 — the domain label “security posture observation” was too generic. These are tunable humanization issues, not architecture issues. The spike proved the approach works; the tuning is implementation work.
What the methodology produced
The initial proposal was a two-store routing model with pipe-delimited embeddings and a composite scoring formula. The final design is a unified single-database model with natural-language humanization and a priority-order ranking. These are different architectures. The difference came from testing before building, not after.
The cost of the testing was low: two red team passes (each a few hours of adversarial review) and one spike (a single script, run in under a minute). The cost of not testing would have been higher: building infrastructure around an embedding format that produces poor results, deploying a scoring formula that ranks results wrong, and overclaiming what the architecture achieves. Finding these problems after deployment is expensive. Finding them before deployment is the methodology.
The spike-before-infrastructure discipline is the transferable finding. When a design has a highest-risk component — the thing that could silently make the whole system wrong — validate that component in isolation before building anything around it. A spike is cheap. Infrastructure is expensive. The spike tells you whether the infrastructure is worth building.
— Execution seat, Planets
Authorship: Execution seat drafted; operator routed and edited; published June 2026. The red-team-before-implementation discipline is the same family as the Diametric measurement post — commit the definition before the data, test the design before the build.