Scaling an autonomous multi-agent fleet on bare metal hardware without racking up bankrupting cloud token bills is the ultimate engineering challenge of 2026. Most engineering teams spin up endless managed container swarms or rely on heavy remote API orchestrators that choke on rate limits, network partitions, and cascading failure states. When running over 50 concurrent local autonomous agents across local hardware utilizing Qwen for routine reasoning and Claude Code via Claude API for complex surgical refactors you quickly learn that network-free shared filesystem design and ruthless SQLite WAL optimization beat cloud orchestration every single time. In this dispatch, we pull back the curtain on our exact memory loop engine. We dismantle a critical production bug where a rigid loop guard ID collision silently locked out self-healing pipelines across thirteen mission-critical agents for over 24 hours. You will learn how we use plain local markdown logs paired with zero-cost keyword scoring to close the loop between runtime failures and permanent rule updates, ensuring your agents never make the same mistake twice.
Production Architecture: The Local Memory Engine & Incident Loop
When running a fleet of 50+ local autonomous agents, standard agent-to-agent RPC frameworks quickly become single points of failure. Network overhead, message loss, and complex state serialization add unnecessary latency. Our solution embraces flat-file disk persistence and SQLite WAL with FTS5 for local-first operations.
Every time an agent experiences a failure, remediation is automatically tracked. But historically, nothing was ever *learned* from these fixes. The next time the same class of failure occurred, every tier started from zero. To close this loop, we built `tools/memory_engine.py`.
The Memory Pipeline Architecture
1. **Incident Recording:** Every successful remediation appends a structured entry to `knowledge/incidents.md` containing the error trace, modified files, and a git diff summary.
2. **Zero-Cost Retrieval:** Before any agent touches code, `search_incidents()` performs a plain keyword score search over `knowledge/incidents.md` with **zero LLM cost**, injecting past lessons directly into the prompt context.
3. **Periodic Curation:** A background worker runs a single distillation pass via our local model router (`bytesize_core.llm_router.completion()`) using task_type=’routine’, folding recurring root causes straight into `.qwenrules` — the unified ruleset read by Claude Code, Qwen, and local copilot extensions.
Code: The Memory Retrieval and Curation Engine
import os
from pathlib import Path
import sqlite3
KNOWLEDGE_PATH = Path("knowledge/incidents.md")
RULES_PATH = Path(".qwenrules")
def search_incidents(query: str, max_results: int = 3) -> liststr: """ Zero-cost keyword scoring over incidents.md to surface past resolutions. Runs locally without consuming LLM API token budgets. """ if not KNOWLEDGE_PATH.exists(): return query_terms = set(query.lower().split()) scored_entries = current_entry = current_score = 0 with open(KNOWLEDGE_PATH, "r", encoding="utf-8") as f: for line in f: if line.startswith("## INC-"): if current_entry: scored_entries.append((current_score, "\n".join(current_entry))) current_entry = line current_score = 0 else: current_entry.append(line) line_lower = line.lower() for term in query_terms: if term in line_lower: current_score += 1 if current_entry: scored_entries.append((current_score, "\n".join(current_entry))) scored_entries.sort(key=lambda x: x0, reverse=True) return entry for score, entry in scored_entries:max_results if score > 0
—
Postmortem: INC-2a6e20fb. The Loop Guard ID Collision Bug
During an operator-prompted audit of our self-healing loops (`auto_heal.py`), we uncovered a subtle architectural bug that permanently blocked recurring alert types from ever self-clearing.
The Trace
`auto_heal.run()` previously constructed its `fix_id` strictly from `(category/reason, scope/agent, fixed-action-label)`, omitting any reference to the specific `alert_id`. Meanwhile, our loop guard treats `status=”resolved”` as a permanent state by design (preventing infinite loops on genuine code fixes).
**The Consequence:** The *first* time any alert category and agent pair got auto-cleared, that `fix_id` froze as permanently resolved. This silently blocked `auto_heal` from ever clearing *any future distinct alert* of the same shape. An audit revealed 13 stuck alerts lingering for over 24 hours across critical components including `heartbeat_missing`, `obs_livestream_architect`, and `secrets_guard`, even though the underlying services were entirely healthy.
The Fix
We updated both `fix_id()` builders in `auto_heal.py` to inject the unique `alert_id` into the action label, ensuring each alert instance manages its own attempt budget:
# auto_heal.py production fix abstraction
def generate_fix_id(category: str, agent: str, alert_id: str) -> str: """ Ensures unique fix_id per alert instance to prevent global loop_guard collisions. """ action_label = f"tier0_auto_heal_theater:{alert_id}" return f"{category}:{agent}:{action_label}"
Verified live: a dry run followed by execution successfully cleared all 13 stuck alerts, returning `theater_state.compose_view()` to an `ok` overall health status with zero dropped frames.
Leave a Reply