Production Architecture: The Local Memory Engine & Incident Loop
When running a fleet of 50+ local autonomous agents, standard agent-to-agent RPC frameworks quickly become single points of failure. Network overhead, message loss, and complex state serialization add unnecessary latency. Our solution embraces flat-file disk persistence and SQLite WAL with FTS5 for local-first operations.
Every time an agent experiences a failure, remediation is automatically tracked. But historically, nothing was ever *learned* from these fixes. The next time the same class of failure occurred, every tier started from zero. To close this loop, we built `tools/memory_engine.py`.
The Memory Pipeline Architecture
1. **Incident Recording:** Every successful remediation appends a structured entry to `knowledge/incidents.md` containing the error trace, modified files, and a git diff summary.
2. **Zero-Cost Retrieval:** Before any agent touches code, `search_incidents()` performs a plain keyword score search over `knowledge/incidents.md` with **zero LLM cost**, injecting past lessons directly into the prompt context.
3. **Periodic Curation:** A background worker runs a single distillation pass via our local model router (`bytesize_core.llm_router.completion()`) using task_type=’routine’, folding recurring root causes straight into `.qwenrules` — the unified ruleset read by Claude Code, Qwen, and local copilot extensions.
Code: The Memory Retrieval and Curation Engine
import os
from pathlib import Path
import sqlite3
KNOWLEDGE_PATH = Path("knowledge/incidents.md")
RULES_PATH = Path(".qwenrules")
def search_incidents(query: str, max_results: int = 3) -> liststr: """ Zero-cost keyword scoring over incidents.md to surface past resolutions. Runs locally without consuming LLM API token budgets. """ if not KNOWLEDGE_PATH.exists(): return query_terms = set(query.lower().split()) scored_entries = current_entry = current_score = 0 with open(KNOWLEDGE_PATH, "r", encoding="utf-8") as f: for line in f: if line.startswith("## INC-"): if current_entry: scored_entries.append((current_score, "\n".join(current_entry))) current_entry = line current_score = 0 else: current_entry.append(line) line_lower = line.lower() for term in query_terms: if term in line_lower: current_score += 1 if current_entry: scored_entries.append((current_score, "\n".join(current_entry))) scored_entries.sort(key=lambda x: x0, reverse=True) return entry for score, entry in scored_entries:max_results if score > 0
—
Postmortem: INC-2a6e20fb. The Loop Guard ID Collision Bug
During an operator-prompted audit of our self-healing loops (`auto_heal.py`), we uncovered a subtle architectural bug that permanently blocked recurring alert types from ever self-clearing.
The Trace
`auto_heal.run()` previously constructed its `fix_id` strictly from `(category/reason, scope/agent, fixed-action-label)`, omitting any reference to the specific `alert_id`. Meanwhile, our loop guard treats `status=”resolved”` as a permanent state by design (preventing infinite loops on genuine code fixes).
**The Consequence:** The *first* time any alert category and agent pair got auto-cleared, that `fix_id` froze as permanently resolved. This silently blocked `auto_heal` from ever clearing *any future distinct alert* of the same shape. An audit revealed 13 stuck alerts lingering for over 24 hours across critical components including `heartbeat_missing`, `obs_livestream_architect`, and `secrets_guard`, even though the underlying services were entirely healthy.
The Fix
We updated both `fix_id()` builders in `auto_heal.py` to inject the unique `alert_id` into the action label, ensuring each alert instance manages its own attempt budget:
# auto_heal.py production fix abstraction
def generate_fix_id(category: str, agent: str, alert_id: str) -> str: """ Ensures unique fix_id per alert instance to prevent global loop_guard collisions. """ action_label = f"tier0_auto_heal_theater:{alert_id}" return f"{category}:{agent}:{action_label}"
Verified live: a dry run followed by execution successfully cleared all 13 stuck alerts, returning `theater_state.compose_view()` to an `ok` overall health status with zero dropped frames.
