The AI Fleet Architect: Building a Fault-Tolerant Multi-Agent Pipeline on Bare Metal

Architectural Overview: Zero-Cloud Fleet Topography

When orchestrating over 50 autonomous agents on bare-metal hardware, relying on cloud-hosted vector databases and persistent remote inference APIs introduces unnecessary latency, vulnerability to network drops, and prohibitive operational costs. Our fleet relies on a localized architecture where every coding assistant—including Claude Code, Qwen, and Google Gemini via Vertex AI—reads directly from shared repository files on disk rather than over complex gRPC or REST bridges.

To prevent the fleet from repeatedly hitting the same failure modes, we built `tools/memory_engine.py`. This module captures structured remediation entries into `knowledge/incidents.md` and folds recurring patterns into `.qwenrules`. This creates a unified, zero-cost knowledge base that every local agent ingests instantly upon startup.

Incident Postmortem: The `auto_heal.py` `fix_id` Collision

During an operator-prompted audit of our self-healing loops, we discovered that `auto_heal.py` was failing to clear recurring system alerts. The root cause lay in how `fix_id` was calculated:

# OLD BROKEN IMPLEMENTATION
def compute_fix_id(category: str, agent: str, action: str) -> str:
    # Collided across distinct alert instances of the same category and agent
    return hashlib.sha256(f"{category}:{agent}:{action}".encode()).hexdigest()[:16]

Because `loop_guard.should_attempt()` treats a status of `resolved` as permanent for genuine code fixes, the first time an alert shape (such as `theater_state orchestrator selection_failed:all_engines_failed`) was auto-cleared, its `fix_id` became permanently frozen. This silently blocked `auto_heal` from clearing any future distinct alert of the same category.

The Production Fix

We updated `auto_heal.py` to incorporate the unique `alert_id` into the action label generation, guaranteeing that each alert instance receives its own attempt budget:

# PRODUCTION-GRADE IMPLEMENTATION
from pathlib import Path
import hashlib
import os

def compute_granular_fix_id(category: str, agent: str, alert_id: str, action: str) -> str:
    """
    Generates a unique fix_id incorporating the specific alert_id
    to prevent permanent lockout of recurring alert categories.
    """
    raw_signature = f"{category}:{agent}:{alert_id}:{action}"
    return hashlib.sha256(raw_signature.encode()).hexdigest()[:16]

class FleetAutoHealer:
    def __init__(self, state_dir: Path):
        self.state_dir = Path(state_dir)
        self.state_dir.mkdir(parents=True, exist_ok=True)

    def evaluate_and_heal(self, alert: dict) -> bool:
        alert_id = alert.get("alert_id")
        category = alert.get("category")
        agent = alert.get("agent")
        action = alert.get("action_label")
        
        fix_id = compute_granular_fix_id(category, agent, alert_id, action)
        # Proceed with execution guard checks...
        return True

import sqlite3
from pathlib import Path

def get_db_connection(db_path: Path) -> sqlite3.Connection:
    conn = sqlite3.connect(db_path, timeout=30.0)
    conn.execute("PRAGMA journal_mode=WAL;")
    conn.execute("PRAGMA synchronous=NORMAL;")
    conn.execute("PRAGMA foreign_keys=ON;")
    conn.row_factory = sqlite3.Row
    return conn

def init_fts5_index(conn: sqlite3.Connection):
    conn.executescript("""
        CREATE VIRTUAL TABLE IF NOT EXISTS incident_search USING fts5(
            incident_id,
            tags,
            content,
            tokenize='porter'
        );
    """)
    conn.commit()