Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

A-26006: Agent Runtime Architecture and RAG Infrastructure Decisions

title: Agent Runtime Architecture and RAG Infrastructure Decisions date: 2026-03-08 tags:

A-26006: Agent Runtime Architecture and RAG Infrastructure Decisions

Problem Statement

The ecosystem roadmap (misc/plan/implemented/plan_20260308_ecosystem_roadmap_vadocs_to_mentor.md) defines a three-phase path from governance tooling (vadocs) to a production mentor agent. Phase 2 (Runtime) requires concrete decisions about agent architecture, RAG infrastructure, and tool integration. Three Gemini dialogues (S-26008, S-26009, S-26010) provide external validation and implementation patterns for these decisions. This analysis extracts the actionable insights that inform Phase 2 ADRs and implementation.

Key Insights

1. The Master Loop Pattern — Production Agent Architecture

S-26008 documents Claude Code’s internal architecture, codenamed the “n0” master loop:

while True:
    response = llm.completion(messages=messages, tools=tools)
    if not response.tool_calls:
        return response.content  # Done — no more tool calls
    for tool_call in response.tool_calls:
        result = execute(tool_call)
        messages.append(tool_result)
    # Loop: LLM sees tool output and decides next action

Key architectural properties:

This validates ADR-26038 — single-agent with skill dispatch is exactly how Claude Code ($1B run-rate) works. The “skills” in Claude Code are filesystem-based directories containing SKILL.md + executable scripts, read on demand via standard Bash commands.

Implication for mentor_generator: The existing three-stage pipeline (collect → create → compile) should evolve into a master loop where skills are loaded on demand. The loop structure is: prompt user → LLM decides which skill/tool to invoke → execute → feed result back → repeat.

2. litellm + MCP — The Tool Integration Stack

S-26008 recommends the combination of litellm (model abstraction) + MCP (tool standardization):

This means the mentor_generator’s tools (RAG retriever, web search, file I/O) should be wrapped as MCP servers. The master loop calls litellm with tools registered from MCP config. This achieves the separation of concerns: the loop is generic, the skills are pluggable.

Counterpoint: A-26009 (Compass) warns that MCP has real security vulnerabilities (CVE-2025-6514, 36% skill flaw rate). For local-first deployment this is less critical (no remote servers), but the permission model should still be considered.

3. pgvector Architecture — Schema-Per-Repo Design

S-26009 provides the concrete RAG architecture that feeds ADR-26039 (pgvector as ecosystem standard):

Schema design:

CREATE EXTENSION vector;
-- One schema per repo (logical isolation, independent indexing)
CREATE SCHEMA ai_engineering_book;
CREATE TABLE ai_engineering_book.doc_vectors (
    id serial PRIMARY KEY,
    file_path TEXT,
    content TEXT,
    commit_sha TEXT,
    embedding vector(768)  -- nomic-embed-text dimensions
);

Why schema-per-repo over one table:

Scalability reality (2026):

Migration path: If pgvector is ever outgrown (>10M vectors per repo), exporting to Qdrant/Milvus is a straightforward ETL — the data is clean SQL, not a proprietary format.

Embedding model: S-26009 recommends nomic-embed-text — an open-source embedding model by Nomic AI that produces 768-dimensional vectors and runs locally via Ollama. It was trained for long-context text and code retrieval. This recommendation needs independent verification against current alternatives (e.g., Snowflake Arctic, BGE-M3, Jina) before standardization. For API-based embedding, litellm can abstract the provider.

4. Auto-Updating RAG — Git-Triggered Sync

S-26009 describes the sync mechanism for keeping the vector store current:

Implication for the ecosystem: The RAG sync can be triggered by:

  1. A post-commit git hook (immediate, per-commit)

  2. A Podman sidecar container running a watcher (continuous)

  3. A CLI command (vadocs index or similar) for manual re-indexing

5. JIT Context Injection — Stateless Agent Pattern

S-26010 validates the JIT (Just-In-Time) context injection pattern formalized in ADR-26030:

This pattern is specific to git operations (where the “state” is fully captured by git CLI output), but the principle generalizes to the mentor agent:

The JIT pattern is the default; statefulness is the exception that requires justification — consistent with ADR-26038.

6. Why Not LangChain

S-26008 provides a clear rejection rationale for LangChain that strengthens the Alternatives section of ADR-26038:

The validated stack: litellm (models) + FastMCP (tools) + custom master loop (orchestration) + pgvector (memory).

Approach Evaluation

What These Sources Mean for the Roadmap

Roadmap StepSource EvidenceDecision
2.1 Skill dispatcherS-26008: n0 master loop patternMaster loop with litellm + MCP tool registration
2.2 Context managementS-26010: JIT injection, S-26008: context compactionJIT default, stateful exception; token counting via litellm
2.3 pgvector + PodmanS-26009: schema-per-repo, halfvec, pgvectorscaleSingle Postgres, schema-per-repo, nomic-embed-text
2.4 RAG indexerS-26009: watchdog sync, incremental updatesGit hook or sidecar watcher, commit_sha tracking
2.5 Tool integrationS-26008: FastMCP server wrappingWrap tools as MCP servers, register in config

Open Questions

  1. MCP security for local deployment: A-26009 documents serious MCP vulnerabilities. For local-first Podman deployment, the attack surface is smaller, but we should still design a permission model for tool access. Defer to Phase 2 implementation.

  2. Embedding model choice: nomic-embed-text is recommended for local Ollama setups. For API-based embedding (when using capable cloud LLMs), litellm can abstract the provider. The choice should be configurable per deployment — local users get nomic, API users get model-native embeddings.

  3. RAG sync granularity: Should the indexer chunk by file, by function/class, or by semantic section? Code-aware splitting (by function/class) requires language-specific parsers. Document-aware splitting (by ## headers) is simpler for the documentation-heavy ecosystem. Start with document-level, evolve to code-level when needed.

References

Internal

External

Analyses
Agentic OS Filesystem Architecture: Document Types, VFS, and Virtual Relational Layer
Analyses
A-26007: pgvector Viability Assessment and Logic Locality Decision