The RAG Pipeline

Retrieval-Augmented Generation is how the agent answers questions about documents it wasn't trained on. Drop a PDF in, ask a question, get a cited answer. The pipeline that makes that work is one of the most engineered parts of the codebase — let's walk through it.

What problem RAG solves

An LLM only knows what it saw during training. Ask it about your internal architecture doc from last week and it'll make something up. RAG fixes this by retrieving relevant chunks from your documents at query time, augmenting the prompt with them, and letting the LLM generate an answer grounded in real text.

The "RAG pipeline" is the machinery that does this well — at scale, with good citations, and without retrieving garbage.

The five stages

1 · Ingestion

Documents come in (PDFs, Markdown, HTML, code). The ingestion phase:

1. Parses by file type into a uniform RagDocument.
2. Chunks using one of three strategies — structure-aware (respects headings/code blocks), fixed-size (deterministic), or semantic (splits on topic shifts). Strategy is configurable per source via AppConfig.AI.Rag.Ingestion.
3. Enriches chunks with contextual headers — a few sentences of "where in the document this came from", inspired by the Anthropic contextual-retrieval pattern.
4. RAPTOR summarization (optional) — builds a hierarchical tree of chunk summaries, so the agent can retrieve at either fine-grained or whole-section granularity.
5. Embeds each chunk via the configured embedding model.
6. Indexes into both a vector store (Azure AI Search or FAISS) and a sparse keyword store (Azure AI Search or SQLite FTS5).

2 · Query transformation

Before retrieving, the user's question gets refined. Two patterns are implemented:

• RAG Fusion — generate N paraphrased variants of the question, retrieve for each, fuse the results. Beats a single embedding for ambiguous queries.
• HyDE (Hypothetical Document Embedding) — generate a fake "ideal answer" to the question, embed that, retrieve chunks similar to the fake answer. Surprisingly effective when the question wording doesn't match the document wording.

A query classifier decides which transformation (or none) to apply per question.

3 · Hybrid retrieval

Dense vector search alone misses queries with rare keywords. BM25 alone misses semantic paraphrases. The harness runs both in parallel and fuses results with Reciprocal Rank Fusion (RRF):

score(chunk) = Σ over rank-lists  ( 1 / (k + rank_in_list) )

# k is typically 60. Each retrieval method ranks the chunk;
# RRF blends them so a chunk doesn't need to be top-1 in either list,
# just present and well-ranked in both.

4 · Reranking and quality

RRF gives a candidate set (typically 20–50 chunks). The reranker scores them more carefully — three strategies are registered under keyed DI:

• AzureSemanticReranker — uses Azure AI Search's semantic ranking.
• CrossEncoderReranker — runs a cross-encoder model (query+chunk pairs) for high-precision reordering.
• NoOpReranker — pass-through, for benchmarks.

After reranking, the CRAG evaluator (Corrective RAG) scores how confidence-worthy the top results are. Configurable accept/refine/reject thresholds determine whether to return the chunks as-is, trigger a query refinement loop, or reject the retrieval and surface a "I don't have good sources for this" response.

5 · Assembly

Finally, the surviving chunks are stitched into context blocks under a token budget. Key features:

• Citation IDs tracked per chunk — the LLM can quote them in its answer for traceability.
• Pointer expansion — if a chunk's parent section or sibling chunks are relevant and under-budget, they're pulled in to provide more context.
• Token budget enforcement — the assembler stops before exceeding AppConfig.Agent.DefaultTokenBudget minus the conversation history reserve.

Alternative backend: Azure AI Search agentic retrieval

The whole ingest → hybrid-retrieve → rerank pipeline above is the harness's portable default — it runs anywhere, against FAISS/SQLite or Azure. But if you're already invested in Azure AI Search, you can hand the entire "query → ranked results" step to a server-side knowledge base instead. That backend is AzureKnowledgeBaseRetriever, and it implements the same IHybridRetriever interface as the local pipeline — so it drops in transparently.

• Opt-in, off by default. Enable it with AppConfig:AI:Rag:AgenticRetrieval:Enabled = true (config class AgenticRetrievalConfig.cs). When it's off, IHybridRetriever resolves to the local dense + sparse + RRF retriever exactly as described above; when it's on, it resolves to the Azure knowledge-base implementation.
• Server-side execution. It calls the stable Azure.Search.Documents GA surface (KnowledgeBaseRetrievalClient, API version 2026-04-01), which runs parallel extractive retrieval with built-in semantic ranking on the service. The LLM query-planning and answer-synthesis features of agentic retrieval are preview-only and deliberately not used, so the template takes on no preview dependency.
• Bring-your-own knowledge base. The referenced knowledge base, its sources, and the underlying index must already exist on the Azure AI Search service — provisioning is an Azure-side concern, not something the harness does for you. Point it at a service via Endpoint + KnowledgeBaseName, with the API key in User Secrets or Key Vault.
• Fails soft. A missing configuration or an Azure request failure returns no results rather than throwing into the pipeline — mirroring the local retriever's graceful-degradation contract.

How an agent uses RAG

From the agent's perspective, RAG is just another ITool — typically document_search. A research skill declares it in allowed-tools; when the LLM calls it, the full pipeline above runs and returns formatted context blocks.

---
allowed-tools:
  - document_search
  - web_search
---

When the user asks about internal documentation, use document_search first.
Cite chunk IDs in the format [doc:chunk-id] when quoting passages.

Configuration

RAG config is one of the deepest sections of AppConfig. Each stage has its own subsection:

Each section maps to a class under Domain.Common/Config/AI/RAG/. Default values are tuned for "works well out of the box on the included demo corpus."

One knob that used to live here has moved: model tiering (which model handles embedding vs. synthesis) is now its own top-level section at AppConfig:AI:ModelRouting (Domain.Common/Config/AI/Routing/ModelRoutingConfig.cs), so routing decisions are shared across the whole harness rather than being RAG-specific.

Where the code lives

Advanced pipeline phases

The five stages above are the baseline. Three additional capabilities ship alongside them and change how the pipeline behaves on harder workloads:

• Complexity routing (Phase A) — an LLM classifier scores each query as simple / moderate / complex and routes it to a tiered pipeline. Simple lookups skip reranking and CRAG; complex queries get the full treatment. Saves 30–50% of retrieval cost on mixed workloads without hurting quality on the hard ones.
• Multi-hop retrieval (Phase B) — for questions that need information chained across sources, the pipeline decomposes the query, retrieves iteratively, and evaluates sufficiency between hops. An answer-faithfulness evaluator runs at the end to catch hallucinations before the response leaves the agent.
• Full autonomy (Phase D) — parallel orchestration across vector, BM25, and graph sources, with retrieval-cost tracking and quality gates at every stage. This is the mode the agent uses when given a research task instead of a single question.

Each phase is gated behind config — you opt into them per agent or per skill. See AppConfig.AI.Rag.ComplexityRouting, AppConfig.AI.Rag.MultiHop, and AppConfig.AI.Rag.Autonomy for the knobs.

Knowledge graph layer

Infrastructure.AI.KnowledgeGraph ships a working graph-backed retrieval layer — not a stub. It exposes the same surface as the vector pipeline (entities and relationships instead of chunks and embeddings) and is wired through DI alongside the rest of RAG.

What's in the box today:

• Graph stores — Neo4jGraphStore and PostgreSqlGraphStore for production, InMemoryGraphStore for tests and local dev, all behind one interface so you can swap without touching consumers.
• Community detection — Leiden algorithm clusters the graph into thematic communities, used by graph-RAG to retrieve at community granularity rather than node-by-node.
• Feedback-weighted retrieval — GraphFeedbackStore + LlmFeedbackDetector blend historical "this node/edge helped answer" weights into ranking, so the graph learns which paths are useful over time.
• Cross-session memory — KnowledgeMemoryService exposes Remember(), Recall(), Forget(), Improve() on top of the graph, with InMemorySessionCache for fast reads and configurable decay tiers (CRITICAL / STANDARD / EPHEMERAL).
• Harmonic memory (opt-in, Memora-style) — indexes a lightweight abstraction + cue-anchor scaffolding over each remembered fact so recall matches on meaning and topic, not just literal text, and related facts cluster together. Recall fuses those matches with the legacy path via Reciprocal Rank Fusion. Gated behind AppConfig:AI:HarmonicMemory:Mode (Off by default — it costs an LLM call per write); Off is the unchanged legacy behavior.
• Provenance + compliance — DefaultProvenanceStamper tags every node and edge with source, pipeline, and timestamp. ComplianceAwareGraphStore enforces retention; DefaultErasureOrchestrator handles right-to-erasure with audited ErasureReceipt records.
• Multi-tenant isolation — TenantIsolatedGraphStore enforces scope boundaries (user → dataset → owner) so multiple agents/users share infrastructure without leaking knowledge.

If you're extending the harness with domain-specific intelligence, this is where richer reasoning patterns plug in. See CLAUDE.md for the full capability list and the Cognee-inspired design lineage.

Where to go from here