Data & Retrieval

The harness has three distinct data systems: the RAG pipeline for document retrieval, the knowledge graph for entity relationships, and the plan store for agent execution state. Each maps to different Azure services.

RAG pipeline on Azure

Documents flow through seven stages from raw files to assembled context. Each stage maps to a specific Azure service (or runs in-process on the Container App).

Ingestion

Documents land in Azure Blob Storage (container: rag-documents). The harness supports three chunking strategies: structure-aware (respects document headings and sections), fixed-size (configurable overlap), and semantic (embedding-based boundary detection). Chunking runs in-process on the Container App.

Embedding

Each chunk is sent to your Azure OpenAI embedding deployment. Embeddings are generated in batches (configurable batch size via AppConfig.AI.Rag.Embeddings.BatchSize). Output dimensions depend on the model you deploy (configurable via AppConfig.AI.Rag.Embeddings.Dimensions).

Indexing

Vectors plus metadata are pushed to Azure AI Search. The index stores both dense vectors (for semantic search) and the raw text (for BM25 keyword search). This enables hybrid retrieval via Reciprocal Rank Fusion (RRF).

Retrieval

At query time, the harness runs both vector similarity and BM25 keyword search in parallel, merges results via RRF, then optionally reranks using Azure AI Search's semantic ranker (cross-encoder model). Query transformation (RAG Fusion, HyDE) happens before the search call.

AI Search configuration

resource searchIndex 'Microsoft.Search/searchServices@2024-06-01-preview' = {
  name: searchServiceName
  location: location
  sku: { name: searchSku }  // 'basic' for dev, 'standard' for prod
  properties: {
    replicaCount: searchReplicas  // 1 for dev, 3+ for prod HA
    partitionCount: 1
    semanticSearch: 'standard'  // enables semantic reranking
  }
}

Knowledge graph backends

The knowledge graph stores entity relationships extracted from documents and conversations. Storage sits behind two interfaces: the embedded Kuzu backend implements IGraphDatabaseBackend, while Neo4j, PostgreSQL, and the dev-only in-memory store implement IKnowledgeGraphStore (with Feedback and TenantIsolated decorators layered on top). Choose based on your operational maturity and query complexity.

What the graph stores

• Entities — extracted from documents and conversations (people, concepts, systems)
• Relationships — typed edges between entities with provenance stamps
• Communities — detected via Leiden algorithm, used for summarization
• Feedback weights — historical retrieval quality scores per node/edge
• Harmonic scaffolding — when harmonic memory is enabled, each trusted memory node also carries a primary abstraction and cue anchors (short entity+aspect phrases) in its properties. On write, related facts can consolidate onto a shared abstraction (topic-adoption, not physical merge); on read, the query matches those fields and the result is fused with legacy recall. Off by default (AppConfig:AI:HarmonicMemory:Mode).

Knowledge graph data flow

Plan state persistence

The harness uses EF Core with SQLite for plan execution state — DAG plans, step status, checkpoints. For production, swap to Azure SQL (or keep SQLite on a persistent volume if single-instance).

Plan store configuration

• ConnectionStrings:PlanState — SQLite connection string (dev) or Azure SQL connection string (prod)
• SqliteVersionInterceptor handles optimistic concurrency automatically
• IDbContextFactory<T> pattern ensures short-lived contexts, safe for concurrent agent operations

Blob Storage layout

A single Storage Account hosts all blob containers. Organize by purpose so you can apply lifecycle policies and access controls independently.

harness-storage (Storage Account)
├── rag-documents/        Raw documents for ingestion
├── rag-indices/          FAISS index snapshots (if using FAISS)
├── audit-logs/           JSONL governance audit trail
├── agent-artifacts/      Sandbox execution outputs
└── knowledge-exports/    Graph export snapshots