Compute & AI Services

Where the harness actually runs, and how it talks to large language models. The agent client is provider-agnostic — the same code path works against OpenRouter, Azure OpenAI, plain OpenAI, or any OpenAI-compatible gateway, selected by a single configuration switch.

Hosting options

The harness is a standard ASP.NET Core application, so it can run on any Azure compute service that supports .NET 10 containers. The choice comes down to how much operational overhead you want to take on, what scaling model fits your workload, and whether you need Docker-in-Docker for the sandbox executor.

Container Apps layout

The harness deploys as three Container Apps, each with a distinct responsibility. Splitting them lets you scale the API and SignalR hub independently and isolate the MCP server for external consumers.

Bicep example

A representative Container App resource definition for the main API. Note how secrets reference Key Vault via managed identity rather than embedding values.

resource harnessApi 'Microsoft.App/containerApps@2024-03-01' = {
  name: 'harness-api'
  location: location
  properties: {
    environmentId: containerAppEnv.id
    configuration: {
      ingress: {
        external: true
        targetPort: 8080
        transport: 'auto'
      }
      secrets: [
        {
          name: 'openai-key'
          keyVaultUrl: '${keyVault.properties.vaultUri}secrets/AzureOpenAI-ApiKey'
          identity: managedIdentity.id
        }
      ]
    }
    template: {
      containers: [
        {
          name: 'harness-api'
          image: '${containerRegistry}/harness-api:${imageTag}'
          resources: { cpu: json('1.0'), memory: '2Gi' }
          env: [
            {
              name: 'AI__AgentFramework__AzureOpenAI__Endpoint'
              value: openAiEndpoint
            }
            {
              name: 'AI__AgentFramework__AzureOpenAI__ApiKey'
              secretRef: 'openai-key'
            }
          ]
        }
      ]
      scale: {
        minReplicas: 1
        maxReplicas: 10
        rules: [
          {
            name: 'http-rule'
            http: { metadata: { concurrentRequests: '50' } }
          }
        ]
      }
    }
  }
}

AI providers

The harness needs two model surfaces: a chat-completion model for the agent runtime, and an embedding model for the RAG pipeline. The chat surface is provider-agnostic — one configuration block (AI:AgentFramework) selects between OpenRouter, Azure OpenAI, plain OpenAI, Anthropic, or any OpenAI-compatible gateway. Embeddings use a separate block (AI:Embedding) and currently expect an OpenAI or Azure OpenAI endpoint.

Agent model (chat completion)

Powers conversation, tool selection, and reasoning — the model that decides when to call tools, how to interpret results, and what to respond. The ClientType switch picks the SDK path; everything else is just an endpoint, a model identifier, and credentials.

Embedding model

Generates vector representations for document chunks during ingestion and queries during retrieval. Configured separately from the chat client and currently expects an OpenAI or Azure OpenAI endpoint — OpenRouter does not yet expose an embeddings surface, so even with OpenRouter selected as the chat provider you still need an OpenAI-style embedding endpoint if you want RAG or the knowledge graph.

Configuration mapping

Each code-level config path maps to a specific resource or setting. Use this table to trace from appsettings.json (or user-secrets) to the upstream provider.

Azure-specific patterns

When the chat provider is Azure OpenAI (directly or via Azure AI Foundry), two Azure-native controls apply that the third-party gateways don't expose.