Observability

Two observability stacks, one OpenTelemetry pipeline. Whether you choose Azure Monitor or the Grafana OSS stack, the harness exports the same traces, metrics, and logs.

Two paths, same data

The harness uses OpenTelemetry (OTel) as its instrumentation standard. This means you choose where the telemetry goes — Azure-native or open-source — without changing any application code. The same spans, metrics, and log records flow regardless of the exporter.

Path 1 — Azure Native

Path 2 — OSS Stack

What the harness instruments

The harness emits structured spans for every major subsystem. These spans carry semantic attributes that let you slice traces by model, tool, quality score, or token count.

Distributed traces (spans)

Custom metrics (counters & histograms)

Azure Monitor setup

For the Azure-native path, configure the OTel exporter to send data to Application Insights via its connection string.

{
  "OpenTelemetry": {
    "Exporter": "AzureMonitor",
    "AzureMonitor": {
      "ConnectionString": "InstrumentationKey=xxx;IngestionEndpoint=https://eastus-8.in.applicationinsights.azure.com/"
    }
  }
}

Application Insights provides built-in dashboards for request rates, failure rates, and response times. For agent-specific views, create custom Azure Workbooks that query the customEvents and dependencies tables in Log Analytics.

KQL query example

Agent turn analysis — average and p95 duration by span name, bucketed by hour:

dependencies
| where name in ("invoke_agent", "chat", "execute_tool", "embeddings")
| summarize avg(duration), percentile(duration, 95), count()
  by name, bin(timestamp, 1h)
| render timechart

Grafana + Tempo + Prometheus setup

For the OSS path, point the OTel exporter at your OTel Collector endpoint.

{
  "OpenTelemetry": {
    "Exporter": "Otlp",
    "Otlp": {
      "Endpoint": "http://otel-collector:4317",
      "Protocol": "grpc"
    }
  }
}

The OTel Collector receives spans, metrics, and logs, then routes them to Tempo (traces), Prometheus (metrics), and Loki (logs). Grafana provides the unified dashboard layer across all three backends. This stack runs as sidecar containers in the Container Apps environment or as a separate deployment.

Trace sampling — do it at the collector

High-traffic production traces are expensive to store. You keep the ones that matter (errors, slow requests, agent runs) and sample down the routine ones. The right place to make that decision is tail-based sampling at the OpenTelemetry Collector, not in application code.

Add a tail_sampling processor to your collector pipeline (scripts/otel-collector/config.yaml). The policy below reproduces the intent of the removed in-app sampler: always keep error traces, traces slower than 5s, and agent-execution traces; probabilistically sample ~10% of everything else. Policies are OR-combined — a trace matching any keep policy is retained.

processors:
  tail_sampling:
    # Buffer a trace this long after its last span before deciding.
    decision_wait: 10s
    # Cap on traces held in memory at once (tune for your throughput).
    num_traces: 50000
    expected_new_traces_per_second: 200
    policies:
      # 1. Always keep error traces.
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]

      # 2. Always keep slow traces (> 5s end-to-end).
      - name: keep-slow
        type: latency
        latency:
          threshold_ms: 5000

      # 3. Always keep agent-execution traces (gen_ai.system set by the harness).
      - name: keep-agent-by-genai-system
        type: string_attribute
        string_attribute:
          key: gen_ai.system
          values: [az.ai.agent, microsoft.extensions.ai, semantic_kernel]
          enabled_regex_matching: false

      # 3b. Also keep any trace that carries the agent.phase attribute.
      - name: keep-agent-by-phase
        type: string_attribute
        string_attribute:
          key: agent.phase
          values: ['.+']
          enabled_regex_matching: true

      # 4. Sample ~10% of everything that matched none of the above.
      - name: sample-the-rest
        type: probabilistic
        probabilistic:
          sampling_percentage: 10

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp/tempo]   # or your backend of choice

Tracing an agent conversation

When you open a trace in your viewer (Jaeger, Tempo, or Application Insights Transaction search), here is what you see for a typical agent turn that involves RAG retrieval and tool use:

Each span carries attributes like gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.tool.name, and error.type that let you diagnose exactly where time and tokens went.

Alerting

Recommended alert rules for production deployments. These apply to both Azure Monitor (metric alerts) and Grafana (Prometheus alert rules).