Observability & Safety

Agents are non-deterministic systems that call APIs, write files, and answer with confidence even when they're wrong. You need to see inside them — and stop them when they go astray. This page covers both: the observability pipeline and the safety layers that wrap every turn.

The observability pipeline

Everything in the harness is instrumented with OpenTelemetry. Every MediatR command opens a span. Every agent turn opens a child span. Every tool call opens a child of that. Every LLM request opens a child of that. You can drill from "this conversation failed" all the way down to "the third sub-call to GPT-4o returned a malformed JSON".

What gets instrumented

• Every MediatR request — via RequestTracingBehavior.
• Every agent turn — span attributes include agent name, conversation ID, turn number, input/output tokens, cost.
• Every tool call — span attributes include tool name, operation, latency, success/failure.
• Every LLM request — via the Microsoft.Extensions.AI built-in tracing, augmented by our custom LlmTokenTrackingProcessor that recognizes spans from Agents.AI / Semantic Kernel and enriches them with agentic context (including cache-read / cache-write token counts).
• RAG retrievals — span per query transform, retrieval, rerank, assembly.
• Tool output compression — ToolOutputCompressionBehavior logs compression events, strategy used, and size reduction ratio.
• Prompt-cache usage — PromptCachingPipelinePolicy + TokenUsageMetrics track cache-read and cache-write token counts, emitted as the agent.tokens.cache_read / agent.tokens.cache_write instruments.
• Real token streaming — when a turn runs via RunStreamingAsync, deltas surface through IAgentTurnStreamSink as they arrive; the turn span captures the streamed run.

Setting up Jaeger locally

Run Jaeger with one Docker command, then point the OTLP exporter at it:

# Start Jaeger all-in-one
docker run -d --name jaeger \
  -p 16686:16686 \
  -p 4317:4317 \
  jaegertracing/all-in-one:latest

# Browse to http://localhost:16686

Configure the harness to export to Jaeger via the OTLP endpoint in AppConfig.Observability.Exporters.Otlp.Endpoint. Run an agent example. Refresh Jaeger and you'll see the conversation timeline span-by-span.

The PostgreSQL audit store

OTel traces are great for debugging but ephemeral. For long-term audit (compliance, post-incident review, governance), the harness writes a structured record per turn to PostgreSQL via IObservabilityStore. The connection string lives in AppConfig.Observability.PostgresConnectionString. Schema includes turn-level token usage, cost, tool invocations, message previews truncated to 500 chars, plus the original message body and the args + stdout for every captured tool call. Those last two are what the deep-link endpoints below serve.

Tamper-evident audit hash-chain

For audit records that must be provably un-altered, the harness writes a hash-chained JSONL log via HashChainedJsonlWriter: each entry carries a hash over its own content plus the previous entry's hash, so any retroactive edit or deletion breaks the chain. AuditChainVerificationService (behind IVerifiableAuditChain) walks the file and reports the first broken link. This gives you a cheap, file-based tamper-evidence guarantee without a database — useful for governance and compliance trails where "nobody quietly changed the record" needs to be demonstrable.

Deep-link endpoints for the dashboard

The harness ships a web dashboard (Foresight, in Presentation.Dashboard) that lists past sessions. Because the audit store truncates previews to 500 characters, the dashboard needs a way to fetch the untruncated payload when you click into a row. Two scoped endpoints on SessionsController do that:

• GET /api/sessions/{id}/tools/{invocationId} — returns ToolInvocationDetailDto with the JSON args the LLM passed to the tool, the full stdout returned to the model, the LLM-supplied CallId, plus the standard metrics (duration, status, error type, result size). Both args and stdout are populated by ToolDiagnosticsMiddleware, which intercepts FunctionCallContent + FunctionResultContent and pairs them by CallId through the scoped ILlmUsageCapture. Args pass through the optional ISecretRedactor before storage.
• GET /api/sessions/{id}/messages/{messageId} — returns MessageBodyDto with the full content_full body captured before the 500-char preview truncation. The list endpoints still return only the preview to keep payloads cheap; the detail endpoint serves the full body on demand.

Both endpoints scope the lookup to (sessionId, id), so a forged invocationId or messageId from a different session returns 404 rather than leaking content across session boundaries. The Dashboard wires the deep-links from ToolsTable (tool name → /sessions/:id/tools/:invocationId) and SessionTimeline ("view full →" link → /sessions/:id/files/:messageId).

Schema migration: nothing to run by hand. The content_full, call_id, args and stdout columns arrive from migration 004_message_and_tool_bodies.sql, which the harness applies itself on the first database connection it opens. Rows recorded before that migration ran return contentFull: null; the dashboard renders a banner explaining the preview fallback in that case.

This page previously told you to apply that file yourself with psql, because the schema used to be delivered by SQL mounted into the Postgres container — and Postgres runs those scripts only when it first creates an empty data directory. A database that already held data could not receive a schema change at all, so hand-applying was the only route. That is fixed: schema now ships as numbered migrations embedded in the application, applied under a lock, with a ledger table recording what has already run, so an installation that has been live for months picks up exactly the changes it is missing.

Two things worth knowing if you are adapting this for your own deployment. To add a schema change, drop a numbered .sql file into Infrastructure.Observability/Migrations/ — it is embedded automatically and applied in numeric order; write it to be safely re-runnable. And Dashboards/postgres-bootstrap/ is a separate, once-per-cluster step that creates the read-only role Grafana connects as. That one is not applied by the application, because creating a role needs a privilege no least-privilege application account should hold — against a managed Postgres you run it yourself, once.

Safety layers, top to bottom

"Safety" in this codebase isn't one thing. It's a stack of independent layers, each catching a different class of problem.

1 · Prompt injection detection

PromptInjectionBehavior runs deterministic detectors over every user input — looking for known patterns ("ignore previous instructions", suspicious base64 blobs, instruction-resembling content in unexpected places). Threats above InjectionBlockThreshold halt the request before it reaches the LLM.

2 · Content safety middleware

Wrapped around the chat client itself (in AgentFactory). Filters configurable content categories — PII, profanity, classified information — both on user input and LLM output. The harness ships with sane defaults; tighten or loosen via AppConfig.AI.Governance.

3 · Tool-invocation governance — GovernedAIFunction

This is the live tool-gating layer, and it runs on the tool-execution path, not the MediatR request pipeline. Every converted tool is wrapped by GovernedAIFunction, whose InvokeCoreAsync runs three ambient gates in order before the tool executes:

1. IToolInvocationGovernor — authorization. Internally chains IToolPermissionService (ThreePhasePermissionResolver) → graded-autonomy risk gate → capability enforcement → the YAML policy engine. Opt-in via GovernanceConfig.EnforceToolInvocation.
2. IToolClassificationGate — Purview data-classification DLP; opt-in via AppConfig:AI:Governance:DataClassification:Mode (Off/Audit/Enforce).
3. IProgressEvaluator — spin / no-progress guard.

4 · Permission resolution & autonomy tiers

The first stage of the governor is IToolPermissionService (ThreePhasePermissionResolver), which evaluates each tool call against the agent's autonomy tier (Restricted, Supervised, Autonomous). Rules are resolved across 9 sources including PluginDeclaration, through a 3-phase resolver (Deny gates → Ask rules → Allow rules). When the decision is "requires approval" the governor records a PendingApproval and blocks fail-closed — live mid-call human escalation is deliberately deferred, so it does not route to IEscalationService in the middle of a tool call. The tool's RiskTier (a BlastRadius value) feeds the graded-autonomy gate: higher tiers may auto-approve low-radius tools while still gating high-radius ones.

4b · Plugin-boundary governance

When a skill comes from a local plugin, additional restrictions apply from the plugin's PluginDeclaration:

• AllowedTools — whitelist of tools the plugin's skills can access. Tools not on the list are filtered out during context assembly.
• DeniedTools — blacklist that is bypass-immune. Even when the agent runs in Autonomous mode, DeniedTools are always blocked. Use this for hard security boundaries.
• AutonomyLevel — overrides the agent's default autonomy tier for this plugin's tool calls.

This lets harness operators grant plugins access to capabilities while constraining their blast radius — a third-party plugin can read files but never write them, regardless of the agent's own permissions.

5 · The sandbox

Tools that touch the filesystem do so through path validators. The agent thinks it has a file system; it actually has a fenced subset. Covered in detail on Tools & Keyed DI.

6 · MCP security scanning

Tools from external MCP servers are scanned at registration time. Suspicious schemas are logged and skipped. See MCP Server & Client.

7 · Response sanitization

ResponseSanitizationBehavior redacts known PII / secret patterns from tool responses before they reach the LLM. Findings above ResponseBlockThreshold block the response entirely.

8 · Human escalation

When the governance engine decides a decision exceeds the agent's authority, it raises an escalation. Escalations have a priority level (Informational, Blocking, Critical) and configurable timeouts and approval strategies (AnyOf, AllOf). Records persist under .agent-sessions/escalations/ for audit. See AppConfig.AI.Governance.Escalation.

The audit trail

AuditTrailBehavior writes a tamper-evident record of every governance decision — what was requested, what the rule said, who approved or denied, when. Combined with the PostgreSQL observability store, you have a full reconstruction path for any conversation: who said what, what the agent decided, what it called, what came back.

Drift detection and learnings

Two longer-term feedback loops:

• Drift detection — EWMA-based scoring of quality regressions across runs. When per-task scores trend down, the system flags a drift event. Config: AppConfig.AI.DriftDetection.
• Learnings — cross-session feedback that biases retrieval, skill loading, and tool selection. Has decay (older learnings weight less) and pruning (forgotten if no longer relevant). Config: AppConfig.AI.Learnings.

The meta-harness loop

Beyond per-run safety, the harness can optimize itself. The meta-harness loop (RunHarnessOptimizationCommand) does this iteratively:

1. Snapshot — capture current skill files as the baseline.
2. Propose — a coding agent reads recent execution traces (JSONL files in .meta-harness/traces/), reasons about why turns failed, and outputs a proposal — which skill files to change, how, and what it observed.
3. Evaluate — run the proposed skill files against the benchmark task suite in eval-tasks/ and score by regex match.
4. Regression gate — verify the candidate doesn't regress on tasks prior winners already solved (threshold default 80%).
5. Record — promote to new best if score improved and regression gate passed.

This is the system that turns drift detection into improvement: signals from observability feed into proposals, which feed back into the skills the agent uses next time.

Reading a Jaeger trace

Describing a trace is much less useful than seeing one, so here is roughly what a two-turn conversation looks like in the Jaeger UI — one where the second turn went wrong:

RunConversationCommand                                   2.41s  ← root: one whole conversation
├─ ExecuteAgentTurnCommand                               0.83s  ← turn 1
│  ├─ chat  gen_ai.usage.input_tokens=1204               0.79s  ← the LLM call itself
│  │        gen_ai.usage.output_tokens=88
│  └─ (no tool calls this turn)
└─ ExecuteAgentTurnCommand                               1.57s  ← turn 2, the one that failed
   ├─ chat  gen_ai.usage.input_tokens=1533               0.61s
   │        gen_ai.usage.output_tokens=142
   ├─ execute_tool  tool=file_system                     0.04s
   │                otel.status_code=ERROR                      ← START HERE
   │                error=Path escapes sandbox root
   └─ chat  gen_ai.usage.input_tokens=1791               0.88s  ← the model retrying
            gen_ai.usage.output_tokens=64                         after seeing the error

Four things to read off that, in order:

1. Find the root. RunConversationCommand is the whole conversation. Its duration is what the user actually waited. (MediatR spans are named after the C# request type, which is why they carry the Command suffix.)
2. Find the failing turn. Each ExecuteAgentTurnCommand child is one turn. Scan for the red one, or the slow one.
3. Look at the turn's children. A chat span is the model thinking; an execute_tool span is the harness doing. In the trace above, the tool span carries otel.status_code=ERROR and an error tag naming the sandbox rejection — that is the actual defect, and everything after it is consequence.
4. Check the token tags when the model, not a tool, is the problem. The gen_ai.usage.* attributes tell you whether input grew unexpectedly (context bloat) or output was truncated.

Where to go from here